From patchwork Tue Nov 5 14:56:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Filipe Manana X-Patchwork-Id: 11228029 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 83824112B for ; Tue, 5 Nov 2019 14:56:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 57DCB21D71 for ; Tue, 5 Nov 2019 14:56:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572965793; bh=W5uVvSmMjCeg/kPsx/uOm38SeQSZeJOHRYzDWiGJKUE=; h=From:To:Cc:Subject:Date:List-ID:From; b=c7aXcfTOxabYxtKXnJ28F+jjOzhMebDKjWzeQZFUgQT5D0qs4GKGf9uZ0NkvH9NpF OPdHnBiAk9NpfOwvpm4o+7ew5BrHyhvR4U8yt+YL5UtniLNIoPkhSanus6OFNL2u9G RVTbgPNJZG50zsPzI+XI0SqV47yUX7m9c37183mE= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389032AbfKEO4d (ORCPT ); Tue, 5 Nov 2019 09:56:33 -0500 Received: from mail.kernel.org ([198.145.29.99]:58578 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388635AbfKEO4d (ORCPT ); Tue, 5 Nov 2019 09:56:33 -0500 Received: from localhost.localdomain (bl8-197-74.dsl.telepac.pt [85.241.197.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AADB1217F5; Tue, 5 Nov 2019 14:56:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572965792; bh=W5uVvSmMjCeg/kPsx/uOm38SeQSZeJOHRYzDWiGJKUE=; h=From:To:Cc:Subject:Date:From; b=iPFFZuPfhxj/APLOQUMeU2N4Srzy99Ake3mtCmEYsro8/1/vS6bNP7j/IQyb+q9xY ciwGv+01UJGbldCbgBh/Gg9rvFJfY2b6P9BBdURy+duqR1jG/TTH5fFKAB6ProTwo6 tvizmu2R7FA5/l8FwGlYlV6/6bbbiMs9/y/gyo2I= From: fdmanana@kernel.org To: fstests@vger.kernel.org Cc: linux-btrfs@vger.kernel.org, Filipe Manana Subject: [PATCH 1/3] common: open files in ro mode for extent and hole count helpers Date: Tue, 5 Nov 2019 14:56:22 +0000 Message-Id: <20191105145622.11181-1-fdmanana@kernel.org> X-Mailer: git-send-email 2.11.0 Sender: fstests-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org From: Filipe Manana The helper functions _count_extents() and _count_holes() open their input file in RW mode to call fiemap, however opening it in RO mode is enough. By opening them in RW mode it also makes it not possible to use them against files residing in btrfs readonly snapshots for example. So just open the files in RO mode in these functions. Signed-off-by: Filipe Manana --- common/rc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/common/rc b/common/rc index ef0c2509..238ffef9 100644 --- a/common/rc +++ b/common/rc @@ -3199,12 +3199,12 @@ _require_metadata_journaling() _count_extents() { - $XFS_IO_PROG -c "fiemap" $1 | tail -n +2 | grep -v hole | wc -l + $XFS_IO_PROG -r -c "fiemap" $1 | tail -n +2 | grep -v hole | wc -l } _count_holes() { - $XFS_IO_PROG -c "fiemap" $1 | tail -n +2 | grep hole | wc -l + $XFS_IO_PROG -r -c "fiemap" $1 | tail -n +2 | grep hole | wc -l } _count_attr_extents() From patchwork Tue Nov 5 14:56:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Filipe Manana X-Patchwork-Id: 11228035 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2BE2C112B for ; Tue, 5 Nov 2019 14:56:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0152A217F5 for ; Tue, 5 Nov 2019 14:56:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572965808; bh=iUobjqWyiOHpQFuEuNChvGI6bw32368NHyot+RqS4gE=; h=From:To:Cc:Subject:Date:List-ID:From; b=Vunkjo2fMZtsaKJkFG93gNqRwGolpk8guaWpk+foQawxuyBHbd/Q9hUNvALiUN2kF X0wVsSGqkXYQNFhjj2snTJ2QbvKe2eQrzUmjkZ/vTBHH8LGuy6wSphVqNTAbSLVWNM fk2m+O8Mfpxih4TXF7Q1QGOiNL8JbbwLzftf+83s= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389598AbfKEO4q (ORCPT ); Tue, 5 Nov 2019 09:56:46 -0500 Received: from mail.kernel.org ([198.145.29.99]:58648 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727889AbfKEO4q (ORCPT ); Tue, 5 Nov 2019 09:56:46 -0500 Received: from localhost.localdomain (bl8-197-74.dsl.telepac.pt [85.241.197.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id F3EAD217F5; Tue, 5 Nov 2019 14:56:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572965805; bh=iUobjqWyiOHpQFuEuNChvGI6bw32368NHyot+RqS4gE=; h=From:To:Cc:Subject:Date:From; b=BuqTCV1+UmdoY5Xt2Db1tDhx1/FAkYb3t+4ACXlm12sEvfQNa5Dhx79U+zQ39c3Rw 8Xfi560QftGAk74/FJkaNL33N+75VHHmGcdKeKYO40BRWNPJ/GGrCxCmaIyROLaO3U 8vO8xlz6+o7F26gEPAgsVSrlsIImPRDVUmXHlBRs= From: fdmanana@kernel.org To: fstests@vger.kernel.org Cc: linux-btrfs@vger.kernel.org, Filipe Manana Subject: [PATCH 2/3] common: add helper to count number of exclusive extents in a file Date: Tue, 5 Nov 2019 14:56:40 +0000 Message-Id: <20191105145640.11231-1-fdmanana@kernel.org> X-Mailer: git-send-email 2.11.0 Sender: fstests-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org From: Filipe Manana Add a new helper that is similar to _count_extents() except that extents that are shared several times by the file with itself (reflinked at different file offsets) are accounted 1 time only, instead of N times. This is motivated by a subsequent test for btrfs that will use this new helper, to verify that send streams are issuing reflink operations. Signed-off-by: Filipe Manana --- common/rc | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/common/rc b/common/rc index 238ffef9..3c412178 100644 --- a/common/rc +++ b/common/rc @@ -3202,6 +3202,15 @@ _count_extents() $XFS_IO_PROG -r -c "fiemap" $1 | tail -n +2 | grep -v hole | wc -l } +# Similar to _count_extents() but if any extent is shared multiples times in +# the file (reflinked to different file offsets), it is accounted as 1 extent +# instead of N extents. +_count_exclusive_extents() +{ + $XFS_IO_PROG -r -c "fiemap" $1 | tail -n +2 | grep -v hole | \ + cut -d ' ' -f 3 | sort | uniq | wc -l +} + _count_holes() { $XFS_IO_PROG -r -c "fiemap" $1 | tail -n +2 | grep hole | wc -l From patchwork Tue Nov 5 14:57:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Filipe Manana X-Patchwork-Id: 11228037 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B18B414E5 for ; Tue, 5 Nov 2019 14:57:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 908A221D6C for ; Tue, 5 Nov 2019 14:57:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572965826; bh=XUeUBwRSU90NHuQYOTLWZCHE0RTyqgxJTHCTijCAmaI=; h=From:To:Cc:Subject:Date:List-ID:From; b=o710w3Mtr9KvFVn8CIbzgHDZ309eU0B7JlvEJNckDYhfMkROdWkUilUut4Dc7gM50 9rmbYvwV6Oi4jP5cPEBnZM28haBL8jbLkcGan93PGO7kx0jZY199gsNlqUn59QFgDQ xSP2F+j6xheeyW5xZJTumWeobbEOWEvpDeHH8Zwo= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389616AbfKEO5G (ORCPT ); Tue, 5 Nov 2019 09:57:06 -0500 Received: from mail.kernel.org ([198.145.29.99]:58744 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389183AbfKEO5G (ORCPT ); Tue, 5 Nov 2019 09:57:06 -0500 Received: from localhost.localdomain (bl8-197-74.dsl.telepac.pt [85.241.197.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id EB690217F5; Tue, 5 Nov 2019 14:57:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572965824; bh=XUeUBwRSU90NHuQYOTLWZCHE0RTyqgxJTHCTijCAmaI=; h=From:To:Cc:Subject:Date:From; b=DeTCHwh5N5HO0JFPHo/rn0Dgm0B1PJOE4IWSHkT4wdkEJSTS7eLhAP15l3eXTs4ym 26dJgiPxe5R6KtGl1d9nnz4iWGyw5lUy3YHh6KaIcgoNcX7wX2IG8jzpdBLSsR8rjh aO5fZK/9OeixRPzbZRZlmZZuyprjGlotiMKMaYmM= From: fdmanana@kernel.org To: fstests@vger.kernel.org Cc: linux-btrfs@vger.kernel.org, Filipe Manana Subject: [PATCH 3/3] btrfs: test that send can issue clone operations within the same file Date: Tue, 5 Nov 2019 14:57:00 +0000 Message-Id: <20191105145700.11281-1-fdmanana@kernel.org> X-Mailer: git-send-email 2.11.0 Sender: fstests-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org From: Filipe Manana Verify that both full and incremental send operations can issue clone operations when a file has extents that are shared with itself (at different offsets of course). This currently fails on btrfs but is addressed by a patch for the kernel titled: "Btrfs: send, allow clone operations within the same file" Signed-off-by: Filipe Manana --- tests/btrfs/200 | 122 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/200.out | 17 ++++++++ tests/btrfs/group | 1 + 3 files changed, 140 insertions(+) create mode 100755 tests/btrfs/200 create mode 100644 tests/btrfs/200.out diff --git a/tests/btrfs/200 b/tests/btrfs/200 new file mode 100755 index 00000000..8661f446 --- /dev/null +++ b/tests/btrfs/200 @@ -0,0 +1,122 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test No. 200 +# +# Check that send operations (full and incremental) are able to issue clone +# operations for extents that are shared between the same file. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* + rm -fr $send_files_dir +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/reflink +. ./common/punch + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_fssum +_require_test +_require_scratch_reflink +_require_xfs_io_command "fiemap" + +send_files_dir=$TEST_DIR/btrfs-test-$seq + +rm -f $seqres.full +rm -fr $send_files_dir +mkdir $send_files_dir + +_scratch_mkfs >>$seqres.full 2>&1 +_scratch_mount + +# Create our first test file, which has an extent that is shared only with +# itself and no other files. We want to verify a full send operation will +# clone the extent. +$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \ + | _filter_xfs_io +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \ + | _filter_xfs_io + +# Create out second test file which initially, for the first send operation, +# only has a single extent that is not shared. +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \ + | _filter_xfs_io + +$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base 2>&1 \ + | _filter_scratch + +$BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \ + | _filter_scratch + +# Now clone the existing extent in file bar to itself at a different offset. +# We want to verify the incremental send operation below will issue a clone +# operation instead of a write operation. +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \ + | _filter_xfs_io + +$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr 2>&1 \ + | _filter_scratch + +$BTRFS_UTIL_PROG send -p $SCRATCH_MNT/base -f $send_files_dir/2.snap \ + $SCRATCH_MNT/incr 2>&1 | _filter_scratch + +# Compute digests of the snapshot trees so that later we can compare against +# digests of the trees in the new filesystem, to see if they match (no data or +# metadata corruption happened). +$FSSUM_PROG -A -f -w $send_files_dir/base.fssum $SCRATCH_MNT/base +$FSSUM_PROG -A -f -w $send_files_dir/incr.fssum \ + -x $SCRATCH_MNT/incr/base $SCRATCH_MNT/incr + +# Now recreate the filesystem by receiving both send streams and verify we get +# the same file contents that the original filesystem had and that files foo +# and bar have shared extents. +_scratch_unmount +_scratch_mkfs >>$seqres.full 2>&1 +_scratch_mount + +$BTRFS_UTIL_PROG receive -f $send_files_dir/1.snap $SCRATCH_MNT +$BTRFS_UTIL_PROG receive -f $send_files_dir/2.snap $SCRATCH_MNT + +# Compute digests of the snapshot trees in the new filesystem and compare them +# to the ones in the original filesystem, they must match. +$FSSUM_PROG -r $send_files_dir/base.fssum $SCRATCH_MNT/base +$FSSUM_PROG -r $send_files_dir/incr.fssum $SCRATCH_MNT/incr + +num_extents=$(_count_extents $SCRATCH_MNT/base/foo) +num_exclusive_extents=$(_count_exclusive_extents $SCRATCH_MNT/base/foo) +if [ $num_extents -ne 2 ] || [ $num_exclusive_extents -ne 1 ]; then + echo "File foo does not have 2 shared extents in the base snapshot" + $XFS_IO_PROG -r -c "fiemap" $SCRATCH_MNT/base/foo +fi + +num_extents=$(_count_extents $SCRATCH_MNT/incr/foo) +num_exclusive_extents=$(_count_exclusive_extents $SCRATCH_MNT/incr/foo) +if [ $num_extents -ne 2 ] || [ $num_exclusive_extents -ne 1 ]; then + echo "File foo does not have 2 shared extents in the incr snapshot" + $XFS_IO_PROG -r -c "fiemap" $SCRATCH_MNT/incr/foo +fi + +num_extents=$(_count_extents $SCRATCH_MNT/incr/bar) +num_exclusive_extents=$(_count_exclusive_extents $SCRATCH_MNT/incr/bar) +if [ $num_extents -ne 2 ] || [ $num_exclusive_extents -ne 1 ]; then + echo "File bar does not have 2 shared extents in the incr snapshot" + $XFS_IO_PROG -r -c "fiemap" $SCRATCH_MNT/incr/bar +fi + +status=0 +exit diff --git a/tests/btrfs/200.out b/tests/btrfs/200.out new file mode 100644 index 00000000..3eec567e --- /dev/null +++ b/tests/btrfs/200.out @@ -0,0 +1,17 @@ +QA output created by 200 +wrote 65536/65536 bytes at offset 0 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +linked 65536/65536 bytes at offset 65536 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 65536/65536 bytes at offset 0 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Create a readonly snapshot of 'SCRATCH_MNT' in 'SCRATCH_MNT/base' +At subvol SCRATCH_MNT/base +linked 65536/65536 bytes at offset 65536 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Create a readonly snapshot of 'SCRATCH_MNT' in 'SCRATCH_MNT/incr' +At subvol SCRATCH_MNT/incr +At subvol base +At snapshot incr +OK +OK diff --git a/tests/btrfs/group b/tests/btrfs/group index c7ab129e..d56dcafa 100644 --- a/tests/btrfs/group +++ b/tests/btrfs/group @@ -202,3 +202,4 @@ 197 auto quick volume 198 auto quick volume 199 auto quick trim +200 auto quick send clone