| Message ID | 20200113032409.11501-1-jencce.kernel@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | generic/175, generic/176: cleanup testdir before exit | expand |
On Mon, Jan 13, 2020 at 11:24:09AM +0800, Murphy Zhou wrote: > Usually the _mkfs helper will cleanup these directories at the > beginning of testcase. However, when testing on NFS, the cleanup > could be very slow and it is confusing that: We have already > started to run generic/176 but we get stuck in _mkfs, cleaning > up files left by the previous testcase generic/175. Isn't this a general problem with the way nfs handles "mkfs" on scratch devices? So you'd want to fix this once by doing the cleanup between tests instead of playing whackamole with whatever crazy file tests we think of next? > To be clear, cleanup testdir before exit. > > Signed-off-by: Murphy Zhou <jencce.kernel@gmail.com> > --- > tests/generic/175 | 1 + > tests/generic/176 | 1 + > 2 files changed, 2 insertions(+) > > diff --git a/tests/generic/175 b/tests/generic/175 > index 79e5b3d6..bd966a28 100755 > --- a/tests/generic/175 > +++ b/tests/generic/175 > @@ -61,6 +61,7 @@ bytes=$((blks * blksz)) > echo "reflinking $blks blocks, $bytes bytes" >> "$seqres.full" > _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full" > > +rm -rf $testdir Or put another way, this probably ought to be in _try_wipe_scratch_devs() --D > # success, all done > status=0 > exit > diff --git a/tests/generic/176 b/tests/generic/176 > index a084578a..bc83762e 100755 > --- a/tests/generic/176 > +++ b/tests/generic/176 > @@ -73,6 +73,7 @@ bytes=$((blocks_needed * blksz)) > echo "reflinking $((blocks_needed / 2)) blocks, $((bytes / 2)) bytes" >> "$seqres.full" > _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full" > > +rm -rf $testdir > # success, all done > status=0 > exit > -- > 2.20.1 >
On Mon, Jan 13, 2020 at 02:42:03PM -0800, Darrick J. Wong wrote: > On Mon, Jan 13, 2020 at 11:24:09AM +0800, Murphy Zhou wrote: > > Usually the _mkfs helper will cleanup these directories at the > > beginning of testcase. However, when testing on NFS, the cleanup > > could be very slow and it is confusing that: We have already > > started to run generic/176 but we get stuck in _mkfs, cleaning > > up files left by the previous testcase generic/175. > > Isn't this a general problem with the way nfs handles "mkfs" on scratch > devices? So you'd want to fix this once by doing the cleanup between > tests instead of playing whackamole with whatever crazy file tests we > think of next? Hmm, it's hard to tell. Deleting these files at anywhere makes sense. This patch is hardly a "fix", just trying to make it more clear. > > > To be clear, cleanup testdir before exit. > > > > Signed-off-by: Murphy Zhou <jencce.kernel@gmail.com> > > --- > > tests/generic/175 | 1 + > > tests/generic/176 | 1 + > > 2 files changed, 2 insertions(+) > > > > diff --git a/tests/generic/175 b/tests/generic/175 > > index 79e5b3d6..bd966a28 100755 > > --- a/tests/generic/175 > > +++ b/tests/generic/175 > > @@ -61,6 +61,7 @@ bytes=$((blks * blksz)) > > echo "reflinking $blks blocks, $bytes bytes" >> "$seqres.full" > > _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full" > > > > +rm -rf $testdir > > Or put another way, this probably ought to be in _try_wipe_scratch_devs() That's for devices I believe.. Thanks, Murphy > > --D > > > # success, all done > > status=0 > > exit > > diff --git a/tests/generic/176 b/tests/generic/176 > > index a084578a..bc83762e 100755 > > --- a/tests/generic/176 > > +++ b/tests/generic/176 > > @@ -73,6 +73,7 @@ bytes=$((blocks_needed * blksz)) > > echo "reflinking $((blocks_needed / 2)) blocks, $((bytes / 2)) bytes" >> "$seqres.full" > > _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full" > > > > +rm -rf $testdir > > # success, all done > > status=0 > > exit > > -- > > 2.20.1 > >
On Wed, Jan 15, 2020 at 12:21:32PM +0800, Murphy Zhou wrote: > On Mon, Jan 13, 2020 at 02:42:03PM -0800, Darrick J. Wong wrote: > > On Mon, Jan 13, 2020 at 11:24:09AM +0800, Murphy Zhou wrote: > > > Usually the _mkfs helper will cleanup these directories at the > > > beginning of testcase. However, when testing on NFS, the cleanup > > > could be very slow and it is confusing that: We have already > > > started to run generic/176 but we get stuck in _mkfs, cleaning > > > up files left by the previous testcase generic/175. > > > > Isn't this a general problem with the way nfs handles "mkfs" on scratch > > devices? So you'd want to fix this once by doing the cleanup between > > tests instead of playing whackamole with whatever crazy file tests we > > think of next? > > Hmm, it's hard to tell. Deleting these files at anywhere makes sense. > This patch is hardly a "fix", just trying to make it more clear. > > > > > > To be clear, cleanup testdir before exit. > > > > > > Signed-off-by: Murphy Zhou <jencce.kernel@gmail.com> > > > --- > > > tests/generic/175 | 1 + > > > tests/generic/176 | 1 + > > > 2 files changed, 2 insertions(+) > > > > > > diff --git a/tests/generic/175 b/tests/generic/175 > > > index 79e5b3d6..bd966a28 100755 > > > --- a/tests/generic/175 > > > +++ b/tests/generic/175 > > > @@ -61,6 +61,7 @@ bytes=$((blks * blksz)) > > > echo "reflinking $blks blocks, $bytes bytes" >> "$seqres.full" > > > _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full" > > > > > > +rm -rf $testdir > > > > Or put another way, this probably ought to be in _try_wipe_scratch_devs() > > That's for devices I believe.. It exists to zap the scratch devices into enough of a clean state that whatever contents a test wrote to the scratch devices cannot easily (e.g. mount) leak into the next test. IOWs, exactly where you'd want FSTYP=NFS to delete whatever junk is in the scratch mount. Plus then you get a clean nfs scratch mount at the start of each test without needing to whackamole each test. > Thanks, > Murphy > > > > > --D > > > > > # success, all done > > > status=0 > > > exit > > > diff --git a/tests/generic/176 b/tests/generic/176 > > > index a084578a..bc83762e 100755 > > > --- a/tests/generic/176 > > > +++ b/tests/generic/176 > > > @@ -73,6 +73,7 @@ bytes=$((blocks_needed * blksz)) > > > echo "reflinking $((blocks_needed / 2)) blocks, $((bytes / 2)) bytes" >> "$seqres.full" > > > _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full" > > > > > > +rm -rf $testdir Also this will massively slow down generic/17[56] on btrfs/XFS considering that we don't care about deleting the massively reflinked file we created. --D > > > # success, all done > > > status=0 > > > exit > > > -- > > > 2.20.1 > > >
On Wed, Jan 15, 2020 at 09:19:23AM -0800, Darrick J. Wong wrote: > On Wed, Jan 15, 2020 at 12:21:32PM +0800, Murphy Zhou wrote: > > On Mon, Jan 13, 2020 at 02:42:03PM -0800, Darrick J. Wong wrote: > > > On Mon, Jan 13, 2020 at 11:24:09AM +0800, Murphy Zhou wrote: > > > > Usually the _mkfs helper will cleanup these directories at the > > > > beginning of testcase. However, when testing on NFS, the cleanup > > > > could be very slow and it is confusing that: We have already > > > > started to run generic/176 but we get stuck in _mkfs, cleaning > > > > up files left by the previous testcase generic/175. > > > > > > Isn't this a general problem with the way nfs handles "mkfs" on scratch > > > devices? So you'd want to fix this once by doing the cleanup between > > > tests instead of playing whackamole with whatever crazy file tests we > > > think of next? > > > > Hmm, it's hard to tell. Deleting these files at anywhere makes sense. > > This patch is hardly a "fix", just trying to make it more clear. > > > > > > > > > To be clear, cleanup testdir before exit. > > > > > > > > Signed-off-by: Murphy Zhou <jencce.kernel@gmail.com> > > > > --- > > > > tests/generic/175 | 1 + > > > > tests/generic/176 | 1 + > > > > 2 files changed, 2 insertions(+) > > > > > > > > diff --git a/tests/generic/175 b/tests/generic/175 > > > > index 79e5b3d6..bd966a28 100755 > > > > --- a/tests/generic/175 > > > > +++ b/tests/generic/175 > > > > @@ -61,6 +61,7 @@ bytes=$((blks * blksz)) > > > > echo "reflinking $blks blocks, $bytes bytes" >> "$seqres.full" > > > > _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full" > > > > > > > > +rm -rf $testdir > > > > > > Or put another way, this probably ought to be in _try_wipe_scratch_devs() > > > > That's for devices I believe.. > > It exists to zap the scratch devices into enough of a clean state that > whatever contents a test wrote to the scratch devices cannot easily > (e.g. mount) leak into the next test. IOWs, exactly where you'd want > FSTYP=NFS to delete whatever junk is in the scratch mount. > > Plus then you get a clean nfs scratch mount at the start of each test > without needing to whackamole each test. > > > Thanks, > > Murphy > > > > > > > > --D > > > > > > > # success, all done > > > > status=0 > > > > exit > > > > diff --git a/tests/generic/176 b/tests/generic/176 > > > > index a084578a..bc83762e 100755 > > > > --- a/tests/generic/176 > > > > +++ b/tests/generic/176 > > > > @@ -73,6 +73,7 @@ bytes=$((blocks_needed * blksz)) > > > > echo "reflinking $((blocks_needed / 2)) blocks, $((bytes / 2)) bytes" >> "$seqres.full" > > > > _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full" > > > > > > > > +rm -rf $testdir > > Also this will massively slow down generic/17[56] on btrfs/XFS > considering that we don't care about deleting the massively reflinked > file we created. Deleting them would be a good testcase to me. After all the tests, deleting them should not cause any problem, or spending to much time. Leaving them behind and letting them get formatted by mkfs, definitely will be faster, but may make potential problems not discovered. If you think the deleting will be extremely slow, maybe it's time to optimize how xfs deleting reflinked files. Thanks, Murphy > > --D > > > > > # success, all done > > > > status=0 > > > > exit > > > > -- > > > > 2.20.1 > > > >
On Thu, Jan 16, 2020 at 03:45:09PM +0800, Murphy Zhou wrote: > On Wed, Jan 15, 2020 at 09:19:23AM -0800, Darrick J. Wong wrote: > > On Wed, Jan 15, 2020 at 12:21:32PM +0800, Murphy Zhou wrote: > > > On Mon, Jan 13, 2020 at 02:42:03PM -0800, Darrick J. Wong wrote: > > > > On Mon, Jan 13, 2020 at 11:24:09AM +0800, Murphy Zhou wrote: > > > > > Usually the _mkfs helper will cleanup these directories at the > > > > > beginning of testcase. However, when testing on NFS, the cleanup > > > > > could be very slow and it is confusing that: We have already > > > > > started to run generic/176 but we get stuck in _mkfs, cleaning > > > > > up files left by the previous testcase generic/175. > > > > > > > > Isn't this a general problem with the way nfs handles "mkfs" on scratch > > > > devices? So you'd want to fix this once by doing the cleanup between > > > > tests instead of playing whackamole with whatever crazy file tests we > > > > think of next? > > > > > > Hmm, it's hard to tell. Deleting these files at anywhere makes sense. > > > This patch is hardly a "fix", just trying to make it more clear. > > > > > > > > > > > > To be clear, cleanup testdir before exit. > > > > > > > > > > Signed-off-by: Murphy Zhou <jencce.kernel@gmail.com> > > > > > --- > > > > > tests/generic/175 | 1 + > > > > > tests/generic/176 | 1 + > > > > > 2 files changed, 2 insertions(+) > > > > > > > > > > diff --git a/tests/generic/175 b/tests/generic/175 > > > > > index 79e5b3d6..bd966a28 100755 > > > > > --- a/tests/generic/175 > > > > > +++ b/tests/generic/175 > > > > > @@ -61,6 +61,7 @@ bytes=$((blks * blksz)) > > > > > echo "reflinking $blks blocks, $bytes bytes" >> "$seqres.full" > > > > > _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full" > > > > > > > > > > +rm -rf $testdir > > > > > > > > Or put another way, this probably ought to be in _try_wipe_scratch_devs() > > > > > > That's for devices I believe.. > > > > It exists to zap the scratch devices into enough of a clean state that > > whatever contents a test wrote to the scratch devices cannot easily > > (e.g. mount) leak into the next test. IOWs, exactly where you'd want > > FSTYP=NFS to delete whatever junk is in the scratch mount. > > > > Plus then you get a clean nfs scratch mount at the start of each test > > without needing to whackamole each test. > > > > > Thanks, > > > Murphy > > > > > > > > > > > --D > > > > > > > > > # success, all done > > > > > status=0 > > > > > exit > > > > > diff --git a/tests/generic/176 b/tests/generic/176 > > > > > index a084578a..bc83762e 100755 > > > > > --- a/tests/generic/176 > > > > > +++ b/tests/generic/176 > > > > > @@ -73,6 +73,7 @@ bytes=$((blocks_needed * blksz)) > > > > > echo "reflinking $((blocks_needed / 2)) blocks, $((bytes / 2)) bytes" >> "$seqres.full" > > > > > _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full" > > > > > > > > > > +rm -rf $testdir > > > > Also this will massively slow down generic/17[56] on btrfs/XFS > > considering that we don't care about deleting the massively reflinked > > file we created. > > Deleting them would be a good testcase to me. After all the tests, > deleting them should not cause any problem, or spending to much > time. Leaving them behind and letting them get formatted by mkfs, > definitely will be faster, but may make potential problems not > discovered. In that case, please rewrite the commit message to say that we're enhancing this stress test to include inactivation of the files it creates. > If you think the deleting will be extremely slow, maybe it's time > to optimize how xfs deleting reflinked files. They already exist & have been out for review for years: https://lore.kernel.org/linux-xfs/157784092020.1362752.15046503361741521784.stgit@magnolia/ --D > > Thanks, > Murphy > > > > > --D > > > > > > > # success, all done > > > > > status=0 > > > > > exit > > > > > -- > > > > > 2.20.1 > > > > >
On Thu, Jan 16, 2020 at 07:52:33AM -0800, Darrick J. Wong wrote: > On Thu, Jan 16, 2020 at 03:45:09PM +0800, Murphy Zhou wrote: > > On Wed, Jan 15, 2020 at 09:19:23AM -0800, Darrick J. Wong wrote: > > > On Wed, Jan 15, 2020 at 12:21:32PM +0800, Murphy Zhou wrote: > > > > On Mon, Jan 13, 2020 at 02:42:03PM -0800, Darrick J. Wong wrote: > > > > > On Mon, Jan 13, 2020 at 11:24:09AM +0800, Murphy Zhou wrote: > > > > > > Usually the _mkfs helper will cleanup these directories at the > > > > > > beginning of testcase. However, when testing on NFS, the cleanup > > > > > > could be very slow and it is confusing that: We have already > > > > > > started to run generic/176 but we get stuck in _mkfs, cleaning > > > > > > up files left by the previous testcase generic/175. > > > > > > > > > > Isn't this a general problem with the way nfs handles "mkfs" on scratch > > > > > devices? So you'd want to fix this once by doing the cleanup between > > > > > tests instead of playing whackamole with whatever crazy file tests we > > > > > think of next? > > > > > > > > Hmm, it's hard to tell. Deleting these files at anywhere makes sense. > > > > This patch is hardly a "fix", just trying to make it more clear. > > > > > > > > > > > > > > > To be clear, cleanup testdir before exit. > > > > > > > > > > > > Signed-off-by: Murphy Zhou <jencce.kernel@gmail.com> > > > > > > --- > > > > > > tests/generic/175 | 1 + > > > > > > tests/generic/176 | 1 + > > > > > > 2 files changed, 2 insertions(+) > > > > > > > > > > > > diff --git a/tests/generic/175 b/tests/generic/175 > > > > > > index 79e5b3d6..bd966a28 100755 > > > > > > --- a/tests/generic/175 > > > > > > +++ b/tests/generic/175 > > > > > > @@ -61,6 +61,7 @@ bytes=$((blks * blksz)) > > > > > > echo "reflinking $blks blocks, $bytes bytes" >> "$seqres.full" > > > > > > _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full" > > > > > > > > > > > > +rm -rf $testdir > > > > > > > > > > Or put another way, this probably ought to be in _try_wipe_scratch_devs() > > > > > > > > That's for devices I believe.. > > > > > > It exists to zap the scratch devices into enough of a clean state that > > > whatever contents a test wrote to the scratch devices cannot easily > > > (e.g. mount) leak into the next test. IOWs, exactly where you'd want > > > FSTYP=NFS to delete whatever junk is in the scratch mount. > > > > > > Plus then you get a clean nfs scratch mount at the start of each test > > > without needing to whackamole each test. > > > > > > > Thanks, > > > > Murphy > > > > > > > > > > > > > > --D > > > > > > > > > > > # success, all done > > > > > > status=0 > > > > > > exit > > > > > > diff --git a/tests/generic/176 b/tests/generic/176 > > > > > > index a084578a..bc83762e 100755 > > > > > > --- a/tests/generic/176 > > > > > > +++ b/tests/generic/176 > > > > > > @@ -73,6 +73,7 @@ bytes=$((blocks_needed * blksz)) > > > > > > echo "reflinking $((blocks_needed / 2)) blocks, $((bytes / 2)) bytes" >> "$seqres.full" > > > > > > _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full" > > > > > > > > > > > > +rm -rf $testdir > > > > > > Also this will massively slow down generic/17[56] on btrfs/XFS > > > considering that we don't care about deleting the massively reflinked > > > file we created. > > > > Deleting them would be a good testcase to me. After all the tests, > > deleting them should not cause any problem, or spending to much > > time. Leaving them behind and letting them get formatted by mkfs, > > definitely will be faster, but may make potential problems not > > discovered. > > In that case, please rewrite the commit message to say that we're > enhancing this stress test to include inactivation of the files it > creates. > > > If you think the deleting will be extremely slow, maybe it's time > > to optimize how xfs deleting reflinked files. > > They already exist & have been out for review for years: > > https://lore.kernel.org/linux-xfs/157784092020.1362752.15046503361741521784.stgit@magnolia/ Great! Going to run some tests. Thanks very much for reviewing! Murphy > > --D > > > > > Thanks, > > Murphy > > > > > > > > --D > > > > > > > > > # success, all done > > > > > > status=0 > > > > > > exit > > > > > > -- > > > > > > 2.20.1 > > > > > >
diff --git a/tests/generic/175 b/tests/generic/175 index 79e5b3d6..bd966a28 100755 --- a/tests/generic/175 +++ b/tests/generic/175 @@ -61,6 +61,7 @@ bytes=$((blks * blksz)) echo "reflinking $blks blocks, $bytes bytes" >> "$seqres.full" _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full" +rm -rf $testdir # success, all done status=0 exit diff --git a/tests/generic/176 b/tests/generic/176 index a084578a..bc83762e 100755 --- a/tests/generic/176 +++ b/tests/generic/176 @@ -73,6 +73,7 @@ bytes=$((blocks_needed * blksz)) echo "reflinking $((blocks_needed / 2)) blocks, $((bytes / 2)) bytes" >> "$seqres.full" _reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $bytes >> "$seqres.full" +rm -rf $testdir # success, all done status=0 exit
Usually the _mkfs helper will cleanup these directories at the beginning of testcase. However, when testing on NFS, the cleanup could be very slow and it is confusing that: We have already started to run generic/176 but we get stuck in _mkfs, cleaning up files left by the previous testcase generic/175. To be clear, cleanup testdir before exit. Signed-off-by: Murphy Zhou <jencce.kernel@gmail.com> --- tests/generic/175 | 1 + tests/generic/176 | 1 + 2 files changed, 2 insertions(+)