| Message ID | 168972905626.1698606.12419796694170752316.stgit@frogsfrogsfrogs (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | fstests: testing improvements | expand |
On Tue, Jul 18, 2023 at 06:10:56PM -0700, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > Create a "-smoketest" parameter to check that will run generic > filesystem smoke testing for five minutes apiece. Since there are only > five smoke tests, this is effectively a 16min super-quick test. > > With gcov enabled, running these tests yields about ~75% coverage for > iomap and ~60% for xfs; or ~50% for ext4 and ~75% for ext4; and ~45% for > btrfs. Coverage was about ~65% for the pagecache. > > Cc: tytso@mit.edu > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- > check | 6 +++++- > doc/group-names.txt | 1 + > tests/generic/475 | 2 +- > tests/generic/476 | 2 +- > tests/generic/521 | 2 +- > tests/generic/522 | 2 +- > tests/generic/642 | 2 +- > 7 files changed, 11 insertions(+), 6 deletions(-) > > > diff --git a/check b/check > index 89e7e7bf20..97c7c4c7d1 100755 > --- a/check > +++ b/check > @@ -68,6 +68,7 @@ check options > -pvfs2 test PVFS2 > -tmpfs test TMPFS > -ubifs test ubifs > + -smoketest run smoke tests for 4min each We have both "smoketest" and "smoke", that's a bit confused :) > -l line mode diff > -udiff show unified diff (default) > -n show me, do not run tests > @@ -290,7 +291,10 @@ while [ $# -gt 0 ]; do > FSTYP=overlay > export OVERLAY=true > ;; > - > + -smoketest) Hmm... I'm wondering if it's worth having a specific running option for someone test group. If each "meaningful" testing way need a specific check option, the ./check file will be too complicated. If we need some recommended test ways, how about make some separated wrappers of ./check? For example: # mkdir fstests/runtest/ # cat > fstests/runtest/smoketest <<EOF export SOAK_DURATION="4m" ./check -g smoketest EOF Of course you can write more codes in it. Thanks, Zorro > + SOAK_DURATION="4m" > + GROUP_LIST="smoketest" > + ;; > -g) group=$2 ; shift ; > GROUP_LIST="$GROUP_LIST ${group//,/ }" > ;; > diff --git a/doc/group-names.txt b/doc/group-names.txt > index 1c35a39432..c3dcca3755 100644 > --- a/doc/group-names.txt > +++ b/doc/group-names.txt > @@ -118,6 +118,7 @@ selftest tests with fixed results, used to validate testing setup > send btrfs send/receive > shrinkfs decreasing the size of a filesystem > shutdown FS_IOC_SHUTDOWN ioctl > +smoketest Simple smoke tests > snapshot btrfs snapshots > soak long running soak tests whose runtime can be controlled > directly by setting the SOAK_DURATION variable > diff --git a/tests/generic/475 b/tests/generic/475 > index 0cbf5131c2..ce7fe013b1 100755 > --- a/tests/generic/475 > +++ b/tests/generic/475 > @@ -12,7 +12,7 @@ > # testing efforts. > # > . ./common/preamble > -_begin_fstest shutdown auto log metadata eio recoveryloop > +_begin_fstest shutdown auto log metadata eio recoveryloop smoketest > > # Override the default cleanup function. > _cleanup() > diff --git a/tests/generic/476 b/tests/generic/476 > index 8e93b73457..b1ae4df4d4 100755 > --- a/tests/generic/476 > +++ b/tests/generic/476 > @@ -8,7 +8,7 @@ > # bugs in the write path. > # > . ./common/preamble > -_begin_fstest auto rw long_rw stress soak > +_begin_fstest auto rw long_rw stress soak smoketest > > # Override the default cleanup function. > _cleanup() > diff --git a/tests/generic/521 b/tests/generic/521 > index 22dd31a8ec..0956e50171 100755 > --- a/tests/generic/521 > +++ b/tests/generic/521 > @@ -7,7 +7,7 @@ > # Long-soak directio fsx test > # > . ./common/preamble > -_begin_fstest soak long_rw > +_begin_fstest soak long_rw smoketest > > # Import common functions. > . ./common/filter > diff --git a/tests/generic/522 b/tests/generic/522 > index f0cbcb245c..0e4e6009ed 100755 > --- a/tests/generic/522 > +++ b/tests/generic/522 > @@ -7,7 +7,7 @@ > # Long-soak buffered fsx test > # > . ./common/preamble > -_begin_fstest soak long_rw > +_begin_fstest soak long_rw smoketest > > # Import common functions. > . ./common/filter > diff --git a/tests/generic/642 b/tests/generic/642 > index eba90903a3..e6a475a8b5 100755 > --- a/tests/generic/642 > +++ b/tests/generic/642 > @@ -8,7 +8,7 @@ > # bugs in the xattr code. > # > . ./common/preamble > -_begin_fstest auto soak attr long_rw stress > +_begin_fstest auto soak attr long_rw stress smoketest > > _cleanup() > { >
On Wed, Jul 19, 2023 at 11:10:24PM +0800, Zorro Lang wrote: > On Tue, Jul 18, 2023 at 06:10:56PM -0700, Darrick J. Wong wrote: > > From: Darrick J. Wong <djwong@kernel.org> > > > > Create a "-smoketest" parameter to check that will run generic > > filesystem smoke testing for five minutes apiece. Since there are only > > five smoke tests, this is effectively a 16min super-quick test. > > > > With gcov enabled, running these tests yields about ~75% coverage for > > iomap and ~60% for xfs; or ~50% for ext4 and ~75% for ext4; and ~45% for > > btrfs. Coverage was about ~65% for the pagecache. > > > > Cc: tytso@mit.edu > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > > --- > > check | 6 +++++- > > doc/group-names.txt | 1 + > > tests/generic/475 | 2 +- > > tests/generic/476 | 2 +- > > tests/generic/521 | 2 +- > > tests/generic/522 | 2 +- > > tests/generic/642 | 2 +- > > 7 files changed, 11 insertions(+), 6 deletions(-) > > > > > > diff --git a/check b/check > > index 89e7e7bf20..97c7c4c7d1 100755 > > --- a/check > > +++ b/check > > @@ -68,6 +68,7 @@ check options > > -pvfs2 test PVFS2 > > -tmpfs test TMPFS > > -ubifs test ubifs > > + -smoketest run smoke tests for 4min each > > We have both "smoketest" and "smoke", that's a bit confused :) We do? git grep doesn't show anything other than what I added: $ git grep smoke check:71: -smoketest run smoke tests for 4min each check:294: -smoketest) check:296: GROUP_LIST="smoketest" doc/group-names.txt:123:smoketest Simple smoke tests tests/generic/475:15:_begin_fstest shutdown auto log metadata eio recoveryloop smoketest tests/generic/476:11:_begin_fstest auto rw long_rw stress soak smoketest tests/generic/521:10:_begin_fstest soak long_rw smoketest tests/generic/522:10:_begin_fstest soak long_rw smoketest tests/generic/533:9:# Simple attr smoke tests for user EAs, dereived from generic/097. tests/generic/642:11:_begin_fstest auto soak attr long_rw stress smoketest > > -l line mode diff > > -udiff show unified diff (default) > > -n show me, do not run tests > > @@ -290,7 +291,10 @@ while [ $# -gt 0 ]; do > > FSTYP=overlay > > export OVERLAY=true > > ;; > > - > > + -smoketest) > > Hmm... I'm wondering if it's worth having a specific running option for > someone test group. If each "meaningful" testing way need a specific check > option, the ./check file will be too complicated. > > If we need some recommended test ways, how about make some separated wrappers > of ./check? For example: > > # mkdir fstests/runtest/ > # cat > fstests/runtest/smoketest <<EOF > export SOAK_DURATION="4m" > ./check -g smoketest > EOF > > Of course you can write more codes in it. The goal here was to give casual developers an easy way to run a quick 15 minute exercise *without* having to write wrapper scripts or type all that in every time. Compare: $ ./check -smoketest vs. $ SOAK_DURATION=4m ./check -g smoketest --D > Thanks, > Zorro > > > + SOAK_DURATION="4m" > > + GROUP_LIST="smoketest" > > + ;; > > -g) group=$2 ; shift ; > > GROUP_LIST="$GROUP_LIST ${group//,/ }" > > ;; > > diff --git a/doc/group-names.txt b/doc/group-names.txt > > index 1c35a39432..c3dcca3755 100644 > > --- a/doc/group-names.txt > > +++ b/doc/group-names.txt > > @@ -118,6 +118,7 @@ selftest tests with fixed results, used to validate testing setup > > send btrfs send/receive > > shrinkfs decreasing the size of a filesystem > > shutdown FS_IOC_SHUTDOWN ioctl > > +smoketest Simple smoke tests > > snapshot btrfs snapshots > > soak long running soak tests whose runtime can be controlled > > directly by setting the SOAK_DURATION variable > > diff --git a/tests/generic/475 b/tests/generic/475 > > index 0cbf5131c2..ce7fe013b1 100755 > > --- a/tests/generic/475 > > +++ b/tests/generic/475 > > @@ -12,7 +12,7 @@ > > # testing efforts. > > # > > . ./common/preamble > > -_begin_fstest shutdown auto log metadata eio recoveryloop > > +_begin_fstest shutdown auto log metadata eio recoveryloop smoketest > > > > # Override the default cleanup function. > > _cleanup() > > diff --git a/tests/generic/476 b/tests/generic/476 > > index 8e93b73457..b1ae4df4d4 100755 > > --- a/tests/generic/476 > > +++ b/tests/generic/476 > > @@ -8,7 +8,7 @@ > > # bugs in the write path. > > # > > . ./common/preamble > > -_begin_fstest auto rw long_rw stress soak > > +_begin_fstest auto rw long_rw stress soak smoketest > > > > # Override the default cleanup function. > > _cleanup() > > diff --git a/tests/generic/521 b/tests/generic/521 > > index 22dd31a8ec..0956e50171 100755 > > --- a/tests/generic/521 > > +++ b/tests/generic/521 > > @@ -7,7 +7,7 @@ > > # Long-soak directio fsx test > > # > > . ./common/preamble > > -_begin_fstest soak long_rw > > +_begin_fstest soak long_rw smoketest > > > > # Import common functions. > > . ./common/filter > > diff --git a/tests/generic/522 b/tests/generic/522 > > index f0cbcb245c..0e4e6009ed 100755 > > --- a/tests/generic/522 > > +++ b/tests/generic/522 > > @@ -7,7 +7,7 @@ > > # Long-soak buffered fsx test > > # > > . ./common/preamble > > -_begin_fstest soak long_rw > > +_begin_fstest soak long_rw smoketest > > > > # Import common functions. > > . ./common/filter > > diff --git a/tests/generic/642 b/tests/generic/642 > > index eba90903a3..e6a475a8b5 100755 > > --- a/tests/generic/642 > > +++ b/tests/generic/642 > > @@ -8,7 +8,7 @@ > > # bugs in the xattr code. > > # > > . ./common/preamble > > -_begin_fstest auto soak attr long_rw stress > > +_begin_fstest auto soak attr long_rw stress smoketest > > > > _cleanup() > > { > > >
On Wed, Jul 19, 2023 at 08:29:07AM -0700, Darrick J. Wong wrote: > On Wed, Jul 19, 2023 at 11:10:24PM +0800, Zorro Lang wrote: > > On Tue, Jul 18, 2023 at 06:10:56PM -0700, Darrick J. Wong wrote: > > > From: Darrick J. Wong <djwong@kernel.org> > > > > > > Create a "-smoketest" parameter to check that will run generic > > > filesystem smoke testing for five minutes apiece. Since there are only > > > five smoke tests, this is effectively a 16min super-quick test. > > > > > > With gcov enabled, running these tests yields about ~75% coverage for > > > iomap and ~60% for xfs; or ~50% for ext4 and ~75% for ext4; and ~45% for > > > btrfs. Coverage was about ~65% for the pagecache. > > > > > > Cc: tytso@mit.edu > > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > > > --- > > > check | 6 +++++- > > > doc/group-names.txt | 1 + > > > tests/generic/475 | 2 +- > > > tests/generic/476 | 2 +- > > > tests/generic/521 | 2 +- > > > tests/generic/522 | 2 +- > > > tests/generic/642 | 2 +- > > > 7 files changed, 11 insertions(+), 6 deletions(-) > > > > > > > > > diff --git a/check b/check > > > index 89e7e7bf20..97c7c4c7d1 100755 > > > --- a/check > > > +++ b/check > > > @@ -68,6 +68,7 @@ check options > > > -pvfs2 test PVFS2 > > > -tmpfs test TMPFS > > > -ubifs test ubifs > > > + -smoketest run smoke tests for 4min each > > > > We have both "smoketest" and "smoke", that's a bit confused :) > > We do? git grep doesn't show anything other than what I added: > > $ git grep smoke > check:71: -smoketest run smoke tests for 4min each > check:294: -smoketest) > check:296: GROUP_LIST="smoketest" > doc/group-names.txt:123:smoketest Simple smoke tests > tests/generic/475:15:_begin_fstest shutdown auto log metadata eio recoveryloop smoketest > tests/generic/476:11:_begin_fstest auto rw long_rw stress soak smoketest > tests/generic/521:10:_begin_fstest soak long_rw smoketest > tests/generic/522:10:_begin_fstest soak long_rw smoketest > tests/generic/533:9:# Simple attr smoke tests for user EAs, dereived from generic/097. > tests/generic/642:11:_begin_fstest auto soak attr long_rw stress smoketest Oh, sorry, my memory is a bit of jumbled ... > > > > -l line mode diff > > > -udiff show unified diff (default) > > > -n show me, do not run tests > > > @@ -290,7 +291,10 @@ while [ $# -gt 0 ]; do > > > FSTYP=overlay > > > export OVERLAY=true > > > ;; > > > - > > > + -smoketest) > > > > Hmm... I'm wondering if it's worth having a specific running option for > > someone test group. If each "meaningful" testing way need a specific check > > option, the ./check file will be too complicated. > > > > If we need some recommended test ways, how about make some separated wrappers > > of ./check? For example: > > > > # mkdir fstests/runtest/ > > # cat > fstests/runtest/smoketest <<EOF > > export SOAK_DURATION="4m" > > ./check -g smoketest > > EOF > > > > Of course you can write more codes in it. > > The goal here was to give casual developers an easy way to run a quick > 15 minute exercise *without* having to write wrapper scripts or type > all that in every time. Compare: > > $ ./check -smoketest > > vs. > > $ SOAK_DURATION=4m ./check -g smoketest Oh, I don't mean let users write that wrapper, I mean we provide some wrapper scripts (to be recommended). E.g. # ./runtest/smoaktest If we give "smoaktest" a specific run option, what will we do if more people want to add more options like that? But a wrapper is not an offical running option, it's just a reference which can be used directly or can be copied. Then we can have more wrappers from each fs expert as reference, to recommend other users how to use fstests specially. And I don't need to add options for each of them. What do you think? Thanks, Zorro > > --D > > > Thanks, > > Zorro > > > > > + SOAK_DURATION="4m" > > > + GROUP_LIST="smoketest" > > > + ;; > > > -g) group=$2 ; shift ; > > > GROUP_LIST="$GROUP_LIST ${group//,/ }" > > > ;; > > > diff --git a/doc/group-names.txt b/doc/group-names.txt > > > index 1c35a39432..c3dcca3755 100644 > > > --- a/doc/group-names.txt > > > +++ b/doc/group-names.txt > > > @@ -118,6 +118,7 @@ selftest tests with fixed results, used to validate testing setup > > > send btrfs send/receive > > > shrinkfs decreasing the size of a filesystem > > > shutdown FS_IOC_SHUTDOWN ioctl > > > +smoketest Simple smoke tests > > > snapshot btrfs snapshots > > > soak long running soak tests whose runtime can be controlled > > > directly by setting the SOAK_DURATION variable > > > diff --git a/tests/generic/475 b/tests/generic/475 > > > index 0cbf5131c2..ce7fe013b1 100755 > > > --- a/tests/generic/475 > > > +++ b/tests/generic/475 > > > @@ -12,7 +12,7 @@ > > > # testing efforts. > > > # > > > . ./common/preamble > > > -_begin_fstest shutdown auto log metadata eio recoveryloop > > > +_begin_fstest shutdown auto log metadata eio recoveryloop smoketest > > > > > > # Override the default cleanup function. > > > _cleanup() > > > diff --git a/tests/generic/476 b/tests/generic/476 > > > index 8e93b73457..b1ae4df4d4 100755 > > > --- a/tests/generic/476 > > > +++ b/tests/generic/476 > > > @@ -8,7 +8,7 @@ > > > # bugs in the write path. > > > # > > > . ./common/preamble > > > -_begin_fstest auto rw long_rw stress soak > > > +_begin_fstest auto rw long_rw stress soak smoketest > > > > > > # Override the default cleanup function. > > > _cleanup() > > > diff --git a/tests/generic/521 b/tests/generic/521 > > > index 22dd31a8ec..0956e50171 100755 > > > --- a/tests/generic/521 > > > +++ b/tests/generic/521 > > > @@ -7,7 +7,7 @@ > > > # Long-soak directio fsx test > > > # > > > . ./common/preamble > > > -_begin_fstest soak long_rw > > > +_begin_fstest soak long_rw smoketest > > > > > > # Import common functions. > > > . ./common/filter > > > diff --git a/tests/generic/522 b/tests/generic/522 > > > index f0cbcb245c..0e4e6009ed 100755 > > > --- a/tests/generic/522 > > > +++ b/tests/generic/522 > > > @@ -7,7 +7,7 @@ > > > # Long-soak buffered fsx test > > > # > > > . ./common/preamble > > > -_begin_fstest soak long_rw > > > +_begin_fstest soak long_rw smoketest > > > > > > # Import common functions. > > > . ./common/filter > > > diff --git a/tests/generic/642 b/tests/generic/642 > > > index eba90903a3..e6a475a8b5 100755 > > > --- a/tests/generic/642 > > > +++ b/tests/generic/642 > > > @@ -8,7 +8,7 @@ > > > # bugs in the xattr code. > > > # > > > . ./common/preamble > > > -_begin_fstest auto soak attr long_rw stress > > > +_begin_fstest auto soak attr long_rw stress smoketest > > > > > > _cleanup() > > > { > > > > > >
On Thu, Jul 20, 2023 at 12:11:15AM +0800, Zorro Lang wrote: > On Wed, Jul 19, 2023 at 08:29:07AM -0700, Darrick J. Wong wrote: > > On Wed, Jul 19, 2023 at 11:10:24PM +0800, Zorro Lang wrote: > > > On Tue, Jul 18, 2023 at 06:10:56PM -0700, Darrick J. Wong wrote: > > > > From: Darrick J. Wong <djwong@kernel.org> > > > > > > > > Create a "-smoketest" parameter to check that will run generic > > > > filesystem smoke testing for five minutes apiece. Since there are only > > > > five smoke tests, this is effectively a 16min super-quick test. > > > > > > > > With gcov enabled, running these tests yields about ~75% coverage for > > > > iomap and ~60% for xfs; or ~50% for ext4 and ~75% for ext4; and ~45% for > > > > btrfs. Coverage was about ~65% for the pagecache. > > > > > > > > Cc: tytso@mit.edu > > > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > > > > --- > > > > check | 6 +++++- > > > > doc/group-names.txt | 1 + > > > > tests/generic/475 | 2 +- > > > > tests/generic/476 | 2 +- > > > > tests/generic/521 | 2 +- > > > > tests/generic/522 | 2 +- > > > > tests/generic/642 | 2 +- > > > > 7 files changed, 11 insertions(+), 6 deletions(-) > > > > > > > > > > > > diff --git a/check b/check > > > > index 89e7e7bf20..97c7c4c7d1 100755 > > > > --- a/check > > > > +++ b/check > > > > @@ -68,6 +68,7 @@ check options > > > > -pvfs2 test PVFS2 > > > > -tmpfs test TMPFS > > > > -ubifs test ubifs > > > > + -smoketest run smoke tests for 4min each > > > > > > We have both "smoketest" and "smoke", that's a bit confused :) > > > > We do? git grep doesn't show anything other than what I added: > > > > $ git grep smoke > > check:71: -smoketest run smoke tests for 4min each > > check:294: -smoketest) > > check:296: GROUP_LIST="smoketest" > > doc/group-names.txt:123:smoketest Simple smoke tests > > tests/generic/475:15:_begin_fstest shutdown auto log metadata eio recoveryloop smoketest > > tests/generic/476:11:_begin_fstest auto rw long_rw stress soak smoketest > > tests/generic/521:10:_begin_fstest soak long_rw smoketest > > tests/generic/522:10:_begin_fstest soak long_rw smoketest > > tests/generic/533:9:# Simple attr smoke tests for user EAs, dereived from generic/097. > > tests/generic/642:11:_begin_fstest auto soak attr long_rw stress smoketest > > Oh, sorry, my memory is a bit of jumbled ... > > > > > > > -l line mode diff > > > > -udiff show unified diff (default) > > > > -n show me, do not run tests > > > > @@ -290,7 +291,10 @@ while [ $# -gt 0 ]; do > > > > FSTYP=overlay > > > > export OVERLAY=true > > > > ;; > > > > - > > > > + -smoketest) > > > > > > Hmm... I'm wondering if it's worth having a specific running option for > > > someone test group. If each "meaningful" testing way need a specific check > > > option, the ./check file will be too complicated. > > > > > > If we need some recommended test ways, how about make some separated wrappers > > > of ./check? For example: > > > > > > # mkdir fstests/runtest/ > > > # cat > fstests/runtest/smoketest <<EOF > > > export SOAK_DURATION="4m" > > > ./check -g smoketest > > > EOF > > > > > > Of course you can write more codes in it. > > > > The goal here was to give casual developers an easy way to run a quick > > 15 minute exercise *without* having to write wrapper scripts or type > > all that in every time. Compare: > > > > $ ./check -smoketest > > > > vs. > > > > $ SOAK_DURATION=4m ./check -g smoketest > > Oh, I don't mean let users write that wrapper, I mean we provide some wrapper > scripts (to be recommended). E.g. > > # ./runtest/smoaktest > > If we give "smoaktest" a specific run option, what will we do if more people > want to add more options like that? > > But a wrapper is not an offical running option, it's just a reference which > can be used directly or can be copied. Then we can have more wrappers from > each fs expert as reference, to recommend other users how to use fstests > specially. And I don't need to add options for each of them. What do you think? I disagree -- this is supposed to be a general smoketest that applies to any filesystem. It's easy to discover this option via ./check --help. Adding wrapper scripts means that now we have to find a separate way to advertise them and people have to find the wrapper on their own if they miss the advertising. --D > Thanks, > Zorro > > > > > --D > > > > > Thanks, > > > Zorro > > > > > > > + SOAK_DURATION="4m" > > > > + GROUP_LIST="smoketest" > > > > + ;; > > > > -g) group=$2 ; shift ; > > > > GROUP_LIST="$GROUP_LIST ${group//,/ }" > > > > ;; > > > > diff --git a/doc/group-names.txt b/doc/group-names.txt > > > > index 1c35a39432..c3dcca3755 100644 > > > > --- a/doc/group-names.txt > > > > +++ b/doc/group-names.txt > > > > @@ -118,6 +118,7 @@ selftest tests with fixed results, used to validate testing setup > > > > send btrfs send/receive > > > > shrinkfs decreasing the size of a filesystem > > > > shutdown FS_IOC_SHUTDOWN ioctl > > > > +smoketest Simple smoke tests > > > > snapshot btrfs snapshots > > > > soak long running soak tests whose runtime can be controlled > > > > directly by setting the SOAK_DURATION variable > > > > diff --git a/tests/generic/475 b/tests/generic/475 > > > > index 0cbf5131c2..ce7fe013b1 100755 > > > > --- a/tests/generic/475 > > > > +++ b/tests/generic/475 > > > > @@ -12,7 +12,7 @@ > > > > # testing efforts. > > > > # > > > > . ./common/preamble > > > > -_begin_fstest shutdown auto log metadata eio recoveryloop > > > > +_begin_fstest shutdown auto log metadata eio recoveryloop smoketest > > > > > > > > # Override the default cleanup function. > > > > _cleanup() > > > > diff --git a/tests/generic/476 b/tests/generic/476 > > > > index 8e93b73457..b1ae4df4d4 100755 > > > > --- a/tests/generic/476 > > > > +++ b/tests/generic/476 > > > > @@ -8,7 +8,7 @@ > > > > # bugs in the write path. > > > > # > > > > . ./common/preamble > > > > -_begin_fstest auto rw long_rw stress soak > > > > +_begin_fstest auto rw long_rw stress soak smoketest > > > > > > > > # Override the default cleanup function. > > > > _cleanup() > > > > diff --git a/tests/generic/521 b/tests/generic/521 > > > > index 22dd31a8ec..0956e50171 100755 > > > > --- a/tests/generic/521 > > > > +++ b/tests/generic/521 > > > > @@ -7,7 +7,7 @@ > > > > # Long-soak directio fsx test > > > > # > > > > . ./common/preamble > > > > -_begin_fstest soak long_rw > > > > +_begin_fstest soak long_rw smoketest > > > > > > > > # Import common functions. > > > > . ./common/filter > > > > diff --git a/tests/generic/522 b/tests/generic/522 > > > > index f0cbcb245c..0e4e6009ed 100755 > > > > --- a/tests/generic/522 > > > > +++ b/tests/generic/522 > > > > @@ -7,7 +7,7 @@ > > > > # Long-soak buffered fsx test > > > > # > > > > . ./common/preamble > > > > -_begin_fstest soak long_rw > > > > +_begin_fstest soak long_rw smoketest > > > > > > > > # Import common functions. > > > > . ./common/filter > > > > diff --git a/tests/generic/642 b/tests/generic/642 > > > > index eba90903a3..e6a475a8b5 100755 > > > > --- a/tests/generic/642 > > > > +++ b/tests/generic/642 > > > > @@ -8,7 +8,7 @@ > > > > # bugs in the xattr code. > > > > # > > > > . ./common/preamble > > > > -_begin_fstest auto soak attr long_rw stress > > > > +_begin_fstest auto soak attr long_rw stress smoketest > > > > > > > > _cleanup() > > > > { > > > > > > > > > >
On Wed, Jul 19, 2023 at 07:27:56PM -0700, Darrick J. Wong wrote: > On Thu, Jul 20, 2023 at 12:11:15AM +0800, Zorro Lang wrote: > > On Wed, Jul 19, 2023 at 08:29:07AM -0700, Darrick J. Wong wrote: > > > On Wed, Jul 19, 2023 at 11:10:24PM +0800, Zorro Lang wrote: > > > > On Tue, Jul 18, 2023 at 06:10:56PM -0700, Darrick J. Wong wrote: > > > > > From: Darrick J. Wong <djwong@kernel.org> > > > > > > > > > > Create a "-smoketest" parameter to check that will run generic > > > > > filesystem smoke testing for five minutes apiece. Since there are only > > > > > five smoke tests, this is effectively a 16min super-quick test. > > > > > > > > > > With gcov enabled, running these tests yields about ~75% coverage for > > > > > iomap and ~60% for xfs; or ~50% for ext4 and ~75% for ext4; and ~45% for > > > > > btrfs. Coverage was about ~65% for the pagecache. > > > > > > > > > > Cc: tytso@mit.edu > > > > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > > > > > --- > > > > > check | 6 +++++- > > > > > doc/group-names.txt | 1 + > > > > > tests/generic/475 | 2 +- > > > > > tests/generic/476 | 2 +- > > > > > tests/generic/521 | 2 +- > > > > > tests/generic/522 | 2 +- > > > > > tests/generic/642 | 2 +- > > > > > 7 files changed, 11 insertions(+), 6 deletions(-) > > > > > > > > > > > > > > > diff --git a/check b/check > > > > > index 89e7e7bf20..97c7c4c7d1 100755 > > > > > --- a/check > > > > > +++ b/check > > > > > @@ -68,6 +68,7 @@ check options > > > > > -pvfs2 test PVFS2 > > > > > -tmpfs test TMPFS > > > > > -ubifs test ubifs > > > > > + -smoketest run smoke tests for 4min each > > > > > > > > We have both "smoketest" and "smoke", that's a bit confused :) > > > > > > We do? git grep doesn't show anything other than what I added: > > > > > > $ git grep smoke > > > check:71: -smoketest run smoke tests for 4min each > > > check:294: -smoketest) > > > check:296: GROUP_LIST="smoketest" > > > doc/group-names.txt:123:smoketest Simple smoke tests > > > tests/generic/475:15:_begin_fstest shutdown auto log metadata eio recoveryloop smoketest > > > tests/generic/476:11:_begin_fstest auto rw long_rw stress soak smoketest > > > tests/generic/521:10:_begin_fstest soak long_rw smoketest > > > tests/generic/522:10:_begin_fstest soak long_rw smoketest > > > tests/generic/533:9:# Simple attr smoke tests for user EAs, dereived from generic/097. > > > tests/generic/642:11:_begin_fstest auto soak attr long_rw stress smoketest > > > > Oh, sorry, my memory is a bit of jumbled ... > > > > > > > > > > -l line mode diff > > > > > -udiff show unified diff (default) > > > > > -n show me, do not run tests > > > > > @@ -290,7 +291,10 @@ while [ $# -gt 0 ]; do > > > > > FSTYP=overlay > > > > > export OVERLAY=true > > > > > ;; > > > > > - > > > > > + -smoketest) > > > > > > > > Hmm... I'm wondering if it's worth having a specific running option for > > > > someone test group. If each "meaningful" testing way need a specific check > > > > option, the ./check file will be too complicated. > > > > > > > > If we need some recommended test ways, how about make some separated wrappers > > > > of ./check? For example: > > > > > > > > # mkdir fstests/runtest/ > > > > # cat > fstests/runtest/smoketest <<EOF > > > > export SOAK_DURATION="4m" > > > > ./check -g smoketest > > > > EOF > > > > > > > > Of course you can write more codes in it. > > > > > > The goal here was to give casual developers an easy way to run a quick > > > 15 minute exercise *without* having to write wrapper scripts or type > > > all that in every time. Compare: > > > > > > $ ./check -smoketest > > > > > > vs. > > > > > > $ SOAK_DURATION=4m ./check -g smoketest > > > > Oh, I don't mean let users write that wrapper, I mean we provide some wrapper > > scripts (to be recommended). E.g. > > > > # ./runtest/smoaktest > > > > If we give "smoaktest" a specific run option, what will we do if more people > > want to add more options like that? > > > > But a wrapper is not an offical running option, it's just a reference which > > can be used directly or can be copied. Then we can have more wrappers from > > each fs expert as reference, to recommend other users how to use fstests > > specially. And I don't need to add options for each of them. What do you think? > > I disagree -- this is supposed to be a general smoketest that applies to > any filesystem. It's easy to discover this option via ./check --help. > > Adding wrapper scripts means that now we have to find a separate way to > advertise them and people have to find the wrapper on their own if they > miss the advertising. Hmmm.. OK, but we also have other general test types, e.g. soak test, stress test, rw test. Should we give each of them a specific option if anyone need that? Thanks, Zorro > > --D > > > Thanks, > > Zorro > > > > > > > > --D > > > > > > > Thanks, > > > > Zorro > > > > > > > > > + SOAK_DURATION="4m" > > > > > + GROUP_LIST="smoketest" > > > > > + ;; > > > > > -g) group=$2 ; shift ; > > > > > GROUP_LIST="$GROUP_LIST ${group//,/ }" > > > > > ;; > > > > > diff --git a/doc/group-names.txt b/doc/group-names.txt > > > > > index 1c35a39432..c3dcca3755 100644 > > > > > --- a/doc/group-names.txt > > > > > +++ b/doc/group-names.txt > > > > > @@ -118,6 +118,7 @@ selftest tests with fixed results, used to validate testing setup > > > > > send btrfs send/receive > > > > > shrinkfs decreasing the size of a filesystem > > > > > shutdown FS_IOC_SHUTDOWN ioctl > > > > > +smoketest Simple smoke tests > > > > > snapshot btrfs snapshots > > > > > soak long running soak tests whose runtime can be controlled > > > > > directly by setting the SOAK_DURATION variable > > > > > diff --git a/tests/generic/475 b/tests/generic/475 > > > > > index 0cbf5131c2..ce7fe013b1 100755 > > > > > --- a/tests/generic/475 > > > > > +++ b/tests/generic/475 > > > > > @@ -12,7 +12,7 @@ > > > > > # testing efforts. > > > > > # > > > > > . ./common/preamble > > > > > -_begin_fstest shutdown auto log metadata eio recoveryloop > > > > > +_begin_fstest shutdown auto log metadata eio recoveryloop smoketest > > > > > > > > > > # Override the default cleanup function. > > > > > _cleanup() > > > > > diff --git a/tests/generic/476 b/tests/generic/476 > > > > > index 8e93b73457..b1ae4df4d4 100755 > > > > > --- a/tests/generic/476 > > > > > +++ b/tests/generic/476 > > > > > @@ -8,7 +8,7 @@ > > > > > # bugs in the write path. > > > > > # > > > > > . ./common/preamble > > > > > -_begin_fstest auto rw long_rw stress soak > > > > > +_begin_fstest auto rw long_rw stress soak smoketest > > > > > > > > > > # Override the default cleanup function. > > > > > _cleanup() > > > > > diff --git a/tests/generic/521 b/tests/generic/521 > > > > > index 22dd31a8ec..0956e50171 100755 > > > > > --- a/tests/generic/521 > > > > > +++ b/tests/generic/521 > > > > > @@ -7,7 +7,7 @@ > > > > > # Long-soak directio fsx test > > > > > # > > > > > . ./common/preamble > > > > > -_begin_fstest soak long_rw > > > > > +_begin_fstest soak long_rw smoketest > > > > > > > > > > # Import common functions. > > > > > . ./common/filter > > > > > diff --git a/tests/generic/522 b/tests/generic/522 > > > > > index f0cbcb245c..0e4e6009ed 100755 > > > > > --- a/tests/generic/522 > > > > > +++ b/tests/generic/522 > > > > > @@ -7,7 +7,7 @@ > > > > > # Long-soak buffered fsx test > > > > > # > > > > > . ./common/preamble > > > > > -_begin_fstest soak long_rw > > > > > +_begin_fstest soak long_rw smoketest > > > > > > > > > > # Import common functions. > > > > > . ./common/filter > > > > > diff --git a/tests/generic/642 b/tests/generic/642 > > > > > index eba90903a3..e6a475a8b5 100755 > > > > > --- a/tests/generic/642 > > > > > +++ b/tests/generic/642 > > > > > @@ -8,7 +8,7 @@ > > > > > # bugs in the xattr code. > > > > > # > > > > > . ./common/preamble > > > > > -_begin_fstest auto soak attr long_rw stress > > > > > +_begin_fstest auto soak attr long_rw stress smoketest > > > > > > > > > > _cleanup() > > > > > { > > > > > > > > > > > > > > >
On Thu, Jul 20, 2023 at 10:34:33PM +0800, Zorro Lang wrote: > On Wed, Jul 19, 2023 at 07:27:56PM -0700, Darrick J. Wong wrote: > > On Thu, Jul 20, 2023 at 12:11:15AM +0800, Zorro Lang wrote: > > > On Wed, Jul 19, 2023 at 08:29:07AM -0700, Darrick J. Wong wrote: > > > > On Wed, Jul 19, 2023 at 11:10:24PM +0800, Zorro Lang wrote: > > > > > On Tue, Jul 18, 2023 at 06:10:56PM -0700, Darrick J. Wong wrote: > > > > > > From: Darrick J. Wong <djwong@kernel.org> > > > > > > > > > > > > Create a "-smoketest" parameter to check that will run generic > > > > > > filesystem smoke testing for five minutes apiece. Since there are only > > > > > > five smoke tests, this is effectively a 16min super-quick test. > > > > > > > > > > > > With gcov enabled, running these tests yields about ~75% coverage for > > > > > > iomap and ~60% for xfs; or ~50% for ext4 and ~75% for ext4; and ~45% for > > > > > > btrfs. Coverage was about ~65% for the pagecache. > > > > > > > > > > > > Cc: tytso@mit.edu > > > > > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > > > > > > --- > > > > > > check | 6 +++++- > > > > > > doc/group-names.txt | 1 + > > > > > > tests/generic/475 | 2 +- > > > > > > tests/generic/476 | 2 +- > > > > > > tests/generic/521 | 2 +- > > > > > > tests/generic/522 | 2 +- > > > > > > tests/generic/642 | 2 +- > > > > > > 7 files changed, 11 insertions(+), 6 deletions(-) > > > > > > > > > > > > > > > > > > diff --git a/check b/check > > > > > > index 89e7e7bf20..97c7c4c7d1 100755 > > > > > > --- a/check > > > > > > +++ b/check > > > > > > @@ -68,6 +68,7 @@ check options > > > > > > -pvfs2 test PVFS2 > > > > > > -tmpfs test TMPFS > > > > > > -ubifs test ubifs > > > > > > + -smoketest run smoke tests for 4min each > > > > > > > > > > We have both "smoketest" and "smoke", that's a bit confused :) > > > > > > > > We do? git grep doesn't show anything other than what I added: > > > > > > > > $ git grep smoke > > > > check:71: -smoketest run smoke tests for 4min each > > > > check:294: -smoketest) > > > > check:296: GROUP_LIST="smoketest" > > > > doc/group-names.txt:123:smoketest Simple smoke tests > > > > tests/generic/475:15:_begin_fstest shutdown auto log metadata eio recoveryloop smoketest > > > > tests/generic/476:11:_begin_fstest auto rw long_rw stress soak smoketest > > > > tests/generic/521:10:_begin_fstest soak long_rw smoketest > > > > tests/generic/522:10:_begin_fstest soak long_rw smoketest > > > > tests/generic/533:9:# Simple attr smoke tests for user EAs, dereived from generic/097. > > > > tests/generic/642:11:_begin_fstest auto soak attr long_rw stress smoketest > > > > > > Oh, sorry, my memory is a bit of jumbled ... > > > > > > > > > > > > > -l line mode diff > > > > > > -udiff show unified diff (default) > > > > > > -n show me, do not run tests > > > > > > @@ -290,7 +291,10 @@ while [ $# -gt 0 ]; do > > > > > > FSTYP=overlay > > > > > > export OVERLAY=true > > > > > > ;; > > > > > > - > > > > > > + -smoketest) > > > > > > > > > > Hmm... I'm wondering if it's worth having a specific running option for > > > > > someone test group. If each "meaningful" testing way need a specific check > > > > > option, the ./check file will be too complicated. > > > > > > > > > > If we need some recommended test ways, how about make some separated wrappers > > > > > of ./check? For example: > > > > > > > > > > # mkdir fstests/runtest/ > > > > > # cat > fstests/runtest/smoketest <<EOF > > > > > export SOAK_DURATION="4m" > > > > > ./check -g smoketest > > > > > EOF > > > > > > > > > > Of course you can write more codes in it. > > > > > > > > The goal here was to give casual developers an easy way to run a quick > > > > 15 minute exercise *without* having to write wrapper scripts or type > > > > all that in every time. Compare: > > > > > > > > $ ./check -smoketest > > > > > > > > vs. > > > > > > > > $ SOAK_DURATION=4m ./check -g smoketest > > > > > > Oh, I don't mean let users write that wrapper, I mean we provide some wrapper > > > scripts (to be recommended). E.g. > > > > > > # ./runtest/smoaktest > > > > > > If we give "smoaktest" a specific run option, what will we do if more people > > > want to add more options like that? > > > > > > But a wrapper is not an offical running option, it's just a reference which > > > can be used directly or can be copied. Then we can have more wrappers from > > > each fs expert as reference, to recommend other users how to use fstests > > > specially. And I don't need to add options for each of them. What do you think? > > > > I disagree -- this is supposed to be a general smoketest that applies to > > any filesystem. It's easy to discover this option via ./check --help. > > > > Adding wrapper scripts means that now we have to find a separate way to > > advertise them and people have to find the wrapper on their own if they > > miss the advertising. > > Hmmm.. OK, but we also have other general test types, e.g. soak test, stress > test, rw test. Should we give each of them a specific option if anyone need > that? If someone wants that, then ok. The taret audience for this are the drive-by filesystem patch authors. IOWs, people who have some small bug they want to try to fix and want to run a quick test to see if their change works. I don't think it's reasonable to expect drive-by'ers to know all that much about the fstests groups or spend the hours it takes to run -g all. As a maintainer, I prefer that these folks have done at least a small taste of QA before they start talking to the lists. Most people probably won't know which groups they want unless they're already working with a maintainer on a specific feature. This isn't intended for experts -- we experts have to know about test groups and flakey tests and whatnot, and that's fine. --D > Thanks, > Zorro > > > > > --D > > > > > Thanks, > > > Zorro > > > > > > > > > > > --D > > > > > > > > > Thanks, > > > > > Zorro > > > > > > > > > > > + SOAK_DURATION="4m" > > > > > > + GROUP_LIST="smoketest" > > > > > > + ;; > > > > > > -g) group=$2 ; shift ; > > > > > > GROUP_LIST="$GROUP_LIST ${group//,/ }" > > > > > > ;; > > > > > > diff --git a/doc/group-names.txt b/doc/group-names.txt > > > > > > index 1c35a39432..c3dcca3755 100644 > > > > > > --- a/doc/group-names.txt > > > > > > +++ b/doc/group-names.txt > > > > > > @@ -118,6 +118,7 @@ selftest tests with fixed results, used to validate testing setup > > > > > > send btrfs send/receive > > > > > > shrinkfs decreasing the size of a filesystem > > > > > > shutdown FS_IOC_SHUTDOWN ioctl > > > > > > +smoketest Simple smoke tests > > > > > > snapshot btrfs snapshots > > > > > > soak long running soak tests whose runtime can be controlled > > > > > > directly by setting the SOAK_DURATION variable > > > > > > diff --git a/tests/generic/475 b/tests/generic/475 > > > > > > index 0cbf5131c2..ce7fe013b1 100755 > > > > > > --- a/tests/generic/475 > > > > > > +++ b/tests/generic/475 > > > > > > @@ -12,7 +12,7 @@ > > > > > > # testing efforts. > > > > > > # > > > > > > . ./common/preamble > > > > > > -_begin_fstest shutdown auto log metadata eio recoveryloop > > > > > > +_begin_fstest shutdown auto log metadata eio recoveryloop smoketest > > > > > > > > > > > > # Override the default cleanup function. > > > > > > _cleanup() > > > > > > diff --git a/tests/generic/476 b/tests/generic/476 > > > > > > index 8e93b73457..b1ae4df4d4 100755 > > > > > > --- a/tests/generic/476 > > > > > > +++ b/tests/generic/476 > > > > > > @@ -8,7 +8,7 @@ > > > > > > # bugs in the write path. > > > > > > # > > > > > > . ./common/preamble > > > > > > -_begin_fstest auto rw long_rw stress soak > > > > > > +_begin_fstest auto rw long_rw stress soak smoketest > > > > > > > > > > > > # Override the default cleanup function. > > > > > > _cleanup() > > > > > > diff --git a/tests/generic/521 b/tests/generic/521 > > > > > > index 22dd31a8ec..0956e50171 100755 > > > > > > --- a/tests/generic/521 > > > > > > +++ b/tests/generic/521 > > > > > > @@ -7,7 +7,7 @@ > > > > > > # Long-soak directio fsx test > > > > > > # > > > > > > . ./common/preamble > > > > > > -_begin_fstest soak long_rw > > > > > > +_begin_fstest soak long_rw smoketest > > > > > > > > > > > > # Import common functions. > > > > > > . ./common/filter > > > > > > diff --git a/tests/generic/522 b/tests/generic/522 > > > > > > index f0cbcb245c..0e4e6009ed 100755 > > > > > > --- a/tests/generic/522 > > > > > > +++ b/tests/generic/522 > > > > > > @@ -7,7 +7,7 @@ > > > > > > # Long-soak buffered fsx test > > > > > > # > > > > > > . ./common/preamble > > > > > > -_begin_fstest soak long_rw > > > > > > +_begin_fstest soak long_rw smoketest > > > > > > > > > > > > # Import common functions. > > > > > > . ./common/filter > > > > > > diff --git a/tests/generic/642 b/tests/generic/642 > > > > > > index eba90903a3..e6a475a8b5 100755 > > > > > > --- a/tests/generic/642 > > > > > > +++ b/tests/generic/642 > > > > > > @@ -8,7 +8,7 @@ > > > > > > # bugs in the xattr code. > > > > > > # > > > > > > . ./common/preamble > > > > > > -_begin_fstest auto soak attr long_rw stress > > > > > > +_begin_fstest auto soak attr long_rw stress smoketest > > > > > > > > > > > > _cleanup() > > > > > > { > > > > > > > > > > > > > > > > > > > > >
On Tue, Jul 25, 2023 at 05:05:24PM -0700, Darrick J. Wong wrote: > > If someone wants that, then ok. The taret audience for this are the > drive-by filesystem patch authors. IOWs, people who have some small bug > they want to try to fix and want to run a quick test to see if their > change works. Zorro, FYI, the context behind this was a comment I had made to Darrick that the time necessary to run "-g quick" had been getting longer and longer, and it might be nice to create a manually curated "-g smoke" that was good enough for drive-by patch authors. I was originally thinking about a cut-down set of tests by selecting a subset of "-g quick", but Darrick suggested that instead, we just run a very small set of tests (mostly based on fsstress / fsx) and just run them in a loop for 4 minutes or so. We also talked about having a time budget (say, 15 minutes) and then just dividing 15 time by the number of tests, and just run them in for a specified soak time, so that the total time is known ahead of time. To be honest, I was a bit dubious it could be that simple, but that's where using kcov to show that you get a pretty good code coverage using something that simple comes from. > I don't think it's reasonable to expect drive-by'ers to know all that > much about the fstests groups or spend the hours it takes to run -g all. > As a maintainer, I prefer that these folks have done at least a small > taste of QA before they start talking to the lists. A big problem for the drive-by'ers is that that the top-level xfstests README file is just plain scary, and has far too many steps for a drive-by patch author to follow. What I plan to add to a maintainer-entry-file.rst file for ext4 in the kernel docs is to tell that drive-by posters that should run "kvm-xfstests smoke" before submitting a patch, and setting up kvm-xfstess is dead simple easy: 1) Install kvm-xfstests --- you only have to run this once % git clone https://github.com/tytso/xfstests-bld fstests % cd fstests % make ; make install # Optional, if your file system you are developing isn't ext4; # change f2fs to the file system of your choice % echo PRIMARY_FSTYPE=f2fs >> ~/.config/kvm-xfstests 2) Build the kernel suitable for use with kvm-xfstests % cd /path/to/your/kernel % install-kconfig % kbuild 3) Run the smoke test --- assuming the cwd is /path/to/your/kernel (Note: today this runs -g quick, but it would be good if this could be faster) % kvm-xfstests smoke It's simple, and since the kvm-xfstests script will download a pre-compiled test appliance image automatically, there's no need to require the drive-by tester to figure out how compile xfstests with any of its prerequisites. And once things are set up, then it's just a matter of running "kbuild" to build your kernel after you make changes, and running "kvm-xfstests smoke" to do a quick smoke testing run. No muss, no fuss, no dirty dishes... :-) Cheers, - Ted
On Wed, Jul 26, 2023 at 02:01:02AM -0400, Theodore Ts'o wrote: > On Tue, Jul 25, 2023 at 05:05:24PM -0700, Darrick J. Wong wrote: > > > > If someone wants that, then ok. The taret audience for this are the > > drive-by filesystem patch authors. IOWs, people who have some small bug > > they want to try to fix and want to run a quick test to see if their > > change works. > > Zorro, > > FYI, the context behind this was a comment I had made to Darrick that > the time necessary to run "-g quick" had been getting longer and > longer, and it might be nice to create a manually curated "-g smoke" > that was good enough for drive-by patch authors. I was originally > thinking about a cut-down set of tests by selecting a subset of "-g > quick", but Darrick suggested that instead, we just run a very small > set of tests (mostly based on fsstress / fsx) and just run them in a > loop for 4 minutes or so. > > We also talked about having a time budget (say, 15 minutes) and then > just dividing 15 time by the number of tests, and just run them in for > a specified soak time, so that the total time is known ahead of time. > > To be honest, I was a bit dubious it could be that simple, but that's > where using kcov to show that you get a pretty good code coverage > using something that simple comes from. > > > I don't think it's reasonable to expect drive-by'ers to know all that > > much about the fstests groups or spend the hours it takes to run -g all. > > As a maintainer, I prefer that these folks have done at least a small > > taste of QA before they start talking to the lists. > > A big problem for the drive-by'ers is that that the top-level xfstests > README file is just plain scary, and has far too many steps for a > drive-by patch author to follow. > > What I plan to add to a maintainer-entry-file.rst file for ext4 in the > kernel docs is to tell that drive-by posters that should run > "kvm-xfstests smoke" before submitting a patch, and setting up > kvm-xfstess is dead simple easy: > > > 1) Install kvm-xfstests --- you only have to run this once > > % git clone https://github.com/tytso/xfstests-bld fstests > % cd fstests > % make ; make install > > # Optional, if your file system you are developing isn't ext4; > # change f2fs to the file system of your choice > % echo PRIMARY_FSTYPE=f2fs >> ~/.config/kvm-xfstests > > > 2) Build the kernel suitable for use with kvm-xfstests > > % cd /path/to/your/kernel > % install-kconfig > % kbuild > > 3) Run the smoke test --- assuming the cwd is /path/to/your/kernel > > (Note: today this runs -g quick, but it would be good if this could be > faster) > > % kvm-xfstests smoke > > > It's simple, and since the kvm-xfstests script will download a > pre-compiled test appliance image automatically, there's no need to > require the drive-by tester to figure out how compile xfstests with > any of its prerequisites. > > And once things are set up, then it's just a matter of running > "kbuild" to build your kernel after you make changes, and running > "kvm-xfstests smoke" to do a quick smoke testing run. > > No muss, no fuss, no dirty dishes... :-) Hi Ted, Thanks for this detailed explanation! Ahaha, I'm just waiting for Darrick wake up, then ask him is there any requirement/context about this patch. Due to he (looks like) a bit hurry to push this patch :) If most of you prefer this way (an ./check option, not a separated wrapper script), I'm OK with that. Just recently I'm a bit worry about the ./check code, it's becoming more and more complex. I hope to separate something from it, but many things entwined, and growing. Anyway that's another story, I'll look into this patchset and review it soon. Thanks, Zorro > > Cheers, > > - Ted >
On Wed, Jul 26, 2023 at 10:54:41PM +0800, Zorro Lang wrote: > > Ahaha, I'm just waiting for Darrick wake up, then ask him is there any > requirement/context about this patch. Due to he (looks like) a bit > hurry to push this patch :) > > If most of you prefer this way (an ./check option, not a separated wrapper > script), I'm OK with that. I'm agnostic on that front, since I already *have* my own wrapper script. So if we need to do it in the wrapper script, I'm certainly OK with that. OTOH, if we think it's a feature which is generally interesting to multiple developers and/or test wrappers, maybe it makes sense to push things into the ./check sccript. So I certainly don't have any objections to adding support to my /root/runtests.sh so that "{gce,kvm,android}-xfstests smoke" gets ends up running the moral equivalent of: SOAK_DURATION=4m ./check -g smoketest ... and adding extra special case support in the check script just for this use case. I'm doing enough other stuff in runtests.sh[1] that it's really not a big deal for me. :-) [1] https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/root/runtests.sh More generally, there are some "intresting" hacks --- for example, I want to be able to run code in between every single test run, and the way I do it is a big ugly, but effective. I basically set LOGGER_PROG to my own special script, gce-logger[2] [2] https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/usr/local/lib/gce-logger and this allows the user to upload a script which will get run in between every single individual fstest (e.g., to grab information from BPF, or grab and reset lockstats, etc.). This script also updates the VM metadata so someone can query the VM to find out what test it's currently running, and the percentage completion for that VM. I could have asked for extra features in check, but whenever possible I try to work around it to limit the number of special things just for my set of wrapper scripts. > Just recently I'm a bit worry about the ./check code, it's becoming more > and more complex. I hope to separate something from it, but many things > entwined, and growing. Anyway that's another story, I'll look into this > patchset and review it soon. Well, I don't use the config sections feature at all, because my wrapper script has a lot more functionality than what you can get with the config sections, so I just pass in TEST_DEV, SCRATCH_DEV, MKFS_OPTIONS, etc., via environment variables, and I have my own set of scripts to set up te test parameters. So if you were going to simplify things by removing config sections, *I* wouldn't care. Enough other people might be using it that changing the fstests interface for this might raise a lot of objections from other folks, though. Cheers, - Ted
As an aside, while I was testing my updates to the kvm-quickstart[1] documentation, I timed how long it takes to run "-g quick" for a basic ext4 file system config with 4k blocks using a desktop NVMe SSD for the test and scratch devices. [1] https://github.com/tytso/xfstests-bld/blob/test/Documentation/kvm-quickstart.md It took 62 minutes, or a little over an hour. Yowza! I hadn't realized that "kvm-xfstests smoke" was now taking that long. It used to be that using a slower SSD (an Intel SATA-attached SSD dating from 2008) I could run "-g quick" in 15 minutes. Clearly, things were a lot simpler back then. :-) Anyway, I definitely need to replace what "kvm-xfstests smoke" does with something else much more abbrevuated before I start requesting drive-by patch submitters to run an fstests "smoke test". Because an hour isn't it. Ideally, I'd like to keep it under 10 minutes if at all possible, but we still want the testing to be likely to detect most of the sort of simple problems that a drive-by patch submitter might be likely find.... The fundamental question is how to do get the maximal amount of value given a limited test budget. - Ted
On Wed, Jul 26, 2023 at 09:36:27PM -0400, Theodore Ts'o wrote: > As an aside, while I was testing my updates to the kvm-quickstart[1] > documentation, I timed how long it takes to run "-g quick" for a basic > ext4 file system config with 4k blocks using a desktop NVMe SSD for > the test and scratch devices. > > [1] https://github.com/tytso/xfstests-bld/blob/test/Documentation/kvm-quickstart.md > > It took 62 minutes, or a little over an hour. Yowza! I hadn't > realized that "kvm-xfstests smoke" was now taking that long. It used > to be that using a slower SSD (an Intel SATA-attached SSD dating from > 2008) I could run "-g quick" in 15 minutes. Clearly, things were a > lot simpler back then. :-) > > Anyway, I definitely need to replace what "kvm-xfstests smoke" does > with something else much more abbrevuated before I start requesting > drive-by patch submitters to run an fstests "smoke test". Because an > hour isn't it. Ideally, I'd like to keep it under 10 minutes if at > all possible, but we still want the testing to be likely to detect > most of the sort of simple problems that a drive-by patch submitter > might be likely find.... > > The fundamental question is how to do get the maximal amount of value > given a limited test budget. Just out of curiosity, if you apply this patch and change kvm-xfstests smoke to run ./check -smoketest, how long does that actually take on your infrastructure? --D > - Ted
On Wed, Jul 26, 2023 at 04:59:30PM -0400, Theodore Ts'o wrote: > On Wed, Jul 26, 2023 at 10:54:41PM +0800, Zorro Lang wrote: > > > > Ahaha, I'm just waiting for Darrick wake up, then ask him is there any > > requirement/context about this patch. Due to he (looks like) a bit > > hurry to push this patch :) > > > > If most of you prefer this way (an ./check option, not a separated wrapper > > script), I'm OK with that. > > I'm agnostic on that front, since I already *have* my own wrapper > script. So if we need to do it in the wrapper script, I'm certainly > OK with that. OTOH, if we think it's a feature which is generally > interesting to multiple developers and/or test wrappers, maybe it > makes sense to push things into the ./check sccript. > > So I certainly don't have any objections to adding support to my > /root/runtests.sh so that "{gce,kvm,android}-xfstests smoke" gets ends > up running the moral equivalent of: > > SOAK_DURATION=4m ./check -g smoketest Hi Ted, Now we provide two ways to help to customize testing in fstests: 1) https://lore.kernel.org/fstests/20230727030529.r4ivp6dmtrht5zo2@zlang-mailbox/T/#mc5cdb59344f4cd681515bf0fab501d7f30f1e263 2) https://lore.kernel.org/fstests/169033660570.3222210.3010411210438664310.stgit@frogsfrogsfrogs/T/#u Which one do you like to use? I'd like to hear more review points before I choose one to merge. Thanks, Zorro > > ... and adding extra special case support in the check script just for > this use case. I'm doing enough other stuff in runtests.sh[1] that > it's really not a big deal for me. :-) > > [1] https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/root/runtests.sh > > > More generally, there are some "intresting" hacks --- for example, I > want to be able to run code in between every single test run, and the > way I do it is a big ugly, but effective. I basically set > > LOGGER_PROG to my own special script, gce-logger[2] > > [2] https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/usr/local/lib/gce-logger > > and this allows the user to upload a script which will get run in > between every single individual fstest (e.g., to grab information from > BPF, or grab and reset lockstats, etc.). This script also updates the > VM metadata so someone can query the VM to find out what test it's > currently running, and the percentage completion for that VM. > > I could have asked for extra features in check, but whenever possible > I try to work around it to limit the number of special things just for > my set of wrapper scripts. > > > > Just recently I'm a bit worry about the ./check code, it's becoming more > > and more complex. I hope to separate something from it, but many things > > entwined, and growing. Anyway that's another story, I'll look into this > > patchset and review it soon. > > Well, I don't use the config sections feature at all, because my > wrapper script has a lot more functionality than what you can get with > the config sections, so I just pass in TEST_DEV, SCRATCH_DEV, > MKFS_OPTIONS, etc., via environment variables, and I have my own set > of scripts to set up te test parameters. > > So if you were going to simplify things by removing config sections, > *I* wouldn't care. Enough other people might be using it that > changing the fstests interface for this might raise a lot of > objections from other folks, though. > > Cheers, > > - Ted >
On Thu, Jul 27, 2023 at 11:25:37AM +0800, Zorro Lang wrote: > > SOAK_DURATION=4m ./check -g smoketest > > Now we provide two ways to help to customize testing in fstests: > > 1) > https://lore.kernel.org/fstests/20230727030529.r4ivp6dmtrht5zo2@zlang-mailbox/T/#mc5cdb59344f4cd681515bf0fab501d7f30f1e263 > > 2) > https://lore.kernel.org/fstests/169033660570.3222210.3010411210438664310.stgit@frogsfrogsfrogs/T/#u > > Which one do you like to use? I'd like to hear more review points before I > choose one to merge. (1) is the "./check -t smoketest" option, and it provides a more generic way of adding new templates. On the positive side it allows more of this kind of simple "configuration" style options where "-t smoketest" is essentially syntactic sugar for: SOAK_DURATION=${SOAK_DURATION:-4m} ./check -g smoketest" The potential disadvantage of (1) is that it seems like extra complexity for what is really simple. (2) is "./check -smoketest" option. Its advantage is that it might easier for a drive-by patcher to type. The disadvantage is that it's adding Yet Another Option to the ./check script. I also will note that we have some "long options" which use a single hypen (e.g., -overlay and -udiff) but newer "long options" seem to use the double hypehn approach (e.g., --exact-order and --large-fs). My personal preference is for the newer GNU getopt style of using double hyphens, but the fact that we have both types of long options is... unfortunate. I guess I have a slight preference for (1), but I'm really not sure either is really necessary. My view is that for a drive-by tester, trying to set up xfstests is Too Hard. So the reality is they will be using some kind of wrapper script --- either one that they've written for their own, such as what Darrick (and I assume other XFS developers have their own), or they're using something like kdevops or kvm-xfstests. From *my* perspective, I have absolutely *no* problem with having my wrapper script use: SOACK_DURATION=4m ./check -g smoketest because I only have to do it once, and no end-user is ever going to see it. They will just use "kvm-xfstests smoke", and all of the magic will be hidden from them. The main advantage of having some kind of "official" top-level way of specifying the smoke test is that it makes it more likely that different wrapper scripts will converge on the same kind of smoke test, and it becomes easier for fstests developers to communicate with each other because the concept of what a "smoke test" is has been well defined in the fstests source code. And for that purpose, I think the "./check -t smoketest" approach works just fine. But really, I can live with either. :-) Cheers, - Ted
On Thu, Jul 27, 2023 at 10:33:26AM -0400, Theodore Ts'o wrote: > On Thu, Jul 27, 2023 at 11:25:37AM +0800, Zorro Lang wrote: > > > SOAK_DURATION=4m ./check -g smoketest > > > > Now we provide two ways to help to customize testing in fstests: > > > > 1) > > https://lore.kernel.org/fstests/20230727030529.r4ivp6dmtrht5zo2@zlang-mailbox/T/#mc5cdb59344f4cd681515bf0fab501d7f30f1e263 > > > > 2) > > https://lore.kernel.org/fstests/169033660570.3222210.3010411210438664310.stgit@frogsfrogsfrogs/T/#u > > > > Which one do you like to use? I'd like to hear more review points before I > > choose one to merge. > > (1) is the "./check -t smoketest" option, and it provides a more > generic way of adding new templates. On the positive side it allows > more of this kind of simple "configuration" style options where "-t > smoketest" is essentially syntactic sugar for: > > SOAK_DURATION=${SOAK_DURATION:-4m} ./check -g smoketest" > > The potential disadvantage of (1) is that it seems like extra > complexity for what is really simple. > > > (2) is "./check -smoketest" option. Its advantage is that it might > easier for a drive-by patcher to type. The disadvantage is that it's > adding Yet Another Option to the ./check script. > > I also will note that we have some "long options" which use a single > hypen (e.g., -overlay and -udiff) but newer "long options" seem to use > the double hypehn approach (e.g., --exact-order and --large-fs). My > personal preference is for the newer GNU getopt style of using double > hyphens, but the fact that we have both types of long options > is... unfortunate. Yeah, I'd like to tidy the ./check, include the option names. But change the check option format will affect many users, cause most of their scripts go wrong suddently, then they need to check and use new option format. That's why I still not touch this part. > > > I guess I have a slight preference for (1), but I'm really not sure > either is really necessary. My view is that for a drive-by tester, > trying to set up xfstests is Too Hard. So the reality is they will be > using some kind of wrapper script --- either one that they've written > for their own, such as what Darrick (and I assume other XFS developers > have their own), or they're using something like kdevops or > kvm-xfstests. Sure, you're right. fstests can be used to do a simple test, but for regular test, a wrapper is needed. Darrick has his wrapper, Dave might has his wrapper too. My team also has our wrapper, (we also have wrappers to run ltp and others). Different users might have different testing environment, so they might build different wrappers to connect fstests with their own environment/requirement. So I'd like to keep fstests as simple underlying test cases, provide basic testing functions to anyone who would like to run it, try to not limit much things. But I'd like to let fstests provides more help to each of your testing requirement. That's my current crude thought :-P > > From *my* perspective, I have absolutely *no* problem with having my > wrapper script use: > > SOACK_DURATION=4m ./check -g smoketest > > because I only have to do it once, and no end-user is ever going to > see it. They will just use "kvm-xfstests smoke", and all of the magic > will be hidden from them. > > The main advantage of having some kind of "official" top-level way of > specifying the smoke test is that it makes it more likely that > different wrapper scripts will converge on the same kind of smoke > test, and it becomes easier for fstests developers to communicate with > each other because the concept of what a "smoke test" is has been well > defined in the fstests source code. And for that purpose, I think the > "./check -t smoketest" approach works just fine. OK, thanks for your reply. I'll double check with Darrick, then merge one of them :) Thanks, Zorro > > But really, I can live with either. :-) > > Cheers, > > - Ted >
On Thu, Jul 27, 2023 at 11:30:46PM +0800, Zorro Lang wrote: > > Yeah, I'd like to tidy the ./check, include the option names. But change the > check option format will affect many users, cause most of their scripts go > wrong suddently, then they need to check and use new option format. That's > why I still not touch this part. What if we add support for the double-hyphen variant for those options that use a syingle-hyphen, and then wait say, a year or so, and then add a deprecation warning to the single hyphen options, and then a year latter drop support the single-hypen variant? I can send a patch for the first part of the process if you think that makes sense.... - Ted
On Fri, Jul 28, 2023 at 11:53:07AM -0400, Theodore Ts'o wrote: > On Thu, Jul 27, 2023 at 11:30:46PM +0800, Zorro Lang wrote: > > > > Yeah, I'd like to tidy the ./check, include the option names. But change the > > check option format will affect many users, cause most of their scripts go > > wrong suddently, then they need to check and use new option format. That's > > why I still not touch this part. > > What if we add support for the double-hyphen variant for those options > that use a syingle-hyphen, and then wait say, a year or so, and then > add a deprecation warning to the single hyphen options, and then a > year latter drop support the single-hypen variant? > > I can send a patch for the first part of the process if you think that > makes sense.... Sure, that makes sense to me. Let's take the first step, we'll see the influence :) And I'd like to use bash buildin getopts() or other standard ways to deal with ./check options, when we can replace those single-hyphen options totally. If you have any better idea, feel free to share with me :) BTW, several month ago I tried to check with you that: https://lore.kernel.org/fstests/20230207182656.ekjt2lhbid6xf2cs@zlang-mailbox/ Any thoughts? Thanks, Zorro > > - Ted >
diff --git a/check b/check index 89e7e7bf20..97c7c4c7d1 100755 --- a/check +++ b/check @@ -68,6 +68,7 @@ check options -pvfs2 test PVFS2 -tmpfs test TMPFS -ubifs test ubifs + -smoketest run smoke tests for 4min each -l line mode diff -udiff show unified diff (default) -n show me, do not run tests @@ -290,7 +291,10 @@ while [ $# -gt 0 ]; do FSTYP=overlay export OVERLAY=true ;; - + -smoketest) + SOAK_DURATION="4m" + GROUP_LIST="smoketest" + ;; -g) group=$2 ; shift ; GROUP_LIST="$GROUP_LIST ${group//,/ }" ;; diff --git a/doc/group-names.txt b/doc/group-names.txt index 1c35a39432..c3dcca3755 100644 --- a/doc/group-names.txt +++ b/doc/group-names.txt @@ -118,6 +118,7 @@ selftest tests with fixed results, used to validate testing setup send btrfs send/receive shrinkfs decreasing the size of a filesystem shutdown FS_IOC_SHUTDOWN ioctl +smoketest Simple smoke tests snapshot btrfs snapshots soak long running soak tests whose runtime can be controlled directly by setting the SOAK_DURATION variable diff --git a/tests/generic/475 b/tests/generic/475 index 0cbf5131c2..ce7fe013b1 100755 --- a/tests/generic/475 +++ b/tests/generic/475 @@ -12,7 +12,7 @@ # testing efforts. # . ./common/preamble -_begin_fstest shutdown auto log metadata eio recoveryloop +_begin_fstest shutdown auto log metadata eio recoveryloop smoketest # Override the default cleanup function. _cleanup() diff --git a/tests/generic/476 b/tests/generic/476 index 8e93b73457..b1ae4df4d4 100755 --- a/tests/generic/476 +++ b/tests/generic/476 @@ -8,7 +8,7 @@ # bugs in the write path. # . ./common/preamble -_begin_fstest auto rw long_rw stress soak +_begin_fstest auto rw long_rw stress soak smoketest # Override the default cleanup function. _cleanup() diff --git a/tests/generic/521 b/tests/generic/521 index 22dd31a8ec..0956e50171 100755 --- a/tests/generic/521 +++ b/tests/generic/521 @@ -7,7 +7,7 @@ # Long-soak directio fsx test # . ./common/preamble -_begin_fstest soak long_rw +_begin_fstest soak long_rw smoketest # Import common functions. . ./common/filter diff --git a/tests/generic/522 b/tests/generic/522 index f0cbcb245c..0e4e6009ed 100755 --- a/tests/generic/522 +++ b/tests/generic/522 @@ -7,7 +7,7 @@ # Long-soak buffered fsx test # . ./common/preamble -_begin_fstest soak long_rw +_begin_fstest soak long_rw smoketest # Import common functions. . ./common/filter diff --git a/tests/generic/642 b/tests/generic/642 index eba90903a3..e6a475a8b5 100755 --- a/tests/generic/642 +++ b/tests/generic/642 @@ -8,7 +8,7 @@ # bugs in the xattr code. # . ./common/preamble -_begin_fstest auto soak attr long_rw stress +_begin_fstest auto soak attr long_rw stress smoketest _cleanup() {