| Message ID | 20180505001951.5665-1-david@fromorbit.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Sat, May 5, 2018 at 3:19 AM, Dave Chinner <david@fromorbit.com> wrote: > From: Dave Chinner <dchinner@redhat.com> > > Also document the new way to run all tests (i.e. check -g all) and > clean up all the stray whitespace in the readme file. > > Signed-Off-By: Dave Chinner <dchinner@redhat.com> [...] > + and it excludes tests that exercise conditions known to cause machine > + failures (i.e. the "dangerous" tests). That would have been nice.. if it was implemented... I counted 22 dangerous tests, 20 of which are in auto group, unlike the 87 dangerous_* tests, none of which are in auto group. If running ./check would exclude 'dangerous' tests by default, should it also exclude dangerous_* tests? Or should we just remove the 20 'dangerous' tests from auto group and try not to add new auto&dangerous tests in the future? Thanks, Amir. -- To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sat, May 05, 2018 at 09:55:54AM +0300, Amir Goldstein wrote: > On Sat, May 5, 2018 at 3:19 AM, Dave Chinner <david@fromorbit.com> wrote: > > From: Dave Chinner <dchinner@redhat.com> > > > > Also document the new way to run all tests (i.e. check -g all) and > > clean up all the stray whitespace in the readme file. > > > > Signed-Off-By: Dave Chinner <dchinner@redhat.com> > [...] > > + and it excludes tests that exercise conditions known to cause machine > > + failures (i.e. the "dangerous" tests). > > That would have been nice.. if it was implemented... > > I counted 22 dangerous tests, 20 of which are in auto group, unlike the > 87 dangerous_* tests, none of which are in auto group. The 'dangerous' group and 'auto' group are not mutually exclusive when it was introduced, see commit 3f28d55c3954 ("add freeze and dangerous groups"). But from previous discussions[1][2], they should be mutually exclusive. [1] https://www.spinics.net/lists/fstests/msg08652.html [2] https://www.spinics.net/lists/fstests/msg08663.html > > If running ./check would exclude 'dangerous' tests by default, should it > also exclude dangerous_* tests? Or should we just remove the 20 > 'dangerous' tests from auto group and try not to add new auto&dangerous > tests in the future? I agreed, I think we should just remove the 20 'dangerous' tests from auto group (AFAICT, they are not dangerous anymore). And I'll avoid adding new tests that are in both dangerous & auto group. Thanks, Eryu -- To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sat, May 05, 2018 at 10:09:11PM +0800, Eryu Guan wrote: > t > The 'dangerous' group and 'auto' group are not mutually exclusive when > it was introduced, see commit 3f28d55c3954 ("add freeze and dangerouss > groups"). But from previous discussions[1][2], they should be mutually > exclusive. What might be nice is if there was some way to declare that a test is dangerous given a particular kernel version (or some combination of kernel version and file system type). Right now I have to manually exclude tests when testing stable kernels, e.g.: gce-xfstests full --kernel gs://gce-xfstests/bzImage-3.18 -X generic/269 -X ext4/022 It's not a huge deal, and it's been on my todo list to add into the {gce,kvm,android}-xfstests framework, but perhaps it belongs in the core xfstests framework. - Ted -- To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sat, May 05, 2018 at 10:09:11PM +0800, Eryu Guan wrote: > On Sat, May 05, 2018 at 09:55:54AM +0300, Amir Goldstein wrote: > > On Sat, May 5, 2018 at 3:19 AM, Dave Chinner <david@fromorbit.com> wrote: > > If running ./check would exclude 'dangerous' tests by default, should it > > also exclude dangerous_* tests? Or should we just remove the 20 > > 'dangerous' tests from auto group and try not to add new auto&dangerous > > tests in the future? > > I agreed, I think we should just remove the 20 'dangerous' tests from > auto group (AFAICT, they are not dangerous anymore). And I'll avoid > adding new tests that are in both dangerous & auto group. If the test is not dangerous anymore, then remove the dangerous tag from the test. FWIW: $ git grep auto tests/*group |grep dangerous tests/btrfs/group:150 auto quick dangerous tests/ext4/group:022 auto quick attr dangerous tests/ext4/group:024 auto quick encrypt dangerous tests/ext4/group:025 auto quick fuzzers dangerous tests/generic/group:068 other auto freeze dangerous stress tests/generic/group:280 auto quota freeze dangerous tests/generic/group:390 auto freeze stress dangerous tests/generic/group:406 auto quick dangerous tests/generic/group:421 auto quick encrypt dangerous tests/generic/group:446 auto quick rw dangerous tests/generic/group:459 auto dangerous tests/generic/group:463 auto quick clone dangerous tests/xfs/group:115 auto quick fuzzers dangerous tests/xfs/group:118 auto quick fsr dangerous tests/xfs/group:119 log v2log auto freeze dangerous tests/xfs/group:294 auto dir metadata dangerous tests/xfs/group:306 auto dangerous quick punch tests/xfs/group:311 auto dangerous quick tests/xfs/group:431 auto quick dangerous tests/xfs/group:438 auto quick quota dangerous $ sudo ./run_check.sh "" "" "-b -s xfs generic/068 generic/280 generic/390 generic/406 generic/421 generic/446 generic/459 generic/463 xfs/115 xfs/118 xfs/119 xfs/294 xfs/306 xfs/311 xfs/431 xfs/438" [....] SECTION -- xfs FSTYP -- xfs (debug) PLATFORM -- Linux/x86_64 test4 4.17.0-rc3-dgc+ MKFS_OPTIONS -- -f -m rmapbt=1,reflink=1 -i sparse=1 /dev/pmem1 MOUNT_OPTIONS -- /dev/pmem1 /mnt/scratch generic/068 49s ... 45s generic/280 2s ... 3s generic/390 25s ... 19s generic/406 2s ... 2s generic/421 [not run] No encryption support for xfs generic/446 13s ... 13s generic/459 15s ... 14s generic/463 5s ... 1s xfs/115 1s ... [not run] test requires XFS bug_on_assert to be off, turn it off to run the test xfs/118 15s ... 15s xfs/119 11s ... 12s xfs/294 58s ... 58s xfs/306 39s ... 41s xfs/311 23s ... 23s xfs/431 1s ... 1s xfs/438 5s ... 5s Passed all 16 tests I can't speak for the btrfs and ext4 tests or generic/421 (encryption test), but none of the other generic or xfs tests marked dangerous appear to be dangerous on a 4.17-rc3 kernel.... Cheers, Dave.
On Sat, May 05, 2018 at 06:23:47PM -0400, Theodore Y. Ts'o wrote: > On Sat, May 05, 2018 at 10:09:11PM +0800, Eryu Guan wrote: > > t > > The 'dangerous' group and 'auto' group are not mutually exclusive when > > it was introduced, see commit 3f28d55c3954 ("add freeze and dangerouss > > groups"). But from previous discussions[1][2], they should be mutually > > exclusive. > > What might be nice is if there was some way to declare that a test is > dangerous given a particular kernel version (or some combination of > kernel version and file system type). Right now I have to manually > exclude tests when testing stable kernels, e.g.: > > gce-xfstests full --kernel gs://gce-xfstests/bzImage-3.18 -X generic/269 -X ext4/022 That's what expunge files are for. Cheers, Dave.
diff --git a/README b/README index 50c68afaf0c2..73f057f3d403 100644 --- a/README +++ b/README @@ -22,7 +22,7 @@ _______________________ - create fsgqa test user ("sudo useradd fsgqa") - create fsgqa group ("sudo groupadd fsgqa") - create 123456-fsgqa test user ("sudo useradd 123456-fsgqa") - + ______________________ USING THE FSQA SUITE ______________________ @@ -36,11 +36,10 @@ Preparing system for tests: from http://www.extra.research.philips.com/udf/, then copy the udf_test binary to xfstests/src/. If you wish to disable UDF verification test set the environment variable DISABLE_UDF_TEST to 1. - - + - create one or two partitions to use for testing - one TEST partition - - format as XFS, mount & optionally populate with + - format as XFS, mount & optionally populate with NON-IMPORTANT stuff - one SCRATCH partition (optional) - leave empty and expect this partition to be clobbered @@ -54,14 +53,14 @@ Preparing system for tests: SCRATCH_DEV should be unused by the tester, and for the legacy support SCRATCH_DEV will be set to the first disk of the SCRATCH_DEV_POOL by xfstests script. - + - setup your environment Quick start: - copy local.config.example to local.config and edit as needed Or: - setenv TEST_DEV "device containing TEST PARTITION" - - setenv TEST_DIR "mount point of TEST PARTITION" - - optionally: + - setenv TEST_DIR "mount point of TEST PARTITION" + - optionally: - setenv SCRATCH_DEV "device containing SCRATCH PARTITION" OR (btrfs only) setenv SCRATCH_DEV_POOL "to 3 or more SCRATCH disks for testing btrfs raid concepts" @@ -109,18 +108,23 @@ Preparing system for tests: - if testing xfsdump, make sure the tape devices have a tape which can be overwritten. - + - make sure $TEST_DEV is a mounted XFS partition - make sure that $SCRATCH_DEV or $SCRATCH_DEV_POOL contains nothing useful - + Running tests: - cd xfstests - - By default the tests suite will run xfs tests: + - By default the tests suite will run all the tests in the auto group. These + are the tests that are expected to function correctly as regression tests, + and it excludes tests that exercise conditions known to cause machine + failures (i.e. the "dangerous" tests). - ./check '*/001' '*/002' '*/003' - ./check '*/06?' - Groups of tests maybe ran by: ./check -g [group(s)] See the 'group' file for details on groups + - If you want to run all tests regardless of what group they are in + (including dangerous tests), use the "all" group: ./check -g all - To randomize test order: ./check -r [test(s)] - You can explicitly specify NFS/CIFS/OVERLAY, otherwise the filesystem type will be autodetected from $TEST_DEV: @@ -130,16 +134,16 @@ Running tests: The TEST and SCRATCH partitions should be pre-formatted with another base fs, where the overlay dirs will be created - + The check script tests the return value of each script, and compares the output against the expected output. If the output is not as expected, a diff will be output and an .out.bad file will be produced for the failing test. - + Unexpected console messages, crashes and hangs may be considered to be failures but are not necessarily detected by the QA system. -__________________________ +__________________________ ADDING TO THE FSQA SUITE __________________________ @@ -168,11 +172,11 @@ Test script environment: $TEST_DEV. (b) mkfs a new XFS filesystem on $SCRATCH_DEV, and mount this - on $SCRATCH_MNT. Call the the _require_scratch function + on $SCRATCH_MNT. Call the the _require_scratch function on startup if you require use of the scratch partition. - _require_scratch does some checks on $SCRATCH_DEV & - $SCRATCH_MNT and makes sure they're unmounted. You should - cleanup when your test is done, and in particular unmount + _require_scratch does some checks on $SCRATCH_DEV & + $SCRATCH_MNT and makes sure they're unmounted. You should + cleanup when your test is done, and in particular unmount $SCRATCH_MNT. Tests can make use of $SCRATCH_LOGDEV and $SCRATCH_RTDEV for testing external log and realtime volumes - however,