| Message ID | 1486932224-17075-10-git-send-email-amir73il@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Sun, Feb 12, 2017 at 10:43:44PM +0200, Amir Goldstein wrote: > When TEST/SCRATCH_DEV are configured to the base fs block device, > use this information to mount base fs before running tests, > unmount it after running tests and cycle on _test_cycle_mount > along with the overlay mounts. > > This helps catching overlayfs bugs related to leaking objects in > underlying (base) fs. > > To preserve expected tests behavior, the semantics are: > - _scratch_mkfs mounts the base fs, cleans all files, creates > lower/upper dirs and keeps base fs mounted > - _scratch_mount mounts base fs (if needed) and mounts overlay > - _scratch_unmount unmounts overlay and base fs > > Tests that use _scratch_unmount to unmount a custom overlay mount > and expect to have access to overlay base dir, were fixed to use > explicit umount $SCRATCH_MNT instead. > > The overlay test itself, does not support formatting the base fs. > However, it is possible to add overlay sections to a multi section > config file after every base fs configuration, to run the overlay > tests with the file system that was created on the test partitions > in the previous section (see example in README.config-sections). > > Signed-off-by: Amir Goldstein <amir73il@gmail.com> > --- > README.config-sections | 6 ++++++ > common/rc | 55 ++++++++++++++++++++++++++++++++++++++++++++++++-- > tests/overlay/003 | 3 ++- > tests/overlay/004 | 3 ++- > tests/overlay/014 | 5 +++-- > 5 files changed, 66 insertions(+), 6 deletions(-) > > diff --git a/README.config-sections b/README.config-sections > index df7c929..d45d6da 100644 > --- a/README.config-sections > +++ b/README.config-sections > @@ -102,6 +102,9 @@ MKFS_OPTIONS="-q -F -b4096" > FSTYP=ext4 > RESULT_BASE="`pwd`/results/`date +%d%m%y_%H%M%S`" > > +[ext4_overlay] > +FSTYP=overlay > + > [ext4_1k_block_size] > MKFS_OPTIONS="-q -F -b1024" > > @@ -112,6 +115,9 @@ MKFS_OPTIONS="-q -F -b4096 -O ^has_journal" > MKFS_OPTIONS="-f" > FSTYP=xfs > > +[xfs_overlay] > +FSTYP=overlay > + > [ext3_filesystem] > FSTYP=ext3 > MOUNT_OPTIONS="-o noatime" > diff --git a/common/rc b/common/rc > index 74168ea..e9ac0df 100644 > --- a/common/rc > +++ b/common/rc > @@ -328,24 +328,70 @@ _overlay_mount() > $SELINUX_MOUNT_OPTIONS $* $dir $mnt > } > > +_overlay_base_test_mount() > +{ > + if [ -z "$OVL_BASE_TEST_DEV" -o -z "$OVL_BASE_TEST_DIR" ] || \ > + _check_mounted_on OVL_BASE_TEST_DEV $OVL_BASE_TEST_DEV \ > + OVL_BASE_TEST_DIR $OVL_BASE_TEST_DIR > + then > + return 0 > + fi > + > + _mount $OVL_BASE_MOUNT_OPTIONS \ OVL_BASE_MOUNT_OPTIONS is from MOUNT_OPTIONS, which is used for scratch mount. Do we need a ovl base counter part of TEST_FS_MOUNT_OPTS? Thanks, Eryu -- To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Feb 13, 2017 at 1:31 PM, Eryu Guan <eguan@redhat.com> wrote: > On Sun, Feb 12, 2017 at 10:43:44PM +0200, Amir Goldstein wrote: >> When TEST/SCRATCH_DEV are configured to the base fs block device, >> use this information to mount base fs before running tests, >> unmount it after running tests and cycle on _test_cycle_mount >> along with the overlay mounts. >> >> This helps catching overlayfs bugs related to leaking objects in >> underlying (base) fs. >> >> To preserve expected tests behavior, the semantics are: >> - _scratch_mkfs mounts the base fs, cleans all files, creates >> lower/upper dirs and keeps base fs mounted >> - _scratch_mount mounts base fs (if needed) and mounts overlay >> - _scratch_unmount unmounts overlay and base fs >> >> Tests that use _scratch_unmount to unmount a custom overlay mount >> and expect to have access to overlay base dir, were fixed to use >> explicit umount $SCRATCH_MNT instead. >> >> The overlay test itself, does not support formatting the base fs. >> However, it is possible to add overlay sections to a multi section >> config file after every base fs configuration, to run the overlay >> tests with the file system that was created on the test partitions >> in the previous section (see example in README.config-sections). >> >> Signed-off-by: Amir Goldstein <amir73il@gmail.com> >> --- >> README.config-sections | 6 ++++++ >> common/rc | 55 ++++++++++++++++++++++++++++++++++++++++++++++++-- >> tests/overlay/003 | 3 ++- >> tests/overlay/004 | 3 ++- >> tests/overlay/014 | 5 +++-- >> 5 files changed, 66 insertions(+), 6 deletions(-) >> >> diff --git a/README.config-sections b/README.config-sections >> index df7c929..d45d6da 100644 >> --- a/README.config-sections >> +++ b/README.config-sections >> @@ -102,6 +102,9 @@ MKFS_OPTIONS="-q -F -b4096" >> FSTYP=ext4 >> RESULT_BASE="`pwd`/results/`date +%d%m%y_%H%M%S`" >> >> +[ext4_overlay] >> +FSTYP=overlay >> + >> [ext4_1k_block_size] >> MKFS_OPTIONS="-q -F -b1024" >> >> @@ -112,6 +115,9 @@ MKFS_OPTIONS="-q -F -b4096 -O ^has_journal" >> MKFS_OPTIONS="-f" >> FSTYP=xfs >> >> +[xfs_overlay] >> +FSTYP=overlay >> + >> [ext3_filesystem] >> FSTYP=ext3 >> MOUNT_OPTIONS="-o noatime" >> diff --git a/common/rc b/common/rc >> index 74168ea..e9ac0df 100644 >> --- a/common/rc >> +++ b/common/rc >> @@ -328,24 +328,70 @@ _overlay_mount() >> $SELINUX_MOUNT_OPTIONS $* $dir $mnt >> } >> >> +_overlay_base_test_mount() >> +{ >> + if [ -z "$OVL_BASE_TEST_DEV" -o -z "$OVL_BASE_TEST_DIR" ] || \ >> + _check_mounted_on OVL_BASE_TEST_DEV $OVL_BASE_TEST_DEV \ >> + OVL_BASE_TEST_DIR $OVL_BASE_TEST_DIR >> + then >> + return 0 >> + fi >> + >> + _mount $OVL_BASE_MOUNT_OPTIONS \ > > OVL_BASE_MOUNT_OPTIONS is from MOUNT_OPTIONS, which is used for scratch > mount. Do we need a ovl base counter part of TEST_FS_MOUNT_OPTS? > I suggest that we change this line to: + _mount $TEST_FS_MOUNT_OPTS \ MOUNT_OPTIONS in config is used for base scratch mount options TEST_FS_MOUNT_OPTS in config is used for base test mount options. OVERLAY_MOUNT_OPTIONS in config is used for both scratch and test overlay mounts The only case where TEST_FS_MOUNT_OPTS is configured not by user is for cifs and cheph (in _test_mount_opts) and I don't think that those fs can be used as overlay base fs. can they? Good enough? -- To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, Feb 12, 2017 at 10:43:44PM +0200, Amir Goldstein wrote: > When TEST/SCRATCH_DEV are configured to the base fs block device, > use this information to mount base fs before running tests, > unmount it after running tests and cycle on _test_cycle_mount > along with the overlay mounts. So I don't normally use the multi-section config file --- and I confess the exact semantics of the config file, and how things are inherited across sections has always been confusing to me (there are bits of README.config-sections which seem to be mutually contradictory[1]). So apologies in advance, but when you have something like this: > diff --git a/README.config-sections b/README.config-sections > index df7c929..d45d6da 100644 > --- a/README.config-sections > +++ b/README.config-sections > @@ -102,6 +102,9 @@ MKFS_OPTIONS="-q -F -b4096" > FSTYP=ext4 > RESULT_BASE="`pwd`/results/`date +%d%m%y_%H%M%S`" > > +[ext4_overlay] > +FSTYP=overlay > + > [ext4_1k_block_size] > MKFS_OPTIONS="-q -F -b1024" How do you know that the base file system type should be ext4? Is it because the previous file system type in a previous config section (OLD_FSTYP in the check script) is ext4? If so, what happens if the first sectionis FSTYP=overlay. Also, what happens previous sections are skipped? Suppose you have a config file that looks like this: [ext4] ... FSTYP=ext4 [ext4_overlay] FSTYP=overlay [xfs] ... FSTYP=xfs [xfs_overlay] FSTYP=overlay What is supposed to happen (and what does happen) if the user runs: ./check -s xfs_overlay or ./check -s ext4 -s xfs_overlay ? Speaking more generally, I'm not a big fan of the config file approach for handling iterations, because of the fact that previous sections will have side effects on follow-on sections, and I'm interested in adding support for test sharding, where different file system test scenarios are run on different GCE VM's, and the ambiguities of how variables are carried over from one section to another makes life hard. It also makes it hard to have multiple file system developers editing a single config file since you have to worry about side effects. Having separate files and separate directories for differnt file system types means that patch collisions are much less likely to have unanticipated side effects, or cause merge conflicts for that matter. I recognize that the local config file is not something that is intended to be managed centrally, but I acutally *like* the fact that I can separate file system test scenarios (and where I want to have a common understanding across ext4 file system developers for what the "bigalloc_1k" test scenario means), from the details of the local hardware configuration. All of this being said, I doubt I'll be able to convince others about changing how the local config system works. I do want to be sure I understand what are the supported way of testing overlayfs (e.g., will the "deprecated" way continue to work forever, or is it going to disappear eventually), and while I'd prefer to not have to play games with the config file if I want to test using overlayfs, if I *do* get forced to do it, it would be useful if there were a bit more explicit description of how things like the mkfs mount options, etc. are side effected by previous config sections, and how to set up overlayfs correctly using such a scheme. (e.g., more documentation than just an a few lines demonstration of what might go in the config file without any detailed semantic explanation of how it all works.) - Ted [1] The ambiguity I was taking about. In one part of the README.config-state file, it states: Note that options are carried between sections so the same options does not have to be specified in each and every sections. However caution should be exercised not to leave unwanted options set from previous sections. (What does this mean when stanzas are skipped?) and later on, it says this: Multiple file systems --------------------- Having different file systems in different config sections is allowed. When FSTYP differs in the following section the FSTYP file system will be created automatically before running the test. Note that if MOUNT_OPTIONS, MKFS_OPTIONS, or FSCK_OPTIONS are not directly specified in the section it will be reset to the default for a given file system. This seems to imply that configuration paramters such as MKFS_OPTIONS do *not* carry over from one config section to another, and is in direct contradiction to the above paragraph. -- To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Feb 13, 2017 at 07:23:56PM -0500, Theodore Ts'o wrote: [snip] > Speaking more generally, I'm not a big fan of the config file approach > for handling iterations, because of the fact that previous sections > will have side effects on follow-on sections, and I'm interested in > adding support for test sharding, where different file system test > scenarios are run on different GCE VM's, and the ambiguities of how > variables are carried over from one section to another makes life hard. I agreed, the section configs are not perfect, sometimes I have SCRATCH_RTDEV "leaked" to other non-rt test sections. I was considering fixing this problem, but haven't got enough time to figure out a reasonable idea yet. > > It also makes it hard to have multiple file system developers editing > a single config file since you have to worry about side effects. > Having separate files and separate directories for differnt file > system types means that patch collisions are much less likely to have > unanticipated side effects, or cause merge conflicts for that matter. > I recognize that the local config file is not something that is > intended to be managed centrally, but I acutally *like* the fact that > I can separate file system test scenarios (and where I want to have a > common understanding across ext4 file system developers for what the > "bigalloc_1k" test scenario means), from the details of the local > hardware configuration. > > All of this being said, I doubt I'll be able to convince others about > changing how the local config system works. I do want to be sure I > understand what are the supported way of testing overlayfs (e.g., will > the "deprecated" way continue to work forever, or is it going to > disappear eventually), and while I'd prefer to not have to play games I'm considering making the old way unsupported eventually at some point, after a long enough soak time. But if people still want it, I can live with it too :) > with the config file if I want to test using overlayfs, if I *do* get > forced to do it, it would be useful if there were a bit more explicit With this update, things get simpler if you're not using multi-section config file, there's an example in commit log of patch 6/9, README is also updated, but seems we need that kind of detailed example in README too. " For example, the following config file can be used to run tests on xfs test/scratch partitions: TEST_DEV=/dev/sda5 TEST_DIR=/mnt/test SCRATCH_DEV=/dev/sda6 SCRATCH_MNT=/mnt/scratch FSTYP=xfs Using the same config file, but executing './check -overlay'... " > description of how things like the mkfs mount options, etc. are side > effected by previous config sections, and how to set up overlayfs FWIW, Multi-section configs are optional not mandatory for testing overlayfs. But I agreed that we need more documentation. > correctly using such a scheme. (e.g., more documentation than just an > a few lines demonstration of what might go in the config file without > any detailed semantic explanation of how it all works.) > > - Ted > > [1] The ambiguity I was taking about. In one part of the > README.config-state file, it states: > > Note that options are carried between sections so the same options does not > have to be specified in each and every sections. However caution should be > exercised not to leave unwanted options set from previous sections. > > (What does this mean when stanzas are skipped?) > > and later on, it says this: > > Multiple file systems > --------------------- > > Having different file systems in different config sections is allowed. When > FSTYP differs in the following section the FSTYP file system will be created > automatically before running the test. > > Note that if MOUNT_OPTIONS, MKFS_OPTIONS, or FSCK_OPTIONS are not directly > specified in the section it will be reset to the default for a given file > system. > > This seems to imply that configuration paramters such as MKFS_OPTIONS > do *not* carry over from one config section to another, and is in Yes, four paramters are not carried across sections, they're unset at the beginning of each section. In common/config we have unset MOUNT_OPTIONS unset MKFS_OPTIONS unset FSCK_OPTIONS unset USE_EXTERNAL Thanks, Eryu -- To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Feb 14, 2017 at 2:23 AM, Theodore Ts'o <tytso@mit.edu> wrote: > On Sun, Feb 12, 2017 at 10:43:44PM +0200, Amir Goldstein wrote: >> When TEST/SCRATCH_DEV are configured to the base fs block device, >> use this information to mount base fs before running tests, >> unmount it after running tests and cycle on _test_cycle_mount >> along with the overlay mounts. > > So I don't normally use the multi-section config file --- and I > confess the exact semantics of the config file, and how things are > inherited across sections has always been confusing to me (there are > bits of README.config-sections which seem to be mutually > contradictory[1]). So apologies in advance, but when you have > something like this: > Indeed. Multi section config does not give the tester super power. Tester needs to know what she is doing. As I replied on the linux-fsdevel thread, I found out that interleaving overlay sections post non-overlay sections work and thought it may be useful for some, so I added the example. I do not intend to fix problems with multi section config + overlay so I may as well remove that example and leave this a useful but undocumented feature. Having written that disclaimer, I will go on to answer your questions. >> diff --git a/README.config-sections b/README.config-sections >> index df7c929..d45d6da 100644 >> --- a/README.config-sections >> +++ b/README.config-sections >> @@ -102,6 +102,9 @@ MKFS_OPTIONS="-q -F -b4096" >> FSTYP=ext4 >> RESULT_BASE="`pwd`/results/`date +%d%m%y_%H%M%S`" >> >> +[ext4_overlay] >> +FSTYP=overlay >> + >> [ext4_1k_block_size] >> MKFS_OPTIONS="-q -F -b1024" > > How do you know that the base file system type should be ext4? Is it > because the previous file system type in a previous config section > (OLD_FSTYP in the check script) is ext4? > Correct. The only thing that overlay does not do is format the base fs, as the README says: - for overlay tests: ./check -overlay [test(s)] The TEST and SCRATCH partitions should be pre-formatted with another base fs, where the overlay dirs will be created > If so, what happens if the first sectionis FSTYP=overlay. Also, what > happens previous sections are skipped? Suppose you have a config file > that looks like this: > > [ext4] > ... > FSTYP=ext4 > > [ext4_overlay] > FSTYP=overlay > > [xfs] > ... > FSTYP=xfs > > [xfs_overlay] > FSTYP=overlay > > What is supposed to happen (and what does happen) if the user runs: > > ./check -s xfs_overlay > > or > > ./check -s ext4 -s xfs_overlay > > ? You know what will happen :-) only bad things. > Speaking more generally, I'm not a big fan of the config file approach > for handling iterations, because of the fact that previous sections > will have side effects on follow-on sections, and I'm interested in > adding support for test sharding, where different file system test > scenarios are run on different GCE VM's, and the ambiguities of how > variables are carried over from one section to another makes life hard. > > It also makes it hard to have multiple file system developers editing > a single config file since you have to worry about side effects. > Having separate files and separate directories for differnt file > system types means that patch collisions are much less likely to have > unanticipated side effects, or cause merge conflicts for that matter. > I recognize that the local config file is not something that is > intended to be managed centrally, but I acutally *like* the fact that > I can separate file system test scenarios (and where I want to have a > common understanding across ext4 file system developers for what the > "bigalloc_1k" test scenario means), from the details of the local > hardware configuration. > It seems to me that multi section config file is not the right tool for the job to test many types of file systems, but it may be useful for testing different MKFS/MOUNT options for the same FSTYP. For that purpose, if interleaving overlay section is helpful, it could be used, but what won't work correctly is './check -overlay' because it will not mkfs base fs when changing sections. Actually, it may not be that hard to add support for mkfs of base fs so check -overlay on multi section config will work, but I did not know if that was an important use case. Therefore, the feedback from prospect overlay testers and the configurations that they use is important. What would be your config if you were to test overlay? P.S. I still did not get around to migrate kvm-xfstests to use new config, but I will. For now patch 8/9 breaks some tests in kvm-xfstests (old config) so I still need to fix that. Amir. -- To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Feb 14, 2017 at 08:43:36AM +0200, Amir Goldstein wrote: > > Correct. The only thing that overlay does not do is format the base fs, > as the README says: > - for overlay tests: ./check -overlay [test(s)] > The TEST and SCRATCH partitions should be pre-formatted > with another base fs, where the overlay dirs will be created > So this means that if I only want to do the equivalent of: gce-xfstests -c overlay generic/013 as part of a bisection search, it's impossible to do this via the config file and "./check -s ext4-overlay generic/013" alone. Someone who wants to do this will have to use their own wrapper script to format the base file system with the proper file system. Correct? Which is fine, I have my own wrapper script system. :-) - Ted -- To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Feb 14, 2017 at 7:07 PM, Theodore Ts'o <tytso@mit.edu> wrote: > On Tue, Feb 14, 2017 at 08:43:36AM +0200, Amir Goldstein wrote: >> >> Correct. The only thing that overlay does not do is format the base fs, >> as the README says: >> - for overlay tests: ./check -overlay [test(s)] >> The TEST and SCRATCH partitions should be pre-formatted >> with another base fs, where the overlay dirs will be created >> > > So this means that if I only want to do the equivalent of: > > gce-xfstests -c overlay generic/013 > > as part of a bisection search, it's impossible to do this via the > config file and "./check -s ext4-overlay generic/013" alone. Someone > who wants to do this will have to use their own wrapper script to > format the base file system with the proper file system. > > Correct? > Correct. I guess it wouldn't be that hard to add mkfs/fsck support for base fs, but for now my budget to deal with this is over. I may get to it another time. > Which is fine, I have my own wrapper script system. :-) > Do we have something like: ./check -s ext4 --setup-only Should we have something like this? Will this do the trick? ./check -s ext4 -n generic/013 -- To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Feb 14, 2017 at 7:55 PM, Amir Goldstein <amir73il@gmail.com> wrote: > On Tue, Feb 14, 2017 at 7:07 PM, Theodore Ts'o <tytso@mit.edu> wrote: >> On Tue, Feb 14, 2017 at 08:43:36AM +0200, Amir Goldstein wrote: >>> >>> Correct. The only thing that overlay does not do is format the base fs, >>> as the README says: >>> - for overlay tests: ./check -overlay [test(s)] >>> The TEST and SCRATCH partitions should be pre-formatted >>> with another base fs, where the overlay dirs will be created >>> >> >> So this means that if I only want to do the equivalent of: >> >> gce-xfstests -c overlay generic/013 >> >> as part of a bisection search, it's impossible to do this via the >> config file and "./check -s ext4-overlay generic/013" alone. Someone >> who wants to do this will have to use their own wrapper script to >> format the base file system with the proper file system. >> >> Correct? >> > > Correct. > > I guess it wouldn't be that hard to add mkfs/fsck support for base fs, > but for now my budget to deal with this is over. I may get to it another > time. > >> Which is fine, I have my own wrapper script system. :-) >> > > Do we have something like: > ./check -s ext4 --setup-only > > Should we have something like this? > > Will this do the trick? > ./check -s ext4 -n generic/013 FYI, for upcoming V4, I recalled the support for overlay sections (i.e. it is an undocumented feature), but verified that multi section can run with -overlay, as long as the sections only change mount options, e.g.: [xfs] TEST_DEV=/dev/sda5 TEST_DIR=/mnt/test SCRATCH_DEV=/dev/sda6 SCRATCH_MNT=/mnt/scratch MKFS_OPTIONS="-f -m rmapbt=1,reflink=1" FSTYP=xfs [xfs_pquota] MOUNT_OPTIONS="-o pquota" TEST_FS_MOUNT_OPTS="-o noatime" OVERLAY_MOUNT_OPTIONS="-o redirect_dir=off" I will collect some of these example in README.overlay -- To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/README.config-sections b/README.config-sections index df7c929..d45d6da 100644 --- a/README.config-sections +++ b/README.config-sections @@ -102,6 +102,9 @@ MKFS_OPTIONS="-q -F -b4096" FSTYP=ext4 RESULT_BASE="`pwd`/results/`date +%d%m%y_%H%M%S`" +[ext4_overlay] +FSTYP=overlay + [ext4_1k_block_size] MKFS_OPTIONS="-q -F -b1024" @@ -112,6 +115,9 @@ MKFS_OPTIONS="-q -F -b4096 -O ^has_journal" MKFS_OPTIONS="-f" FSTYP=xfs +[xfs_overlay] +FSTYP=overlay + [ext3_filesystem] FSTYP=ext3 MOUNT_OPTIONS="-o noatime" diff --git a/common/rc b/common/rc index 74168ea..e9ac0df 100644 --- a/common/rc +++ b/common/rc @@ -328,24 +328,70 @@ _overlay_mount() $SELINUX_MOUNT_OPTIONS $* $dir $mnt } +_overlay_base_test_mount() +{ + if [ -z "$OVL_BASE_TEST_DEV" -o -z "$OVL_BASE_TEST_DIR" ] || \ + _check_mounted_on OVL_BASE_TEST_DEV $OVL_BASE_TEST_DEV \ + OVL_BASE_TEST_DIR $OVL_BASE_TEST_DIR + then + return 0 + fi + + _mount $OVL_BASE_MOUNT_OPTIONS \ + $SELINUX_MOUNT_OPTIONS \ + $OVL_BASE_TEST_DEV $OVL_BASE_TEST_DIR +} + _overlay_test_mount() { - _overlay_mount $OVL_BASE_TEST_DIR $TEST_DIR $* + _overlay_base_test_mount && \ + _overlay_mount $OVL_BASE_TEST_DIR $TEST_DIR $* +} + +_overlay_base_scratch_mount() +{ + if [ -z "$OVL_BASE_SCRATCH_DEV" -o -z "$OVL_BASE_SCRATCH_MNT" ] || \ + _check_mounted_on OVL_BASE_SCRATCH_DEV $OVL_BASE_SCRATCH_DEV \ + OVL_BASE_SCRATCH_MNT $OVL_BASE_SCRATCH_MNT + then + return 0 + fi + + _mount $OVL_BASE_MOUNT_OPTIONS \ + $SELINUX_MOUNT_OPTIONS \ + $OVL_BASE_SCRATCH_DEV $OVL_BASE_SCRATCH_MNT +} + +_overlay_base_scratch_unmount() +{ + [ -n "$OVL_BASE_SCRATCH_DEV" -a -n "$OVL_BASE_SCRATCH_MNT" ] || return 0 + + $UMOUNT_PROG $OVL_BASE_SCRATCH_MNT } _overlay_scratch_mount() { - _overlay_mount $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT $* + _overlay_base_scratch_mount && \ + _overlay_mount $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT $* +} + +_overlay_base_test_unmount() +{ + [ -n "$OVL_BASE_TEST_DEV" -a -n "$OVL_BASE_TEST_DIR" ] || return 0 + + $UMOUNT_PROG $OVL_BASE_TEST_DIR } _overlay_test_unmount() { $UMOUNT_PROG $TEST_DIR + _overlay_base_test_unmount } _overlay_scratch_unmount() { $UMOUNT_PROG $SCRATCH_MNT + _overlay_base_scratch_unmount } _scratch_mount() @@ -652,7 +698,12 @@ _scratch_cleanup_files() overlay) # Avoid rm -rf /* if we messed up [ -n "$OVL_BASE_SCRATCH_MNT" ] || return 1 + # overlay 'mkfs' needs to make sure base fs is mounted and clean + _overlay_base_scratch_unmount 2>/dev/null + _overlay_base_scratch_mount rm -rf $OVL_BASE_SCRATCH_MNT/* + _overlay_mkdirs $OVL_BASE_SCRATCH_MNT + # leave base fs mouted so tests can setup lower/upper dir files ;; *) [ -n "$SCRATCH_MNT" ] || return 1 diff --git a/tests/overlay/003 b/tests/overlay/003 index 7c7f41b..f980edb 100755 --- a/tests/overlay/003 +++ b/tests/overlay/003 @@ -89,7 +89,8 @@ rm -rf ${SCRATCH_MNT}/* # nothing should be listed ls ${SCRATCH_MNT}/ -_scratch_unmount +# unmount overlayfs but not base fs +$UMOUNT_PROG $SCRATCH_MNT rm -rf $lowerdir echo "Silence is golden" diff --git a/tests/overlay/004 b/tests/overlay/004 index bc08f34..611847a 100755 --- a/tests/overlay/004 +++ b/tests/overlay/004 @@ -85,7 +85,8 @@ _user_do "chmod g+t ${SCRATCH_MNT}/attr_file2 > /dev/null 2>&1" _user_do "chmod u-X ${SCRATCH_MNT}/attr_file2 > /dev/null 2>&1" stat -c %a ${SCRATCH_MNT}/attr_file2 -_scratch_unmount +# unmount overlayfs but not base fs +$UMOUNT_PROG $SCRATCH_MNT # check mode bits of the file that has been copied up, and # the file that should not have been copied up. diff --git a/tests/overlay/014 b/tests/overlay/014 index 491f735..653b7d6 100755 --- a/tests/overlay/014 +++ b/tests/overlay/014 @@ -73,7 +73,8 @@ mkdir -p $lowerdir1/testdir/d _overlay_mount_dirs $lowerdir1 $lowerdir2 $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT rm -rf $SCRATCH_MNT/testdir mkdir -p $SCRATCH_MNT/testdir/visibledir -_scratch_unmount +# unmount overlayfs but not base fs +$UMOUNT_PROG $SCRATCH_MNT # mount overlay again, with lowerdir1 and lowerdir2 as multiple lowerdirs, # and create a new file in testdir, triggers copyup from lowerdir, @@ -84,7 +85,7 @@ touch $SCRATCH_MNT/testdir/visiblefile # umount and mount overlay again, buggy kernel treats the copied-up dir as # opaque, visibledir is not seen in merged dir. -_scratch_unmount +$UMOUNT_PROG $SCRATCH_MNT _overlay_mount_dirs "$lowerdir2:$lowerdir1" $upperdir $workdir \ $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT ls $SCRATCH_MNT/testdir
When TEST/SCRATCH_DEV are configured to the base fs block device, use this information to mount base fs before running tests, unmount it after running tests and cycle on _test_cycle_mount along with the overlay mounts. This helps catching overlayfs bugs related to leaking objects in underlying (base) fs. To preserve expected tests behavior, the semantics are: - _scratch_mkfs mounts the base fs, cleans all files, creates lower/upper dirs and keeps base fs mounted - _scratch_mount mounts base fs (if needed) and mounts overlay - _scratch_unmount unmounts overlay and base fs Tests that use _scratch_unmount to unmount a custom overlay mount and expect to have access to overlay base dir, were fixed to use explicit umount $SCRATCH_MNT instead. The overlay test itself, does not support formatting the base fs. However, it is possible to add overlay sections to a multi section config file after every base fs configuration, to run the overlay tests with the file system that was created on the test partitions in the previous section (see example in README.config-sections). Signed-off-by: Amir Goldstein <amir73il@gmail.com> --- README.config-sections | 6 ++++++ common/rc | 55 ++++++++++++++++++++++++++++++++++++++++++++++++-- tests/overlay/003 | 3 ++- tests/overlay/004 | 3 ++- tests/overlay/014 | 5 +++-- 5 files changed, 66 insertions(+), 6 deletions(-)