| Message ID | 1309275199-10801-5-git-send-email-josef@redhat.com (mailing list archive) |
|---|---|
| State | Not Applicable |
| Headers | show |
On Tue, Jun 28, 2011 at 11:33:19AM -0400, Josef Bacik wrote: > This is a test to make sure seek_data/seek_hole is acting like it does on > Solaris. It will check to see if the fs supports finding a hole or not and will > adjust as necessary. So I just looked at this with an eye to validating an XFS implementation, and I came up with this list of stuff that the test does not cover that I'd need to test in some way: - files with clean unwritten extents. Are they a hole or data? What's SEEK_DATA supposed to return on layout like hole-unwritten-data? i.e. needs to add fallocate to the picture... - files with dirty unwritten extents (i.e. dirty in memory, not on disk). They are most definitely data, and most filesystems will need a separate lookup path to detect dirty unwritten ranges because the state is kept separately (page cache vs extent cache). Plenty of scope for filesystem specific bugs here so needs a roubust test. - cold cache behaviour - all dirty data ranges the test creates are hot in cache and not even forced to disk, so it is not testing the no-page-cache-over-the-data-range case. i.e. it tests delalloc state tracking but not data-extent-already exists lookups during a seek. - assumes that allocation size is the block size and that holes follows block size alignment. We already know that ext4 does not follow that rule when doing small sparse writes close together in a file, and XFS is also known to fill holes when doing sparse writes past EOF. - only tests single block data extents ?o doesn't cover corner cases like skipping over multiple fragmented data extents to the next hole. Some more comments in line.... > +_cleanup() > +{ > + rm -f $tmp.* > +} > + > +trap "_cleanup ; exit \$status" 0 1 2 3 15 > + > +# get standard environment, filters and checks > +. ./common.rc > +. ./common.filter > + > +# real QA test starts here > +_supported_fs generic > +_supported_os Linux > + > +testfile=$TEST_DIR/seek_test.$$ > +logfile=$TEST_DIR/seek_test.$$.log The log file is usually named $seq.full, and doesn't get placed in the filesystem being tested. It gets saved in the xfstests directory along side $seq.out.bad for analysis whenteh test fails... > +[ -x $here/src/seek-tester ] || _notrun "seek-tester not built" > + > +_cleanup() > +{ > + rm -f $testfile > + rm -f $logfile > +} > +trap "_cleanup; exit \$status" 0 1 2 3 15 > + > +echo "Silence is golden" > +$here/src/seek-tester -q $testfile 2>&1 | tee -a $logfile Personally I'd prefer the test to be a bit noisy about what it is running, especially when there are so many subtests the single invocation is running. It makes no difference to the run time ofthe test, or the output when something fails, but it at least allows you to run the test manually and see what it is doing easily... > + > +if grep -q "SEEK_HOLE is not supported" $logfile; then > + _notrun "SEEK_HOLE/SEEK_DATA not supported by this kernel" > +fi > + > +rm -f $logfile > +rm -f $testfile > + > +status=0 ; exit > diff --git a/255.out b/255.out > new file mode 100644 > index 0000000..7eefb82 > --- /dev/null > +++ b/255.out > @@ -0,0 +1,2 @@ > +QA output created by 255 > +Silence is golden > diff --git a/group b/group > index 1f86075..c045e70 100644 > --- a/group > +++ b/group > @@ -368,3 +368,4 @@ deprecated > 252 auto quick prealloc > 253 auto quick > 254 auto quick > +255 auto quick I'd suggest that rw and prealloc (once unwritten extent testing is added) groups should also be defined for this test. Otherwise, the test code looks ok if a bit over-engineered.... > +struct testrec { > + int test_num; > + int (*test_func)(int fd, int testnum); > + char *test_desc; > +}; > + > +struct testrec seek_tests[] = { > + { 1, test01, "Test basic support" }, > + { 2, test02, "Test an empty file" }, > + { 3, test03, "Test a full file" }, > + { 4, test04, "Test file hole at beg, data at end" }, > + { 5, test05, "Test file data at beg, hole at end" }, > + { 6, test06, "Test file hole data hole data" }, So, to take from the hole punch test matrix, it covers a bunch more file state transitions and cases that are just as relevant to SEEK_HOLE/SEEK_DATA. Those cases are: # 1. into a hole # 2. into allocated space # 3. into unwritten space # 4. hole -> data # 5. hole -> unwritten # 6. data -> hole # 7. data -> unwritten # 8. unwritten -> hole # 9. unwritten -> data # 10. hole -> data -> hole # 11. data -> hole -> data # 12. unwritten -> data -> unwritten # 13. data -> unwritten -> data # 14. data -> hole @ EOF # 15. data -> hole @ 0 # 16. data -> cache cold ->hole # 17. data -> hole in single block file I thikn we also need to cover most of these same cases, right? And SEEK_HOLE/SEEK data also need to explicitly separate the unwritten tests into "clean unwritten" and "dirty unwritten" and cover the transitions between regions of those states as well, right? Cheers, Dave.
On Wed, Jun 29, 2011 at 04:53:07PM +1000, Dave Chinner wrote: > On Tue, Jun 28, 2011 at 11:33:19AM -0400, Josef Bacik wrote: > > This is a test to make sure seek_data/seek_hole is acting like it does on > > Solaris. It will check to see if the fs supports finding a hole or not and will > > adjust as necessary. > > So I just looked at this with an eye to validating an XFS > implementation, and I came up with this list of stuff that the test > does not cover that I'd need to test in some way: > > - files with clean unwritten extents. Are they a hole or > data? What's SEEK_DATA supposed to return on layout like > hole-unwritten-data? i.e. needs to add fallocate to the > picture... > > - files with dirty unwritten extents (i.e. dirty in memory, > not on disk). They are most definitely data, and most > filesystems will need a separate lookup path to detect > dirty unwritten ranges because the state is kept > separately (page cache vs extent cache). Plenty of scope > for filesystem specific bugs here so needs a roubust test. The discussion leading up to the resurrection of SEEK_HOLE/SEEK_DATA was pretty much about that point. The conclusion based on the Sun documentation and common sense was that SEEK_DATA may only consider unwritten extents as hole if the filesystem has a way to distinguish plain unwritten extents and those that have been dirtied. Else it should be considered data. Testing for making sure dirty preallocated areas aren't wrongly reported sounds relatively easy, the rest falls into implementation details, which imho is fine. Not reporting preallocated extents as holes just is a quality of implementation issue and not a bug. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 29/06/11 08:40, Christoph Hellwig wrote: > On Wed, Jun 29, 2011 at 04:53:07PM +1000, Dave Chinner wrote: >> On Tue, Jun 28, 2011 at 11:33:19AM -0400, Josef Bacik wrote: >>> This is a test to make sure seek_data/seek_hole is acting like it does on >>> Solaris. It will check to see if the fs supports finding a hole or not and will >>> adjust as necessary. >> >> So I just looked at this with an eye to validating an XFS >> implementation, and I came up with this list of stuff that the test >> does not cover that I'd need to test in some way: >> >> - files with clean unwritten extents. Are they a hole or >> data? What's SEEK_DATA supposed to return on layout like >> hole-unwritten-data? i.e. needs to add fallocate to the >> picture... >> >> - files with dirty unwritten extents (i.e. dirty in memory, >> not on disk). They are most definitely data, and most >> filesystems will need a separate lookup path to detect >> dirty unwritten ranges because the state is kept >> separately (page cache vs extent cache). Plenty of scope >> for filesystem specific bugs here so needs a roubust test. > > The discussion leading up to the resurrection of SEEK_HOLE/SEEK_DATA > was pretty much about that point. The conclusion based on the Sun > documentation and common sense was that SEEK_DATA may only consider > unwritten extents as hole if the filesystem has a way to distinguish > plain unwritten extents and those that have been dirtied. Else it > should be considered data. > > Testing for making sure dirty preallocated areas aren't wrongly > reported sounds relatively easy, the rest falls into implementation > details, which imho is fine. Not reporting preallocated extents > as holes just is a quality of implementation issue and not a bug. There is the argument, that if this interface can distinguish these dirty unwritten extents, then why can't the fiemap interface too? The advantage of the fiemap interface is that it can distinguish empty extents vs holes. Empty extents will become increasingly common I think, given the fragmentation and space guarantee benefits they give. It would be cool for cp for example to be able to efficiently copy empty extents from source to dest. cheers, Pádraig. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 06/29/2011 02:53 AM, Dave Chinner wrote: > On Tue, Jun 28, 2011 at 11:33:19AM -0400, Josef Bacik wrote: >> This is a test to make sure seek_data/seek_hole is acting like it does on >> Solaris. It will check to see if the fs supports finding a hole or not and will >> adjust as necessary. > > So I just looked at this with an eye to validating an XFS > implementation, and I came up with this list of stuff that the test > does not cover that I'd need to test in some way: > > - files with clean unwritten extents. Are they a hole or > data? What's SEEK_DATA supposed to return on layout like > hole-unwritten-data? i.e. needs to add fallocate to the > picture... > > - files with dirty unwritten extents (i.e. dirty in memory, > not on disk). They are most definitely data, and most > filesystems will need a separate lookup path to detect > dirty unwritten ranges because the state is kept > separately (page cache vs extent cache). Plenty of scope > for filesystem specific bugs here so needs a roubust test. > > - cold cache behaviour - all dirty data ranges the test > creates are hot in cache and not even forced to disk, so > it is not testing the no-page-cache-over-the-data-range > case. i.e. it tests delalloc state tracking but not > data-extent-already exists lookups during a seek. > > - assumes that allocation size is the block size and that > holes follows block size alignment. We already know that > ext4 does not follow that rule when doing small sparse > writes close together in a file, and XFS is also known to > fill holes when doing sparse writes past EOF. > > - only tests single block data extents ?o doesn't cover > corner cases like skipping over multiple fragmented data > extents to the next hole. > Yeah I intentionally left out preallocated stuff because these are going to be implementation specific, so I was going to leave that for a later exercise when people actually start doing proper implementations. > Some more comments in line.... > >> +_cleanup() >> +{ >> + rm -f $tmp.* >> +} >> + >> +trap "_cleanup ; exit \$status" 0 1 2 3 15 >> + >> +# get standard environment, filters and checks >> +. ./common.rc >> +. ./common.filter >> + >> +# real QA test starts here >> +_supported_fs generic >> +_supported_os Linux >> + >> +testfile=$TEST_DIR/seek_test.$$ >> +logfile=$TEST_DIR/seek_test.$$.log > > The log file is usually named $seq.full, and doesn't get placed in > the filesystem being tested. It gets saved in the xfstests directory > along side $seq.out.bad for analysis whenteh test fails... > I only want it to see if SEEK_HOLE fails so I can say it didn't run. I followed the same example as the fiemap test that Eric wrote. >> +[ -x $here/src/seek-tester ] || _notrun "seek-tester not built" >> + >> +_cleanup() >> +{ >> + rm -f $testfile >> + rm -f $logfile >> +} >> +trap "_cleanup; exit \$status" 0 1 2 3 15 >> + >> +echo "Silence is golden" >> +$here/src/seek-tester -q $testfile 2>&1 | tee -a $logfile > > Personally I'd prefer the test to be a bit noisy about what it is > running, especially when there are so many subtests the single > invocation is running. It makes no difference to the run time ofthe > test, or the output when something fails, but it at least allows you > to run the test manually and see what it is doing easily... > Right, the problem with this test is it will run differently depending on the implementation. I agree, I really like the noisy output tests, but unfortunately if I run this test on ext4 where it currently treats the entire file as data, and then run it on btrfs where it is smarter and actually recognizes holes, we end up with two different outputs that are both correct. >> + >> +if grep -q "SEEK_HOLE is not supported" $logfile; then >> + _notrun "SEEK_HOLE/SEEK_DATA not supported by this kernel" >> +fi >> + >> +rm -f $logfile >> +rm -f $testfile >> + >> +status=0 ; exit >> diff --git a/255.out b/255.out >> new file mode 100644 >> index 0000000..7eefb82 >> --- /dev/null >> +++ b/255.out >> @@ -0,0 +1,2 @@ >> +QA output created by 255 >> +Silence is golden >> diff --git a/group b/group >> index 1f86075..c045e70 100644 >> --- a/group >> +++ b/group >> @@ -368,3 +368,4 @@ deprecated >> 252 auto quick prealloc >> 253 auto quick >> 254 auto quick >> +255 auto quick > > I'd suggest that rw and prealloc (once unwritten extent > testing is added) groups should also be defined for this test. > > Otherwise, the test code looks ok if a bit over-engineered.... > >> +struct testrec { >> + int test_num; >> + int (*test_func)(int fd, int testnum); >> + char *test_desc; >> +}; >> + >> +struct testrec seek_tests[] = { >> + { 1, test01, "Test basic support" }, >> + { 2, test02, "Test an empty file" }, >> + { 3, test03, "Test a full file" }, >> + { 4, test04, "Test file hole at beg, data at end" }, >> + { 5, test05, "Test file data at beg, hole at end" }, >> + { 6, test06, "Test file hole data hole data" }, > > So, to take from the hole punch test matrix, it covers a bunch more > file state transitions and cases that are just as relevant to > SEEK_HOLE/SEEK_DATA. Those cases are: > > # 1. into a hole > # 2. into allocated space > # 3. into unwritten space > # 4. hole -> data > # 5. hole -> unwritten > # 6. data -> hole > # 7. data -> unwritten > # 8. unwritten -> hole > # 9. unwritten -> data > # 10. hole -> data -> hole > # 11. data -> hole -> data > # 12. unwritten -> data -> unwritten > # 13. data -> unwritten -> data > # 14. data -> hole @ EOF > # 15. data -> hole @ 0 > # 16. data -> cache cold ->hole > # 17. data -> hole in single block file > > I thikn we also need to cover most of these same cases, right? And > SEEK_HOLE/SEEK data also need to explicitly separate the unwritten > tests into "clean unwritten" and "dirty unwritten" and cover the > transitions between regions of those states as well, right? > Yeah you are right, but again doing preallocated stuff is tricky, but I can expand it now if that's what we want. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 06/29/2011 12:40 AM, Christoph Hellwig wrote: > On Wed, Jun 29, 2011 at 04:53:07PM +1000, Dave Chinner wrote: >> On Tue, Jun 28, 2011 at 11:33:19AM -0400, Josef Bacik wrote: >>> This is a test to make sure seek_data/seek_hole is acting like it does on >>> Solaris. It will check to see if the fs supports finding a hole or not and will >>> adjust as necessary. >> So I just looked at this with an eye to validating an XFS >> implementation, and I came up with this list of stuff that the test >> does not cover that I'd need to test in some way: >> >> - files with clean unwritten extents. Are they a hole or >> data? What's SEEK_DATA supposed to return on layout like >> hole-unwritten-data? i.e. needs to add fallocate to the >> picture... >> >> - files with dirty unwritten extents (i.e. dirty in memory, >> not on disk). They are most definitely data, and most >> filesystems will need a separate lookup path to detect >> dirty unwritten ranges because the state is kept >> separately (page cache vs extent cache). Plenty of scope >> for filesystem specific bugs here so needs a roubust test. > The discussion leading up to the resurrection of SEEK_HOLE/SEEK_DATA > was pretty much about that point. The conclusion based on the Sun > documentation and common sense was that SEEK_DATA may only consider > unwritten extents as hole if the filesystem has a way to distinguish > plain unwritten extents and those that have been dirtied. Else it > should be considered data. > > Testing for making sure dirty preallocated areas aren't wrongly > reported sounds relatively easy, the rest falls into implementation > details, which imho is fine. Not reporting preallocated extents > as holes just is a quality of implementation issue and not a bug. I agree. And if I might add my 2 cents that it would be much easier if we added another test that created files with all the worrisome boundary conditions and used SEEK_DATA/HOLE to copy the files and compared using md5sum. This would be far easier than one that expects a certain pos for each operation. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 06/29/2011 03:42 AM, Pádraig Brady wrote: > There is the argument, that if this interface can distinguish > these dirty unwritten extents, then why can't the fiemap interface too? > The advantage of the fiemap interface is that it can distinguish > empty extents vs holes. Empty extents will become increasingly common > I think, given the fragmentation and space guarantee benefits they give. > It would be cool for cp for example to be able to efficiently copy > empty extents from source to dest. I'm not too sure about that. Atleast not enabled by default. Most users use cp to backup data. Not empty space. In this case, this empty extent may not even be de-dupable. Frankly I'd be happier of cp started to exploited fallocate() to create larger extents before copying data into them. Atleast for the large files. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Jun 29, 2011 at 10:29:02AM -0700, Sunil Mushran wrote: > I'm not too sure about that. Atleast not enabled by default. Most users > use cp to backup data. Not empty space. In this case, this empty extent > may not even be de-dupable. > > Frankly I'd be happier of cp started to exploited fallocate() to create larger > extents before copying data into them. Atleast for the large files. That's what delayed allocation is for. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 06/29/2011 10:36 AM, Christoph Hellwig wrote: > On Wed, Jun 29, 2011 at 10:29:02AM -0700, Sunil Mushran wrote: >> I'm not too sure about that. Atleast not enabled by default. Most users >> use cp to backup data. Not empty space. In this case, this empty extent >> may not even be de-dupable. >> >> Frankly I'd be happier of cp started to exploited fallocate() to create larger >> extents before copying data into them. Atleast for the large files. > That's what delayed allocation is for. A feature fewer file systems support than fallocate(). ;) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 06/29/2011 01:10 PM, Sunil Mushran wrote: > On 06/29/2011 12:40 AM, Christoph Hellwig wrote: >> On Wed, Jun 29, 2011 at 04:53:07PM +1000, Dave Chinner wrote: >>> On Tue, Jun 28, 2011 at 11:33:19AM -0400, Josef Bacik wrote: >>>> This is a test to make sure seek_data/seek_hole is acting like it >>>> does on >>>> Solaris. It will check to see if the fs supports finding a hole or >>>> not and will >>>> adjust as necessary. >>> So I just looked at this with an eye to validating an XFS >>> implementation, and I came up with this list of stuff that the test >>> does not cover that I'd need to test in some way: >>> >>> - files with clean unwritten extents. Are they a hole or >>> data? What's SEEK_DATA supposed to return on layout like >>> hole-unwritten-data? i.e. needs to add fallocate to the >>> picture... >>> >>> - files with dirty unwritten extents (i.e. dirty in memory, >>> not on disk). They are most definitely data, and most >>> filesystems will need a separate lookup path to detect >>> dirty unwritten ranges because the state is kept >>> separately (page cache vs extent cache). Plenty of scope >>> for filesystem specific bugs here so needs a roubust test. >> The discussion leading up to the resurrection of SEEK_HOLE/SEEK_DATA >> was pretty much about that point. The conclusion based on the Sun >> documentation and common sense was that SEEK_DATA may only consider >> unwritten extents as hole if the filesystem has a way to distinguish >> plain unwritten extents and those that have been dirtied. Else it >> should be considered data. >> >> Testing for making sure dirty preallocated areas aren't wrongly >> reported sounds relatively easy, the rest falls into implementation >> details, which imho is fine. Not reporting preallocated extents >> as holes just is a quality of implementation issue and not a bug. > > I agree. And if I might add my 2 cents that it would be much easier > if we added another test that created files with all the worrisome boundary > conditions and used SEEK_DATA/HOLE to copy the files and compared > using md5sum. This would be far easier than one that expects a certain > pos for each operation. That's a great point, I think I will rig something like that up. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 29/06/11 18:29, Sunil Mushran wrote: > On 06/29/2011 03:42 AM, Pádraig Brady wrote: >> There is the argument, that if this interface can distinguish >> these dirty unwritten extents, then why can't the fiemap interface too? >> The advantage of the fiemap interface is that it can distinguish >> empty extents vs holes. Empty extents will become increasingly common >> I think, given the fragmentation and space guarantee benefits they give. >> It would be cool for cp for example to be able to efficiently copy >> empty extents from source to dest. > > I'm not too sure about that. Atleast not enabled by default. Most users > use cp to backup data. Not empty space. In this case, this empty extent > may not even be de-dupable. That's a fair point. On the other hand if you wanted to start working with the backup copy, you might want it allocated to avoid fragmentation and ENOSPC issues. What we were going with was empty -> hole with cp --sparse=always and empty -> empty otherwise. If empty and hole can not be distinguished though, then this process will be impacted. > > Frankly I'd be happier of cp started to exploited fallocate() to create > larger > extents before copying data into them. Atleast for the large files. Yes we definitely will start doing that. That will help fragmentation and give early ENOSPC. We can't use fiemap for this at the moment (on XSF or ext4 (without a sync)) but the seek_data interface should allow us to do this to some extent (pardon the pun). cheers, Pádraig. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Jun 29, 2011 at 11:42:38AM +0100, P?draig Brady wrote: > There is the argument, that if this interface can distinguish > these dirty unwritten extents, then why can't the fiemap interface too? > The advantage of the fiemap interface is that it can distinguish > empty extents vs holes. Empty extents will become increasingly common > I think, given the fragmentation and space guarantee benefits they give. > It would be cool for cp for example to be able to efficiently copy > empty extents from source to dest. That brings us back to square one. FIEMAP is supposed to tell you about the physical layout on disk. Unwritten extents physically always are there, but whether they might have to be copied depends entirely on in-core state. Finding that incore state in addition is not all that easy compared to simply walking the extents. People might decide it's worth for an interface like SEEK_HOLE specificly asking for that, but grafting it into FIEMAP through the backdoor is a horrible idea. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Jun 28, 2011 at 11:33:19AM -0400, Josef Bacik wrote: > This is a test to make sure seek_data/seek_hole is acting like it does on > Solaris. It will check to see if the fs supports finding a hole or not and will > adjust as necessary. Can you resend this with any updates that happened in the meantime? Dave also still had some comments about semantics, so it might be worth to incorporate that as well. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Aug 25, 2011 at 02:06:32AM -0400, Christoph Hellwig wrote: > On Tue, Jun 28, 2011 at 11:33:19AM -0400, Josef Bacik wrote: > > This is a test to make sure seek_data/seek_hole is acting like it does on > > Solaris. It will check to see if the fs supports finding a hole or not and will > > adjust as necessary. > > Can you resend this with any updates that happened in the meantime? > > Dave also still had some comments about semantics, so it might be worth > to incorporate that as well. The main questions I had when looking at this was how we should handle unwritten extents - the only answer I got was along the lines of "we'll deal with that once filesystems have implemented something". That's a bit of a chicken-and-egg situation, and doesn't help me decide what is the best thing to do. I don't want to have to re-implement this code when it's decided i did the wrong thing initially. The most basic question I really want answered is this: - is preallocated space a HOLE, or is it DATA? Whatever the answer, I think it should be consistently presented by all filesystems that support preallocation, and it should be encoded into the generic SEEK_HOLE/SEEK_DATA tests.... Answering that question then helps answer the more complex questions I had, like: - what does SEEK_DATA return when you have a file layout like "hole-prealloc-data"? Answers to that sort of question need to be known so we can write corner-case tests to correctly verify the filesystem implementation. I like to do better than present userspace with an interface that behaves vastly different depending on the underlying filesystem, but if the answer is "definition and implementation is entirely filesystem specific" then I'll just go make something up.... Cheers, Dave.
On 2011-08-25, at 12:40 AM, Dave Chinner wrote: > On Thu, Aug 25, 2011 at 02:06:32AM -0400, Christoph Hellwig wrote: >> On Tue, Jun 28, 2011 at 11:33:19AM -0400, Josef Bacik wrote: >>> This is a test to make sure seek_data/seek_hole is acting like it does on >>> Solaris. It will check to see if the fs supports finding a hole or not and will adjust as necessary. >> >> Can you resend this with any updates that happened in the meantime? >> >> Dave also still had some comments about semantics, so it might be worth >> to incorporate that as well. > > The main questions I had when looking at this was how we should > handle unwritten extents - the only answer I got was along the lines > of "we'll deal with that once filesystems have implemented > something". That's a bit of a chicken-and-egg situation, and doesn't > help me decide what is the best thing to do. I don't want to have to > re-implement this code when it's decided i did the wrong thing > initially. Let's first clarify what you mean by an unwritten extent? Do you mean a preallocated extent that returns 0 when read, or do you mean a delayed allocation extent that was written by the application that is still in memory but not yet written to disk? Unfortunately, ZFS has no concept of preallocated extents, so we can't look to it for precedent, but it definitely has delayed allocation. Possibly if UFS on Solaris has SEEK_HOLE and also preallocated extents (I have no idea) it could be tested? > The most basic question I really want answered is this: > > - is preallocated space a HOLE, or is it DATA? > > Whatever the answer, I think it should be consistently > presented by all filesystems that support preallocation, and it > should be encoded into the generic SEEK_HOLE/SEEK_DATA tests.... My thought would be that a preallocated extent is still a HOLE, because it doesn't contain data that an application actually cares about. On the other hand, a delalloc extent is DATA because it has something that an application cares about. The original reason SEEK_HOLE/SEEK_DATA were brought up over FIEMAP was "cp" being able to consistently access delalloc data that was only in the page cache, so if we don't get that right it will have been a pointless exercise. > Answering that question then helps answer the more complex questions > I had, like: > > - what does SEEK_DATA return when you have a file layout > like "hole-prealloc-data"? I would think only the "data" part, since that is what the definition of "SEEK_DATA" is IMHO. > Answers to that sort of question need to be known so we can write > corner-case tests to correctly verify the filesystem implementation. > > I like to do better than present userspace with an interface that > behaves vastly different depending on the underlying filesystem, but > if the answer is "definition and implementation is entirely > filesystem specific" then I'll just go make something up.... > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > -- > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html Cheers, Andreas -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Aug 25, 2011 at 12:51:56AM -0600, Andreas Dilger wrote: > On 2011-08-25, at 12:40 AM, Dave Chinner wrote: > > On Thu, Aug 25, 2011 at 02:06:32AM -0400, Christoph Hellwig wrote: > >> On Tue, Jun 28, 2011 at 11:33:19AM -0400, Josef Bacik wrote: > >>> This is a test to make sure seek_data/seek_hole is acting like it does on > >>> Solaris. It will check to see if the fs supports finding a hole or not and will adjust as necessary. > >> > >> Can you resend this with any updates that happened in the meantime? > >> > >> Dave also still had some comments about semantics, so it might be worth > >> to incorporate that as well. > > > > The main questions I had when looking at this was how we should > > handle unwritten extents - the only answer I got was along the lines > > of "we'll deal with that once filesystems have implemented > > something". That's a bit of a chicken-and-egg situation, and doesn't > > help me decide what is the best thing to do. I don't want to have to > > re-implement this code when it's decided i did the wrong thing > > initially. > > Let's first clarify what you mean by an unwritten extent? Do you mean a > preallocated extent that returns 0 when read, Exactly that. > or do you mean a delayed > allocation extent that was written by the application that is still in > memory but not yet written to disk? That's not an unwritten extent - that's a delayed allocation extent ;) > Unfortunately, ZFS has no concept of preallocated extents, so we can't > look to it for precedent, but it definitely has delayed allocation. > Possibly if UFS on Solaris has SEEK_HOLE and also preallocated extents > (I have no idea) it could be tested? > > > The most basic question I really want answered is this: > > > > - is preallocated space a HOLE, or is it DATA? > > > > Whatever the answer, I think it should be consistently > > presented by all filesystems that support preallocation, and it > > should be encoded into the generic SEEK_HOLE/SEEK_DATA tests.... > > My thought would be that a preallocated extent is still a HOLE, because > it doesn't contain data that an application actually cares about. On > the other hand, a delalloc extent is DATA because it has something that > an application cares about. OK, that's the way I'd expect to treat both preallocated and delalloc space. > > Answering that question then helps answer the more complex questions > > I had, like: > > > > - what does SEEK_DATA return when you have a file layout > > like "hole-prealloc-data"? > > I would think only the "data" part, since that is what the definition > of "SEEK_DATA" is IMHO. Agreed, that's the way I'd interpret it, too. So perhaps we need to ensure that this interpretation is actually tested by this test? How about some definitions to work by: Data: a range of the file that contains valid data, regardless of whether it exists in memory or on disk. The valid data can be preceeded and/or followed by an arbitrary number of zero bytes dependent on the underlying implementation of hole detection. Hole: a range of the file that contains no data or is made up entirely of NULL (zero) data. Holes include preallocated ranges of files that have not had actual data written to them. Does that make sense? It has sufficient flexibility in it for the existing generic "non-implementation", allows for filesystems to define their own hole detection boundaries (e.g. filesystem block size), and effectively defines how preallocated ranges from fallocate() should be treated (i.e. as holes). If we can agree on those definitions, I think that we should document them in both the kernel and the man page that defines SEEK_HOLE/SEEK_DATA so everyone is on the same page... Cheers, Dave.
2011/8/26 Dave Chinner <david@fromorbit.com>: > On Thu, Aug 25, 2011 at 12:51:56AM -0600, Andreas Dilger wrote: >> On 2011-08-25, at 12:40 AM, Dave Chinner wrote: >> > On Thu, Aug 25, 2011 at 02:06:32AM -0400, Christoph Hellwig wrote: >> >> On Tue, Jun 28, 2011 at 11:33:19AM -0400, Josef Bacik wrote: >> >>> This is a test to make sure seek_data/seek_hole is acting like it does on >> >>> Solaris. It will check to see if the fs supports finding a hole or not and will adjust as necessary. >> >> >> >> Can you resend this with any updates that happened in the meantime? >> >> >> >> Dave also still had some comments about semantics, so it might be worth >> >> to incorporate that as well. >> > >> > The main questions I had when looking at this was how we should >> > handle unwritten extents - the only answer I got was along the lines >> > of "we'll deal with that once filesystems have implemented >> > something". That's a bit of a chicken-and-egg situation, and doesn't >> > help me decide what is the best thing to do. I don't want to have to >> > re-implement this code when it's decided i did the wrong thing >> > initially. >> >> Let's first clarify what you mean by an unwritten extent? Do you mean a >> preallocated extent that returns 0 when read, > > Exactly that. > >> or do you mean a delayed >> allocation extent that was written by the application that is still in >> memory but not yet written to disk? > > That's not an unwritten extent - that's a delayed allocation extent ;) > >> Unfortunately, ZFS has no concept of preallocated extents, so we can't >> look to it for precedent, but it definitely has delayed allocation. >> Possibly if UFS on Solaris has SEEK_HOLE and also preallocated extents >> (I have no idea) it could be tested? >> >> > The most basic question I really want answered is this: >> > >> > - is preallocated space a HOLE, or is it DATA? >> > >> > Whatever the answer, I think it should be consistently >> > presented by all filesystems that support preallocation, and it >> > should be encoded into the generic SEEK_HOLE/SEEK_DATA tests.... >> >> My thought would be that a preallocated extent is still a HOLE, because >> it doesn't contain data that an application actually cares about. On >> the other hand, a delalloc extent is DATA because it has something that >> an application cares about. > > OK, that's the way I'd expect to treat both preallocated and > delalloc space. > >> > Answering that question then helps answer the more complex questions >> > I had, like: >> > >> > - what does SEEK_DATA return when you have a file layout >> > like "hole-prealloc-data"? >> >> I would think only the "data" part, since that is what the definition >> of "SEEK_DATA" is IMHO. > > Agreed, that's the way I'd interpret it, too. So perhaps we need to > ensure that this interpretation is actually tested by this test? > > How about some definitions to work by: > > Data: a range of the file that contains valid data, regardless of > whether it exists in memory or on disk. The valid data can be > preceeded and/or followed by an arbitrary number of zero bytes > dependent on the underlying implementation of hole detection. > > Hole: a range of the file that contains no data or is made up > entirely of NULL (zero) data. Holes include preallocated ranges of > files that have not had actual data written to them. > > Does that make sense? It has sufficient flexibility in it for the No for me. A hole is made up of zero data? It's a strange definition for me. A hole is simply no data. With this definition a hole can be start from a block with data but partially filled, and fs should check instead of presence of a block, the data content. I don't like it very much. For the fallocate case, we need only a simple rule: no check is done beyond eof. At this point it's clear that we can check the file block allocation till i_size, don't care if there are other blocks allocated. I don't know if it's applicable even for unwritten extents case. Marco -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Il 26/08/2011 16:41, Zach Brown ha scritto: >>> Hole: a range of the file that contains no data or is made up >>> entirely of NULL (zero) data. Holes include preallocated ranges of >>> files that have not had actual data written to them. > >> No for me. A hole is made up of zero data? It's a strange definition >> for me. > > It's a very natural definition for me. It mirrors the behaviour of > read() of sparse data inside i_size that file system authors already > have to consider. > > It's also a reminder for people that this interface is about avoiding > reading zeros. Systems that track contents can do this for files that > had tons of zeros written. The data is there but the app is > specifically asking us to skip it by using SEEK_DATA. > > - z > I think we need to consider a hole and "data not present/not written yet" as two different cases even they are related. For example, if I do an fallocate without keep size option, then I do a read, I have the same behavior of sparse data inside i_size, but the blocks are allocated so no sparse data in this case. Simply there are no difference from app point of view. Marco -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Il 27/08/2011 10:30, Marco Stornelli ha scritto: > Il 26/08/2011 16:41, Zach Brown ha scritto: >>>> Hole: a range of the file that contains no data or is made up >>>> entirely of NULL (zero) data. Holes include preallocated ranges of >>>> files that have not had actual data written to them. >> >>> No for me. A hole is made up of zero data? It's a strange definition >>> for me. >> >> It's a very natural definition for me. It mirrors the behaviour of >> read() of sparse data inside i_size that file system authors already >> have to consider. >> >> It's also a reminder for people that this interface is about avoiding >> reading zeros. Systems that track contents can do this for files that >> had tons of zeros written. The data is there but the app is >> specifically asking us to skip it by using SEEK_DATA. >> >> - z >> > > I think we need to consider a hole and "data not present/not written > yet" as two different cases even they are related. For example, if I do > an fallocate without keep size option, then I do a read, I have the same > behavior of sparse data inside i_size, but the blocks are allocated so > no sparse data in this case. Simply there are no difference from app > point of view. > > Marco Please don't care about the last part, when reading in this case the app will have a return value different from zero obviously, I was under the effect of a beer :) However I'd add to the definition, that we consider holes only inside i_size, as Zack said. In this way, we haven't got any ambiguity for preallocated space beyond eof. Marco -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 08/27/2011 01:30 AM, Marco Stornelli wrote: > Il 26/08/2011 16:41, Zach Brown ha scritto: >>>> Hole: a range of the file that contains no data or is made up >>>> entirely of NULL (zero) data. Holes include preallocated ranges of >>>> files that have not had actual data written to them. >> >>> No for me. A hole is made up of zero data? It's a strange definition >>> for me. >> >> It's a very natural definition for me. It mirrors the behaviour of >> read() of sparse data inside i_size that file system authors already >> have to consider. >> >> It's also a reminder for people that this interface is about avoiding >> reading zeros. Systems that track contents can do this for files that >> had tons of zeros written. The data is there but the app is >> specifically asking us to skip it by using SEEK_DATA. >> >> - z >> > > I think we need to consider a hole and "data not present/not written yet" as two different cases even they are related. For example, if I do an fallocate without keep size option, then I do a read, I have the same behavior of sparse data inside i_size, but the blocks are allocated so no sparse data in this case. Simply there are no difference from app point of view. Exactly. That's why seek_hole should identify them alike, if possible. But that should not be a requirement because the sole aim here is to improve performance. Identifying a hole as data is not the end of the world. In some cases it may be more efficient. We just have to ensure that we don't identify data as a hole. BTW, we still have the fiemap interface that allows users to identify unwritten extents, etc. Use that if you want that kind of detail. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 08/25/2011 06:35 PM, Dave Chinner wrote: > Agreed, that's the way I'd interpret it, too. So perhaps we need to > ensure that this interpretation is actually tested by this test? > > How about some definitions to work by: > > Data: a range of the file that contains valid data, regardless of > whether it exists in memory or on disk. The valid data can be > preceeded and/or followed by an arbitrary number of zero bytes > dependent on the underlying implementation of hole detection. > > Hole: a range of the file that contains no data or is made up > entirely of NULL (zero) data. Holes include preallocated ranges of > files that have not had actual data written to them. > > Does that make sense? It has sufficient flexibility in it for the > existing generic "non-implementation", allows for filesystems to > define their own hole detection boundaries (e.g. filesystem block > size), and effectively defines how preallocated ranges from > fallocate() should be treated (i.e. as holes). If we can agree on > those definitions, I think that we should document them in both the > kernel and the man page that defines SEEK_HOLE/SEEK_DATA so everyone > is on the same page... We should not tie in the definition to existing fs technologies. Instead we should let the fs weigh the cost of providing accurate information with the possible gain in performance. Data: A range in a file that could contain something other than nulls. If in doubt, it is data. Hole: A range in a file that only contains nulls. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Aug 30, 2011 at 06:17:02PM -0700, Sunil Mushran wrote: > On 08/25/2011 06:35 PM, Dave Chinner wrote: > >Agreed, that's the way I'd interpret it, too. So perhaps we need to > >ensure that this interpretation is actually tested by this test? > > > >How about some definitions to work by: > > > >Data: a range of the file that contains valid data, regardless of > >whether it exists in memory or on disk. The valid data can be > >preceeded and/or followed by an arbitrary number of zero bytes > >dependent on the underlying implementation of hole detection. > > > >Hole: a range of the file that contains no data or is made up > >entirely of NULL (zero) data. Holes include preallocated ranges of > >files that have not had actual data written to them. > > > >Does that make sense? It has sufficient flexibility in it for the > >existing generic "non-implementation", allows for filesystems to > >define their own hole detection boundaries (e.g. filesystem block > >size), and effectively defines how preallocated ranges from > >fallocate() should be treated (i.e. as holes). If we can agree on > >those definitions, I think that we should document them in both the > >kernel and the man page that defines SEEK_HOLE/SEEK_DATA so everyone > >is on the same page... > > We should not tie in the definition to existing fs technologies. Such as? If we don't use well known, well defined terminology, we end up with ambiguous, vague functionality and inconsistent implementations. > Instead > we should let the fs weigh the cost of providing accurate information > with the possible gain in performance. > > Data: > A range in a file that could contain something other than nulls. > If in doubt, it is data. > > Hole: > A range in a file that only contains nulls. And that's -exactly- the ambiguous, vague definition that has raised all these questions in the first place. I was in doubt about whether unwritten extents can be considered a hole, and by your definition that means it should be data. But Andreas seems to be in no doubt it should be considered a hole. Hence if I implement XFS support and Andreas implements ext4 support by your defintion, we end with vastly different behaviour even though the two filesystems use the same underlying technology for preallocated ranges. That's exactly the inconsistency in implementation that I'd like us to avoid. IOWs, the definition needs to be clear enough to prevent these inconsistencies from occurring. Indeed, the phrase "preallocated ranges that have not had data written to them" is as independent of filesystem implementation or technologies as possible. However, because Linux supports preallocation (unlike our reference platform), and we encourage developers to use it where appropriate, it is best that we define how we expect such ranges to behave clearly. That makes life easier for everyone. Cheers, Dave.
On Wed, 31 Aug 2011, Dave Chinner wrote: > On Tue, Aug 30, 2011 at 06:17:02PM -0700, Sunil Mushran wrote: >> On 08/25/2011 06:35 PM, Dave Chinner wrote: >>> Agreed, that's the way I'd interpret it, too. So perhaps we need to >>> ensure that this interpretation is actually tested by this test? >>> >>> How about some definitions to work by: >>> >>> Data: a range of the file that contains valid data, regardless of >>> whether it exists in memory or on disk. The valid data can be >>> preceeded and/or followed by an arbitrary number of zero bytes >>> dependent on the underlying implementation of hole detection. >>> >>> Hole: a range of the file that contains no data or is made up >>> entirely of NULL (zero) data. Holes include preallocated ranges of >>> files that have not had actual data written to them. >>> >>> Does that make sense? It has sufficient flexibility in it for the >>> existing generic "non-implementation", allows for filesystems to >>> define their own hole detection boundaries (e.g. filesystem block >>> size), and effectively defines how preallocated ranges from >>> fallocate() should be treated (i.e. as holes). If we can agree on >>> those definitions, I think that we should document them in both the >>> kernel and the man page that defines SEEK_HOLE/SEEK_DATA so everyone >>> is on the same page... >> >> We should not tie in the definition to existing fs technologies. > > Such as? If we don't use well known, well defined terminology, we > end up with ambiguous, vague functionality and inconsistent > implementations. > >> Instead >> we should let the fs weigh the cost of providing accurate information >> with the possible gain in performance. >> >> Data: >> A range in a file that could contain something other than nulls. >> If in doubt, it is data. >> >> Hole: >> A range in a file that only contains nulls. > > And that's -exactly- the ambiguous, vague definition that has raised > all these questions in the first place. I was in doubt about whether > unwritten extents can be considered a hole, and by your definition > that means it should be data. But Andreas seems to be in no doubt it > should be considered a hole. > > Hence if I implement XFS support and Andreas implements ext4 support > by your defintion, we end with vastly different behaviour even > though the two filesystems use the same underlying technology for > preallocated ranges. That's exactly the inconsistency in > implementation that I'd like us to avoid. > > IOWs, the definition needs to be clear enough to prevent these > inconsistencies from occurring. Indeed, the phrase "preallocated > ranges that have not had data written to them" is as independent of > filesystem implementation or technologies as possible. However, > because Linux supports preallocation (unlike our reference > platform), and we encourage developers to use it where appropriate, > it is best that we define how we expect such ranges to behave > clearly. That makes life easier for everyone. Since a sparse file has the holes filled by nulls by definition, it seems fairly clear that they chould count as holes. In fact, I would not be surprised to see some filesystem _only_ report the unwritten pieces of sparse files as holes (not any other ranges of nulls) the question I have is how large does the range of nulls need to be before it's reported as a hole? disk sectors, filesystem blocks, other? David Lang -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 8/30/2011 8:29 PM, Dave Chinner wrote: > And that's -exactly- the ambiguous, vague definition that has raised > all these questions in the first place. I was in doubt about whether > unwritten extents can be considered a hole, and by your definition > that means it should be data. But Andreas seems to be in no doubt it > should be considered a hole. Fair enough. Let me rephrase. Data: A range in a file when read could return something other than nulls. Hole: A range in a file when read only returns nulls. Considering preallocated extents only return null, they should be considered holes. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Aug 30, 2011 at 11:29 PM, Dave Chinner <david@fromorbit.com> wrote: > On Tue, Aug 30, 2011 at 06:17:02PM -0700, Sunil Mushran wrote: >> Instead >> we should let the fs weigh the cost of providing accurate information >> with the possible gain in performance. >> >> Data: >> A range in a file that could contain something other than nulls. >> If in doubt, it is data. >> >> Hole: >> A range in a file that only contains nulls. > > And that's -exactly- the ambiguous, vague definition that has raised > all these questions in the first place. I was in doubt about whether > unwritten extents can be considered a hole, and by your definition > that means it should be data. But Andreas seems to be in no doubt it > should be considered a hole. That's fine, though. Different filesystems have different abilities to recognize a data hole - FAT can't do it at all. Perhaps the requirements would be better stated in reverse: If the filesystem knows that a read() will return nulls (for whatever reason based on it's internal knowledge), it can report a hole. If it can't guarantee that, it's data. It's an absolute requirement that SEEK_DATA never miss data. SEEK_HOLE working is a nicety that userspace would appreciate - remember that the consumer here is cp(1), using it to skip empty portions of files and create sparse destination files. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 08/31/2011 05:43 AM, Sunil Mushran wrote: > On 8/30/2011 8:29 PM, Dave Chinner wrote: >> And that's -exactly- the ambiguous, vague definition that has raised >> all these questions in the first place. I was in doubt about whether >> unwritten extents can be considered a hole, and by your definition >> that means it should be data. But Andreas seems to be in no doubt it >> should be considered a hole. > > Fair enough. Let me rephrase. > > Data: > A range in a file when read could return something other than nulls. > > Hole: > A range in a file when read only returns nulls. > > Considering preallocated extents only return null, they should > be considered holes. I would concur with this. Treating preallocated extents as holes will allow them to be processed quickly by `cp` etc. What we lose is the ability to copy the allocation from src to dst. But that's less important in comparison, and could probably be done on a per file rather than per extent basis anyway. Note preallocation would be good for disk layout and timely ENOSPC. Note fiemap gives us greater control by distinguishing holes and empty extents, thus allowing us to both efficiently copy and maintain allocation. But that interface currently requires a sync of the file on some file systems, which we couldn't enable for performance reasons in cp. cheers, Pádraig. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/255 b/255 new file mode 100755 index 0000000..4bb4d0b --- /dev/null +++ b/255 @@ -0,0 +1,71 @@ +#! /bin/bash +# FS QA Test No. 255 +# +# Test SEEK_DATA and SEEK_HOLE +# +#----------------------------------------------------------------------- +# Copyright (c) 2011 Red Hat. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +# +#----------------------------------------------------------------------- +# +# creator +owner=josef@redhat.com + +seq=`basename $0` +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! + +_cleanup() +{ + rm -f $tmp.* +} + +trap "_cleanup ; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +# real QA test starts here +_supported_fs generic +_supported_os Linux + +testfile=$TEST_DIR/seek_test.$$ +logfile=$TEST_DIR/seek_test.$$.log + +[ -x $here/src/seek-tester ] || _notrun "seek-tester not built" + +_cleanup() +{ + rm -f $testfile + rm -f $logfile +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +echo "Silence is golden" +$here/src/seek-tester -q $testfile 2>&1 | tee -a $logfile + +if grep -q "SEEK_HOLE is not supported" $logfile; then + _notrun "SEEK_HOLE/SEEK_DATA not supported by this kernel" +fi + +rm -f $logfile +rm -f $testfile + +status=0 ; exit diff --git a/255.out b/255.out new file mode 100644 index 0000000..7eefb82 --- /dev/null +++ b/255.out @@ -0,0 +1,2 @@ +QA output created by 255 +Silence is golden diff --git a/group b/group index 1f86075..c045e70 100644 --- a/group +++ b/group @@ -368,3 +368,4 @@ deprecated 252 auto quick prealloc 253 auto quick 254 auto quick +255 auto quick diff --git a/src/Makefile b/src/Makefile index 91088bf..ccdaeec 100644 --- a/src/Makefile +++ b/src/Makefile @@ -17,7 +17,7 @@ LINUX_TARGETS = xfsctl bstat t_mtab getdevicesize preallo_rw_pattern_reader \ preallo_rw_pattern_writer ftrunc trunc fs_perms testx looptest \ locktest unwritten_mmap bulkstat_unlink_test t_stripealign \ bulkstat_unlink_test_modified t_dir_offset t_futimens t_immutable \ - stale_handle pwrite_mmap_blocked fstrim + stale_handle pwrite_mmap_blocked fstrim seek-tester SUBDIRS = diff --git a/src/seek-tester.c b/src/seek-tester.c new file mode 100644 index 0000000..5141b45 --- /dev/null +++ b/src/seek-tester.c @@ -0,0 +1,475 @@ +/* + * Copyright (C) 2011 Oracle. All rights reserved. + * Copyright (C) 2011 Red Hat. All rights reserved. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public + * License v2 as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + * + * You should have received a copy of the GNU General Public + * License along with this program; if not, write to the + * Free Software Foundation, Inc., 59 Temple Place - Suite 330, + * Boston, MA 021110-1307, USA. + */ + +#define _XOPEN_SOURCE 500 +#include <sys/types.h> +#include <sys/stat.h> +#include <errno.h> +#include <fcntl.h> +#include <stdio.h> +#include <string.h> +#include <unistd.h> +#include <stdlib.h> + +#ifndef SEEK_DATA +#define SEEK_DATA 3 +#define SEEK_HOLE 4 +#endif + +#define FS_NO_HOLES (1 << 0) +#define QUIET (1 << 1) + +static blksize_t alloc_size; +static unsigned flags = 0; + +static int get_io_sizes(int fd) +{ + struct stat buf; + int ret; + + ret = fstat(fd, &buf); + if (ret) + fprintf(stderr, " ERROR %d: Failed to find io blocksize\n", + errno); + + /* st_blksize is typically also the allocation size */ + alloc_size = buf.st_blksize; + + if (!(flags & QUIET)) + printf("Allocation size: %ld\n", alloc_size); + + return ret; +} + +#define do_free(x) do { if(x) free(x); } while(0); + +static void *do_malloc(size_t size) +{ + void *buf; + + buf = malloc(size); + if (!buf) + fprintf(stderr, " ERROR: Unable to allocate %ld bytes\n", + (long)size); + + return buf; +} + +static int do_truncate(int fd, off_t length) +{ + int ret; + + ret = ftruncate(fd, length); + if (ret) + fprintf(stderr, " ERROR %d: Failed to extend file " + "to %ld bytes\n", errno, (long)length); + return ret; +} + +static ssize_t do_pwrite(int fd, const void *buf, size_t count, off_t offset) +{ + ssize_t ret, written = 0; + + while (count > written) { + ret = pwrite(fd, buf + written, count - written, offset + written); + if (ret < 0) { + fprintf(stderr, " ERROR %d: Failed to write %ld " + "bytes\n", errno, (long)count); + return ret; + } + written += ret; + } + + return 0; +} + +static int do_lseek(int testnum, int subtest, int fd, int origin, off_t set, + off_t exp) +{ + off_t pos; + int ret = -1; + + pos = lseek(fd, set, origin); + + if (pos != exp) { + fprintf(stderr, " ERROR in Test %d.%d: POS expected %ld, " + "got %ld\n", testnum, subtest, (long)exp, (long)pos); + goto out; + } + + if (pos == -1 && errno != ENXIO) { + fprintf(stderr, " ERROR in Test %d.%d: ERRNO expected %d, " + "got %d\n", testnum, subtest, ENXIO, errno); + goto out; + } + + ret = 0; + +out: + return ret; +} + +static int get_flags(int fd) +{ + const char *buf = "ABCDEF"; + ssize_t written; + off_t pos; + int ret; + + ret = do_truncate(fd, alloc_size * 2); + if (ret) + return ret; + + written = do_pwrite(fd, buf, strlen(buf), 0); + if (written) + return -1; + + pos = lseek(fd, 0, SEEK_HOLE); + if (pos == alloc_size * 2) { + if (!(flags & QUIET)) + printf("File system does not recognize holes, the only " + "hole found will be at the end.\n"); + flags |= FS_NO_HOLES; + } else if (pos == (off_t)-1) { + fprintf(stderr, "SEEK_HOLE is not supported\n"); + return -1; + } + + return 0; +} + +/* test hole data hole data */ +static int test06(int fd, int testnum) +{ + int ret = 0; + char *buf = NULL; + int bufsz = alloc_size; + int filsz = bufsz * 4; + int off; + + if (flags & FS_NO_HOLES) + return 1; + + /* HOLE - DATA - HOLE - DATA */ + /* Each unit is bufsz */ + + buf = do_malloc(bufsz); + if (!buf) + goto out; + memset(buf, 'a', bufsz); + + ret = do_pwrite(fd, buf, bufsz, bufsz); + if (!ret) + do_pwrite(fd, buf, bufsz, bufsz * 3); + if (ret) + goto out; + + /* offset at the beginning */ + ret += do_lseek(testnum, 1, fd, SEEK_HOLE, 0, 0); + ret += do_lseek(testnum, 2, fd, SEEK_HOLE, 1, 1); + ret += do_lseek(testnum, 3, fd, SEEK_DATA, 0, bufsz); + ret += do_lseek(testnum, 4, fd, SEEK_DATA, 1, bufsz); + + /* offset around first hole-data boundary */ + off = bufsz; + ret += do_lseek(testnum, 5, fd, SEEK_HOLE, off - 1, off - 1); + ret += do_lseek(testnum, 6, fd, SEEK_DATA, off - 1, off); + ret += do_lseek(testnum, 7, fd, SEEK_HOLE, off, bufsz * 2); + ret += do_lseek(testnum, 8, fd, SEEK_DATA, off, off); + ret += do_lseek(testnum, 9, fd, SEEK_HOLE, off + 1, bufsz * 2); + ret += do_lseek(testnum, 10, fd, SEEK_DATA, off + 1, off + 1); + + /* offset around data-hole boundary */ + off = bufsz * 2; + ret += do_lseek(testnum, 11, fd, SEEK_HOLE, off - 1, off); + ret += do_lseek(testnum, 12, fd, SEEK_DATA, off - 1, off - 1); + ret += do_lseek(testnum, 13, fd, SEEK_HOLE, off, off); + ret += do_lseek(testnum, 14, fd, SEEK_DATA, off, bufsz * 3); + ret += do_lseek(testnum, 15, fd, SEEK_HOLE, off + 1, off + 1); + ret += do_lseek(testnum, 16, fd, SEEK_DATA, off + 1, bufsz * 3); + + /* offset around second hole-data boundary */ + off = bufsz * 3; + ret += do_lseek(testnum, 17, fd, SEEK_HOLE, off - 1, off - 1); + ret += do_lseek(testnum, 18, fd, SEEK_DATA, off - 1, off); + ret += do_lseek(testnum, 19, fd, SEEK_HOLE, off, filsz); + ret += do_lseek(testnum, 20, fd, SEEK_DATA, off, off); + ret += do_lseek(testnum, 21, fd, SEEK_HOLE, off + 1, filsz); + ret += do_lseek(testnum, 22, fd, SEEK_DATA, off + 1, off + 1); + + /* offset around the end of file */ + off = filsz; + ret += do_lseek(testnum, 23, fd, SEEK_HOLE, off - 1, filsz); + ret += do_lseek(testnum, 24, fd, SEEK_DATA, off - 1, filsz - 1); + ret += do_lseek(testnum, 25, fd, SEEK_HOLE, off, -1); + ret += do_lseek(testnum, 26, fd, SEEK_DATA, off, -1); + ret += do_lseek(testnum, 27, fd, SEEK_HOLE, off + 1, -1); + ret += do_lseek(testnum, 28, fd, SEEK_DATA, off + 1, -1); + +out: + do_free(buf); + return ret; +} + +/* test file with data at the beginning and a hole at the end */ +static int test05(int fd, int testnum) +{ + int ret = -1; + char *buf = NULL; + int bufsz = alloc_size; + int filsz = bufsz * 4; + + if (flags & FS_NO_HOLES) + return 1; + + /* DATA - HOLE */ + /* Each unit is bufsz */ + + buf = do_malloc(bufsz); + if (!buf) + goto out; + memset(buf, 'a', bufsz); + + ret = do_truncate(fd, filsz); + if (!ret) + ret = do_pwrite(fd, buf, bufsz, 0); + if (ret) + goto out; + + /* offset at the beginning */ + ret += do_lseek(testnum, 1, fd, SEEK_HOLE, 0, bufsz); + ret += do_lseek(testnum, 2, fd, SEEK_HOLE, 1, bufsz); + ret += do_lseek(testnum, 3, fd, SEEK_DATA, 0, 0); + ret += do_lseek(testnum, 4, fd, SEEK_DATA, 1, 1); + + /* offset around data-hole boundary */ + ret += do_lseek(testnum, 5, fd, SEEK_HOLE, bufsz - 1, bufsz); + ret += do_lseek(testnum, 6, fd, SEEK_DATA, bufsz - 1, bufsz - 1); + ret += do_lseek(testnum, 7, fd, SEEK_HOLE, bufsz, bufsz); + ret += do_lseek(testnum, 8, fd, SEEK_DATA, bufsz, -1); + ret += do_lseek(testnum, 9, fd, SEEK_HOLE, bufsz + 1, bufsz + 1); + ret += do_lseek(testnum, 10, fd, SEEK_DATA, bufsz + 1, -1); + + /* offset around eof */ + ret += do_lseek(testnum, 11, fd, SEEK_HOLE, filsz - 1, filsz - 1); + ret += do_lseek(testnum, 12, fd, SEEK_DATA, filsz - 1, -1); + ret += do_lseek(testnum, 13, fd, SEEK_HOLE, filsz, -1); + ret += do_lseek(testnum, 14, fd, SEEK_DATA, filsz, -1); + ret += do_lseek(testnum, 15, fd, SEEK_HOLE, filsz + 1, -1); + ret += do_lseek(testnum, 16, fd, SEEK_DATA, filsz + 1, -1); + +out: + do_free(buf); + return ret; +} + +/* test hole begin and data end */ +static int test04(int fd, int testnum) +{ + int ret; + char *buf = "ABCDEFGH"; + int bufsz = sizeof(buf); + int holsz = alloc_size * 2; + int filsz = holsz + bufsz; + + if (flags & FS_NO_HOLES) + return 1; + + /* HOLE - DATA */ + + ret = do_pwrite(fd, buf, bufsz, holsz); + if (ret) + goto out; + + /* offset at the beginning */ + ret += do_lseek(testnum, 1, fd, SEEK_HOLE, 0, 0); + ret += do_lseek(testnum, 2, fd, SEEK_HOLE, 1, 1); + ret += do_lseek(testnum, 3, fd, SEEK_DATA, 0, holsz); + ret += do_lseek(testnum, 4, fd, SEEK_DATA, 1, holsz); + + /* offset around hole-data boundary */ + ret += do_lseek(testnum, 5, fd, SEEK_HOLE, holsz - 1, holsz - 1); + ret += do_lseek(testnum, 6, fd, SEEK_DATA, holsz - 1, holsz); + ret += do_lseek(testnum, 7, fd, SEEK_HOLE, holsz, filsz); + ret += do_lseek(testnum, 8, fd, SEEK_DATA, holsz, holsz); + ret += do_lseek(testnum, 9, fd, SEEK_HOLE, holsz + 1, filsz); + ret += do_lseek(testnum, 10, fd, SEEK_DATA, holsz + 1, holsz + 1); + + /* offset around eof */ + ret += do_lseek(testnum, 11, fd, SEEK_HOLE, filsz - 1, filsz); + ret += do_lseek(testnum, 12, fd, SEEK_DATA, filsz - 1, filsz - 1); + ret += do_lseek(testnum, 13, fd, SEEK_HOLE, filsz, -1); + ret += do_lseek(testnum, 14, fd, SEEK_DATA, filsz, -1); + ret += do_lseek(testnum, 15, fd, SEEK_HOLE, filsz + 1, -1); + ret += do_lseek(testnum, 16, fd, SEEK_DATA, filsz + 1, -1); +out: + return ret; +} + +/* test full file */ +static int test03(int fd, int testnum) +{ + char *buf = NULL; + int bufsz = alloc_size + 100; + int ret = -1; + + buf = do_malloc(bufsz); + if (!buf) + goto out; + memset(buf, 'a', bufsz); + + ret = do_pwrite(fd, buf, bufsz, 0); + if (ret) + goto out; + + /* offset at the beginning */ + ret += do_lseek(testnum, 1, fd, SEEK_HOLE, 0, bufsz); + ret += do_lseek(testnum, 2, fd, SEEK_HOLE, 1, bufsz); + ret += do_lseek(testnum, 3, fd, SEEK_DATA, 0, 0); + ret += do_lseek(testnum, 4, fd, SEEK_DATA, 1, 1); + + /* offset around eof */ + ret += do_lseek(testnum, 5, fd, SEEK_HOLE, bufsz - 1, bufsz); + ret += do_lseek(testnum, 6, fd, SEEK_DATA, bufsz - 1, bufsz - 1); + ret += do_lseek(testnum, 7, fd, SEEK_HOLE, bufsz, -1); + ret += do_lseek(testnum, 8, fd, SEEK_DATA, bufsz, -1); + ret += do_lseek(testnum, 9, fd, SEEK_HOLE, bufsz + 1, -1); + ret += do_lseek(testnum, 10, fd, SEEK_DATA, bufsz + 1, -1); + +out: + do_free(buf); + return ret; +} + +/* test empty file */ +static int test02(int fd, int testnum) +{ + int ret = 0; + + ret += do_lseek(testnum, 1, fd, SEEK_DATA, 0, -1); + ret += do_lseek(testnum, 2, fd, SEEK_HOLE, 0, -1); + ret += do_lseek(testnum, 3, fd, SEEK_HOLE, 1, -1); + + return ret; +} + +/* test feature support */ +static int test01(int fd, int testnum) +{ + int ret; + char buf[] = "ABCDEFGH"; + int bufsz = sizeof(buf); + + ret = do_pwrite(fd, buf, bufsz, 0); + if (ret) + goto out; + + ret += do_lseek(testnum, 1, fd, SEEK_DATA, 0, 0); + ret += do_lseek(testnum, 2, fd, SEEK_HOLE, 0, bufsz); + +out: + return ret; +} + +struct testrec { + int test_num; + int (*test_func)(int fd, int testnum); + char *test_desc; +}; + +struct testrec seek_tests[] = { + { 1, test01, "Test basic support" }, + { 2, test02, "Test an empty file" }, + { 3, test03, "Test a full file" }, + { 4, test04, "Test file hole at beg, data at end" }, + { 5, test05, "Test file data at beg, hole at end" }, + { 6, test06, "Test file hole data hole data" }, +}; + +static int run_test(int fd, struct testrec *tr) +{ + int ret; + + ret = tr->test_func(fd, tr->test_num); + if (!(flags & QUIET)) + printf("%02d. %-50s\t%s\n", tr->test_num, tr->test_desc, + ret < 0 ? "FAIL" : (ret == 0 ? "SUCC" : "NOT RUN")); + return ret; +} + +void print_help() +{ + printf("seek-test [-h] [-q] filename\n"); + printf("\t-h - this message\n"); + printf("\t-q - quiet, no output\n"); + printf("\tfilename - file to use for the test\n"); +} + +int main(int argc, char **argv) +{ + int ret = -1; + int i, fd = -1; + int c; + int numtests = sizeof(seek_tests) / sizeof(struct testrec); + + while ((c = getopt(argc, argv, "qh")) != -1) { + switch (c) { + case 'q': + flags |= QUIET; + break; + case 'h': + print_help(); + exit(0); + default: + print_help(); + exit(1); + } + } + + if (optind >= argc) { + print_help(); + exit(1); + } + + fd = open(argv[optind], O_RDWR|O_CREAT|O_TRUNC, 0644); + if (fd < 0) { + fprintf(stderr, "Failed to open testfile: %d\n", errno); + goto out; + } + + ret = get_io_sizes(fd); + if (ret) + goto out; + + ret = get_flags(fd); + if (ret) + goto out; + + for (i = 0; i < numtests; ++i) { + ret = do_truncate(fd, 0); + if (ret) + goto out; + run_test(fd, &seek_tests[i]); + } + +out: + if (fd > -1) + close(fd); + return ret; +}
This is a test to make sure seek_data/seek_hole is acting like it does on Solaris. It will check to see if the fs supports finding a hole or not and will adjust as necessary. Signed-off-by: Josef Bacik <josef@redhat.com> --- 255 | 71 ++++++++ 255.out | 2 + group | 1 + src/Makefile | 2 +- src/seek-tester.c | 475 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 550 insertions(+), 1 deletions(-) create mode 100755 255 create mode 100644 255.out create mode 100644 src/seek-tester.c