| Message ID | 20170417032642.30770-10-quwenruo@cn.fujitsu.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
Nice. With this new option, it should be possible to achieve something like RAID56 corruption test-script say, https://patchwork.kernel.org/patch/9583455/ right? ---- Cheers, Lakshmipathi.G FOSS Programmer. http://www.giis.co.in http://www.webminal.org On Mon, Apr 17, 2017 at 8:56 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote: > Introduce a new option, --pattern, to specify the pattern to fill > mirror. > > Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com> > --- > Documentation/btrfs-modify.asciidoc | 5 +++++ > modify/mirror.c | 11 ++++++++++- > 2 files changed, 15 insertions(+), 1 deletion(-) > > diff --git a/Documentation/btrfs-modify.asciidoc b/Documentation/btrfs-modify.asciidoc > index ae2ada65..d82fb0df 100644 > --- a/Documentation/btrfs-modify.asciidoc > +++ b/Documentation/btrfs-modify.asciidoc > @@ -41,6 +41,11 @@ stripe number 1. > P for RAID56 1st parity stripe. Only RAID5 and RAID6 support stripe number P. > Q for RAID6 2nd parity stripe. > > +--pattern <pattern>:::: > +Specify to pattern to modify the mirror data5f536652. > ++ > +Default value is "0x66524842" (ascii "BHRf", part of the btrfs magic "_BHRfS_M"). > + > EXIT STATUS > ----------- > *btrfs-modify* returns a zero exit status if all its operations succeed. > diff --git a/modify/mirror.c b/modify/mirror.c > index c89927f6..dd717fbe 100644 > --- a/modify/mirror.c > +++ b/modify/mirror.c > @@ -373,13 +373,15 @@ int modify_mirror(int argc, char **argv) > char *device; > u64 length = (u64)-1; > u64 logical = (u64)-1; > + u32 pattern = 0x66524842; > int stripe = STRIPE_UNINITILIZED; > int ret; > > while (1) { > int c; > enum { GETOPT_VAL_LOGICAL = 257, GETOPT_VAL_LENGTH, > - GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET }; > + GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET, > + GETOPT_VAL_PATTERN }; > static const struct option long_options[] = { > { "logical", required_argument, NULL, > GETOPT_VAL_LOGICAL }, > @@ -388,6 +390,8 @@ int modify_mirror(int argc, char **argv) > { "stripe", required_argument, NULL, GETOPT_VAL_STRIPE }, > { "root-ino-offset", required_argument, NULL, > GETOPT_VAL_ROOT_INO_OFFSET}, > + { "pattern", required_argument, NULL, > + GETOPT_VAL_PATTERN}, > { NULL, 0, NULL, 0 } > }; > > @@ -407,6 +411,9 @@ int modify_mirror(int argc, char **argv) > case GETOPT_VAL_ROOT_INO_OFFSET: > parse_root_ino_offset(&dest, optarg); > break; > + case GETOPT_VAL_PATTERN: > + pattern = arg_strtou32(optarg); > + break; > case '?': > case 'h': > usage(modify_mirror_usage); > @@ -439,6 +446,8 @@ int modify_mirror(int argc, char **argv) > stripe = 0; > } > > + memset(write_buf, pattern, sizeof(write_buf) / sizeof(pattern)); > + > fs_info = open_ctree_fs_info(device, 0, 0, 0, OPEN_CTREE_WRITES); > if (!fs_info) { > error("failed to open btrfs on device %s\n", device); > -- > 2.12.2 > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Thanks for the example and details. I understood some and need to
re-read couple of more times to understand the remaining.
btw, I created a corruption framework(with previous org), the sample
usage and example is below. It looks similar to Btrfs corruption tool.
thanks.
--
corrupt.py --help
Usage: corrupt.py [options] <arg1> <arg2> <arg3> ...]
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-t object, --object=object
The object type to be corrupted [ choose from (inode,
l_inner,l_master,l_leaf,fileinode_inner,fileinode_leaf,dirinode_in
ner,dirinode_leaf,data_block,fec_block,baddr)
default: inode]
-p /fs/path, --path=/fs/path
The file or directory path in /fs
-q optype, --optype=optype
The corruption operation on the object [choose from
(set,xor,add) default: set]
-o offset, --offset=offset
The valid offset value of the object [default:0]
-m mirror, --mirror=mirror
The mirror number [default: 1]
-s size, --size=size The offset field/byte to be corrupted [default: 1]
-c fixcrc, --fixcrc=fixcrc
This fixes the CRC's of the corrupted objects
-n lbn, --lbn=lbn The lbn of the data block to be corrupted
-b baddr, --baddr=baddr
The baddr to be corrupted
--log=LOG_FILE This logs the result into the log file
Example usage:
corrupt.py --object=inode --offset=0 --size=4 --path=/fs/path/to/file.txt
Will corrupt first 4 bytes of inode structure.
corrupt.py --object=dirinode_inner --offset=0 --size=4
--path=/fs/path/to/file.txt
Will corrupt first 4 bytes of directory-inode structure.
corrupt.py --object=dirinode_leaf --offset=0 --size=4
--path=/fs/path/to/file.txt
Will corrupt first 4 bytes of directory-inode leaf structure.
(file.txt should be
large enough to contain leafs).
corrupt.py --object=inode --offset=0 --size=4
--path=/fs/path/to/file.txt --mirror=2
Will corrupt first 4 bytes of inode structure that reside on Node/drive-2.
--optype=optype ==> default is set. It sets '0' to specified offset.
add => Will try given value to existing value. For example.
offset=(nlink's offset) --optype=add will increase the nlink count by
1.
(running fsck, expected to find this and fix it)
where as optype xor => should flip the bits at given offset.
--
----
Cheers,
Lakshmipathi.G
FOSS Programmer.
http://www.giis.co.in http://www.webminal.org
On Thu, Apr 20, 2017 at 6:30 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
>
> At 04/19/2017 03:46 PM, Lakshmipathi.G wrote:
>>
>> Thanks. This should make easier to create corruption test scripts.
>>
>> when using --root-ino-offset 5,257,0 Does final entry 0 has to be default value ? Does it represent any value at
>> this moment?
>
>
> For case 5,257,0, it means the start point of the corruption is root 5, inode 257 and file offset 0.
>
> So let's take the following file layout as an example (a little complex one, to show off all the ability)
>
> Root 5 inode 257:
>
> \/ This is file offset
> Extent 0: disk bytenr X, disk len 4K, nr_bytes 4K offset 0 plain
> Extent 4K: disk bytenr Y, disk len 8K, nr_bytes 4K offset 0 plain
> Extent 8K: disk bytenr Z, disk len 8K, nr_bytes 4K offset 4K plain
>
> And if you specified --root-ino-offset 5,257,0 and --length 12288
>
> Then the following on-disk range will be corrupted:
> [X, X+4K)
> [Y, Y+4K)
> [Z+4K, Z+8K)
>
> Making all the 0~12K of root5, inode257 get corrupted, while unrelated ranges like [Z, Z+4K) and [Y+4K, Y+8K) not modified.
>
> While for RAID1/RAID10/DUP usage, just create a file on a new fs, then pass --root-ino-offset 5,257,0 with --length <file_size> will corrupt the whole file for you, allowing us to test recovery.
>
>>
>> One more thing, does btrfs-modify command has option to print out existing parity value (all parity of a file or specific a parity)?
>> something like get_parity option?
>
>
> 1) For print parity value
> I didn't see the need to print parity value.
> As if you want to verify if the parity is good or not, just use offline scrub.
> It will detect parity/data corruption.
>
> 2) For printing/corrupting value of a file in RAID56 profile
> Not as easy as one may think.
>
> Think about the following case:
>
> Root 5 ino 257:
> Extent 0: disk bytenr X(rounded to stripe boundary), disk len 32K, nr_bytes 32K, offset 0.
> Extent 1: disk bytenr X+32K, disk len 64K, nr_bytes 64K, offset 0
>
> On disk it will be:
>
> | Dev 1 | Dev 2 | Dev 3 |
> X X+64K X+128K X+192K
> |<-Ext 0->|<----- Ext 1---->| |///// Parity /////|
>
> For Extent 0, it's OK to just print out/corrupt its value, as it's completely inside one data stripe.
>
> But for Extent 1, it crosses 2 device.
> If we corrupt on disk data of Extent1, we will corrupt both data stripe, leaving the full stripe unrecoverable.
>
> So for P/Q corrupting, I'm considering to record which full stripe we have corrupted and never corrupt any full stripe already damaged in next version.
>
> Thanks,
> Qu
>>
>>
>>
>>
>> ----
>> Cheers,
>> Lakshmipathi.G
>> FOSS Programmer.
>> http://www.giis.co.in http://www.webminal.org
>>
>> On Tue, Apr 18, 2017 at 12:43 PM, Qu Wenruo <quwenruo@cn.fujitsu.com <mailto:quwenruo@cn.fujitsu.com>> wrote:
>>
>>
>>
>> At 04/18/2017 03:05 PM, Lakshmipathi.G wrote:
>>
>> Nice. With this new option, it should be possible to achieve
>> something
>> like RAID56 corruption test-script say,
>> https://patchwork.kernel.org/patch/9583455/
>> <https://patchwork.kernel.org/patch/9583455/> right?
>>
>>
>> Yes, just for your use case!
>>
>> It would be much more simpler to craft such recovery test case.
>>
>> Just create a file in default subvolume (5), write large enough
>> data, get the file ino (257 for first inode).
>>
>> Then pass --root-ino-offset 5,257,0 --length 64k --stripe 0 to
>> corrupt data stripe.
>> Or change to --stripe P to corrupt parity.
>>
>> Although it's still a little tricky to use current "--stripe" option
>> to only corrupt data stripe (above 64K may crosses 2 data stripes).
>> It shouldn't to be hard to add a new parameter --raid-stripe for
>> such usage.
>>
>> But it should be good enough for DUP/RAID1/RAID10.
>>
>> Thanks,
>> Qu
>>
>>
>> ----
>> Cheers,
>> Lakshmipathi.G
>> FOSS Programmer.
>> http://www.giis.co.in http://www.webminal.org
>>
>>
>> On Mon, Apr 17, 2017 at 8:56 AM, Qu Wenruo
>> <quwenruo@cn.fujitsu.com <mailto:quwenruo@cn.fujitsu.com>> wrote:
>>
>> Introduce a new option, --pattern, to specify the pattern to
>> fill
>> mirror.
>>
>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com
>> <mailto:quwenruo@cn.fujitsu.com>>
>>
>> ---
>> Documentation/btrfs-modify.asciidoc | 5 +++++
>> modify/mirror.c | 11 ++++++++++-
>> 2 files changed, 15 insertions(+), 1 deletion(-)
>>
>> diff --git a/Documentation/btrfs-modify.asciidoc
>> b/Documentation/btrfs-modify.asciidoc
>> index ae2ada65..d82fb0df 100644
>> --- a/Documentation/btrfs-modify.asciidoc
>> +++ b/Documentation/btrfs-modify.asciidoc
>> @@ -41,6 +41,11 @@ stripe number 1.
>> P for RAID56 1st parity stripe. Only RAID5 and RAID6
>> support stripe number P.
>> Q for RAID6 2nd parity stripe.
>>
>> +--pattern <pattern>::::
>> +Specify to pattern to modify the mirror data5f536652.
>> ++
>> +Default value is "0x66524842" (ascii "BHRf", part of the
>> btrfs magic "_BHRfS_M").
>> +
>> EXIT STATUS
>> -----------
>> *btrfs-modify* returns a zero exit status if all its
>> operations succeed.
>> diff --git a/modify/mirror.c b/modify/mirror.c
>> index c89927f6..dd717fbe 100644
>> --- a/modify/mirror.c
>> +++ b/modify/mirror.c
>> @@ -373,13 +373,15 @@ int modify_mirror(int argc, char **argv)
>> char *device;
>> u64 length = (u64)-1;
>> u64 logical = (u64)-1;
>> + u32 pattern = 0x66524842;
>> int stripe = STRIPE_UNINITILIZED;
>> int ret;
>>
>> while (1) {
>> int c;
>> enum { GETOPT_VAL_LOGICAL = 257,
>> GETOPT_VAL_LENGTH,
>> - GETOPT_VAL_STRIPE,
>> GETOPT_VAL_ROOT_INO_OFFSET };
>> + GETOPT_VAL_STRIPE,
>> GETOPT_VAL_ROOT_INO_OFFSET,
>> + GETOPT_VAL_PATTERN };
>> static const struct option long_options[] = {
>> { "logical", required_argument, NULL,
>> GETOPT_VAL_LOGICAL },
>> @@ -388,6 +390,8 @@ int modify_mirror(int argc, char **argv)
>> { "stripe", required_argument,
>> NULL, GETOPT_VAL_STRIPE },
>> { "root-ino-offset",
>> required_argument, NULL,
>> GETOPT_VAL_ROOT_INO_OFFSET},
>> + { "pattern", required_argument, NULL,
>> + GETOPT_VAL_PATTERN},
>> { NULL, 0, NULL, 0 }
>> };
>>
>> @@ -407,6 +411,9 @@ int modify_mirror(int argc, char **argv)
>> case GETOPT_VAL_ROOT_INO_OFFSET:
>> parse_root_ino_offset(&dest, optarg);
>> break;
>> + case GETOPT_VAL_PATTERN:
>> + pattern = arg_strtou32(optarg);
>> + break;
>> case '?':
>> case 'h':
>> usage(modify_mirror_usage);
>> @@ -439,6 +446,8 @@ int modify_mirror(int argc, char **argv)
>> stripe = 0;
>> }
>>
>> + memset(write_buf, pattern, sizeof(write_buf) /
>> sizeof(pattern));
>> +
>> fs_info = open_ctree_fs_info(device, 0, 0, 0,
>> OPEN_CTREE_WRITES);
>> if (!fs_info) {
>> error("failed to open btrfs on device
>> %s\n", device);
>> --
>> 2.12.2
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe
>> linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> <mailto:majordomo@vger.kernel.org>
>> More majordomo info at
>> http://vger.kernel.org/majordomo-info.html
>> <http://vger.kernel.org/majordomo-info.html>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe
>> linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> <mailto:majordomo@vger.kernel.org>
>> More majordomo info at
>> http://vger.kernel.org/majordomo-info.html
>> <http://vger.kernel.org/majordomo-info.html>
>>
>>
>>
>>
>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, Apr 23, 2017 at 01:12:42PM +0530, Lakshmipathi.G wrote: > Thanks for the example and details. I understood some and need to > re-read couple of more times to understand the remaining. > > btw, I created a corruption framework(with previous org), the sample > usage and example is below. It looks similar to Btrfs corruption tool. > thanks. > > -- > corrupt.py --help [...] Interesting, can you please share the script? This is another alternative that seems more plausible for rapid prototyping of various corruption scenarios. The C utility (either existing btrfs-corrupt-block or the proposed btrfs-modify) can become tedious to change, but can be compiled and distributed without the python dependency. I wanted to use something python-based for tests when Hans announced the python-btrfs project, but it has broader goals than just the testsuite needs. So we could have our own corrupt.py, just for our internal use. I'm not sure if a compiled tool like btrfs-modify is really needed, but why we can't have both. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
okay, but I'm not sure whether it can be useful here other than the
sample cli usage. The reason, this tool relies heavily on Python/C
layer[1]. The framework works like this:
C-api(fs-progs) <--> Python/C api <--> Python program.
C-api is typical fs library (like libext2fs/libbtrfs) and C/Python
used to export every on-disk fs structure as an python-object along
with few fs-library functions. Finally in python script we can perform
stuffs like:
--
import fslib as fs
disk_object=fs.get_inode("/file/path")
#set few values finally make a call
retval = fs.corrupt_object(disk_object, offset, size, optype, fixcrc)
---
at C-python layer above python-object converted into C appropriate
data-type and passed on to C-layer.
hard-part will be troubleshooting issues, tracking them is difficult
because both gdb and pdb can help only upto certain point.
[1]: https://docs.python.org/2/extending/extending.html
Cheers.
Lakshmipathi.G
On 5/5/17, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
> At 05/05/2017 12:53 AM, David Sterba wrote:
>> On Sun, Apr 23, 2017 at 01:12:42PM +0530, Lakshmipathi.G wrote:
>>> Thanks for the example and details. I understood some and need to
>>> re-read couple of more times to understand the remaining.
>>>
>>> btw, I created a corruption framework(with previous org), the sample
>>> usage and example is below. It looks similar to Btrfs corruption tool.
>>> thanks.
>>>
>>> --
>>> corrupt.py --help
>> [...]
>>
>> Interesting, can you please share the script? This is another
>> alternative that seems more plausible for rapid prototyping of various
>> corruption scenarios. The C utility (either existing btrfs-corrupt-block
>> or the proposed btrfs-modify) can become tedious to change, but can be
>> compiled and distributed without the python dependency.
>>
>> I wanted to use something python-based for tests when Hans announced the
>> python-btrfs project, but it has broader goals than just the testsuite
>> needs. So we could have our own corrupt.py, just for our internal use.
>>
>> I'm not sure if a compiled tool like btrfs-modify is really needed, but
>> why we can't have both.
>>
>>
> Python based tool is always a good idea.
>
> If python based lib can provide the same capability of current C
> facilities (btrfs_search_slot() at least) and have better encapsulation,
> I'm totally committed to python based one.
>
> Although I'm a little afraid that the python script is not using the low
> level btrfs_search_slot(), but just getting chunk layout then do
> mathematics to get stripe location.
>
> Thanks,
> Qu
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/Documentation/btrfs-modify.asciidoc b/Documentation/btrfs-modify.asciidoc index ae2ada65..d82fb0df 100644 --- a/Documentation/btrfs-modify.asciidoc +++ b/Documentation/btrfs-modify.asciidoc @@ -41,6 +41,11 @@ stripe number 1. P for RAID56 1st parity stripe. Only RAID5 and RAID6 support stripe number P. Q for RAID6 2nd parity stripe. +--pattern <pattern>:::: +Specify to pattern to modify the mirror data5f536652. ++ +Default value is "0x66524842" (ascii "BHRf", part of the btrfs magic "_BHRfS_M"). + EXIT STATUS ----------- *btrfs-modify* returns a zero exit status if all its operations succeed. diff --git a/modify/mirror.c b/modify/mirror.c index c89927f6..dd717fbe 100644 --- a/modify/mirror.c +++ b/modify/mirror.c @@ -373,13 +373,15 @@ int modify_mirror(int argc, char **argv) char *device; u64 length = (u64)-1; u64 logical = (u64)-1; + u32 pattern = 0x66524842; int stripe = STRIPE_UNINITILIZED; int ret; while (1) { int c; enum { GETOPT_VAL_LOGICAL = 257, GETOPT_VAL_LENGTH, - GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET }; + GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET, + GETOPT_VAL_PATTERN }; static const struct option long_options[] = { { "logical", required_argument, NULL, GETOPT_VAL_LOGICAL }, @@ -388,6 +390,8 @@ int modify_mirror(int argc, char **argv) { "stripe", required_argument, NULL, GETOPT_VAL_STRIPE }, { "root-ino-offset", required_argument, NULL, GETOPT_VAL_ROOT_INO_OFFSET}, + { "pattern", required_argument, NULL, + GETOPT_VAL_PATTERN}, { NULL, 0, NULL, 0 } }; @@ -407,6 +411,9 @@ int modify_mirror(int argc, char **argv) case GETOPT_VAL_ROOT_INO_OFFSET: parse_root_ino_offset(&dest, optarg); break; + case GETOPT_VAL_PATTERN: + pattern = arg_strtou32(optarg); + break; case '?': case 'h': usage(modify_mirror_usage); @@ -439,6 +446,8 @@ int modify_mirror(int argc, char **argv) stripe = 0; } + memset(write_buf, pattern, sizeof(write_buf) / sizeof(pattern)); + fs_info = open_ctree_fs_info(device, 0, 0, 0, OPEN_CTREE_WRITES); if (!fs_info) { error("failed to open btrfs on device %s\n", device);
Introduce a new option, --pattern, to specify the pattern to fill mirror. Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com> --- Documentation/btrfs-modify.asciidoc | 5 +++++ modify/mirror.c | 11 ++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-)