| Message ID | 20120710041625.GA8909@sli.dy.fi (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
Thanks for doing this, I'll start looking into it right away. One thing though: it's probably not the contents of struct eb that's being corrupted, but the data the eb points to. See for example read_extent_buffer to see how to access it. Sorry for being unclear on this. Thanks, Arne On 10.07.2012 06:16, Sami Liedes wrote: > On Mon, Jul 09, 2012 at 11:05:47AM +0200, Arne Jansen wrote: >>> * Just before the crash: >>> btrfs: invalid parameters for read_extent_buffer: start (32771) > eb->len (32768). eb start is 2261163409408, level 100, generation 4412718571037421157, nritems 538968254. len param 17. debug 2/989/538968254/4412718571037421157/0x0/0/0x0/0x0 >>> >> >> At a first glance: the generation converted to ascii is: "ent() ==", >> so someone is patching the memory with ascii text, possibly C source. >> It might be interesting to dump the full contents of the eb, to get >> a clue on the source of the data. > > I changed the code to dump the contents of the eb struct at the point > where that error (btrfs: invalid parameters...) is printed, at the > "checksum mismatch 4" site and at the "node seems invalid now" site. > Now I have a big log of 1795 corrupted ebs. So far nothing that looks > remotely like ascii text, though. But I have two different versions of > the eb that caused that warning, a less corrupted one and a more > corrupted one: > > ------------------------------------------------------------ > btrfs: --- start eb contents at ffff8801b13cc4c8 --- > btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00 ...f............ > btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff .......0....... > btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff X.<.....X.<..... > btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff ........p.<..... > btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 p.<............. > btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff ._.............. > btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff @L.......f...... > btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff ........@....... > btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff @............... > btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 ..<............. > btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00 ........ > btrfs: --- end eb contents at ffff8801b13cc4c8 --- > btrfs: dm-6 checksum verify failed on 2239404212224 wanted B5F632BC found 3579FB59 level 160 > btrfs: node seems invalid now. checksum ok = 1 > btrfs: --- start eb contents at ffff8801b13cc4c8 --- > [... identical dump to above ...] > btrfs: --- end eb contents at ffff8801b13cc4c8 --- > btrfs: invalid parameters for read_extent_buffer: start (32771) > eb->len (32768). eb start is 2239404212224, level 160, generation 4716553384049587249, nritems 295705211. len param 17. debug 2/989/295705211/4716553384049587249/0x0/0/0x0/0x0 > btrfs: --- start eb contents at ffff8801b13cc4c8 --- > btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00 ...f............ > btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff .......0....... > btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff X.<.....X.<..... > btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff ........p.<..... > btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 p.<............. > btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff ._.............. > btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff @L.......f...... > btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff ........@....... > btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff @............... > btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00 ..<............. > btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00 ........{....... > btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00 14q<P.tA........ > btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00 ........ > btrfs: --- end eb contents at ffff8801b13cc4c8 --- > ------------[ cut here ]------------ > WARNING: at fs/btrfs/extent_io.c:4533 read_extent_buffer+0x1ef/0x220 [btrfs]() > Hardware name: System Product Name > Modules linked in: <omitted> [last unloaded: scsi_wait_scan] > Pid: 25829, comm: btrfs-endio-met Tainted: G W 3.4.4+btrfsdebug3+ #4 > [...] > ------------------------------------------------------------ > > Here's the difference (lines reordered from diff to make comparing > easier): > > ------------------------------------------------------------ > -btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 ..<............. > +btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00 ..<............. > -btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00 ........{....... > -btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00 14q<P.tA........ > ------------------------------------------------------------ > > If there's one pattern that catches eye in the dumps, it's that in > many places where the same eb is dumped multiple times due to multiple > "checksum mismatch 4"s, there are bytes at offsets 0x30 and 0x31 that > always seem to have the same value and both separately increase > between the dumps, usually(?) by two. Then that might be normal, I > haven't looked at what should be at a struct eb :) There are some > instances where they wrap, for example > > f8 f8 -> fa fa -> fc fc -> fe fe -> 00 00 -> 02 02 -> ..., > > with no other changes visible in the corrupted ebs. But that is by no > means the only change in the struct eb contents that can be observed, > only the most obvious pattern. > > For example, here's a 11-long string of eb contents. First the dmesg: > > ------------------------------------------------------------ > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 1] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 2] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 3] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 4] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 5] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 6] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 7] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 8] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 9] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 10] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 11] > ------------------------------------------------------------ > > Where DUMP 1 contents (file cc1.txt if you'd want to apply the diffs > below) is > > ------------------------------------------------------------ > btrfs: --- start eb contents at ffff8801cafbacc0 --- > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > btrfs: ffff8801cafbad30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad40: 00 00 10 00 00 00 00 00 04 04 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad50: 50 ad fb ca 01 88 ff ff 50 ad fb ca 01 88 ff ff P.......P....... > btrfs: ffff8801cafbad60: 02 02 00 00 00 00 00 00 68 ad fb ca 01 88 ff ff ........h....... > btrfs: ffff8801cafbad70: 68 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 h............... > btrfs: ffff8801cafbad80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad90: 00 a2 20 08 00 ea ff ff c0 2a f4 07 00 ea ff ff .. ......*...... > btrfs: ffff8801cafbada0: 00 ac 89 07 00 ea ff ff c0 42 79 07 00 ea ff ff .........By..... > btrfs: ffff8801cafbadb0: 80 af 77 07 00 ea ff ff 40 37 49 08 00 ea ff ff ..w.....@7I..... > btrfs: ffff8801cafbadc0: 80 6d f3 07 00 ea ff ff c0 a7 e8 07 00 ea ff ff .m.............. > btrfs: ffff8801cafbadd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 ........ > btrfs: --- end eb contents at ffff8801cafbacc0 --- > ------------------------------------------------------------ > > And the successive diffs are > > ------------------------------------------------------------ > --- cc1.txt 2012-07-10 06:57:21.564665577 +0300 > +++ cc2.txt 2012-07-10 06:59:10.272634578 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > @@ -20,9 +20,9 @@ > btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > -btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 ................ > -btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > -btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 01 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbae20: 4d 00 00 00 00 00 00 00 4e 00 00 00 00 00 00 00 M.......N....... > +btrfs: ffff8801cafbae30: 07 78 02 00 00 00 00 00 00 00 00 00 00 00 00 00 .x.............. > btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 ........ > btrfs: --- end eb contents at ffff8801cafbacc0 --- > --- cc2.txt 2012-07-10 06:59:10.272634578 +0300 > +++ cc3.txt 2012-07-10 06:59:10.016634663 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc3.txt 2012-07-10 06:59:10.016634663 +0300 > +++ cc4.txt 2012-07-10 06:59:09.752634749 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc4.txt 2012-07-10 06:59:09.752634749 +0300 > +++ cc5.txt 2012-07-10 06:59:09.504634831 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc5.txt 2012-07-10 06:59:09.504634831 +0300 > +++ cc6.txt 2012-07-10 06:59:09.240634917 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc6.txt 2012-07-10 06:59:09.240634917 +0300 > +++ cc7.txt 2012-07-10 06:59:08.944635015 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc7.txt 2012-07-10 06:59:08.944635015 +0300 > +++ cc8.txt 2012-07-10 06:59:08.656635109 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 !!.............. > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc8.txt 2012-07-10 06:59:08.656635109 +0300 > +++ cc9.txt 2012-07-10 06:59:08.328635216 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 !!.............. > +btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ##.............. > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc9.txt 2012-07-10 06:59:08.328635216 +0300 > +++ cc10.txt 2012-07-10 06:58:33.180646115 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ##.............. > +btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 %%.............. > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc10.txt 2012-07-10 06:58:33.180646115 +0300 > +++ cc11.txt 2012-07-10 06:58:41.340643696 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 %%.............. > +btrfs: ffff8801cafbacf0: 27 27 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ''.............. > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > ------------------------------------------------------------ > > And since it's of course possible there's something wrong with my > modifications, here's the patch I used: > > ------------------------------------------------------------ > diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c > index 7452ecb..aadb82c 100644 > --- a/fs/btrfs/extent_io.c > +++ b/fs/btrfs/extent_io.c > @@ -4527,6 +4527,9 @@ void read_extent_buffer(struct extent_buffer *eb, void *dstv, > len, > eb->debug[0], eb->debug[1], eb->debug[2], eb->debug[3], > eb->debug[4], eb->debug[5], eb->debug[6], eb->debug[7]); > + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); > + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); > + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); > WARN_ON(1); > } > WARN_ON(start + len > eb->start + eb->len); > diff --git a/fs/btrfs/reada.c b/fs/btrfs/reada.c > index ea81bd4..663d6c4 100644 > --- a/fs/btrfs/reada.c > +++ b/fs/btrfs/reada.c > @@ -130,9 +130,13 @@ static int __readahead_hook(struct btrfs_root *root, struct extent_buffer *eb, > kref_get(&re->refcnt); > spin_unlock(&fs_info->reada_lock); > > - if (!err && btrfs_csum_tree_block(root, eb)) > + if (!err && btrfs_csum_tree_block(root, eb)) { > printk(KERN_ERR "btrfs: checksum mismatch 4 on %llu\n", > eb->start); > + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); > + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); > + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); > + } > > if (!re) > return -1; > @@ -150,9 +154,14 @@ static int __readahead_hook(struct btrfs_root *root, struct extent_buffer *eb, > if (err == 0) { > nritems = level ? btrfs_header_nritems(eb) : 0; > if (level > BTRFS_MAX_LEVEL || > - nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) > + nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) { > printk(KERN_ERR "btrfs: node seems invalid now. checksum ok = %d\n", > btrfs_csum_tree_block(root, eb)); > + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); > + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); > + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); > + > + } > generation = btrfs_header_generation(eb); > /* > * FIXME: currently we just set nritems to 0 if this is a leaf, > > ------------------------------------------------------------ > > Sami -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 10.07.2012 06:16, Sami Liedes wrote: > On Mon, Jul 09, 2012 at 11:05:47AM +0200, Arne Jansen wrote: >>> * Just before the crash: >>> btrfs: invalid parameters for read_extent_buffer: start (32771) > eb->len (32768). eb start is 2261163409408, level 100, generation 4412718571037421157, nritems 538968254. len param 17. debug 2/989/538968254/4412718571037421157/0x0/0/0x0/0x0 >>> >> >> At a first glance: the generation converted to ascii is: "ent() ==", >> so someone is patching the memory with ascii text, possibly C source. >> It might be interesting to dump the full contents of the eb, to get >> a clue on the source of the data. > > I changed the code to dump the contents of the eb struct at the point > where that error (btrfs: invalid parameters...) is printed, at the > "checksum mismatch 4" site and at the "node seems invalid now" site. > Now I have a big log of 1795 corrupted ebs. So far nothing that looks > remotely like ascii text, though. But I have two different versions of > the eb that caused that warning, a less corrupted one and a more > corrupted one: > > ------------------------------------------------------------ > btrfs: --- start eb contents at ffff8801b13cc4c8 --- > btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00 ...f............ > btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff .......0....... > btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff X.<.....X.<..... > btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff ........p.<..... > btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 p.<............. > btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff ._.............. > btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff @L.......f...... > btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff ........@....... > btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff @............... > btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 ..<............. > btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00 ........ this one looks good so far. > btrfs: --- end eb contents at ffff8801b13cc4c8 --- > btrfs: dm-6 checksum verify failed on 2239404212224 wanted B5F632BC found 3579FB59 level 160 > btrfs: node seems invalid now. checksum ok = 1 > btrfs: --- start eb contents at ffff8801b13cc4c8 --- > [... identical dump to above ...] > btrfs: --- end eb contents at ffff8801b13cc4c8 --- > btrfs: invalid parameters for read_extent_buffer: start (32771) > eb->len (32768). eb start is 2239404212224, level 160, generation 4716553384049587249, nritems 295705211. len param 17. debug 2/989/295705211/4716553384049587249/0x0/0/0x0/0x0 > btrfs: --- start eb contents at ffff8801b13cc4c8 --- > btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00 ...f............ > btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff .......0....... > btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff X.<.....X.<..... > btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff ........p.<..... > btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 p.<............. > btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff ._.............. > btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff @L.......f...... > btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff ........@....... > btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff @............... > btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00 ..<............. > btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00 ........{....... > btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00 14q<P.tA........ > btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00 ........ > btrfs: --- end eb contents at ffff8801b13cc4c8 --- > ------------[ cut here ]------------ > WARNING: at fs/btrfs/extent_io.c:4533 read_extent_buffer+0x1ef/0x220 [btrfs]() > Hardware name: System Product Name > Modules linked in: <omitted> [last unloaded: scsi_wait_scan] > Pid: 25829, comm: btrfs-endio-met Tainted: G W 3.4.4+btrfsdebug3+ #4 > [...] > ------------------------------------------------------------ > > Here's the difference (lines reordered from diff to make comparing > easier): > > ------------------------------------------------------------ > -btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 ..<............. > +btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00 ..<............. > -btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00 ........{....... > -btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00 14q<P.tA........ > ------------------------------------------------------------ The diff is just the eb->debug[] Jan added. > > If there's one pattern that catches eye in the dumps, it's that in > many places where the same eb is dumped multiple times due to multiple > "checksum mismatch 4"s, there are bytes at offsets 0x30 and 0x31 that > always seem to have the same value and both separately increase > between the dumps, usually(?) by two. Then that might be normal, I > haven't looked at what should be at a struct eb :) There are some > instances where they wrap, for example > > f8 f8 -> fa fa -> fc fc -> fe fe -> 00 00 -> 02 02 -> ..., this is the spinlock to protect the refcnt, so some reference taking/releasing has been going on. > > with no other changes visible in the corrupted ebs. But that is by no > means the only change in the struct eb contents that can be observed, > only the most obvious pattern. > > For example, here's a 11-long string of eb contents. First the dmesg: > > ------------------------------------------------------------ > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 1] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 2] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 3] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 4] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 5] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 6] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 7] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 8] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 9] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 10] > btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 > btrfs: checksum mismatch 4 on 1127200522240 > [DUMP 11] > ------------------------------------------------------------ > > Where DUMP 1 contents (file cc1.txt if you'd want to apply the diffs > below) is > > ------------------------------------------------------------ > btrfs: --- start eb contents at ffff8801cafbacc0 --- > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > btrfs: ffff8801cafbad30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad40: 00 00 10 00 00 00 00 00 04 04 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad50: 50 ad fb ca 01 88 ff ff 50 ad fb ca 01 88 ff ff P.......P....... > btrfs: ffff8801cafbad60: 02 02 00 00 00 00 00 00 68 ad fb ca 01 88 ff ff ........h....... > btrfs: ffff8801cafbad70: 68 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 h............... > btrfs: ffff8801cafbad80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad90: 00 a2 20 08 00 ea ff ff c0 2a f4 07 00 ea ff ff .. ......*...... > btrfs: ffff8801cafbada0: 00 ac 89 07 00 ea ff ff c0 42 79 07 00 ea ff ff .........By..... > btrfs: ffff8801cafbadb0: 80 af 77 07 00 ea ff ff 40 37 49 08 00 ea ff ff ..w.....@7I..... > btrfs: ffff8801cafbadc0: 80 6d f3 07 00 ea ff ff c0 a7 e8 07 00 ea ff ff .m.............. > btrfs: ffff8801cafbadd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 ........ > btrfs: --- end eb contents at ffff8801cafbacc0 --- > ------------------------------------------------------------ > > And the successive diffs are > > ------------------------------------------------------------ > --- cc1.txt 2012-07-10 06:57:21.564665577 +0300 > +++ cc2.txt 2012-07-10 06:59:10.272634578 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ the spinlock > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > @@ -20,9 +20,9 @@ > btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > -btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 ................ > -btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > -btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 01 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbae20: 4d 00 00 00 00 00 00 00 4e 00 00 00 00 00 00 00 M.......N....... > +btrfs: ffff8801cafbae30: 07 78 02 00 00 00 00 00 00 00 00 00 00 00 00 00 .x.............. and eb->debug > btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 ........ > btrfs: --- end eb contents at ffff8801cafbacc0 --- > --- cc2.txt 2012-07-10 06:59:10.272634578 +0300 > +++ cc3.txt 2012-07-10 06:59:10.016634663 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc3.txt 2012-07-10 06:59:10.016634663 +0300 > +++ cc4.txt 2012-07-10 06:59:09.752634749 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc4.txt 2012-07-10 06:59:09.752634749 +0300 > +++ cc5.txt 2012-07-10 06:59:09.504634831 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc5.txt 2012-07-10 06:59:09.504634831 +0300 > +++ cc6.txt 2012-07-10 06:59:09.240634917 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc6.txt 2012-07-10 06:59:09.240634917 +0300 > +++ cc7.txt 2012-07-10 06:59:08.944635015 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc7.txt 2012-07-10 06:59:08.944635015 +0300 > +++ cc8.txt 2012-07-10 06:59:08.656635109 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > +btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 !!.............. > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc8.txt 2012-07-10 06:59:08.656635109 +0300 > +++ cc9.txt 2012-07-10 06:59:08.328635216 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 !!.............. > +btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ##.............. > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc9.txt 2012-07-10 06:59:08.328635216 +0300 > +++ cc10.txt 2012-07-10 06:58:33.180646115 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ##.............. > +btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 %%.............. > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > --- cc10.txt 2012-07-10 06:58:33.180646115 +0300 > +++ cc11.txt 2012-07-10 06:58:41.340643696 +0300 > @@ -2,7 +2,7 @@ > btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ > btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... > -btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 %%.............. > +btrfs: ffff8801cafbacf0: 27 27 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ''.............. > btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... > ------------------------------------------------------------ All spinlocks. I'll go through this again with Jan to see if his eb->debug might give any clue here. It might be good to have a look at the contents of the pages though, as written in the previous mail. -Arne > > And since it's of course possible there's something wrong with my > modifications, here's the patch I used: > > ------------------------------------------------------------ > diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c > index 7452ecb..aadb82c 100644 > --- a/fs/btrfs/extent_io.c > +++ b/fs/btrfs/extent_io.c > @@ -4527,6 +4527,9 @@ void read_extent_buffer(struct extent_buffer *eb, void *dstv, > len, > eb->debug[0], eb->debug[1], eb->debug[2], eb->debug[3], > eb->debug[4], eb->debug[5], eb->debug[6], eb->debug[7]); > + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); > + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); > + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); > WARN_ON(1); > } > WARN_ON(start + len > eb->start + eb->len); > diff --git a/fs/btrfs/reada.c b/fs/btrfs/reada.c > index ea81bd4..663d6c4 100644 > --- a/fs/btrfs/reada.c > +++ b/fs/btrfs/reada.c > @@ -130,9 +130,13 @@ static int __readahead_hook(struct btrfs_root *root, struct extent_buffer *eb, > kref_get(&re->refcnt); > spin_unlock(&fs_info->reada_lock); > > - if (!err && btrfs_csum_tree_block(root, eb)) > + if (!err && btrfs_csum_tree_block(root, eb)) { > printk(KERN_ERR "btrfs: checksum mismatch 4 on %llu\n", > eb->start); > + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); > + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); > + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); > + } > > if (!re) > return -1; > @@ -150,9 +154,14 @@ static int __readahead_hook(struct btrfs_root *root, struct extent_buffer *eb, > if (err == 0) { > nritems = level ? btrfs_header_nritems(eb) : 0; > if (level > BTRFS_MAX_LEVEL || > - nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) > + nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) { > printk(KERN_ERR "btrfs: node seems invalid now. checksum ok = %d\n", > btrfs_csum_tree_block(root, eb)); > + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); > + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); > + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); > + > + } > generation = btrfs_header_generation(eb); > /* > * FIXME: currently we just set nritems to 0 if this is a leaf, > > ------------------------------------------------------------ > > Sami -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Any news on this? I you give me some hints, I can try to reproduce it here. -Arne On 10.07.2012 08:57, Arne Jansen wrote: > On 10.07.2012 06:16, Sami Liedes wrote: >> On Mon, Jul 09, 2012 at 11:05:47AM +0200, Arne Jansen wrote: >>>> * Just before the crash: >>>> btrfs: invalid parameters for read_extent_buffer: start (32771) > eb->len (32768). eb start is 2261163409408, level 100, generation 4412718571037421157, nritems 538968254. len param 17. debug 2/989/538968254/4412718571037421157/0x0/0/0x0/0x0 >>>> >>> >>> At a first glance: the generation converted to ascii is: "ent() ==", >>> so someone is patching the memory with ascii text, possibly C source. >>> It might be interesting to dump the full contents of the eb, to get >>> a clue on the source of the data. >> >> I changed the code to dump the contents of the eb struct at the point >> where that error (btrfs: invalid parameters...) is printed, at the >> "checksum mismatch 4" site and at the "node seems invalid now" site. >> Now I have a big log of 1795 corrupted ebs. So far nothing that looks >> remotely like ascii text, though. But I have two different versions of >> the eb that caused that warning, a less corrupted one and a more >> corrupted one: >> >> ------------------------------------------------------------ >> btrfs: --- start eb contents at ffff8801b13cc4c8 --- >> btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00 ...f............ >> btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff .......0....... >> btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff X.<.....X.<..... >> btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff ........p.<..... >> btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 p.<............. >> btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff ._.............. >> btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff @L.......f...... >> btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff ........@....... >> btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff @............... >> btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 ..<............. >> btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00 ........ > > this one looks good so far. > >> btrfs: --- end eb contents at ffff8801b13cc4c8 --- >> btrfs: dm-6 checksum verify failed on 2239404212224 wanted B5F632BC found 3579FB59 level 160 >> btrfs: node seems invalid now. checksum ok = 1 >> btrfs: --- start eb contents at ffff8801b13cc4c8 --- >> [... identical dump to above ...] >> btrfs: --- end eb contents at ffff8801b13cc4c8 --- >> btrfs: invalid parameters for read_extent_buffer: start (32771) > eb->len (32768). eb start is 2239404212224, level 160, generation 4716553384049587249, nritems 295705211. len param 17. debug 2/989/295705211/4716553384049587249/0x0/0/0x0/0x0 >> btrfs: --- start eb contents at ffff8801b13cc4c8 --- >> btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00 ...f............ >> btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff .......0....... >> btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff X.<.....X.<..... >> btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff ........p.<..... >> btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 p.<............. >> btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff ._.............. >> btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff @L.......f...... >> btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff ........@....... >> btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff @............... >> btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00 ..<............. >> btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00 ........{....... >> btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00 14q<P.tA........ >> btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00 ........ >> btrfs: --- end eb contents at ffff8801b13cc4c8 --- >> ------------[ cut here ]------------ >> WARNING: at fs/btrfs/extent_io.c:4533 read_extent_buffer+0x1ef/0x220 [btrfs]() >> Hardware name: System Product Name >> Modules linked in: <omitted> [last unloaded: scsi_wait_scan] >> Pid: 25829, comm: btrfs-endio-met Tainted: G W 3.4.4+btrfsdebug3+ #4 >> [...] >> ------------------------------------------------------------ >> >> Here's the difference (lines reordered from diff to make comparing >> easier): >> >> ------------------------------------------------------------ >> -btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 ..<............. >> +btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00 ..<............. >> -btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00 ........{....... >> -btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00 14q<P.tA........ >> ------------------------------------------------------------ > > The diff is just the eb->debug[] Jan added. > >> >> If there's one pattern that catches eye in the dumps, it's that in >> many places where the same eb is dumped multiple times due to multiple >> "checksum mismatch 4"s, there are bytes at offsets 0x30 and 0x31 that >> always seem to have the same value and both separately increase >> between the dumps, usually(?) by two. Then that might be normal, I >> haven't looked at what should be at a struct eb :) There are some >> instances where they wrap, for example >> >> f8 f8 -> fa fa -> fc fc -> fe fe -> 00 00 -> 02 02 -> ..., > > this is the spinlock to protect the refcnt, so some reference taking/releasing > has been going on. > >> >> with no other changes visible in the corrupted ebs. But that is by no >> means the only change in the struct eb contents that can be observed, >> only the most obvious pattern. >> >> For example, here's a 11-long string of eb contents. First the dmesg: >> >> ------------------------------------------------------------ >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 1] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 2] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 3] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 4] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 5] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 6] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 7] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 8] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 9] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 10] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 11] >> ------------------------------------------------------------ >> >> Where DUMP 1 contents (file cc1.txt if you'd want to apply the diffs >> below) is >> >> ------------------------------------------------------------ >> btrfs: --- start eb contents at ffff8801cafbacc0 --- >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> btrfs: ffff8801cafbad30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad40: 00 00 10 00 00 00 00 00 04 04 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad50: 50 ad fb ca 01 88 ff ff 50 ad fb ca 01 88 ff ff P.......P....... >> btrfs: ffff8801cafbad60: 02 02 00 00 00 00 00 00 68 ad fb ca 01 88 ff ff ........h....... >> btrfs: ffff8801cafbad70: 68 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 h............... >> btrfs: ffff8801cafbad80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad90: 00 a2 20 08 00 ea ff ff c0 2a f4 07 00 ea ff ff .. ......*...... >> btrfs: ffff8801cafbada0: 00 ac 89 07 00 ea ff ff c0 42 79 07 00 ea ff ff .........By..... >> btrfs: ffff8801cafbadb0: 80 af 77 07 00 ea ff ff 40 37 49 08 00 ea ff ff ..w.....@7I..... >> btrfs: ffff8801cafbadc0: 80 6d f3 07 00 ea ff ff c0 a7 e8 07 00 ea ff ff .m.............. >> btrfs: ffff8801cafbadd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 ........ >> btrfs: --- end eb contents at ffff8801cafbacc0 --- >> ------------------------------------------------------------ >> >> And the successive diffs are >> >> ------------------------------------------------------------ >> --- cc1.txt 2012-07-10 06:57:21.564665577 +0300 >> +++ cc2.txt 2012-07-10 06:59:10.272634578 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > > the spinlock > >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> @@ -20,9 +20,9 @@ >> btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> -btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 ................ >> -btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> -btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 01 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbae20: 4d 00 00 00 00 00 00 00 4e 00 00 00 00 00 00 00 M.......N....... >> +btrfs: ffff8801cafbae30: 07 78 02 00 00 00 00 00 00 00 00 00 00 00 00 00 .x.............. > > and eb->debug > >> btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 ........ >> btrfs: --- end eb contents at ffff8801cafbacc0 --- >> --- cc2.txt 2012-07-10 06:59:10.272634578 +0300 >> +++ cc3.txt 2012-07-10 06:59:10.016634663 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc3.txt 2012-07-10 06:59:10.016634663 +0300 >> +++ cc4.txt 2012-07-10 06:59:09.752634749 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc4.txt 2012-07-10 06:59:09.752634749 +0300 >> +++ cc5.txt 2012-07-10 06:59:09.504634831 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc5.txt 2012-07-10 06:59:09.504634831 +0300 >> +++ cc6.txt 2012-07-10 06:59:09.240634917 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc6.txt 2012-07-10 06:59:09.240634917 +0300 >> +++ cc7.txt 2012-07-10 06:59:08.944635015 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc7.txt 2012-07-10 06:59:08.944635015 +0300 >> +++ cc8.txt 2012-07-10 06:59:08.656635109 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 !!.............. >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc8.txt 2012-07-10 06:59:08.656635109 +0300 >> +++ cc9.txt 2012-07-10 06:59:08.328635216 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 !!.............. >> +btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ##.............. >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc9.txt 2012-07-10 06:59:08.328635216 +0300 >> +++ cc10.txt 2012-07-10 06:58:33.180646115 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ##.............. >> +btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 %%.............. >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc10.txt 2012-07-10 06:58:33.180646115 +0300 >> +++ cc11.txt 2012-07-10 06:58:41.340643696 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 %%.............. >> +btrfs: ffff8801cafbacf0: 27 27 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ''.............. >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> ------------------------------------------------------------ > > All spinlocks. > > I'll go through this again with Jan to see if his eb->debug might give > any clue here. It might be good to have a look at the contents of the > pages though, as written in the previous mail. > > -Arne > >> >> And since it's of course possible there's something wrong with my >> modifications, here's the patch I used: >> >> ------------------------------------------------------------ >> diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c >> index 7452ecb..aadb82c 100644 >> --- a/fs/btrfs/extent_io.c >> +++ b/fs/btrfs/extent_io.c >> @@ -4527,6 +4527,9 @@ void read_extent_buffer(struct extent_buffer *eb, void *dstv, >> len, >> eb->debug[0], eb->debug[1], eb->debug[2], eb->debug[3], >> eb->debug[4], eb->debug[5], eb->debug[6], eb->debug[7]); >> + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); >> + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); >> + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); >> WARN_ON(1); >> } >> WARN_ON(start + len > eb->start + eb->len); >> diff --git a/fs/btrfs/reada.c b/fs/btrfs/reada.c >> index ea81bd4..663d6c4 100644 >> --- a/fs/btrfs/reada.c >> +++ b/fs/btrfs/reada.c >> @@ -130,9 +130,13 @@ static int __readahead_hook(struct btrfs_root *root, struct extent_buffer *eb, >> kref_get(&re->refcnt); >> spin_unlock(&fs_info->reada_lock); >> >> - if (!err && btrfs_csum_tree_block(root, eb)) >> + if (!err && btrfs_csum_tree_block(root, eb)) { >> printk(KERN_ERR "btrfs: checksum mismatch 4 on %llu\n", >> eb->start); >> + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); >> + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); >> + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); >> + } >> >> if (!re) >> return -1; >> @@ -150,9 +154,14 @@ static int __readahead_hook(struct btrfs_root *root, struct extent_buffer *eb, >> if (err == 0) { >> nritems = level ? btrfs_header_nritems(eb) : 0; >> if (level > BTRFS_MAX_LEVEL || >> - nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) >> + nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) { >> printk(KERN_ERR "btrfs: node seems invalid now. checksum ok = %d\n", >> btrfs_csum_tree_block(root, eb)); >> + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); >> + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); >> + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); >> + >> + } >> generation = btrfs_header_generation(eb); >> /* >> * FIXME: currently we just set nritems to 0 if this is a leaf, >> >> ------------------------------------------------------------ >> >> Sami > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Jul 16, 2012 at 10:20:28AM +0200, Arne Jansen wrote: > Any news on this? I you give me some hints, I can try to reproduce > it here. I've been planning for a few days now to try if it's reproducible in a virtual machine with the same filesystem images. However I haven't gotten around to doing this yet... I tried to get KMEMCHECK working on my computer, but failed (and generated a bug report). It seems to work in KVM though. So, currently my idea is to boot the machine with a live USB stick, install kvm and make qemu qcow images backed by the real (2*1.1T) devices, but writing changes to the qcow images (I dare not mess with the actual devices, and don't happen to have quite 2.2T extra space outside of them...), and try to run scrub there. If that succeeds and the bug happens there too, debugging *should* be easier, and it *should* be possible to run it under KMEMCHECK too. If the bug doesn't happen inside a virtual machine, that would be interesting information too. Another idea might be to use LVM snapshots of the actual devices, but recent messages have made be wary of that approach - plus it's somewhat of a pain, because the snapshotting would have to be done before mounting anyway, since I have two devices and I doubt LVM supports snapshotting two separate devices at the exact same moment. However, I'm a bit ill at the moment, so I assume it might be at least a few days until I get to actually implementing this... Sami
On Tue, Jul 17, 2012 at 12:29:33AM +0300, Sami Liedes wrote: > So, currently my idea is to boot the machine with a live USB stick, > install kvm and make qemu qcow images backed by the real (2*1.1T) > devices, but writing changes to the qcow images (I dare not mess with > the actual devices, and don't happen to have quite 2.2T extra space > outside of them...), and try to run scrub there. If that succeeds and > the bug happens there too, debugging *should* be easier, and it > *should* be possible to run it under KMEMCHECK too. If the bug doesn't > happen inside a virtual machine, that would be interesting information > too. I have now been able to reproduce the bug in KVM with the setup described above. I think it's safe to say now that the bug depends on some kind of interaction between btrfs and dm-crypt. With the following setup, the bug does NOT happen: * kvm, single cpu * sees 3 disks, /dev/vda=root, /dev/vdb=btrfs-dev1, /dev/vdc=btrfs-dev2 * The btrfs devices are essentially snapshots of the real btrfs devices in raid-1 configuration (2*1.1T). As the real devices are encrypted, the decryption is done outside the KVM, i.e. the KVM snapshots are backed by the decrypted devices. With the following setup, the bug DOES happen: * kvm, single cpu * sees 3 disks, /dev/vda=root, /dev/vdb=part1, /dev/vdc=part2, where part[12] is are LUKS containers containing the individual btrfs devices * inside kvm, they are opened using cryptsetup luksOpen /dev/vdb root1 cryptsetup luksOpen /dev/vdc root2 * after this, the filesystem is mounted with mount /dev/mapper/root1 /media -o device=/dev/mapper/root1,device=/dev/mapper/root2 * The devices are snapshots of the actual physical encrypted partitions containing the btrfs devices. I have not yet figured out if this can be reproduced using a pristine, smaller btrfs filesystem in raid-1 configuration inside KVM or if there's something about my specific filesystem that causes this. I can investigate that too; it's easier to do for me than the above testing, as I don't need to have continuous physical access to the computer to do that. Here's the .config of the kernel I used inside KVM to reproduce this: http://www.niksula.hut.fi/~sliedes/btrfs/config.3.4.4 I also ran the same tests with KMEMCHECK. Both with and without crypto, there were quite a number of (of course possibly false) warnings from btrfs code. I doubt any of them are related to this bug - there were no KMEMCHECK warnings during the scrub operation. Here are the logs, anyway: http://www.niksula.hut.fi/~sliedes/btrfs/screenlog.nocrypto.gz http://www.niksula.hut.fi/~sliedes/btrfs/screenlog.crypto.gz Sami
On Sat, Jul 28, 2012 at 03:08:47PM +0300, Sami Liedes wrote: > I have not yet figured out if this can be reproduced using a pristine, > smaller btrfs filesystem in raid-1 configuration inside KVM or if > there's something about my specific filesystem that causes this. I can > investigate that too; it's easier to do for me than the above testing, > as I don't need to have continuous physical access to the computer to > do that. It seems the bug doesn't happen with a new filesystem made with mkfs.btrfs -mraid1 -draid1 /dev/mapper/btrfs_crypt /dev/mapper/btrfs2_crypt and subsequently filled with data... Unfortunate. I started to wonder what's so special about the second device of my filesystem. All the errors always seem to come from that device. The only thing that comes to mind is that that device was not originally part of the filesystem; it started as a single-device filesystem formatted with mkfs.btrfs with default options and was subsequently rebalanced under a 3.4.2 kernel. So I started to play with btrfs fi balance in my KVM instance with my two-device filesystem, and hit this oops, which may or may not be related to my previous bug and/or dm-crypt... ------------------------------------------------------------ [ 15.195342] device fsid a844eb60-eb9c-4e24-ae91-c1627bf2d439 devid 1 transid 176 /dev/mapper/btrfs_crypt [ 15.196606] device fsid a844eb60-eb9c-4e24-ae91-c1627bf2d439 devid 2 transid 176 /dev/mapper/btrfs2_crypt [ 15.197895] device fsid a844eb60-eb9c-4e24-ae91-c1627bf2d439 devid 1 transid 176 /dev/mapper/btrfs_crypt [ 15.200202] btrfs: disk space caching is enabled # btrfs device delete [something...] [ 1462.242456] btrfs: unable to go below two devices on raid1 # btrfs fi balance start -mconvert=dup -dconvert=raid0 /media [ 1895.048075] btrfs: unable to start balance with target metadata profile 32 # btrfs fi balance start -dconvert=raid0 /media [ 1917.106536] btrfs: relocating block group 10229907456 flags 17 [ 1929.188609] btrfs: relocating block group 8887730176 flags 17 [ 1944.690916] btrfs: found 2152 extents [ 1947.016210] btrfs: found 2152 extents [ 1947.421397] btrfs: relocating block group 7545552896 flags 17 [ 2024.225203] btrfs: found 36762 extents [ 2094.983055] btrfs: corrupt node block=8830455808,root=1: generation (197) too new in slot 0 (maximum expected 196) [ 2094.984858] ------------[ cut here ]------------ [ 2094.986076] WARNING: at fs/btrfs/super.c:219 __btrfs_abort_transaction+0xa5/0xc0() [ 2094.987912] Hardware name: Bochs [ 2094.988735] btrfs: Transaction abortedPid: 1623, comm: btrfs-transacti Not tainted 3.4.4 #9 [ 2094.988741] Call Trace: [ 2094.989361] [<ffffffff8103da65>] warn_slowpath_common+0x75/0xb0 [ 2094.990829] [<ffffffff8103db11>] warn_slowpath_fmt+0x41/0x50 [ 2094.992243] [<ffffffff8148a2b5>] __btrfs_abort_transaction+0xa5/0xc0 [ 2094.993521] [<ffffffff81498463>] __btrfs_free_extent+0x213/0x7b0 [ 2094.994176] [<ffffffff8149cac7>] ? run_clustered_refs+0xd7/0xa10 [ 2094.994835] [<ffffffff8158d9fd>] ? do_raw_spin_unlock+0x5d/0xb0 [ 2094.995481] [<ffffffff8149ce05>] run_clustered_refs+0x415/0xa10 [ 2094.996143] [<ffffffff814f13e8>] ? find_ref_head+0xb8/0xe0 [ 2094.996806] [<ffffffff8149d499>] ? btrfs_run_delayed_refs+0x99/0x430 [ 2094.997505] [<ffffffff8149d56d>] btrfs_run_delayed_refs+0x16d/0x430 [ 2094.998188] [<ffffffff81497c19>] ? next_block_group.isra.65+0x29/0x80 [ 2094.998890] [<ffffffff8176f326>] ? _raw_spin_unlock+0x26/0x30 [ 2094.999520] [<ffffffff8149d8e0>] btrfs_write_dirty_block_groups+0xb0/0x630 [ 2095.000281] [<ffffffff814d365a>] ? free_extent_buffer+0x1a/0x70 [ 2095.000931] [<ffffffff8176905d>] commit_cowonly_roots+0xe7/0x1c1 [ 2095.001610] [<ffffffff814aed29>] btrfs_commit_transaction+0x519/0xa40 [ 2095.002317] [<ffffffff8105f1d0>] ? abort_exclusive_wait+0xb0/0xb0 [ 2095.002997] [<ffffffff814a7a25>] transaction_kthread+0x245/0x2c0 [ 2095.003673] [<ffffffff814a77e0>] ? check_leaf.isra.105+0x300/0x300 [ 2095.004372] [<ffffffff8105e57e>] kthread+0x8e/0xa0 [ 2095.004902] [<ffffffff81771464>] kernel_thread_helper+0x4/0x10 [ 2095.005549] [<ffffffff8105e4f0>] ? kthread_flush_work_fn+0x10/0x10 [ 2095.006222] [<ffffffff81771460>] ? gs_change+0x13/0x13 [ 2095.006787] ---[ end trace 8341f112debcf176 ]--- [ 2095.007287] BTRFS error (device dm-1) in __btrfs_free_extent:5134: IO failure [ 2095.008059] btrfs is forced readonly [ 2095.008454] btrfs: run_one_delayed_ref returned -5 [ 2095.008455] BTRFS error (device dm-1) in btrfs_run_delayed_refs:2454: IO failure [ 2176.876382] ------------[ cut here ]------------ [ 2176.877217] kernel BUG at fs/btrfs/relocation.c:3733! [ 2176.878139] invalid opcode: 0000 [#1] SMP [ 2176.879049] CPU 5 [ 2176.879435] Pid: 1676, comm: btrfs Tainted: G W 3.4.4 #9 Bochs Bochs [ 2176.880383] RIP: 0010:[<ffffffff814f97d3>] [<ffffffff814f97d3>] relocate_block_group+0x643/0x690 [ 2176.880383] RSP: 0000:ffff880002bfdb08 EFLAGS: 00010206 [ 2176.880383] RAX: ffffffffffffffe2 RBX: ffffffffffffffe2 RCX: ffff880007aade10 [ 2176.880383] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880006fbe800 [ 2176.880383] RBP: ffff880002bfdb88 R08: 0000000000000003 R09: 0000000000000000 [ 2176.880383] R10: 0000000100072905 R11: 0000000000000001 R12: ffff8800037a8020 [ 2176.880383] R13: 0000000000005ee7 R14: ffff880006bbae10 R15: ffff8800037a8000 [ 2176.880383] FS: 0000000000000000(0000) GS:ffff880007d40000(0063) knlGS:00000000f75be720 [ 2176.880383] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b [ 2176.880383] CR2: 00000000f75c2c80 CR3: 0000000006664000 CR4: 00000000000006a0 [ 2176.880383] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2176.880383] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2176.880383] Process btrfs (pid: 1676, threadinfo ffff880002bfc000, task ffff880007aad880) [ 2176.880383] Stack: [ 2176.880383] ffff880005c94fb0 ffff880002bfdb50 0000000000000001 0000000000000001 [ 2176.880383] ffff880005c94bc8 0000000000000000 ffff880002bfdb88 00ffffff814afd9a [ 2176.880383] a800000001ebfdf0 000000000000c000 ffff8800037a8000 ffff8800037a8000 [ 2176.880383] Call Trace: [ 2176.880383] [<ffffffff814f99c4>] btrfs_relocate_block_group+0x1a4/0x2d0 [ 2176.880383] [<ffffffff8106cbad>] ? __wake_up+0x2d/0x70 [ 2176.880383] [<ffffffff814d67e5>] btrfs_relocate_chunk.isra.52+0x65/0x700 [ 2176.880383] [<ffffffff814d365a>] ? free_extent_buffer+0x1a/0x70 [ 2176.880383] [<ffffffff8158d9fd>] ? do_raw_spin_unlock+0x5d/0xb0 [ 2176.880383] [<ffffffff8176f326>] ? _raw_spin_unlock+0x26/0x30 [ 2176.880383] [<ffffffff814ce1e2>] ? release_extent_buffer.isra.41+0x32/0xc0 [ 2176.880383] [<ffffffff814d365a>] ? free_extent_buffer+0x1a/0x70 [ 2176.880383] [<ffffffff814d366a>] ? free_extent_buffer+0x2a/0x70 [ 2176.880383] [<ffffffff814dacbf>] btrfs_balance+0x7ff/0xce0 [ 2176.880383] [<ffffffff814df9d9>] btrfs_ioctl_balance.isra.51+0x139/0x430 [ 2176.880383] [<ffffffff814e3245>] btrfs_ioctl+0x95/0x1260 [ 2176.880383] [<ffffffff81063ace>] ? up_read+0x1e/0x40 [ 2176.880383] [<ffffffff81024abc>] ? do_page_fault+0x1ac/0x490 [ 2176.880383] [<ffffffff810ebf9b>] ? __vma_link_rb+0x2b/0x30 [ 2176.880383] [<ffffffff81152796>] compat_sys_ioctl+0x96/0x1310 [ 2176.880383] [<ffffffff81587a49>] ? lockdep_sys_exit_thunk+0x35/0x67 [ 2176.880383] [<ffffffff81771612>] sysenter_dispatch+0x7/0x25 [ 2176.880383] Code: ff ff 66 0f 1f 44 00 00 41 0f b6 87 59 06 00 00 83 c8 08 41 88 87 59 06 00 00 e9 10 fe ff ff bb f4 ff ff ff e9 88 fc ff ff 0f 0b <0f> 0b c7 45 98 00 00 00 00 4c 89 f7 e8 cc 27 f9 ff 48 83 ca ff [ 2176.880383] RIP [<ffffffff814f97d3>] relocate_block_group+0x643/0x690 [ 2176.880383] RSP <ffff880002bfdb08> [ 2176.922828] ---[ end trace 8341f112debcf177 ]--- ------------------------------------------------------------ Sami
--- cc1.txt 2012-07-10 06:57:21.564665577 +0300 +++ cc2.txt 2012-07-10 06:59:10.272634578 +0300 @@ -2,7 +2,7 @@ btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... -btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ +btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... @@ -20,9 +20,9 @@ btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ -btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 ................ -btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ -btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ +btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 01 00 00 00 00 00 00 00 ................ +btrfs: ffff8801cafbae20: 4d 00 00 00 00 00 00 00 4e 00 00 00 00 00 00 00 M.......N....... +btrfs: ffff8801cafbae30: 07 78 02 00 00 00 00 00 00 00 00 00 00 00 00 00 .x.............. btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 ........ btrfs: --- end eb contents at ffff8801cafbacc0 --- --- cc2.txt 2012-07-10 06:59:10.272634578 +0300 +++ cc3.txt 2012-07-10 06:59:10.016634663 +0300 @@ -2,7 +2,7 @@ btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... -btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ +btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... --- cc3.txt 2012-07-10 06:59:10.016634663 +0300 +++ cc4.txt 2012-07-10 06:59:09.752634749 +0300 @@ -2,7 +2,7 @@ btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... -btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ +btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... --- cc4.txt 2012-07-10 06:59:09.752634749 +0300 +++ cc5.txt 2012-07-10 06:59:09.504634831 +0300 @@ -2,7 +2,7 @@ btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... -btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ +btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... --- cc5.txt 2012-07-10 06:59:09.504634831 +0300 +++ cc6.txt 2012-07-10 06:59:09.240634917 +0300 @@ -2,7 +2,7 @@ btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... -btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ +btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... --- cc6.txt 2012-07-10 06:59:09.240634917 +0300 +++ cc7.txt 2012-07-10 06:59:08.944635015 +0300 @@ -2,7 +2,7 @@ btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... -btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ +btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... --- cc7.txt 2012-07-10 06:59:08.944635015 +0300 +++ cc8.txt 2012-07-10 06:59:08.656635109 +0300 @@ -2,7 +2,7 @@ btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... -btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ +btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 !!.............. btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... --- cc8.txt 2012-07-10 06:59:08.656635109 +0300 +++ cc9.txt 2012-07-10 06:59:08.328635216 +0300 @@ -2,7 +2,7 @@ btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... -btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 !!.............. +btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ##.............. btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... --- cc9.txt 2012-07-10 06:59:08.328635216 +0300 +++ cc10.txt 2012-07-10 06:58:33.180646115 +0300 @@ -2,7 +2,7 @@ btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... -btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ##.............. +btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 %%.............. btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... --- cc10.txt 2012-07-10 06:58:33.180646115 +0300 +++ cc11.txt 2012-07-10 06:58:41.340643696 +0300 @@ -2,7 +2,7 @@ btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... -btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 %%.............. +btrfs: ffff8801cafbacf0: 27 27 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ''.............. btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... ------------------------------------------------------------ And since it's of course possible there's something wrong with my modifications, here's the patch I used: ------------------------------------------------------------ diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index 7452ecb..aadb82c 100644 --- a/fs/btrfs/extent_io.c +++ b/fs/btrfs/extent_io.c @@ -4527,6 +4527,9 @@ void read_extent_buffer(struct extent_buffer *eb, void *dstv, len, eb->debug[0], eb->debug[1], eb->debug[2], eb->debug[3], eb->debug[4], eb->debug[5], eb->debug[6], eb->debug[7]); + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); WARN_ON(1); } WARN_ON(start + len > eb->start + eb->len); diff --git a/fs/btrfs/reada.c b/fs/btrfs/reada.c index ea81bd4..663d6c4 100644 --- a/fs/btrfs/reada.c +++ b/fs/btrfs/reada.c @@ -130,9 +130,13 @@ static int __readahead_hook(struct btrfs_root *root, struct extent_buffer *eb, kref_get(&re->refcnt); spin_unlock(&fs_info->reada_lock); - if (!err && btrfs_csum_tree_block(root, eb)) + if (!err && btrfs_csum_tree_block(root, eb)) { printk(KERN_ERR "btrfs: checksum mismatch 4 on %llu\n", eb->start); + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); + } if (!re) return -1; @@ -150,9 +154,14 @@ static int __readahead_hook(struct btrfs_root *root, struct extent_buffer *eb, if (err == 0) { nritems = level ? btrfs_header_nritems(eb) : 0; if (level > BTRFS_MAX_LEVEL || - nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) + nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) { printk(KERN_ERR "btrfs: node seems invalid now. checksum ok = %d\n", btrfs_csum_tree_block(root, eb)); + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); + + } generation = btrfs_header_generation(eb); /* * FIXME: currently we just set nritems to 0 if this is a leaf,