| Message ID | 1578990137-308222-1-git-send-email-andrey.shinkevich@virtuozzo.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Dump QCOW2 metadata | expand |
On 14.01.20 09:22, Andrey Shinkevich wrote: > The information about QCOW2 metadata allocations in an image ELF-file is > helpful for finding issues with the image data integrity. Sorry that I’m replying only so late – but I don’t know why we need this in qemu, and this cover letter doesn’t provide a justification. I mean, it isn’t too complex (from the diffstat), but wouldn’t it be better to just have a script for this? I suppose one reason to put it in qemu/qemu-img is that a script wouldn’t be packaged by distributions. So if a user has a corrupted image, with this series we could tell them to run qemu-img check -M and put the output somewhere. With a script, we’d first have to tell them to download the script. But then again, downloading a script (that should be part of the qemu repository) doesn’t seem too much trouble to me either. So I’m curious as to whether you had a specific reason in mind when you decided to implement this as part of qemu itself? (I suppose the additional complexity is fully limited to the check infrastructure, so it wouldn’t interfere with the rest of the qcow2 driver. Hm. Fair enough.) Max
20.02.2020 14:58, Max Reitz wrote: > On 14.01.20 09:22, Andrey Shinkevich wrote: >> The information about QCOW2 metadata allocations in an image ELF-file is >> helpful for finding issues with the image data integrity. > > Sorry that I’m replying only so late – but I don’t know why we need this > in qemu, and this cover letter doesn’t provide a justification. I mean, > it isn’t too complex (from the diffstat), but wouldn’t it be better to > just have a script for this? > > I suppose one reason to put it in qemu/qemu-img is that a script > wouldn’t be packaged by distributions. So if a user has a corrupted > image, with this series we could tell them to run qemu-img check -M and > put the output somewhere. With a script, we’d first have to tell them > to download the script. But then again, downloading a script (that > should be part of the qemu repository) doesn’t seem too much trouble to > me either. > > So I’m curious as to whether you had a specific reason in mind when you > decided to implement this as part of qemu itself? > > (I suppose the additional complexity is fully limited to the check > infrastructure, so it wouldn’t interfere with the rest of the qcow2 > driver. Hm. Fair enough.) > Just not to parse qcow2 from scratch. Qemu already can read qcow2, and it looks through all its structures during check, why not to add an ability to represent these structures as a qobject?
Am 20.02.2020 um 12:58 hat Max Reitz geschrieben: > On 14.01.20 09:22, Andrey Shinkevich wrote: > > The information about QCOW2 metadata allocations in an image ELF-file is > > helpful for finding issues with the image data integrity. > > Sorry that I’m replying only so late – but I don’t know why we need this > in qemu, and this cover letter doesn’t provide a justification. I mean, > it isn’t too complex (from the diffstat), but wouldn’t it be better to > just have a script for this? Specifically, we could extend tests/qemu-iotests/qcow2.py. This seems to be debugging output that would be in line with what the script is already used for. Kevin
On 2/20/20 6:28 AM, Kevin Wolf wrote: > Am 20.02.2020 um 12:58 hat Max Reitz geschrieben: >> On 14.01.20 09:22, Andrey Shinkevich wrote: >>> The information about QCOW2 metadata allocations in an image ELF-file is >>> helpful for finding issues with the image data integrity. >> >> Sorry that I’m replying only so late – but I don’t know why we need this >> in qemu, and this cover letter doesn’t provide a justification. I mean, >> it isn’t too complex (from the diffstat), but wouldn’t it be better to >> just have a script for this? > > Specifically, we could extend tests/qemu-iotests/qcow2.py. This seems to > be debugging output that would be in line with what the script is > already used for. I also just discovered GNU poke, http://jemarch.net/poke, which is an arbitrary binary-format editor with a fairly good example of how it can be used to inspect ELF files. I'm wondering if it would be easier to write a pickle describing the qcow2 format that would make it easier to do interactive browsing/editing of a qcow2 file, at the expense of having to depend on poke (which has not yet hit the 1.0 release and is not yet bundled for Fedora).
On 20.02.20 13:10, Vladimir Sementsov-Ogievskiy wrote: > 20.02.2020 14:58, Max Reitz wrote: >> On 14.01.20 09:22, Andrey Shinkevich wrote: >>> The information about QCOW2 metadata allocations in an image ELF-file is >>> helpful for finding issues with the image data integrity. >> >> Sorry that I’m replying only so late – but I don’t know why we need this >> in qemu, and this cover letter doesn’t provide a justification. I mean, >> it isn’t too complex (from the diffstat), but wouldn’t it be better to >> just have a script for this? >> >> I suppose one reason to put it in qemu/qemu-img is that a script >> wouldn’t be packaged by distributions. So if a user has a corrupted >> image, with this series we could tell them to run qemu-img check -M and >> put the output somewhere. With a script, we’d first have to tell them >> to download the script. But then again, downloading a script (that >> should be part of the qemu repository) doesn’t seem too much trouble to >> me either. >> >> So I’m curious as to whether you had a specific reason in mind when you >> decided to implement this as part of qemu itself? >> >> (I suppose the additional complexity is fully limited to the check >> infrastructure, so it wouldn’t interfere with the rest of the qcow2 >> driver. Hm. Fair enough.) >> > > Just not to parse qcow2 from scratch. Qemu already can read qcow2, and > it looks through all its structures during check, why not > to add an ability to represent these structures as a qobject? Because it’d be code in qemu (i.e., a liability) that users are pretty much never going to use. Max
The information about QCOW2 metadata allocations in an image ELF-file is helpful for finding issues with the image data integrity. v3: Descriptions of the new key option were added. The names of identifiers were amended. The QEMU-IMG key options were put in the sorted order in the code. All suggested by Eric. Discussed in the email thread with ID <1577447039-400109-1-git-send-email-andrey.shinkevich@virtuozzo.com> Snapshots dump example: $ sudo ./qemu-img check /.../.../harddisk.hdd -M --output=json { "image-end-offset": 24820842496, "total-clusters": 153600, "check-errors": 0, "viscera": { "refcount-table": { "location": { "offset": 3845128192, "size": 1048576 } }, "active-l1": { "name": "L1 active table", "location": { "offset": 4194304, "size": 16 }, "l2-list": [ { "offset": 619708416, "size": 1048576 }, { "offset": 1156579328, "size": 1048576 } ] }, "qcow2-header": { "location": { "offset": 0, "size": 1048576 }, "version": 3 }, "snapshot-table": { "location": { "offset": 648019968, "size": 191 }, "l1-list": [ { "name": "{3036f6c5-3a1f-44cb-af1f-653cc87fba04}", "location": { "offset": 14680064, "size": 16 }, "l2-list": [ { "offset": 3957325824, "size": 1048576 }, { "offset": 7025459200, "size": 1048576 } ] }, { "name": "{0aa1a7d6-16ee-4b44-a515-b5ecc571c959}", "location": { "offset": 638582784, "size": 16 }, "l2-list": [ { "offset": 3957325824, "size": 1048576 }, { "offset": 7025459200, "size": 1048576 } ] } ] } }, "allocated-clusters": 22485, "filename": "/.../.../harddisk.hdd", "format": "qcow2", "fragmented-clusters": 3549 } Bitmaps dump example: $ ./qemu-img check /home/disk -M --output=json { "image-end-offset": 1441792, "total-clusters": 16, "check-errors": 0, "viscera": { "refcount-table": { "location": { "offset": 65536, "size": 65536 } }, "active-l1": { "name": "L1 active table", "location": { "offset": 196608, "size": 8 }, "l2-list": [ { "offset": 262144, "size": 65536 } ] }, "bitmaps": { "bitmap-dir": { "location": { "offset": 1048576, "size": 64 }, "dir-entries": [ { "bitmap-table": { "location": { "offset": 589824, "size": 8 }, "table-entries": [ { "type": "all-zeros" } ] }, "bitmap-name": "bitmap-1" }, { "bitmap-table": { "location": { "offset": 983040, "size": 8 }, "table-entries": [ { "cluster": { "offset": 655360, "size": 65536 }, "type": "serialized" } ] }, "bitmap-name": "bitmap-2" } ] }, "nb-bitmaps": 2 }, "qcow2-header": { "location": { "offset": 0, "size": 65536 }, "version": 3 } }, "allocated-clusters": 12, "filename": "/home/disk", "format": "qcow2", "fragmented-clusters": 2 } Andrey Shinkevich (3): qcow2: introduce Qcow2Metadata structure qemu-img: sort key options alphabetically qcow2: dump QCOW2 metadata block/qcow2-bitmap.c | 54 ++++++++++++- block/qcow2-refcount.c | 84 ++++++++++++++++---- block/qcow2.c | 30 +++++++ block/qcow2.h | 6 +- include/block/block.h | 3 +- qapi/block-core.json | 209 ++++++++++++++++++++++++++++++++++++++++++++++++- qemu-img.c | 50 +++++++++--- qemu-img.texi | 7 +- 8 files changed, 408 insertions(+), 35 deletions(-)