| Message ID | cover.1588699789.git.berto@igalia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Add subcluster allocation to qcow2 | expand |
Hi, Berto Excuse me, I'd like to test v5, but I failed to apply the series to master branch. Which commit can I use? Thanks. Regards, Derek On 2020/5/6 上午1:38, Alberto Garcia wrote: > Hi, > > here's the new version of the patches to add subcluster allocation > support to qcow2. > > Please refer to the cover letter of the first version for a full > description of the patches: > > https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html > > Important changes here: > > - I fixed hopefully all of the issues mentioned in the previous > review. Thanks to everyone who contributed. > > - There's now support for partial zeroing of clusters (i.e. at the > subcluster level). > > - Many more tests. > > - QCOW_OFLAG_ZERO is simply ignored now and not considered a sign of a > corrupt image anymore. I hesitated about this, but we could still > add that check later. I think there's a case for adding a new > QCOW2_CLUSTER_INVALID type and include this and other scenarios that > we already consider corrupt (for example: clusters with unaligned > offsets). We would need to see if for 'qemu-img check' adding > QCOW2_CLUSTER_INVALID complicates things or not. But I think that > all is material for its own series. > > And I think that's all. See below for the detailed list of changes, > and thanks again for the feedback. > > Berto > > v5: > - Patch 01: Fix indentation [Max], add trace event [Vladimir] > - Patch 02: Add host_cluster_offset variable [Vladirmir] > - Patch 05: Have separate l2_entry and cluster_offset variables [Vladimir] > - Patch 06: Only context changes due to patch 05 > - Patch 11: New patch > - Patch 13: Change documentation of get_l2_entry() > - Patch 14: Add QCOW_OFLAG_SUB_{ALLOC,ZERO}_RANGE [Eric] and rewrite > the other macros. > Ignore QCOW_OFLAG_ZERO on images with subclusters > (i.e. don't treat them as corrupted). > - Patch 15: New patch > - Patch 19: Optimize cow by skipping all leading and trailing zero and > unallocated subclusters [Vladimir] > Return 0 on success [Vladimir] > Squash patch that updated handle_dependencies() [Vladirmir] > - Patch 20: Call count_contiguous_subclusters() after the main switch > in qcow2_get_host_offset() [Vladimir] > Add assertion and remove goto statement [Vladimir] > - Patch 21: Rewrite algorithm. > - Patch 22: Rewrite algorithm. > - Patch 24: Replace loop with the _RANGE macros from patch 14 [Eric] > - Patch 27: New patch > - Patch 28: Update version number and expected output from tests. > - Patch 31: Add many more new tests > > v4: https://lists.gnu.org/archive/html/qemu-block/2020-03/msg00966.html > v3: https://lists.gnu.org/archive/html/qemu-block/2019-12/msg00587.html > v2: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg01642.html > v1: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html > > Output of git backport-diff against v4: > > Key: > [----] : patches are identical > [####] : number of functional differences between upstream/downstream patch > [down] : patch is downstream-only > The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively > > 001/31:[0005] [FC] 'qcow2: Make Qcow2AioTask store the full host offset' > 002/31:[0018] [FC] 'qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()' > 003/31:[----] [--] 'qcow2: Add calculate_l2_meta()' > 004/31:[----] [--] 'qcow2: Split cluster_needs_cow() out of count_cow_clusters()' > 005/31:[0038] [FC] 'qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()' > 006/31:[0004] [FC] 'qcow2: Add get_l2_entry() and set_l2_entry()' > 007/31:[----] [--] 'qcow2: Document the Extended L2 Entries feature' > 008/31:[----] [--] 'qcow2: Add dummy has_subclusters() function' > 009/31:[----] [--] 'qcow2: Add subcluster-related fields to BDRVQcow2State' > 010/31:[----] [--] 'qcow2: Add offset_to_sc_index()' > 011/31:[down] 'qcow2: Add offset_into_subcluster() and size_to_subclusters()' > 012/31:[----] [--] 'qcow2: Add l2_entry_size()' > 013/31:[0003] [FC] 'qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()' > 014/31:[0023] [FC] 'qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()' > 015/31:[down] 'qcow2: Add qcow2_cluster_is_allocated()' > 016/31:[----] [--] 'qcow2: Add cluster type parameter to qcow2_get_host_offset()' > 017/31:[----] [--] 'qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*' > 018/31:[----] [--] 'qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC' > 019/31:[0066] [FC] 'qcow2: Add subcluster support to calculate_l2_meta()' > 020/31:[0022] [FC] 'qcow2: Add subcluster support to qcow2_get_host_offset()' > 021/31:[0040] [FC] 'qcow2: Add subcluster support to zero_in_l2_slice()' > 022/31:[0061] [FC] 'qcow2: Add subcluster support to discard_in_l2_slice()' > 023/31:[----] [--] 'qcow2: Add subcluster support to check_refcounts_l2()' > 024/31:[0019] [FC] 'qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()' > 025/31:[----] [--] 'qcow2: Clear the L2 bitmap when allocating a compressed cluster' > 026/31:[----] [--] 'qcow2: Add subcluster support to handle_alloc_space()' > 027/31:[down] 'qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()' > 028/31:[0105] [FC] 'qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit' > 029/31:[----] [-C] 'qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters' > 030/31:[----] [--] 'qcow2: Add subcluster support to qcow2_measure()' > 031/31:[0694] [FC] 'iotests: Add tests for qcow2 images with extended L2 entries' > > Alberto Garcia (31): > qcow2: Make Qcow2AioTask store the full host offset > qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset() > qcow2: Add calculate_l2_meta() > qcow2: Split cluster_needs_cow() out of count_cow_clusters() > qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied() > qcow2: Add get_l2_entry() and set_l2_entry() > qcow2: Document the Extended L2 Entries feature > qcow2: Add dummy has_subclusters() function > qcow2: Add subcluster-related fields to BDRVQcow2State > qcow2: Add offset_to_sc_index() > qcow2: Add offset_into_subcluster() and size_to_subclusters() > qcow2: Add l2_entry_size() > qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap() > qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type() > qcow2: Add qcow2_cluster_is_allocated() > qcow2: Add cluster type parameter to qcow2_get_host_offset() > qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_* > qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC > qcow2: Add subcluster support to calculate_l2_meta() > qcow2: Add subcluster support to qcow2_get_host_offset() > qcow2: Add subcluster support to zero_in_l2_slice() > qcow2: Add subcluster support to discard_in_l2_slice() > qcow2: Add subcluster support to check_refcounts_l2() > qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2() > qcow2: Clear the L2 bitmap when allocating a compressed cluster > qcow2: Add subcluster support to handle_alloc_space() > qcow2: Add subcluster support to qcow2_co_pwrite_zeroes() > qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit > qcow2: Assert that expand_zero_clusters_in_l1() does not support > subclusters > qcow2: Add subcluster support to qcow2_measure() > iotests: Add tests for qcow2 images with extended L2 entries > > docs/interop/qcow2.txt | 68 ++- > docs/qcow2-cache.txt | 19 +- > qapi/block-core.json | 7 + > block/qcow2.h | 204 ++++++- > include/block/block_int.h | 1 + > block/qcow2-cluster.c | 892 ++++++++++++++++++++----------- > block/qcow2-refcount.c | 38 +- > block/qcow2.c | 283 ++++++---- > block/trace-events | 2 +- > tests/qemu-iotests/031.out | 8 +- > tests/qemu-iotests/036.out | 4 +- > tests/qemu-iotests/049.out | 102 ++-- > tests/qemu-iotests/060.out | 1 + > tests/qemu-iotests/061 | 6 + > tests/qemu-iotests/061.out | 25 +- > tests/qemu-iotests/065 | 18 +- > tests/qemu-iotests/082.out | 48 +- > tests/qemu-iotests/085.out | 38 +- > tests/qemu-iotests/144.out | 4 +- > tests/qemu-iotests/182.out | 2 +- > tests/qemu-iotests/185.out | 8 +- > tests/qemu-iotests/198.out | 2 + > tests/qemu-iotests/206.out | 4 + > tests/qemu-iotests/242.out | 5 + > tests/qemu-iotests/255.out | 8 +- > tests/qemu-iotests/271 | 664 +++++++++++++++++++++++ > tests/qemu-iotests/271.out | 519 ++++++++++++++++++ > tests/qemu-iotests/274.out | 49 +- > tests/qemu-iotests/280.out | 2 +- > tests/qemu-iotests/common.filter | 1 + > tests/qemu-iotests/group | 1 + > 31 files changed, 2459 insertions(+), 574 deletions(-) > create mode 100755 tests/qemu-iotests/271 > create mode 100644 tests/qemu-iotests/271.out >
On Wed 20 May 2020 11:35:09 AM CEST, Derek Su wrote: > Hi, Berto > > Excuse me, I'd like to test v5, but I failed to apply the series to > master branch. Which commit can I use? Try applying the patches on top of commit e4d7019e1a Berto
Hi, here's the new version of the patches to add subcluster allocation support to qcow2. Please refer to the cover letter of the first version for a full description of the patches: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html Important changes here: - I fixed hopefully all of the issues mentioned in the previous review. Thanks to everyone who contributed. - There's now support for partial zeroing of clusters (i.e. at the subcluster level). - Many more tests. - QCOW_OFLAG_ZERO is simply ignored now and not considered a sign of a corrupt image anymore. I hesitated about this, but we could still add that check later. I think there's a case for adding a new QCOW2_CLUSTER_INVALID type and include this and other scenarios that we already consider corrupt (for example: clusters with unaligned offsets). We would need to see if for 'qemu-img check' adding QCOW2_CLUSTER_INVALID complicates things or not. But I think that all is material for its own series. And I think that's all. See below for the detailed list of changes, and thanks again for the feedback. Berto v5: - Patch 01: Fix indentation [Max], add trace event [Vladimir] - Patch 02: Add host_cluster_offset variable [Vladirmir] - Patch 05: Have separate l2_entry and cluster_offset variables [Vladimir] - Patch 06: Only context changes due to patch 05 - Patch 11: New patch - Patch 13: Change documentation of get_l2_entry() - Patch 14: Add QCOW_OFLAG_SUB_{ALLOC,ZERO}_RANGE [Eric] and rewrite the other macros. Ignore QCOW_OFLAG_ZERO on images with subclusters (i.e. don't treat them as corrupted). - Patch 15: New patch - Patch 19: Optimize cow by skipping all leading and trailing zero and unallocated subclusters [Vladimir] Return 0 on success [Vladimir] Squash patch that updated handle_dependencies() [Vladirmir] - Patch 20: Call count_contiguous_subclusters() after the main switch in qcow2_get_host_offset() [Vladimir] Add assertion and remove goto statement [Vladimir] - Patch 21: Rewrite algorithm. - Patch 22: Rewrite algorithm. - Patch 24: Replace loop with the _RANGE macros from patch 14 [Eric] - Patch 27: New patch - Patch 28: Update version number and expected output from tests. - Patch 31: Add many more new tests v4: https://lists.gnu.org/archive/html/qemu-block/2020-03/msg00966.html v3: https://lists.gnu.org/archive/html/qemu-block/2019-12/msg00587.html v2: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg01642.html v1: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html Output of git backport-diff against v4: Key: [----] : patches are identical [####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively 001/31:[0005] [FC] 'qcow2: Make Qcow2AioTask store the full host offset' 002/31:[0018] [FC] 'qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()' 003/31:[----] [--] 'qcow2: Add calculate_l2_meta()' 004/31:[----] [--] 'qcow2: Split cluster_needs_cow() out of count_cow_clusters()' 005/31:[0038] [FC] 'qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()' 006/31:[0004] [FC] 'qcow2: Add get_l2_entry() and set_l2_entry()' 007/31:[----] [--] 'qcow2: Document the Extended L2 Entries feature' 008/31:[----] [--] 'qcow2: Add dummy has_subclusters() function' 009/31:[----] [--] 'qcow2: Add subcluster-related fields to BDRVQcow2State' 010/31:[----] [--] 'qcow2: Add offset_to_sc_index()' 011/31:[down] 'qcow2: Add offset_into_subcluster() and size_to_subclusters()' 012/31:[----] [--] 'qcow2: Add l2_entry_size()' 013/31:[0003] [FC] 'qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()' 014/31:[0023] [FC] 'qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()' 015/31:[down] 'qcow2: Add qcow2_cluster_is_allocated()' 016/31:[----] [--] 'qcow2: Add cluster type parameter to qcow2_get_host_offset()' 017/31:[----] [--] 'qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*' 018/31:[----] [--] 'qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC' 019/31:[0066] [FC] 'qcow2: Add subcluster support to calculate_l2_meta()' 020/31:[0022] [FC] 'qcow2: Add subcluster support to qcow2_get_host_offset()' 021/31:[0040] [FC] 'qcow2: Add subcluster support to zero_in_l2_slice()' 022/31:[0061] [FC] 'qcow2: Add subcluster support to discard_in_l2_slice()' 023/31:[----] [--] 'qcow2: Add subcluster support to check_refcounts_l2()' 024/31:[0019] [FC] 'qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()' 025/31:[----] [--] 'qcow2: Clear the L2 bitmap when allocating a compressed cluster' 026/31:[----] [--] 'qcow2: Add subcluster support to handle_alloc_space()' 027/31:[down] 'qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()' 028/31:[0105] [FC] 'qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit' 029/31:[----] [-C] 'qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters' 030/31:[----] [--] 'qcow2: Add subcluster support to qcow2_measure()' 031/31:[0694] [FC] 'iotests: Add tests for qcow2 images with extended L2 entries' Alberto Garcia (31): qcow2: Make Qcow2AioTask store the full host offset qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset() qcow2: Add calculate_l2_meta() qcow2: Split cluster_needs_cow() out of count_cow_clusters() qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied() qcow2: Add get_l2_entry() and set_l2_entry() qcow2: Document the Extended L2 Entries feature qcow2: Add dummy has_subclusters() function qcow2: Add subcluster-related fields to BDRVQcow2State qcow2: Add offset_to_sc_index() qcow2: Add offset_into_subcluster() and size_to_subclusters() qcow2: Add l2_entry_size() qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap() qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type() qcow2: Add qcow2_cluster_is_allocated() qcow2: Add cluster type parameter to qcow2_get_host_offset() qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_* qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC qcow2: Add subcluster support to calculate_l2_meta() qcow2: Add subcluster support to qcow2_get_host_offset() qcow2: Add subcluster support to zero_in_l2_slice() qcow2: Add subcluster support to discard_in_l2_slice() qcow2: Add subcluster support to check_refcounts_l2() qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2() qcow2: Clear the L2 bitmap when allocating a compressed cluster qcow2: Add subcluster support to handle_alloc_space() qcow2: Add subcluster support to qcow2_co_pwrite_zeroes() qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters qcow2: Add subcluster support to qcow2_measure() iotests: Add tests for qcow2 images with extended L2 entries docs/interop/qcow2.txt | 68 ++- docs/qcow2-cache.txt | 19 +- qapi/block-core.json | 7 + block/qcow2.h | 204 ++++++- include/block/block_int.h | 1 + block/qcow2-cluster.c | 892 ++++++++++++++++++++----------- block/qcow2-refcount.c | 38 +- block/qcow2.c | 283 ++++++---- block/trace-events | 2 +- tests/qemu-iotests/031.out | 8 +- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/049.out | 102 ++-- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061 | 6 + tests/qemu-iotests/061.out | 25 +- tests/qemu-iotests/065 | 18 +- tests/qemu-iotests/082.out | 48 +- tests/qemu-iotests/085.out | 38 +- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/185.out | 8 +- tests/qemu-iotests/198.out | 2 + tests/qemu-iotests/206.out | 4 + tests/qemu-iotests/242.out | 5 + tests/qemu-iotests/255.out | 8 +- tests/qemu-iotests/271 | 664 +++++++++++++++++++++++ tests/qemu-iotests/271.out | 519 ++++++++++++++++++ tests/qemu-iotests/274.out | 49 +- tests/qemu-iotests/280.out | 2 +- tests/qemu-iotests/common.filter | 1 + tests/qemu-iotests/group | 1 + 31 files changed, 2459 insertions(+), 574 deletions(-) create mode 100755 tests/qemu-iotests/271 create mode 100644 tests/qemu-iotests/271.out