| Message ID | cover.1584468723.git.berto@igalia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Add subcluster allocation to qcow2 | expand |
Hello, This work is promising and interesting. I'd like to try this new feature. Could you please export a branch because the patches cannot be applied to current master? Thanks. Regards, Derek On 2020/3/18 上午2:15, Alberto Garcia wrote: > Hi, > > here's the new version of the patches to add subcluster allocation > support to qcow2. > > Please refer to the cover letter of the first version for a full > description of the patches: > > https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html > > I think that this version fixes all the problems pointed out by Max > and Eric during the review a couple of weeks ago. I also dropped the > RFC tag. > > Berto > > v4: > - Patch 01: New patch > - Patch 02: New patch > - Patch 05: Documentation updates [Eric] > - Patch 06: Fix rebase conflicts > - Patch 07: Change bit order in the subcluster allocation bitmap. > Change incompatible bit number. [Max, Eric] > - Patch 09: Rename QCOW_MAX_SUBCLUSTERS_PER_CLUSTER to > QCOW_EXTL2_SUBCLUSTERS_PER_CLUSTER [Eric] > - Patch 13: Change bit order in the subcluster allocation bitmap [Max, Eric] > Add more documentation. > Ignore the subcluster bitmap in the L2 entries of > compressed clusters. > - Patch 14: New patch > - Patch 15: Update to work with the changes from patches 02 and 14. > - Patch 16: Update to work with the changes from patches 02 and 14. > - Patch 18: Update to work with the changes from patches 02 and 14. > Update documentation. > Fix return value on early exit. > - Patch 20: Make sure to clear the subcluster allocation bitmap when a > cluster is unallocated. > - Patch 26: Update to work with the changes from patch 14. > - Patch 27: New patch [Max] > - Patch 28: Update version number, incompatible bit number and test > expectations. > - Patch 30: Add new tests. > Make the test verify its own results. [Max] > > v3: https://lists.gnu.org/archive/html/qemu-block/2019-12/msg00587.html > - Patch 01: Rename host_offset to host_cluster_offset and make 'bytes' > an unsigned int [Max] > - Patch 03: Rename cluster_needs_cow to cluster_needs_new_alloc and > count_cow_clusters to count_single_write_clusters. Update > documentation and add more assertions and checks [Max] > - Patch 09: Update qcow2_co_truncate() to properly support extended L2 > entries [Max] > - Patch 10: Forbid calling set_l2_bitmap() if the image does not have > extended L2 entries [Max] > - Patch 11 (new): Add QCow2SubclusterType [Max] > - Patch 12 (new): Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_* > - Patch 13 (new): Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC > - Patch 14: Use QCow2SubclusterType instead of QCow2ClusterType [Max] > - Patch 15: Use QCow2SubclusterType instead of QCow2ClusterType [Max] > - Patch 19: Don't call set_l2_bitmap() if the image does not have > extended L2 entries [Max] > - Patch 21: Use smaller data types. > - Patch 22: Don't call set_l2_bitmap() if the image does not have > extended L2 entries [Max] > - Patch 23: Use smaller data types. > - Patch 25: Update test results and documentation. Move the check for > the minimum subcluster size to validate_cluster_size(). > - Patch 26 (new): Add subcluster support to qcow2_measure() > - Patch 27: Add more tests > > v2: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg01642.html > - Patch 12: Update after the changes in 88f468e546. > - Patch 21 (new): Clear the L2 bitmap when allocating a compressed > cluster. Compressed clusters should have the bitmap all set to 0. > - Patch 24: Document the new fields in the QAPI documentation [Eric]. > - Patch 25: Allow qcow2 preallocation with backing files. > - Patch 26: Add some tests for qcow2 images with extended L2 entries. > > v1: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html > > Output of git backport-diff against v3: > > Key: > [----] : patches are identical > [####] : number of functional differences between upstream/downstream patch > [down] : patch is downstream-only > The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively > > 001/30:[down] 'qcow2: Make Qcow2AioTask store the full host offset' > 002/30:[down] 'qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()' > 003/30:[----] [-C] 'qcow2: Add calculate_l2_meta()' > 004/30:[----] [--] 'qcow2: Split cluster_needs_cow() out of count_cow_clusters()' > 005/30:[0020] [FC] 'qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()' > 006/30:[0010] [FC] 'qcow2: Add get_l2_entry() and set_l2_entry()' > 007/30:[0020] [FC] 'qcow2: Document the Extended L2 Entries feature' > 008/30:[----] [--] 'qcow2: Add dummy has_subclusters() function' > 009/30:[0004] [FC] 'qcow2: Add subcluster-related fields to BDRVQcow2State' > 010/30:[----] [--] 'qcow2: Add offset_to_sc_index()' > 011/30:[----] [-C] 'qcow2: Add l2_entry_size()' > 012/30:[----] [--] 'qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()' > 013/30:[0046] [FC] 'qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()' > 014/30:[down] 'qcow2: Add cluster type parameter to qcow2_get_host_offset()' > 015/30:[0082] [FC] 'qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*' > 016/30:[0002] [FC] 'qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC' > 017/30:[----] [-C] 'qcow2: Add subcluster support to calculate_l2_meta()' > 018/30:[down] 'qcow2: Add subcluster support to qcow2_get_host_offset()' > 019/30:[----] [--] 'qcow2: Add subcluster support to zero_in_l2_slice()' > 020/30:[0012] [FC] 'qcow2: Add subcluster support to discard_in_l2_slice()' > 021/30:[----] [--] 'qcow2: Add subcluster support to check_refcounts_l2()' > 022/30:[----] [--] 'qcow2: Fix offset calculation in handle_dependencies()' > 023/30:[----] [-C] 'qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()' > 024/30:[----] [--] 'qcow2: Clear the L2 bitmap when allocating a compressed cluster' > 025/30:[----] [--] 'qcow2: Add subcluster support to handle_alloc_space()' > 026/30:[0006] [FC] 'qcow2: Restrict qcow2_co_pwrite_zeroes() to full clusters only' > 027/30:[down] 'qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters' > 028/30:[0019] [FC] 'qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit' > 029/30:[----] [--] 'qcow2: Add subcluster support to qcow2_measure()' > 030/30:[0313] [FC] 'iotests: Add tests for qcow2 images with extended L2 entries' > > Alberto Garcia (30): > qcow2: Make Qcow2AioTask store the full host offset > qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset() > qcow2: Add calculate_l2_meta() > qcow2: Split cluster_needs_cow() out of count_cow_clusters() > qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied() > qcow2: Add get_l2_entry() and set_l2_entry() > qcow2: Document the Extended L2 Entries feature > qcow2: Add dummy has_subclusters() function > qcow2: Add subcluster-related fields to BDRVQcow2State > qcow2: Add offset_to_sc_index() > qcow2: Add l2_entry_size() > qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap() > qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type() > qcow2: Add cluster type parameter to qcow2_get_host_offset() > qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_* > qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC > qcow2: Add subcluster support to calculate_l2_meta() > qcow2: Add subcluster support to qcow2_get_host_offset() > qcow2: Add subcluster support to zero_in_l2_slice() > qcow2: Add subcluster support to discard_in_l2_slice() > qcow2: Add subcluster support to check_refcounts_l2() > qcow2: Fix offset calculation in handle_dependencies() > qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2() > qcow2: Clear the L2 bitmap when allocating a compressed cluster > qcow2: Add subcluster support to handle_alloc_space() > qcow2: Restrict qcow2_co_pwrite_zeroes() to full clusters only > qcow2: Assert that expand_zero_clusters_in_l1() does not support > subclusters > qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit > qcow2: Add subcluster support to qcow2_measure() > iotests: Add tests for qcow2 images with extended L2 entries > > docs/interop/qcow2.txt | 68 ++- > docs/qcow2-cache.txt | 19 +- > qapi/block-core.json | 7 + > block/qcow2.h | 178 +++++++- > include/block/block_int.h | 1 + > block/qcow2-cluster.c | 696 ++++++++++++++++++++----------- > block/qcow2-refcount.c | 38 +- > block/qcow2.c | 257 +++++++----- > tests/qemu-iotests/031.out | 8 +- > tests/qemu-iotests/036.out | 4 +- > tests/qemu-iotests/049.out | 102 ++--- > tests/qemu-iotests/060.out | 1 + > tests/qemu-iotests/061 | 6 + > tests/qemu-iotests/061.out | 25 +- > tests/qemu-iotests/065 | 18 +- > tests/qemu-iotests/082.out | 48 ++- > tests/qemu-iotests/085.out | 38 +- > tests/qemu-iotests/144.out | 4 +- > tests/qemu-iotests/182.out | 2 +- > tests/qemu-iotests/185.out | 8 +- > tests/qemu-iotests/198.out | 2 + > tests/qemu-iotests/206.out | 4 + > tests/qemu-iotests/242.out | 5 + > tests/qemu-iotests/255.out | 8 +- > tests/qemu-iotests/271 | 359 ++++++++++++++++ > tests/qemu-iotests/271.out | 244 +++++++++++ > tests/qemu-iotests/280.out | 2 +- > tests/qemu-iotests/common.filter | 1 + > tests/qemu-iotests/group | 1 + > 29 files changed, 1682 insertions(+), 472 deletions(-) > create mode 100755 tests/qemu-iotests/271 > create mode 100644 tests/qemu-iotests/271.out >
On Tue 21 Apr 2020 07:06:42 AM CEST, Derek Su <dereksu@qnap.com> wrote: > Hello, > > This work is promising and interesting. > I'd like to try this new feature. > Could you please export a branch because the patches cannot be applied > to current master? Hi, you can apply v4 on top of 3189e9d38c82266ea5750a81255fd229c7ddf1e6 I also plan to publish v5 this week. Any feedback is very much appreciated. Thanks! Berto
Hi, here's the new version of the patches to add subcluster allocation support to qcow2. Please refer to the cover letter of the first version for a full description of the patches: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html I think that this version fixes all the problems pointed out by Max and Eric during the review a couple of weeks ago. I also dropped the RFC tag. Berto v4: - Patch 01: New patch - Patch 02: New patch - Patch 05: Documentation updates [Eric] - Patch 06: Fix rebase conflicts - Patch 07: Change bit order in the subcluster allocation bitmap. Change incompatible bit number. [Max, Eric] - Patch 09: Rename QCOW_MAX_SUBCLUSTERS_PER_CLUSTER to QCOW_EXTL2_SUBCLUSTERS_PER_CLUSTER [Eric] - Patch 13: Change bit order in the subcluster allocation bitmap [Max, Eric] Add more documentation. Ignore the subcluster bitmap in the L2 entries of compressed clusters. - Patch 14: New patch - Patch 15: Update to work with the changes from patches 02 and 14. - Patch 16: Update to work with the changes from patches 02 and 14. - Patch 18: Update to work with the changes from patches 02 and 14. Update documentation. Fix return value on early exit. - Patch 20: Make sure to clear the subcluster allocation bitmap when a cluster is unallocated. - Patch 26: Update to work with the changes from patch 14. - Patch 27: New patch [Max] - Patch 28: Update version number, incompatible bit number and test expectations. - Patch 30: Add new tests. Make the test verify its own results. [Max] v3: https://lists.gnu.org/archive/html/qemu-block/2019-12/msg00587.html - Patch 01: Rename host_offset to host_cluster_offset and make 'bytes' an unsigned int [Max] - Patch 03: Rename cluster_needs_cow to cluster_needs_new_alloc and count_cow_clusters to count_single_write_clusters. Update documentation and add more assertions and checks [Max] - Patch 09: Update qcow2_co_truncate() to properly support extended L2 entries [Max] - Patch 10: Forbid calling set_l2_bitmap() if the image does not have extended L2 entries [Max] - Patch 11 (new): Add QCow2SubclusterType [Max] - Patch 12 (new): Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_* - Patch 13 (new): Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC - Patch 14: Use QCow2SubclusterType instead of QCow2ClusterType [Max] - Patch 15: Use QCow2SubclusterType instead of QCow2ClusterType [Max] - Patch 19: Don't call set_l2_bitmap() if the image does not have extended L2 entries [Max] - Patch 21: Use smaller data types. - Patch 22: Don't call set_l2_bitmap() if the image does not have extended L2 entries [Max] - Patch 23: Use smaller data types. - Patch 25: Update test results and documentation. Move the check for the minimum subcluster size to validate_cluster_size(). - Patch 26 (new): Add subcluster support to qcow2_measure() - Patch 27: Add more tests v2: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg01642.html - Patch 12: Update after the changes in 88f468e546. - Patch 21 (new): Clear the L2 bitmap when allocating a compressed cluster. Compressed clusters should have the bitmap all set to 0. - Patch 24: Document the new fields in the QAPI documentation [Eric]. - Patch 25: Allow qcow2 preallocation with backing files. - Patch 26: Add some tests for qcow2 images with extended L2 entries. v1: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html Output of git backport-diff against v3: Key: [----] : patches are identical [####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively 001/30:[down] 'qcow2: Make Qcow2AioTask store the full host offset' 002/30:[down] 'qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()' 003/30:[----] [-C] 'qcow2: Add calculate_l2_meta()' 004/30:[----] [--] 'qcow2: Split cluster_needs_cow() out of count_cow_clusters()' 005/30:[0020] [FC] 'qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()' 006/30:[0010] [FC] 'qcow2: Add get_l2_entry() and set_l2_entry()' 007/30:[0020] [FC] 'qcow2: Document the Extended L2 Entries feature' 008/30:[----] [--] 'qcow2: Add dummy has_subclusters() function' 009/30:[0004] [FC] 'qcow2: Add subcluster-related fields to BDRVQcow2State' 010/30:[----] [--] 'qcow2: Add offset_to_sc_index()' 011/30:[----] [-C] 'qcow2: Add l2_entry_size()' 012/30:[----] [--] 'qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()' 013/30:[0046] [FC] 'qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()' 014/30:[down] 'qcow2: Add cluster type parameter to qcow2_get_host_offset()' 015/30:[0082] [FC] 'qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*' 016/30:[0002] [FC] 'qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC' 017/30:[----] [-C] 'qcow2: Add subcluster support to calculate_l2_meta()' 018/30:[down] 'qcow2: Add subcluster support to qcow2_get_host_offset()' 019/30:[----] [--] 'qcow2: Add subcluster support to zero_in_l2_slice()' 020/30:[0012] [FC] 'qcow2: Add subcluster support to discard_in_l2_slice()' 021/30:[----] [--] 'qcow2: Add subcluster support to check_refcounts_l2()' 022/30:[----] [--] 'qcow2: Fix offset calculation in handle_dependencies()' 023/30:[----] [-C] 'qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()' 024/30:[----] [--] 'qcow2: Clear the L2 bitmap when allocating a compressed cluster' 025/30:[----] [--] 'qcow2: Add subcluster support to handle_alloc_space()' 026/30:[0006] [FC] 'qcow2: Restrict qcow2_co_pwrite_zeroes() to full clusters only' 027/30:[down] 'qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters' 028/30:[0019] [FC] 'qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit' 029/30:[----] [--] 'qcow2: Add subcluster support to qcow2_measure()' 030/30:[0313] [FC] 'iotests: Add tests for qcow2 images with extended L2 entries' Alberto Garcia (30): qcow2: Make Qcow2AioTask store the full host offset qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset() qcow2: Add calculate_l2_meta() qcow2: Split cluster_needs_cow() out of count_cow_clusters() qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied() qcow2: Add get_l2_entry() and set_l2_entry() qcow2: Document the Extended L2 Entries feature qcow2: Add dummy has_subclusters() function qcow2: Add subcluster-related fields to BDRVQcow2State qcow2: Add offset_to_sc_index() qcow2: Add l2_entry_size() qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap() qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type() qcow2: Add cluster type parameter to qcow2_get_host_offset() qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_* qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC qcow2: Add subcluster support to calculate_l2_meta() qcow2: Add subcluster support to qcow2_get_host_offset() qcow2: Add subcluster support to zero_in_l2_slice() qcow2: Add subcluster support to discard_in_l2_slice() qcow2: Add subcluster support to check_refcounts_l2() qcow2: Fix offset calculation in handle_dependencies() qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2() qcow2: Clear the L2 bitmap when allocating a compressed cluster qcow2: Add subcluster support to handle_alloc_space() qcow2: Restrict qcow2_co_pwrite_zeroes() to full clusters only qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit qcow2: Add subcluster support to qcow2_measure() iotests: Add tests for qcow2 images with extended L2 entries docs/interop/qcow2.txt | 68 ++- docs/qcow2-cache.txt | 19 +- qapi/block-core.json | 7 + block/qcow2.h | 178 +++++++- include/block/block_int.h | 1 + block/qcow2-cluster.c | 696 ++++++++++++++++++++----------- block/qcow2-refcount.c | 38 +- block/qcow2.c | 257 +++++++----- tests/qemu-iotests/031.out | 8 +- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/049.out | 102 ++--- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061 | 6 + tests/qemu-iotests/061.out | 25 +- tests/qemu-iotests/065 | 18 +- tests/qemu-iotests/082.out | 48 ++- tests/qemu-iotests/085.out | 38 +- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/185.out | 8 +- tests/qemu-iotests/198.out | 2 + tests/qemu-iotests/206.out | 4 + tests/qemu-iotests/242.out | 5 + tests/qemu-iotests/255.out | 8 +- tests/qemu-iotests/271 | 359 ++++++++++++++++ tests/qemu-iotests/271.out | 244 +++++++++++ tests/qemu-iotests/280.out | 2 +- tests/qemu-iotests/common.filter | 1 + tests/qemu-iotests/group | 1 + 29 files changed, 1682 insertions(+), 472 deletions(-) create mode 100755 tests/qemu-iotests/271 create mode 100644 tests/qemu-iotests/271.out