mbox

[PULL,v2,00/11] bitmaps patches for 2020-05-26

Message ID 20200527213708.1783644-1-eblake@redhat.com (mailing list archive)
State New, archived
Headers show

Pull-request

https://repo.or.cz/qemu/ericb.git tags/pull-bitmaps-2020-05-26-v2

Message

Eric Blake May 27, 2020, 9:36 p.m. UTC
The following changes since commit 06539ebc76b8625587aa78d646a9d8d5fddf84f3:

  Merge remote-tracking branch 'remotes/philmd-gitlab/tags/mips-hw-next-20200526' into staging (2020-05-26 20:25:06 +0100)

are available in the Git repository at:

  https://repo.or.cz/qemu/ericb.git tags/pull-bitmaps-2020-05-26-v2

for you to fetch changes up to 958796e47d3af10ece329294a7bb41d5853667db:

  iotests: Add test 291 to for qemu-img bitmap coverage (2020-05-27 16:19:59 -0500)

v2: fix iotest 190 to not be as sensitive to different sparseness of
qcow2 file on various filesystems, such as FreeBSD (sending only the
changed patch)

----------------------------------------------------------------
bitmaps patches for 2020-05-26

- fix non-blockdev migration of bitmaps when mirror job is in use
- add bitmap sizing to 'qemu-img measure'
- add 'qemu-img convert --bitmaps'

----------------------------------------------------------------
Eric Blake (5):
      iotests: Fix test 178
      qcow2: Expose bitmaps' size during measure
      qemu-img: Factor out code for merging bitmaps
      qemu-img: Add convert --bitmaps option
      iotests: Add test 291 to for qemu-img bitmap coverage

Vladimir Sementsov-Ogievskiy (6):
      migration: refactor init_dirty_bitmap_migration
      block/dirty-bitmap: add bdrv_has_named_bitmaps helper
      migration: fix bitmaps pre-blockdev migration with mirror job
      iotests: 194: test also migration of dirty bitmap
      migration: add_bitmaps_to_list: check disk name once
      migration: forbid bitmap migration by generated node-name

 docs/tools/qemu-img.rst          |  13 +++-
 qapi/block-core.json             |  16 +++--
 block/qcow2.h                    |   2 +
 include/block/dirty-bitmap.h     |   1 +
 block/crypto.c                   |   2 +-
 block/dirty-bitmap.c             |  13 ++++
 block/qcow2-bitmap.c             |  36 ++++++++++
 block/qcow2.c                    |  14 +++-
 block/raw-format.c               |   2 +-
 migration/block-dirty-bitmap.c   | 142 ++++++++++++++++++++++++++++-----------
 qemu-img.c                       | 107 ++++++++++++++++++++++++-----
 qemu-img-cmds.hx                 |   4 +-
 tests/qemu-iotests/178.out.qcow2 |  18 ++++-
 tests/qemu-iotests/178.out.raw   |   2 +-
 tests/qemu-iotests/190           |  47 ++++++++++++-
 tests/qemu-iotests/190.out       |  27 +++++++-
 tests/qemu-iotests/194           |  14 ++--
 tests/qemu-iotests/194.out       |   6 ++
 tests/qemu-iotests/291           | 112 ++++++++++++++++++++++++++++++
 tests/qemu-iotests/291.out       |  80 ++++++++++++++++++++++
 tests/qemu-iotests/group         |   1 +
 21 files changed, 582 insertions(+), 77 deletions(-)
 create mode 100755 tests/qemu-iotests/291
 create mode 100644 tests/qemu-iotests/291.out

Comments

Peter Maydell May 28, 2020, 5:13 p.m. UTC | #1
On Wed, 27 May 2020 at 22:38, Eric Blake <eblake@redhat.com> wrote:
>
> The following changes since commit 06539ebc76b8625587aa78d646a9d8d5fddf84f3:
>
>   Merge remote-tracking branch 'remotes/philmd-gitlab/tags/mips-hw-next-20200526' into staging (2020-05-26 20:25:06 +0100)
>
> are available in the Git repository at:
>
>   https://repo.or.cz/qemu/ericb.git tags/pull-bitmaps-2020-05-26-v2
>
> for you to fetch changes up to 958796e47d3af10ece329294a7bb41d5853667db:
>
>   iotests: Add test 291 to for qemu-img bitmap coverage (2020-05-27 16:19:59 -0500)
>
> v2: fix iotest 190 to not be as sensitive to different sparseness of
> qcow2 file on various filesystems, such as FreeBSD (sending only the
> changed patch)
>
> ----------------------------------------------------------------
> bitmaps patches for 2020-05-26
>
> - fix non-blockdev migration of bitmaps when mirror job is in use
> - add bitmap sizing to 'qemu-img measure'
> - add 'qemu-img convert --bitmaps'
>

Your fix for iotest 190 doesn't seem to work. Here's the new output
(same on openbsd, freebsd, netbsd):

  TEST    iotest-qcow2: 190 [fail]
QEMU          --
"/home/qemu/qemu-test.B2aJfj/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64"
-nodefaults -display none -machine virt -accel qtest
QEMU_IMG      --
"/home/qemu/qemu-test.B2aJfj/build/tests/qemu-iotests/../../qemu-img"
QEMU_IO       --
"/home/qemu/qemu-test.B2aJfj/build/tests/qemu-iotests/../../qemu-io"
--cache writeback --aio threads -f qcow2
QEMU_NBD      --
"/home/qemu/qemu-test.B2aJfj/build/tests/qemu-iotests/../../qemu-nbd"
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- OpenBSD/amd64 openbsd 6.6
TEST_DIR      -- /home/qemu/qemu-test.B2aJfj/build/tests/qemu-iotests/scratch
SOCK_DIR      -- /tmp/tmp.9459w4OTae
SOCKET_SCM_HELPER --

--- /home/qemu/qemu-test.B2aJfj/src/tests/qemu-iotests/190.out  Thu
May 28 16:06:04 2020
+++ /home/qemu/qemu-test.B2aJfj/build/tests/qemu-iotests/190.out.bad
 Thu May 28 16:31:13 2020
@@ -17,7 +17,7 @@
 fully allocated size: 10813440
 required size: 2199023255552
 fully allocated size: 2199023255552
-required size: SIZE
+required size: 17170432
 fully allocated size: 17170432
 required size: 335806464
 fully allocated size: 2199359062016

I think this is an issue with your sed expression:
+$QEMU_IMG measure -O qcow2 -f raw "$TEST_IMG" |
+    sed '/^required size:/ s/[0-9]\+/SIZE/'

BSD sed seems to strictly implement POSIX basic regexes,
in which '+' is not handled. Compare:

$ echo "required size: 17170432" | sed '/^required size:/ s/[0-9]+/SIZE/'
required size: 17170432
$
$ echo "required size: 17170432" | sed '/^required size:/ s/[0-9][0-9]*/SIZE/'
required size: SIZE

thanks
-- PMM
Eric Blake May 28, 2020, 5:49 p.m. UTC | #2
On 5/28/20 12:13 PM, Peter Maydell wrote:

>> v2: fix iotest 190 to not be as sensitive to different sparseness of
>> qcow2 file on various filesystems, such as FreeBSD (sending only the
>> changed patch)
>>
>> ----------------------------------------------------------------
>> bitmaps patches for 2020-05-26
>>
>> - fix non-blockdev migration of bitmaps when mirror job is in use
>> - add bitmap sizing to 'qemu-img measure'
>> - add 'qemu-img convert --bitmaps'
>>
> 
> Your fix for iotest 190 doesn't seem to work. Here's the new output
> (same on openbsd, freebsd, netbsd):

> -required size: SIZE
> +required size: 17170432
>   fully allocated size: 17170432
>   required size: 335806464
>   fully allocated size: 2199359062016
> 
> I think this is an issue with your sed expression:
> +$QEMU_IMG measure -O qcow2 -f raw "$TEST_IMG" |
> +    sed '/^required size:/ s/[0-9]\+/SIZE/'
> 
> BSD sed seems to strictly implement POSIX basic regexes,
> in which '+' is not handled. Compare:
> 
> $ echo "required size: 17170432" | sed '/^required size:/ s/[0-9]+/SIZE/'
> required size: 17170432
> $
> $ echo "required size: 17170432" | sed '/^required size:/ s/[0-9][0-9]*/SIZE/'
> required size: SIZE

D'oh.  v3 coming up with the long-hand more portable sed usage.