mbox series

[v2,0/6] qcow2: advanced compression options

Message ID 1570026166-748566-1-git-send-email-andrey.shinkevich@virtuozzo.com (mailing list archive)
Headers show
Series qcow2: advanced compression options | expand

Message

Andrey Shinkevich Oct. 2, 2019, 2:22 p.m. UTC
New enhancements for writing compressed data to QCOW2 image.

The preceding patches have been queued in the Max's block branch:

Based-on: https://github.com/XanClic/qemu.git block

v2:
    The number of parameters in nbd_export_new() has been reduced by
    the introduced flags for all the boolean ones (suggested by Eric).

Andrey Shinkevich (6):
  qcow2: Allow writing compressed data to multiple clusters
  tests/qemu-iotests: add case of writing compressed data to multiple
    clusters
  qemu-nbd: add compression flag support
  block: support compressed write for copy-on-read
  block-stream: add compress option
  tests/qemu-iotests: add case for block-stream compress

 block/io.c                 |  21 +++++++--
 block/qcow2.c              | 113 ++++++++++++++++++++++++++++++++++-----------
 block/stream.c             |  19 +++++---
 block/trace-events         |   2 +-
 blockdev-nbd.c             |   8 +++-
 blockdev.c                 |  14 +++++-
 hmp-commands.hx            |   4 +-
 include/block/block_int.h  |   3 +-
 include/block/nbd.h        |  11 ++++-
 monitor/hmp-cmds.c         |   5 +-
 nbd/server.c               |  14 ++++--
 qapi/block-core.json       |   5 +-
 qemu-nbd.c                 |  30 ++++++++++--
 qemu-nbd.texi              |   2 +
 tests/qemu-iotests/030     |  49 +++++++++++++++++++-
 tests/qemu-iotests/030.out |   4 +-
 tests/qemu-iotests/214     |   9 ++++
 tests/qemu-iotests/214.out |   6 +++
 18 files changed, 257 insertions(+), 62 deletions(-)

Comments

no-reply@patchew.org Oct. 2, 2019, 3:07 p.m. UTC | #1
Patchew URL: https://patchew.org/QEMU/1570026166-748566-1-git-send-email-andrey.shinkevich@virtuozzo.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      block/parallels.o
  CC      block/blklogwrites.o
  CC      block/block-backend.o
/tmp/qemu-test/src/block/qcow2.c:4077:64: error: unknown type name 'AioTask'
 static coroutine_fn int qcow2_co_pwritev_compressed_task_entry(AioTask *task)
                                                                ^
/tmp/qemu-test/src/block/qcow2.c: In function 'qcow2_co_pwritev_compressed_part':
/tmp/qemu-test/src/block/qcow2.c:4098:5: error: unknown type name 'AioTaskPool'
     AioTaskPool *aio = NULL;
     ^
/tmp/qemu-test/src/block/qcow2.c:4123:5: error: implicit declaration of function 'aio_task_pool_status' [-Werror=implicit-function-declaration]
     while (bytes && aio_task_pool_status(aio) == 0) {
     ^
/tmp/qemu-test/src/block/qcow2.c:4123:5: error: nested extern declaration of 'aio_task_pool_status' [-Werror=nested-externs]
/tmp/qemu-test/src/block/qcow2.c:4130:13: error: implicit declaration of function 'aio_task_pool_new' [-Werror=implicit-function-declaration]
             aio = aio_task_pool_new(QCOW2_MAX_WORKERS);
             ^
/tmp/qemu-test/src/block/qcow2.c:4130:13: error: nested extern declaration of 'aio_task_pool_new' [-Werror=nested-externs]
/tmp/qemu-test/src/block/qcow2.c:4130:37: error: 'QCOW2_MAX_WORKERS' undeclared (first use in this function)
             aio = aio_task_pool_new(QCOW2_MAX_WORKERS);
                                     ^
/tmp/qemu-test/src/block/qcow2.c:4130:37: note: each undeclared identifier is reported only once for each function it appears in
/tmp/qemu-test/src/block/qcow2.c:4133:9: error: implicit declaration of function 'qcow2_add_task' [-Werror=implicit-function-declaration]
         ret = qcow2_add_task(bs, aio, qcow2_co_pwritev_compressed_task_entry,
         ^
/tmp/qemu-test/src/block/qcow2.c:4133:9: error: nested extern declaration of 'qcow2_add_task' [-Werror=nested-externs]
/tmp/qemu-test/src/block/qcow2.c:4133:39: error: 'qcow2_co_pwritev_compressed_task_entry' undeclared (first use in this function)
         ret = qcow2_add_task(bs, aio, qcow2_co_pwritev_compressed_task_entry,
                                       ^
/tmp/qemu-test/src/block/qcow2.c:4145:9: error: implicit declaration of function 'aio_task_pool_wait_all' [-Werror=implicit-function-declaration]
         aio_task_pool_wait_all(aio);
         ^
/tmp/qemu-test/src/block/qcow2.c:4145:9: error: nested extern declaration of 'aio_task_pool_wait_all' [-Werror=nested-externs]
/tmp/qemu-test/src/block/qcow2.c: At top level:
/tmp/qemu-test/src/block/qcow2.c:4012:1: error: 'qcow2_co_pwritev_compressed_task' defined but not used [-Werror=unused-function]
 qcow2_co_pwritev_compressed_task(BlockDriverState *bs,
 ^
cc1: all warnings being treated as errors
make: *** [block/qcow2.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 662, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=4a0fa981cdb74b48946f62d3df956bf1', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-zk1t0pwl/src/docker-src.2019-10-02-11.05.34.28094:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=4a0fa981cdb74b48946f62d3df956bf1
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-zk1t0pwl/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    1m39.268s
user    0m7.984s


The full log is available at
http://patchew.org/logs/1570026166-748566-1-git-send-email-andrey.shinkevich@virtuozzo.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
no-reply@patchew.org Oct. 2, 2019, 3:07 p.m. UTC | #2
Patchew URL: https://patchew.org/QEMU/1570026166-748566-1-git-send-email-andrey.shinkevich@virtuozzo.com/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      block/quorum.o
  CC      block/blkdebug.o
  CC      block/blkverify.o
/tmp/qemu-test/src/block/qcow2.c:4077:64: error: unknown type name 'AioTask'; did you mean 'AioWait'?
 static coroutine_fn int qcow2_co_pwritev_compressed_task_entry(AioTask *task)
                                                                ^~~~~~~
                                                                AioWait
/tmp/qemu-test/src/block/qcow2.c: In function 'qcow2_co_pwritev_compressed_part':
/tmp/qemu-test/src/block/qcow2.c:4098:5: error: unknown type name 'AioTaskPool'
     AioTaskPool *aio = NULL;
     ^~~~~~~~~~~
/tmp/qemu-test/src/block/qcow2.c:4123:21: error: implicit declaration of function 'aio_task_pool_status' [-Werror=implicit-function-declaration]
     while (bytes && aio_task_pool_status(aio) == 0) {
                     ^~~~~~~~~~~~~~~~~~~~
/tmp/qemu-test/src/block/qcow2.c:4123:21: error: nested extern declaration of 'aio_task_pool_status' [-Werror=nested-externs]
/tmp/qemu-test/src/block/qcow2.c:4130:19: error: implicit declaration of function 'aio_task_pool_new'; did you mean 'aio_timer_new'? [-Werror=implicit-function-declaration]
             aio = aio_task_pool_new(QCOW2_MAX_WORKERS);
                   ^~~~~~~~~~~~~~~~~
                   aio_timer_new
/tmp/qemu-test/src/block/qcow2.c:4130:19: error: nested extern declaration of 'aio_task_pool_new' [-Werror=nested-externs]
/tmp/qemu-test/src/block/qcow2.c:4130:37: error: 'QCOW2_MAX_WORKERS' undeclared (first use in this function); did you mean 'QCOW2_MAX_THREADS'?
             aio = aio_task_pool_new(QCOW2_MAX_WORKERS);
                                     ^~~~~~~~~~~~~~~~~
                                     QCOW2_MAX_THREADS
/tmp/qemu-test/src/block/qcow2.c:4130:37: note: each undeclared identifier is reported only once for each function it appears in
/tmp/qemu-test/src/block/qcow2.c:4133:15: error: implicit declaration of function 'qcow2_add_task'; did you mean 'qcow2_do_open'? [-Werror=implicit-function-declaration]
         ret = qcow2_add_task(bs, aio, qcow2_co_pwritev_compressed_task_entry,
               ^~~~~~~~~~~~~~
               qcow2_do_open
/tmp/qemu-test/src/block/qcow2.c:4133:15: error: nested extern declaration of 'qcow2_add_task' [-Werror=nested-externs]
/tmp/qemu-test/src/block/qcow2.c:4133:39: error: 'qcow2_co_pwritev_compressed_task_entry' undeclared (first use in this function); did you mean 'qcow2_co_pwritev_compressed_task'?
         ret = qcow2_add_task(bs, aio, qcow2_co_pwritev_compressed_task_entry,
                                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                       qcow2_co_pwritev_compressed_task
/tmp/qemu-test/src/block/qcow2.c:4145:9: error: implicit declaration of function 'aio_task_pool_wait_all' [-Werror=implicit-function-declaration]
         aio_task_pool_wait_all(aio);
         ^~~~~~~~~~~~~~~~~~~~~~
/tmp/qemu-test/src/block/qcow2.c:4145:9: error: nested extern declaration of 'aio_task_pool_wait_all' [-Werror=nested-externs]
At top level:
/tmp/qemu-test/src/block/qcow2.c:4012:1: error: 'qcow2_co_pwritev_compressed_task' defined but not used [-Werror=unused-function]
 qcow2_co_pwritev_compressed_task(BlockDriverState *bs,
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
make: *** [/tmp/qemu-test/src/rules.mak:69: block/qcow2.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 662, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=b4ead0a324e949c19b89413b42658f34', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-w_sflkq1/src/docker-src.2019-10-02-11.05.59.29878:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=b4ead0a324e949c19b89413b42658f34
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-w_sflkq1/src'
make: *** [docker-run-test-mingw@fedora] Error 2

real    1m50.644s
user    0m7.709s


The full log is available at
http://patchew.org/logs/1570026166-748566-1-git-send-email-andrey.shinkevich@virtuozzo.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Vladimir Sementsov-Ogievskiy Oct. 2, 2019, 3:35 p.m. UTC | #3
Hi,

02.10.2019 18:07, no-reply@patchew.org wrote:
> Patchew URL: https://patchew.org/QEMU/1570026166-748566-1-git-send-email-andrey.shinkevich@virtuozzo.com/
> 
> 
> 
> Hi,
> 
> This series failed the docker-quick@centos7 build test. Please find the testing commands and
> their output below. If you have Docker installed, you can probably reproduce it
> locally.
> 
> === TEST SCRIPT BEGIN ===
> #!/bin/bash
> make docker-image-centos7 V=1 NETWORK=1
> time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
> === TEST SCRIPT END ===
> 
>    CC      block/parallels.o
>    CC      block/blklogwrites.o
>    CC      block/block-backend.o
> /tmp/qemu-test/src/block/qcow2.c:4077:64: error: unknown type name 'AioTask'
>   static coroutine_fn int qcow2_co_pwritev_compressed_task_entry(AioTask *task)
>                                                                  ^
> /tmp/qemu-test/src/block/qcow2.c: In function 'qcow2_co_pwritev_compressed_part':
> /tmp/qemu-test/src/block/qcow2.c:4098:5: error: unknown type name 'AioTaskPool'
>       AioTaskPool *aio = NULL;
>       ^


Who knows, what is wrong with it? Seems patchew ignores Based-on: tag in cover-letter,
which is written as "Based-on: https://github.com/XanClic/qemu.git block"...
These new types and functions are defined in Max's block branch.


> /tmp/qemu-test/src/block/qcow2.c:4123:5: error: implicit declaration of function 'aio_task_pool_status' [-Werror=implicit-function-declaration]
>       while (bytes && aio_task_pool_status(aio) == 0) {
>       ^
> /tmp/qemu-test/src/block/qcow2.c:4123:5: error: nested extern declaration of 'aio_task_pool_status' [-Werror=nested-externs]
> /tmp/qemu-test/src/block/qcow2.c:4130:13: error: implicit declaration of function 'aio_task_pool_new' [-Werror=implicit-function-declaration]
>               aio = aio_task_pool_new(QCOW2_MAX_WORKERS);
>               ^
> /tmp/qemu-test/src/block/qcow2.c:4130:13: error: nested extern declaration of 'aio_task_pool_new' [-Werror=nested-externs]
> /tmp/qemu-test/src/block/qcow2.c:4130:37: error: 'QCOW2_MAX_WORKERS' undeclared (first use in this function)
>               aio = aio_task_pool_new(QCOW2_MAX_WORKERS);
>                                       ^
> /tmp/qemu-test/src/block/qcow2.c:4130:37: note: each undeclared identifier is reported only once for each function it appears in
> /tmp/qemu-test/src/block/qcow2.c:4133:9: error: implicit declaration of function 'qcow2_add_task' [-Werror=implicit-function-declaration]
>           ret = qcow2_add_task(bs, aio, qcow2_co_pwritev_compressed_task_entry,
>           ^
> /tmp/qemu-test/src/block/qcow2.c:4133:9: error: nested extern declaration of 'qcow2_add_task' [-Werror=nested-externs]
> /tmp/qemu-test/src/block/qcow2.c:4133:39: error: 'qcow2_co_pwritev_compressed_task_entry' undeclared (first use in this function)
>           ret = qcow2_add_task(bs, aio, qcow2_co_pwritev_compressed_task_entry,
>                                         ^
> /tmp/qemu-test/src/block/qcow2.c:4145:9: error: implicit declaration of function 'aio_task_pool_wait_all' [-Werror=implicit-function-declaration]
>           aio_task_pool_wait_all(aio);
>           ^
> /tmp/qemu-test/src/block/qcow2.c:4145:9: error: nested extern declaration of 'aio_task_pool_wait_all' [-Werror=nested-externs]
> /tmp/qemu-test/src/block/qcow2.c: At top level:
> /tmp/qemu-test/src/block/qcow2.c:4012:1: error: 'qcow2_co_pwritev_compressed_task' defined but not used [-Werror=unused-function]
>   qcow2_co_pwritev_compressed_task(BlockDriverState *bs,
>   ^
> cc1: all warnings being treated as errors
> make: *** [block/qcow2.o] Error 1
> make: *** Waiting for unfinished jobs....
> Traceback (most recent call last):
>    File "./tests/docker/docker.py", line 662, in <module>
> ---
>      raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=4a0fa981cdb74b48946f62d3df956bf1', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-zk1t0pwl/src/docker-src.2019-10-02-11.05.34.28094:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
> filter=--filter=label=com.qemu.instance.uuid=4a0fa981cdb74b48946f62d3df956bf1
> make[1]: *** [docker-run] Error 1
> make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-zk1t0pwl/src'
> make: *** [docker-run-test-quick@centos7] Error 2
> 
> real    1m39.268s
> user    0m7.984s
> 
> 
> The full log is available at
> http://patchew.org/logs/1570026166-748566-1-git-send-email-andrey.shinkevich@virtuozzo.com/testing.docker-quick@centos7/?type=message.
> ---
> Email generated automatically by Patchew [https://patchew.org/].
> Please send your feedback to patchew-devel@redhat.com
>
Max Reitz Oct. 2, 2019, 3:58 p.m. UTC | #4
On 02.10.19 17:35, Vladimir Sementsov-Ogievskiy wrote:
> Hi,
> 
> 02.10.2019 18:07, no-reply@patchew.org wrote:
>> Patchew URL: https://patchew.org/QEMU/1570026166-748566-1-git-send-email-andrey.shinkevich@virtuozzo.com/
>>
>>
>>
>> Hi,
>>
>> This series failed the docker-quick@centos7 build test. Please find the testing commands and
>> their output below. If you have Docker installed, you can probably reproduce it
>> locally.
>>
>> === TEST SCRIPT BEGIN ===
>> #!/bin/bash
>> make docker-image-centos7 V=1 NETWORK=1
>> time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
>> === TEST SCRIPT END ===
>>
>>    CC      block/parallels.o
>>    CC      block/blklogwrites.o
>>    CC      block/block-backend.o
>> /tmp/qemu-test/src/block/qcow2.c:4077:64: error: unknown type name 'AioTask'
>>   static coroutine_fn int qcow2_co_pwritev_compressed_task_entry(AioTask *task)
>>                                                                  ^
>> /tmp/qemu-test/src/block/qcow2.c: In function 'qcow2_co_pwritev_compressed_part':
>> /tmp/qemu-test/src/block/qcow2.c:4098:5: error: unknown type name 'AioTaskPool'
>>       AioTaskPool *aio = NULL;
>>       ^
> 
> 
> Who knows, what is wrong with it? Seems patchew ignores Based-on: tag in cover-letter,
> which is written as "Based-on: https://github.com/XanClic/qemu.git block"...
> These new types and functions are defined in Max's block branch.

It would be news to me if Patchew supported such URLs.  I just put it
into my cover letter for human reviewers...

(Actually, it would be news to me if Patchew supported Based-on at all
reliably...)

Max
Markus Armbruster Oct. 2, 2019, 7:04 p.m. UTC | #5
Max Reitz <mreitz@redhat.com> writes:

> On 02.10.19 17:35, Vladimir Sementsov-Ogievskiy wrote:
>> Hi,
>> 
>> 02.10.2019 18:07, no-reply@patchew.org wrote:
>>> Patchew URL: https://patchew.org/QEMU/1570026166-748566-1-git-send-email-andrey.shinkevich@virtuozzo.com/
>>>
>>>
>>>
>>> Hi,
>>>
>>> This series failed the docker-quick@centos7 build test. Please find the testing commands and
>>> their output below. If you have Docker installed, you can probably reproduce it
>>> locally.
>>>
>>> === TEST SCRIPT BEGIN ===
>>> #!/bin/bash
>>> make docker-image-centos7 V=1 NETWORK=1
>>> time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
>>> === TEST SCRIPT END ===
>>>
>>>    CC      block/parallels.o
>>>    CC      block/blklogwrites.o
>>>    CC      block/block-backend.o
>>> /tmp/qemu-test/src/block/qcow2.c:4077:64: error: unknown type name 'AioTask'
>>>   static coroutine_fn int qcow2_co_pwritev_compressed_task_entry(AioTask *task)
>>>                                                                  ^
>>> /tmp/qemu-test/src/block/qcow2.c: In function 'qcow2_co_pwritev_compressed_part':
>>> /tmp/qemu-test/src/block/qcow2.c:4098:5: error: unknown type name 'AioTaskPool'
>>>       AioTaskPool *aio = NULL;
>>>       ^
>> 
>> 
>> Who knows, what is wrong with it? Seems patchew ignores Based-on: tag in cover-letter,
>> which is written as "Based-on: https://github.com/XanClic/qemu.git block"...
>> These new types and functions are defined in Max's block branch.
>
> It would be news to me if Patchew supported such URLs.  I just put it
> into my cover letter for human reviewers...
>
> (Actually, it would be news to me if Patchew supported Based-on at all
> reliably...)

https://github.com/patchew-project/patchew/commit/5e461e7c49c1913cb34349f45cc7566627b37288

I use Based-on: <Message-Id> all the time, and it works reliably for me.