mbox series

[v2,0/9] preallocate filter

Message ID 20200814130348.20625-1-vsementsov@virtuozzo.com (mailing list archive)
Headers show
Series preallocate filter | expand

Message

Vladimir Sementsov-Ogievskiy Aug. 14, 2020, 1:03 p.m. UTC 
Hi all!

Here is a filter, which does preallocation on write.

In Virtuozzo we have to deal with some custom distributed storage
solution, where allocation is relatively expensive operation. We have to
workaround it in Qemu, so here is a new filter.

For the details refer to original cover-letter
"[PATCH 0/5] preallocate filter"
https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg06443.html

v2:
1-6 are new and substitutes bdrv_co_range_try_lock mechanism used in v1
07: add note to docs/system/qemu-block-drivers.rst.inc
    add open options
    rebase on new BDRV_REQ_NO_WAIT flag
    drop bs->file check in _co_flush()
08: new
09: use new iotests.verify_o_direct()

Vladimir Sementsov-Ogievskiy (9):
  block: simplify comment to BDRV_REQ_SERIALISING
  block/io.c: drop assertion on double waiting for request serialisation
  block/io: split out bdrv_find_conflicting_request
  block/io: bdrv_wait_serialising_requests_locked: drop extra bs arg
  block: bdrv_mark_request_serialising: split non-waiting function
  block: introduce BDRV_REQ_NO_WAIT flag
  block: introduce preallocate filter
  iotests.py: add verify_o_direct helper
  iotests: add 298 to test new preallocate filter driver

 docs/system/qemu-block-drivers.rst.inc |  26 +++
 qapi/block-core.json                   |  20 +-
 include/block/block.h                  |  20 +-
 include/block/block_int.h              |   3 +-
 block/file-posix.c                     |   2 +-
 block/io.c                             | 131 +++++++-----
 block/preallocate.c                    | 264 +++++++++++++++++++++++++
 block/Makefile.objs                    |   1 +
 tests/qemu-iotests/298                 |  46 +++++
 tests/qemu-iotests/298.out             |   5 +
 tests/qemu-iotests/group               |   1 +
 tests/qemu-iotests/iotests.py          |   6 +
 12 files changed, 457 insertions(+), 68 deletions(-)
 create mode 100644 block/preallocate.c
 create mode 100644 tests/qemu-iotests/298
 create mode 100644 tests/qemu-iotests/298.out

Comments

no-reply@patchew.org Aug. 14, 2020, 4:15 p.m. UTC | #1 
Patchew URL: https://patchew.org/QEMU/20200814130348.20625-1-vsementsov@virtuozzo.com/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      nbd/trace.o
  CC      scsi/trace.o

Warning, treated as error:
../src/docs/system/qemu-block-drivers.rst.inc:980:Duplicate explicit target name: "cmdoption-preallocate-arg-prealloc-align".
  CC      audio/trace.o
  CC      chardev/trace.o
---
  CC      hw/ide/trace.o
  CC      hw/input/trace.o
  CC      hw/intc/trace.o
make: *** [Makefile:1111: .docs_system_qemu.1_docs_system_qemu-block-drivers.7_docs_system_qemu-cpu-models.7.sentinel.] Error 2
make: *** Deleting file '.docs_system_qemu.1_docs_system_qemu-block-drivers.7_docs_system_qemu-cpu-models.7.sentinel.'
make: *** Waiting for unfinished jobs....

Warning, treated as error:
../src/docs/system/qemu-block-drivers.rst.inc:980:Duplicate explicit target name: "cmdoption-preallocate-arg-prealloc-align".
make: *** [Makefile:1100: docs/system/index.html] Error 2
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 709, in <module>
    sys.exit(main())
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=2b8332eeb0c84671bf5f795b37073245', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-msztbvd9/src/docker-src.2020-08-14-12.13.08.24140:/var/tmp/qemu:z,ro', 'qemu/fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=2b8332eeb0c84671bf5f795b37073245
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-msztbvd9/src'
make: *** [docker-run-test-mingw@fedora] Error 2

real    2m29.643s
user    0m7.966s


The full log is available at
http://patchew.org/logs/20200814130348.20625-1-vsementsov@virtuozzo.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
no-reply@patchew.org Aug. 14, 2020, 4:30 p.m. UTC | #2 
Patchew URL: https://patchew.org/QEMU/20200814130348.20625-1-vsementsov@virtuozzo.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  TEST    check-unit: tests/test-char
Unexpected error in object_property_try_add() at /tmp/qemu-test/src/qom/object.c:1181:
attempt to add duplicate property 'serial-id' to object (type 'container')
ERROR test-char - too few tests run (expected 38, got 9)
make: *** [check-unit] Error 1
make: *** Waiting for unfinished jobs....
  TEST    iotest-qcow2: 029
  TEST    check-qtest-x86_64: tests/qtest/hd-geo-test
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=933e39134d33428cb91c95a7f1ece9e6', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-u8odh47c/src/docker-src.2020-08-14-12.16.21.29250:/var/tmp/qemu:z,ro', 'qemu/centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=933e39134d33428cb91c95a7f1ece9e6
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-u8odh47c/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    13m58.743s
user    0m8.567s


The full log is available at
http://patchew.org/logs/20200814130348.20625-1-vsementsov@virtuozzo.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Vladimir Sementsov-Ogievskiy Aug. 14, 2020, 6:30 p.m. UTC | #3 
v3 will come soon, don't look at this.

14.08.2020 16:03, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> Here is a filter, which does preallocation on write.
> 
> In Virtuozzo we have to deal with some custom distributed storage
> solution, where allocation is relatively expensive operation. We have to
> workaround it in Qemu, so here is a new filter.
> 
> For the details refer to original cover-letter
> "[PATCH 0/5] preallocate filter"
> https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg06443.html
> 
> v2:
> 1-6 are new and substitutes bdrv_co_range_try_lock mechanism used in v1
> 07: add note to docs/system/qemu-block-drivers.rst.inc
>      add open options
>      rebase on new BDRV_REQ_NO_WAIT flag
>      drop bs->file check in _co_flush()
> 08: new
> 09: use new iotests.verify_o_direct()
> 
> Vladimir Sementsov-Ogievskiy (9):
>    block: simplify comment to BDRV_REQ_SERIALISING
>    block/io.c: drop assertion on double waiting for request serialisation
>    block/io: split out bdrv_find_conflicting_request
>    block/io: bdrv_wait_serialising_requests_locked: drop extra bs arg
>    block: bdrv_mark_request_serialising: split non-waiting function
>    block: introduce BDRV_REQ_NO_WAIT flag
>    block: introduce preallocate filter
>    iotests.py: add verify_o_direct helper
>    iotests: add 298 to test new preallocate filter driver
> 
>   docs/system/qemu-block-drivers.rst.inc |  26 +++
>   qapi/block-core.json                   |  20 +-
>   include/block/block.h                  |  20 +-
>   include/block/block_int.h              |   3 +-
>   block/file-posix.c                     |   2 +-
>   block/io.c                             | 131 +++++++-----
>   block/preallocate.c                    | 264 +++++++++++++++++++++++++
>   block/Makefile.objs                    |   1 +
>   tests/qemu-iotests/298                 |  46 +++++
>   tests/qemu-iotests/298.out             |   5 +
>   tests/qemu-iotests/group               |   1 +
>   tests/qemu-iotests/iotests.py          |   6 +
>   12 files changed, 457 insertions(+), 68 deletions(-)
>   create mode 100644 block/preallocate.c
>   create mode 100644 tests/qemu-iotests/298
>   create mode 100644 tests/qemu-iotests/298.out
>