mbox

[PULL,00/64] Block layer patches

Message ID 20200907110936.261684-1-kwolf@redhat.com
State New, archived
Headers show

Pull-request

git://repo.or.cz/qemu/kevin.git tags/for-upstream

Message

Kevin Wolf Sept. 7, 2020, 11:08 a.m. UTC
The following changes since commit 7c37270b3fbe3d034ba80e488761461676e21eb4:

  Merge remote-tracking branch 'remotes/kraxel/tags/ui-20200904-pull-request' into staging (2020-09-06 16:23:55 +0100)

are available in the Git repository at:

  git://repo.or.cz/qemu/kevin.git tags/for-upstream

for you to fetch changes up to 4cc0ec9b1b8830f7d1fcf5dfded19ef070f98eaa:

  block/nvme: Pair doorbell registers (2020-09-07 12:47:57 +0200)

----------------------------------------------------------------
Block layer patches:

- qemu-img create: Fail gracefully when backing file is an empty string
- Fixes related to filter block nodes ("Deal with filters" series)
- block/nvme: Various cleanups required to use multiple queues
- block/nvme: Use NvmeBar structure from "block/nvme.h"
- file-win32: Fix "locking" option
- iotests: Allow running from different directory

----------------------------------------------------------------
Connor Kuehl (1):
      block: Raise an error when backing file parameter is an empty string

Kevin Wolf (2):
      iotests: Allow running from different directory
      file-win32: Fix "locking" option

Max Reitz (43):
      block: Add child access functions
      block: Add chain helper functions
      block: bdrv_cow_child() for bdrv_has_zero_init()
      block: bdrv_set_backing_hd() is about bs->backing
      block: Include filters when freezing backing chain
      block: Drop bdrv_is_encrypted()
      block: Add bdrv_supports_compressed_writes()
      throttle: Support compressed writes
      copy-on-read: Support compressed writes
      block: Use bdrv_filter_(bs|child) where obvious
      block: Use CAFs in block status functions
      stream: Deal with filters
      block: Use CAFs when working with backing chains
      block: Use bdrv_cow_child() in bdrv_co_truncate()
      block: Re-evaluate backing file handling in reopen
      block: Flush all children in generic code
      vmdk: Drop vmdk_co_flush()
      block: Iterate over children in refresh_limits
      block: Use CAFs in bdrv_refresh_filename()
      block: Use CAF in bdrv_co_rw_vmstate()
      block/snapshot: Fix fallback
      block: Use CAFs for debug breakpoints
      block: Improve get_allocated_file_size's default
      block/null: Implement bdrv_get_allocated_file_size
      blockdev: Use CAF in external_snapshot_prepare()
      block: Report data child for query-blockstats
      block: Use child access functions for QAPI queries
      block-copy: Use CAF to find sync=top base
      mirror: Deal with filters
      backup: Deal with filters
      commit: Deal with filters
      nbd: Use CAF when looking for dirty bitmap
      qemu-img: Use child access functions
      block: Drop backing_bs()
      blockdev: Fix active commit choice
      block: Inline bdrv_co_block_status_from_*()
      block: Leave BDS.backing_{file,format} constant
      iotests: Test that qcow2's data-file is flushed
      iotests: Let complete_and_wait() work with commit
      iotests: Add filter commit test cases
      iotests: Add filter mirror test cases
      iotests: Add test for commit in sub directory
      iotests: Test committing to overridden backing

Philippe Mathieu-Daudé (18):
      block/nvme: Replace magic value by SCALE_MS definition
      block/nvme: Avoid further processing if trace event not enabled
      block/nvme: Let nvme_create_queue_pair() fail gracefully
      block/nvme: Define INDEX macros to ease code review
      block/nvme: Improve error message when IO queue creation failed
      block/nvme: Use common error path in nvme_add_io_queue()
      block/nvme: Rename local variable
      block/nvme: Use union of NvmeIdCtrl / NvmeIdNs structures
      block/nvme: Replace qemu_try_blockalign0 by qemu_try_blockalign/memset
      block/nvme: Replace qemu_try_blockalign(bs) by qemu_try_memalign(pg_sz)
      block/nvme: Simplify nvme_init_queue() arguments
      block/nvme: Replace BDRV_POLL_WHILE by AIO_WAIT_WHILE
      block/nvme: Simplify nvme_create_queue_pair() arguments
      block/nvme: Extract nvme_poll_queue()
      block/nvme: Use an array of EventNotifier
      block/nvme: Group controller registers in NVMeRegs structure
      block/nvme: Use generic NvmeBar structure
      block/nvme: Pair doorbell registers

 qapi/block-core.json           |  58 +++--
 include/block/block.h          |   2 +-
 include/block/block_int.h      |  95 +++++---
 block.c                        | 492 +++++++++++++++++++++++++++++++++--------
 block/backup-top.c             |   4 +-
 block/backup.c                 |   9 +-
 block/blkdebug.c               |   7 +-
 block/blklogwrites.c           |   1 -
 block/block-backend.c          |   7 +-
 block/block-copy.c             |   4 +-
 block/commit.c                 |  95 +++++---
 block/copy-on-read.c           |  13 +-
 block/file-win32.c             |  22 +-
 block/filter-compress.c        |   2 -
 block/io.c                     | 142 ++++++------
 block/mirror.c                 | 119 +++++++---
 block/monitor/block-hmp-cmds.c |   2 +-
 block/null.c                   |   7 +
 block/nvme.c                   | 248 +++++++++++----------
 block/qapi.c                   |  74 +++++--
 block/snapshot.c               | 104 +++++++--
 block/stream.c                 |  63 ++++--
 block/throttle.c               |  11 +-
 block/vmdk.c                   |  16 --
 blockdev.c                     | 101 +++++++--
 migration/block-dirty-bitmap.c |   8 +-
 nbd/server.c                   |   6 +-
 qemu-img.c                     |  43 ++--
 tests/qemu-iotests/iotests.py  |  10 +-
 tests/qemu-iotests/020         |  44 ++++
 tests/qemu-iotests/020.out     |  10 +
 tests/qemu-iotests/040         | 238 ++++++++++++++++++++
 tests/qemu-iotests/040.out     |   4 +-
 tests/qemu-iotests/041         | 146 +++++++++++-
 tests/qemu-iotests/041.out     |   4 +-
 tests/qemu-iotests/049         |   4 +
 tests/qemu-iotests/049.out     |   5 +
 tests/qemu-iotests/153.out     |   2 +-
 tests/qemu-iotests/184         |   3 +-
 tests/qemu-iotests/184.out     |  14 +-
 tests/qemu-iotests/204.out     |   1 +
 tests/qemu-iotests/228         |   6 +-
 tests/qemu-iotests/228.out     |   6 +-
 tests/qemu-iotests/244         |  49 ++++
 tests/qemu-iotests/244.out     |   7 +
 tests/qemu-iotests/245         |   4 +-
 tests/qemu-iotests/273.out     |   4 +-
 tests/qemu-iotests/check       |   2 +-
 48 files changed, 1763 insertions(+), 555 deletions(-)

Comments

Peter Maydell Sept. 7, 2020, 8:22 p.m. UTC | #1
On Mon, 7 Sep 2020 at 12:09, Kevin Wolf <kwolf@redhat.com> wrote:
>
> The following changes since commit 7c37270b3fbe3d034ba80e488761461676e21eb4:
>
>   Merge remote-tracking branch 'remotes/kraxel/tags/ui-20200904-pull-request' into staging (2020-09-06 16:23:55 +0100)
>
> are available in the Git repository at:
>
>   git://repo.or.cz/qemu/kevin.git tags/for-upstream
>
> for you to fetch changes up to 4cc0ec9b1b8830f7d1fcf5dfded19ef070f98eaa:
>
>   block/nvme: Pair doorbell registers (2020-09-07 12:47:57 +0200)
>
> ----------------------------------------------------------------
> Block layer patches:
>
> - qemu-img create: Fail gracefully when backing file is an empty string
> - Fixes related to filter block nodes ("Deal with filters" series)
> - block/nvme: Various cleanups required to use multiple queues
> - block/nvme: Use NvmeBar structure from "block/nvme.h"
> - file-win32: Fix "locking" option
> - iotests: Allow running from different directory

Fails in make check on iotests 040 and/or 041, various hosts:

s390x linux:

  TEST    iotest-qcow2: 041 [fail]
QEMU          --
"/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-system-s390x"
-nodefaults -display none -accel qtest
QEMU_IMG      -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-img"
QEMU_IO       --
"/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-io"
--cache writeback --aio threads -f qcow2
QEMU_NBD      -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-nbd"
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- Linux/s390x qemu01 4.15.0-72-generic
TEST_DIR      -- /home/ubuntu/qemu/build/all/tests/qemu-iotests/scratch
SOCK_DIR      -- /tmp/tmp.5Tpl6u2SCo
SOCKET_SCM_HELPER --
/home/ubuntu/qemu/build/all/tests/qemu-iotests/socket_scm_helper

--- /home/ubuntu/qemu/tests/qemu-iotests/041.out        2020-09-07
14:29:45.468466636 -0400
+++ /home/ubuntu/qemu/build/all/tests/qemu-iotests/041.out.bad
2020-09-07 14:43:41.494989911 -0400
@@ -1,5 +1,29 @@
-...........................................................................................................
+.FF........................................................................................................
+======================================================================
+FAIL: test_explicit_mirror_filter (__main__.TestFilters)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "041", line 1415, in test_explicit_mirror_filter
+    self.assert_qmp(result, 'return', {})
+  File "/home/ubuntu/qemu/tests/qemu-iotests/iotests.py", line 888,
in assert_qmp
+    result = self.dictpath(d, path)
+  File "/home/ubuntu/qemu/tests/qemu-iotests/iotests.py", line 862, in dictpath
+    self.fail(f'failed path traversal for "{path}" in "{d}"')
+AssertionError: failed path traversal for "return" in "{'error':
{'class': 'GenericError', 'desc': "Device 'virtio-blk-ccw' can't go on
PCI bus"}}"
+
+======================================================================
+FAIL: test_implicit_mirror_filter (__main__.TestFilters)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "041", line 1381, in test_implicit_mirror_filter
+    self.assert_qmp(result, 'return', {})
+  File "/home/ubuntu/qemu/tests/qemu-iotests/iotests.py", line 888,
in assert_qmp
+    result = self.dictpath(d, path)
+  File "/home/ubuntu/qemu/tests/qemu-iotests/iotests.py", line 862, in dictpath
+    self.fail(f'failed path traversal for "{path}" in "{d}"')
+AssertionError: failed path traversal for "return" in "{'error':
{'class': 'GenericError', 'desc': "Device 'virtio-blk-ccw' can't go on
PCI bus"}}"
+
 ----------------------------------------------------------------------
 Ran 107 tests

-OK
+FAILED (failures=2)


freebsd:

  TEST    iotest-qcow2: 040 [fail]
QEMU          --
"/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-system-aarch64"
-nodefaults -display none -accel qtest -machine virt
QEMU_IMG      --
"/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-img"
QEMU_IO       --
"/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-io"
 --cache writeback --aio threads -f qcow2
QEMU_NBD      --
"/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-nbd"
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- FreeBSD/amd64 freebsd 12.1-RELEASE
TEST_DIR      -- /home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/scratch
SOCK_DIR      -- /tmp/tmp.gaJ11NA1
SOCKET_SCM_HELPER --

--- /home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/040.out
2020-09-07 18:29:28.000000000 +0000
+++ /home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/040.out.bad
 2020-09-07 18:37:26.783967000 +0000
@@ -1,5 +1,17 @@
-.................................................................
+....................F............................................
+======================================================================
+FAIL: test_filtered_active_commit_with_filter (__main__.TestCommitWithFilters)
 ----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "040", line 867, in test_filtered_active_commit_with_filter
+    self.assert_qmp(result, 'return', {})
+  File "/usr/home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/iotests.py",
line 888, in assert_qmp
+    result = self.dictpath(d, path)
+  File "/usr/home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/iotests.py",
line 862, in dictpath
+    self.fail(f'failed path traversal for "{path}" in "{d}"')
+AssertionError: failed path traversal for "return" in "{'error':
{'class': 'GenericError', 'desc': "Bus 'pcie.0' does not support
hotplugging"}}"
+
+----------------------------------------------------------------------
 Ran 65 tests

-OK
+FAILED (failures=1)
  TEST    iotest-qcow2: 041 [fail]
QEMU          --
"/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-system-aarch64"
-nodefaults -display none -accel qtest -machine virt
QEMU_IMG      --
"/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-img"
QEMU_IO       --
"/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-io"
 --cache writeback --aio threads -f qcow2
QEMU_NBD      --
"/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-nbd"
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- FreeBSD/amd64 freebsd 12.1-RELEASE
TEST_DIR      -- /home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/scratch
SOCK_DIR      -- /tmp/tmp.gaJ11NA1
SOCKET_SCM_HELPER --

--- /home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/041.out
2020-09-07 18:29:28.000000000 +0000
+++ /home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/041.out.bad
 2020-09-07 18:38:32.186227000 +0000
@@ -1,5 +1,29 @@
-...........................................................................................................
+.FF........................................................................................................
+======================================================================
+FAIL: test_explicit_mirror_filter (__main__.TestFilters)
 ----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "041", line 1415, in test_explicit_mirror_filter
+    self.assert_qmp(result, 'return', {})
+  File "/usr/home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/iotests.py",
line 888, in assert_qmp
+    result = self.dictpath(d, path)
+  File "/usr/home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/iotests.py",
line 862, in dictpath
+    self.fail(f'failed path traversal for "{path}" in "{d}"')
+AssertionError: failed path traversal for "return" in "{'error':
{'class': 'GenericError', 'desc': "Bus 'pci.0' not found"}}"
+
+======================================================================
+FAIL: test_implicit_mirror_filter (__main__.TestFilters)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "041", line 1381, in test_implicit_mirror_filter
+    self.assert_qmp(result, 'return', {})
+  File "/usr/home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/iotests.py",
line 888, in assert_qmp
+    result = self.dictpath(d, path)
+  File "/usr/home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/iotests.py",
line 862, in dictpath
+    self.fail(f'failed path traversal for "{path}" in "{d}"')
+AssertionError: failed path traversal for "return" in "{'error':
{'class': 'GenericError', 'desc': "Bus 'pci.0' not found"}}"
+
+----------------------------------------------------------------------
 Ran 107 tests

-OK
+FAILED (failures=2)

AArch32 and AArch64 Linux:

  TEST    iotest-qcow2: 041 [fail]
QEMU          --
"/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-system-aarch64"
-nodefaults -disp
lay none -accel qtest -machine virt
QEMU_IMG      --
"/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-img"
QEMU_IO       --
"/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-io"
 --cache writeback --aio thre
ads -f qcow2
QEMU_NBD      --
"/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-nbd"
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- Linux/aarch64 mustang-maydell 4.15.0-101-generic
TEST_DIR      --
/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/scratch
SOCK_DIR      -- /tmp/tmp.L8FBRpgYvF
SOCKET_SCM_HELPER --
/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/socket_scm_helper

--- /home/peter.maydell/qemu/tests/qemu-iotests/041.out 2020-09-07
18:29:45.583104907 +0000
+++ /home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/041.out.bad
      2020-09-07 18:45:19.385149062 +0000
@@ -1,5 +1,29 @@
-...........................................................................................................
+.FF........................................................................................................
+======================================================================
+FAIL: test_explicit_mirror_filter (__main__.TestFilters)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "041", line 1415, in test_explicit_mirror_filter
+    self.assert_qmp(result, 'return', {})
+  File "/home/peter.maydell/qemu/tests/qemu-iotests/iotests.py", line
888, in assert_qmp
+    result = self.dictpath(d, path)
+  File "/home/peter.maydell/qemu/tests/qemu-iotests/iotests.py", line
862, in dictpath
+    self.fail(f'failed path traversal for "{path}" in "{d}"')
+AssertionError: failed path traversal for "return" in "{'error':
{'class': 'GenericError', 'desc': "Bus 'pci.0' not fou
nd"}}"
+
+======================================================================
+FAIL: test_implicit_mirror_filter (__main__.TestFilters)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "041", line 1381, in test_implicit_mirror_filter
+    self.assert_qmp(result, 'return', {})
+  File "/home/peter.maydell/qemu/tests/qemu-iotests/iotests.py", line
888, in assert_qmp
+    result = self.dictpath(d, path)
+  File "/home/peter.maydell/qemu/tests/qemu-iotests/iotests.py", line
862, in dictpath
+    self.fail(f'failed path traversal for "{path}" in "{d}"')
+AssertionError: failed path traversal for "return" in "{'error':
{'class': 'GenericError', 'desc': "Bus 'pci.0' not found"}}"
+
 ----------------------------------------------------------------------
 Ran 107 tests

-OK
+FAILED (failures=2)


thanks
-- PMM
Kevin Wolf Sept. 8, 2020, 7:01 a.m. UTC | #2
Am 07.09.2020 um 22:22 hat Peter Maydell geschrieben:
> On Mon, 7 Sep 2020 at 12:09, Kevin Wolf <kwolf@redhat.com> wrote:
> >
> > The following changes since commit 7c37270b3fbe3d034ba80e488761461676e21eb4:
> >
> >   Merge remote-tracking branch 'remotes/kraxel/tags/ui-20200904-pull-request' into staging (2020-09-06 16:23:55 +0100)
> >
> > are available in the Git repository at:
> >
> >   git://repo.or.cz/qemu/kevin.git tags/for-upstream
> >
> > for you to fetch changes up to 4cc0ec9b1b8830f7d1fcf5dfded19ef070f98eaa:
> >
> >   block/nvme: Pair doorbell registers (2020-09-07 12:47:57 +0200)
> >
> > ----------------------------------------------------------------
> > Block layer patches:
> >
> > - qemu-img create: Fail gracefully when backing file is an empty string
> > - Fixes related to filter block nodes ("Deal with filters" series)
> > - block/nvme: Various cleanups required to use multiple queues
> > - block/nvme: Use NvmeBar structure from "block/nvme.h"
> > - file-win32: Fix "locking" option
> > - iotests: Allow running from different directory
> 
> Fails in make check on iotests 040 and/or 041, various hosts:
> 
> s390x linux:
> 
>   TEST    iotest-qcow2: 041 [fail]
> QEMU          --
> "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-system-s390x"
> -nodefaults -display none -accel qtest
> QEMU_IMG      -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-io"
> --cache writeback --aio threads -f qcow2
> QEMU_NBD      -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- Linux/s390x qemu01 4.15.0-72-generic
> TEST_DIR      -- /home/ubuntu/qemu/build/all/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.5Tpl6u2SCo
> SOCKET_SCM_HELPER --
> /home/ubuntu/qemu/build/all/tests/qemu-iotests/socket_scm_helper
> 
> --- /home/ubuntu/qemu/tests/qemu-iotests/041.out        2020-09-07
> 14:29:45.468466636 -0400
> +++ /home/ubuntu/qemu/build/all/tests/qemu-iotests/041.out.bad
> 2020-09-07 14:43:41.494989911 -0400
> @@ -1,5 +1,29 @@
> -...........................................................................................................
> +.FF........................................................................................................
> +======================================================================
> +FAIL: test_explicit_mirror_filter (__main__.TestFilters)
> +----------------------------------------------------------------------
> +Traceback (most recent call last):
> +  File "041", line 1415, in test_explicit_mirror_filter
> +    self.assert_qmp(result, 'return', {})
> +  File "/home/ubuntu/qemu/tests/qemu-iotests/iotests.py", line 888,
> in assert_qmp
> +    result = self.dictpath(d, path)
> +  File "/home/ubuntu/qemu/tests/qemu-iotests/iotests.py", line 862, in dictpath
> +    self.fail(f'failed path traversal for "{path}" in "{d}"')
> +AssertionError: failed path traversal for "return" in "{'error':
> {'class': 'GenericError', 'desc': "Device 'virtio-blk-ccw' can't go on
> PCI bus"}}"

Max, any specific reason you specified the bus in device_add?

This seems to fix it for me. Do you agree with the change?

diff --git a/tests/qemu-iotests/041 b/tests/qemu-iotests/041
index cdbef3ba20..203ed58868 100755
--- a/tests/qemu-iotests/041
+++ b/tests/qemu-iotests/041
@@ -1376,7 +1376,6 @@ class TestFilters(iotests.QMPTestCase):
         result = self.vm.qmp('device_add',
                              driver='virtio-blk',
                              id='virtio',
-                             bus='pci.0',
                              drive='source')
         self.assert_qmp(result, 'return', {})

@@ -1410,7 +1409,6 @@ class TestFilters(iotests.QMPTestCase):
         result = self.vm.qmp('device_add',
                              driver='virtio-blk',
                              id='virtio',
-                             bus='pci.0',
                              drive='source')
         self.assert_qmp(result, 'return', {})


> +======================================================================
> +FAIL: test_implicit_mirror_filter (__main__.TestFilters)
> +----------------------------------------------------------------------
> +Traceback (most recent call last):
> +  File "041", line 1381, in test_implicit_mirror_filter
> +    self.assert_qmp(result, 'return', {})
> +  File "/home/ubuntu/qemu/tests/qemu-iotests/iotests.py", line 888,
> in assert_qmp
> +    result = self.dictpath(d, path)
> +  File "/home/ubuntu/qemu/tests/qemu-iotests/iotests.py", line 862, in dictpath
> +    self.fail(f'failed path traversal for "{path}" in "{d}"')
> +AssertionError: failed path traversal for "return" in "{'error':
> {'class': 'GenericError', 'desc': "Device 'virtio-blk-ccw' can't go on
> PCI bus"}}"
> +
>  ----------------------------------------------------------------------
>  Ran 107 tests
> 
> -OK
> +FAILED (failures=2)
> 
> 
> freebsd:
> 
>   TEST    iotest-qcow2: 040 [fail]
> QEMU          --
> "/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-system-aarch64"
> -nodefaults -display none -accel qtest -machine virt
> QEMU_IMG      --
> "/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-io"
>  --cache writeback --aio threads -f qcow2
> QEMU_NBD      --
> "/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- FreeBSD/amd64 freebsd 12.1-RELEASE
> TEST_DIR      -- /home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.gaJ11NA1
> SOCKET_SCM_HELPER --
> 
> --- /home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/040.out
> 2020-09-07 18:29:28.000000000 +0000
> +++ /home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/040.out.bad
>  2020-09-07 18:37:26.783967000 +0000
> @@ -1,5 +1,17 @@
> -.................................................................
> +....................F............................................
> +======================================================================
> +FAIL: test_filtered_active_commit_with_filter (__main__.TestCommitWithFilters)
>  ----------------------------------------------------------------------
> +Traceback (most recent call last):
> +  File "040", line 867, in test_filtered_active_commit_with_filter
> +    self.assert_qmp(result, 'return', {})
> +  File "/usr/home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/iotests.py",
> line 888, in assert_qmp
> +    result = self.dictpath(d, path)
> +  File "/usr/home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/iotests.py",
> line 862, in dictpath
> +    self.fail(f'failed path traversal for "{path}" in "{d}"')
> +AssertionError: failed path traversal for "return" in "{'error':
> {'class': 'GenericError', 'desc': "Bus 'pcie.0' does not support
> hotplugging"}}"

This one I don't really understand. Why "pcie.0"? Do FreeBSD hosts
default to a different machine type? The default should be the "pc"
machine type, which has pci, not pcie.

> +----------------------------------------------------------------------
>  Ran 65 tests
> 
> -OK
> +FAILED (failures=1)
>   TEST    iotest-qcow2: 041 [fail]
> QEMU          --
> "/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-system-aarch64"
> -nodefaults -display none -accel qtest -machine virt
> QEMU_IMG      --
> "/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-io"
>  --cache writeback --aio threads -f qcow2
> QEMU_NBD      --
> "/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- FreeBSD/amd64 freebsd 12.1-RELEASE
> TEST_DIR      -- /home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.gaJ11NA1
> SOCKET_SCM_HELPER --
> 
> --- /home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/041.out
> 2020-09-07 18:29:28.000000000 +0000
> +++ /home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/041.out.bad
>  2020-09-07 18:38:32.186227000 +0000
> @@ -1,5 +1,29 @@
> -...........................................................................................................
> +.FF........................................................................................................
> +======================================================================
> +FAIL: test_explicit_mirror_filter (__main__.TestFilters)
>  ----------------------------------------------------------------------
> +Traceback (most recent call last):
> +  File "041", line 1415, in test_explicit_mirror_filter
> +    self.assert_qmp(result, 'return', {})
> +  File "/usr/home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/iotests.py",
> line 888, in assert_qmp
> +    result = self.dictpath(d, path)
> +  File "/usr/home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/iotests.py",
> line 862, in dictpath
> +    self.fail(f'failed path traversal for "{path}" in "{d}"')
> +AssertionError: failed path traversal for "return" in "{'error':
> {'class': 'GenericError', 'desc': "Bus 'pci.0' not found"}}"
> +
> +======================================================================
> +FAIL: test_implicit_mirror_filter (__main__.TestFilters)
> +----------------------------------------------------------------------
> +Traceback (most recent call last):
> +  File "041", line 1381, in test_implicit_mirror_filter
> +    self.assert_qmp(result, 'return', {})
> +  File "/usr/home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/iotests.py",
> line 888, in assert_qmp
> +    result = self.dictpath(d, path)
> +  File "/usr/home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/iotests.py",
> line 862, in dictpath
> +    self.fail(f'failed path traversal for "{path}" in "{d}"')
> +AssertionError: failed path traversal for "return" in "{'error':
> {'class': 'GenericError', 'desc': "Bus 'pci.0' not found"}}"
> +
> +----------------------------------------------------------------------
>  Ran 107 tests

I guess this is the same problem.

> -OK
> +FAILED (failures=2)
> 
> AArch32 and AArch64 Linux:
> 
>   TEST    iotest-qcow2: 041 [fail]
> QEMU          --
> "/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-system-aarch64"
> -nodefaults -disp
> lay none -accel qtest -machine virt
> QEMU_IMG      --
> "/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-io"
>  --cache writeback --aio thre
> ads -f qcow2
> QEMU_NBD      --
> "/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- Linux/aarch64 mustang-maydell 4.15.0-101-generic
> TEST_DIR      --
> /home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.L8FBRpgYvF
> SOCKET_SCM_HELPER --
> /home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/socket_scm_helper
> 
> --- /home/peter.maydell/qemu/tests/qemu-iotests/041.out 2020-09-07
> 18:29:45.583104907 +0000
> +++ /home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/041.out.bad
>       2020-09-07 18:45:19.385149062 +0000
> @@ -1,5 +1,29 @@
> -...........................................................................................................
> +.FF........................................................................................................
> +======================================================================
> +FAIL: test_explicit_mirror_filter (__main__.TestFilters)
> +----------------------------------------------------------------------
> +Traceback (most recent call last):
> +  File "041", line 1415, in test_explicit_mirror_filter
> +    self.assert_qmp(result, 'return', {})
> +  File "/home/peter.maydell/qemu/tests/qemu-iotests/iotests.py", line
> 888, in assert_qmp
> +    result = self.dictpath(d, path)
> +  File "/home/peter.maydell/qemu/tests/qemu-iotests/iotests.py", line
> 862, in dictpath
> +    self.fail(f'failed path traversal for "{path}" in "{d}"')
> +AssertionError: failed path traversal for "return" in "{'error':
> {'class': 'GenericError', 'desc': "Bus 'pci.0' not fou
> nd"}}"
> +
> +======================================================================
> +FAIL: test_implicit_mirror_filter (__main__.TestFilters)
> +----------------------------------------------------------------------
> +Traceback (most recent call last):
> +  File "041", line 1381, in test_implicit_mirror_filter
> +    self.assert_qmp(result, 'return', {})
> +  File "/home/peter.maydell/qemu/tests/qemu-iotests/iotests.py", line
> 888, in assert_qmp
> +    result = self.dictpath(d, path)
> +  File "/home/peter.maydell/qemu/tests/qemu-iotests/iotests.py", line
> 862, in dictpath
> +    self.fail(f'failed path traversal for "{path}" in "{d}"')
> +AssertionError: failed path traversal for "return" in "{'error':
> {'class': 'GenericError', 'desc': "Bus 'pci.0' not found"}}"
> +
>  ----------------------------------------------------------------------
>  Ran 107 tests

Probably solved by removing the bus like suggested for s390x above.

Kevin
Max Reitz Sept. 8, 2020, 9:01 a.m. UTC | #3
On 08.09.20 09:01, Kevin Wolf wrote:
> Am 07.09.2020 um 22:22 hat Peter Maydell geschrieben:
>> On Mon, 7 Sep 2020 at 12:09, Kevin Wolf <kwolf@redhat.com> wrote:
>>>
>>> The following changes since commit 7c37270b3fbe3d034ba80e488761461676e21eb4:
>>>
>>>   Merge remote-tracking branch 'remotes/kraxel/tags/ui-20200904-pull-request' into staging (2020-09-06 16:23:55 +0100)
>>>
>>> are available in the Git repository at:
>>>
>>>   git://repo.or.cz/qemu/kevin.git tags/for-upstream
>>>
>>> for you to fetch changes up to 4cc0ec9b1b8830f7d1fcf5dfded19ef070f98eaa:
>>>
>>>   block/nvme: Pair doorbell registers (2020-09-07 12:47:57 +0200)
>>>
>>> ----------------------------------------------------------------
>>> Block layer patches:
>>>
>>> - qemu-img create: Fail gracefully when backing file is an empty string
>>> - Fixes related to filter block nodes ("Deal with filters" series)
>>> - block/nvme: Various cleanups required to use multiple queues
>>> - block/nvme: Use NvmeBar structure from "block/nvme.h"
>>> - file-win32: Fix "locking" option
>>> - iotests: Allow running from different directory
>>
>> Fails in make check on iotests 040 and/or 041, various hosts:
>>
>> s390x linux:
>>
>>   TEST    iotest-qcow2: 041 [fail]
>> QEMU          --
>> "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-system-s390x"
>> -nodefaults -display none -accel qtest
>> QEMU_IMG      -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-img"
>> QEMU_IO       --
>> "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-io"
>> --cache writeback --aio threads -f qcow2
>> QEMU_NBD      -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-nbd"
>> IMGFMT        -- qcow2 (compat=1.1)
>> IMGPROTO      -- file
>> PLATFORM      -- Linux/s390x qemu01 4.15.0-72-generic
>> TEST_DIR      -- /home/ubuntu/qemu/build/all/tests/qemu-iotests/scratch
>> SOCK_DIR      -- /tmp/tmp.5Tpl6u2SCo
>> SOCKET_SCM_HELPER --
>> /home/ubuntu/qemu/build/all/tests/qemu-iotests/socket_scm_helper
>>
>> --- /home/ubuntu/qemu/tests/qemu-iotests/041.out        2020-09-07
>> 14:29:45.468466636 -0400
>> +++ /home/ubuntu/qemu/build/all/tests/qemu-iotests/041.out.bad
>> 2020-09-07 14:43:41.494989911 -0400
>> @@ -1,5 +1,29 @@
>> -...........................................................................................................
>> +.FF........................................................................................................
>> +======================================================================
>> +FAIL: test_explicit_mirror_filter (__main__.TestFilters)
>> +----------------------------------------------------------------------
>> +Traceback (most recent call last):
>> +  File "041", line 1415, in test_explicit_mirror_filter
>> +    self.assert_qmp(result, 'return', {})
>> +  File "/home/ubuntu/qemu/tests/qemu-iotests/iotests.py", line 888,
>> in assert_qmp
>> +    result = self.dictpath(d, path)
>> +  File "/home/ubuntu/qemu/tests/qemu-iotests/iotests.py", line 862, in dictpath
>> +    self.fail(f'failed path traversal for "{path}" in "{d}"')
>> +AssertionError: failed path traversal for "return" in "{'error':
>> {'class': 'GenericError', 'desc': "Device 'virtio-blk-ccw' can't go on
>> PCI bus"}}"
> 
> Max, any specific reason you specified the bus in device_add?

No, I don’t think so.

> This seems to fix it for me. Do you agree with the change?
> 
> diff --git a/tests/qemu-iotests/041 b/tests/qemu-iotests/041
> index cdbef3ba20..203ed58868 100755
> --- a/tests/qemu-iotests/041
> +++ b/tests/qemu-iotests/041
> @@ -1376,7 +1376,6 @@ class TestFilters(iotests.QMPTestCase):
>          result = self.vm.qmp('device_add',
>                               driver='virtio-blk',
>                               id='virtio',
> -                             bus='pci.0',
>                               drive='source')
>          self.assert_qmp(result, 'return', {})
> 
> @@ -1410,7 +1409,6 @@ class TestFilters(iotests.QMPTestCase):
>          result = self.vm.qmp('device_add',
>                               driver='virtio-blk',
>                               id='virtio',
> -                             bus='pci.0',
>                               drive='source')
>          self.assert_qmp(result, 'return', {})

Partially – this still fails on FreeBSD just like 040 fails.  So as
discussed on IRC, we should probably make this add a scsi-hd device and
add a virtio-scsi device to the VM in setUp().

[...]

>> freebsd:
>>
>>   TEST    iotest-qcow2: 040 [fail]
>> QEMU          --
>> "/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-system-aarch64"
>> -nodefaults -display none -accel qtest -machine virt
>> QEMU_IMG      --
>> "/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-img"
>> QEMU_IO       --
>> "/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-io"
>>  --cache writeback --aio threads -f qcow2
>> QEMU_NBD      --
>> "/usr/home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/../../qemu-nbd"
>> IMGFMT        -- qcow2 (compat=1.1)
>> IMGPROTO      -- file
>> PLATFORM      -- FreeBSD/amd64 freebsd 12.1-RELEASE
>> TEST_DIR      -- /home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/scratch
>> SOCK_DIR      -- /tmp/tmp.gaJ11NA1
>> SOCKET_SCM_HELPER --
>>
>> --- /home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/040.out
>> 2020-09-07 18:29:28.000000000 +0000
>> +++ /home/qemu/qemu-test.6pxNB5/build/tests/qemu-iotests/040.out.bad
>>  2020-09-07 18:37:26.783967000 +0000
>> @@ -1,5 +1,17 @@
>> -.................................................................
>> +....................F............................................
>> +======================================================================
>> +FAIL: test_filtered_active_commit_with_filter (__main__.TestCommitWithFilters)
>>  ----------------------------------------------------------------------
>> +Traceback (most recent call last):
>> +  File "040", line 867, in test_filtered_active_commit_with_filter
>> +    self.assert_qmp(result, 'return', {})
>> +  File "/usr/home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/iotests.py",
>> line 888, in assert_qmp
>> +    result = self.dictpath(d, path)
>> +  File "/usr/home/qemu/qemu-test.6pxNB5/src/tests/qemu-iotests/iotests.py",
>> line 862, in dictpath
>> +    self.fail(f'failed path traversal for "{path}" in "{d}"')
>> +AssertionError: failed path traversal for "return" in "{'error':
>> {'class': 'GenericError', 'desc': "Bus 'pcie.0' does not support
>> hotplugging"}}"
> 
> This one I don't really understand. Why "pcie.0"? Do FreeBSD hosts
> default to a different machine type? The default should be the "pc"
> machine type, which has pci, not pcie.

As you’ve pointed out on IRC, this is for aarch64, so probably not so
much about the pc machine type.  Still, it passes on Linux, so there’s a
bit of a mystery still.

Anyway, using virtio-scsi + scsi-hd should make it pass.

I’ve attached a diff that makes “make vm-build-freebsd” (for aarch64)
pass for me.

Max
diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
index 2a54f9ad21..caf286571a 100755
--- a/tests/qemu-iotests/040
+++ b/tests/qemu-iotests/040
@@ -761,7 +761,7 @@ class TestCommitWithFilters(iotests.QMPTestCase):
 
         self.do_test_io('write')
 
-        self.vm = iotests.VM()
+        self.vm = iotests.VM().add_device('virtio-scsi,id=vio-scsi')
         self.vm.launch()
 
         result = self.vm.qmp('object-add', qom_type='throttle-group', id='tg')
@@ -862,8 +862,8 @@ class TestCommitWithFilters(iotests.QMPTestCase):
         # Add a device, so the commit job finds a parent it can change
         # to point to the base node (so we can test that top-filter is
         # dropped from the graph)
-        result = self.vm.qmp('device_add', id='drv0', driver='virtio-blk',
-                             drive='top-filter')
+        result = self.vm.qmp('device_add', id='drv0', driver='scsi-hd',
+                             bus='vio-scsi.0', drive='top-filter')
         self.assert_qmp(result, 'return', {})
 
         # Try to release our reference to top-filter; that should not
@@ -889,7 +889,7 @@ class TestCommitWithFilters(iotests.QMPTestCase):
 
         # Check that drv0 is now connected to cow-2
         blockdevs = self.vm.qmp('query-block')['return']
-        drv0 = next(dev for dev in blockdevs if '/drv0' in dev['qdev'])
+        drv0 = next(dev for dev in blockdevs if dev['qdev'] == 'drv0')
         self.assertEqual(drv0['inserted']['node-name'], 'cow-2')
 
         # 3 has been comitted into 2
diff --git a/tests/qemu-iotests/041 b/tests/qemu-iotests/041
index cdbef3ba20..a7780853cd 100755
--- a/tests/qemu-iotests/041
+++ b/tests/qemu-iotests/041
@@ -1300,7 +1300,7 @@ class TestFilters(iotests.QMPTestCase):
         qemu_io('-c', 'write -P 1 0 512k', backing_img)
         qemu_io('-c', 'write -P 2 512k 512k', test_img)
 
-        self.vm = iotests.VM()
+        self.vm = iotests.VM().add_device('virtio-scsi,id=vio-scsi')
         self.vm.launch()
 
         result = self.vm.qmp('blockdev-add', **{
@@ -1374,9 +1374,9 @@ class TestFilters(iotests.QMPTestCase):
 
         # We need this so we can query from above the mirror node
         result = self.vm.qmp('device_add',
-                             driver='virtio-blk',
+                             driver='scsi-hd',
                              id='virtio',
-                             bus='pci.0',
+                             bus='vio-scsi.0',
                              drive='source')
         self.assert_qmp(result, 'return', {})
 
@@ -1389,8 +1389,8 @@ class TestFilters(iotests.QMPTestCase):
 
         # The mirror filter is now an implicit node, so it should be
         # invisible when querying the backing chain
-        device_info = self.vm.qmp('query-block')['return'][0]
-        assert device_info['qdev'] == '/machine/peripheral/virtio/virtio-backend'
+        blockdevs = self.vm.qmp('query-block')['return']
+        device_info = next(dev for dev in blockdevs if dev['qdev'] == 'virtio')
 
         assert device_info['inserted']['node-name'] == 'source'
 
@@ -1408,9 +1408,9 @@ class TestFilters(iotests.QMPTestCase):
 
         # We need this so we can query from above the mirror node
         result = self.vm.qmp('device_add',
-                             driver='virtio-blk',
+                             driver='scsi-hd',
                              id='virtio',
-                             bus='pci.0',
+                             bus='vio-scsi.0',
                              drive='source')
         self.assert_qmp(result, 'return', {})
 
@@ -1424,8 +1424,8 @@ class TestFilters(iotests.QMPTestCase):
 
         # With a node-name given to it, the mirror filter should now
         # be visible
-        device_info = self.vm.qmp('query-block')['return'][0]
-        assert device_info['qdev'] == '/machine/peripheral/virtio/virtio-backend'
+        blockdevs = self.vm.qmp('query-block')['return']
+        device_info = next(dev for dev in blockdevs if dev['qdev'] == 'virtio')
 
         assert device_info['inserted']['node-name'] == 'mirror-filter'