From patchwork Fri Dec 21 23:47:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 10741147 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1560613B5 for ; Fri, 21 Dec 2018 23:50:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 01BC228807 for ; Fri, 21 Dec 2018 23:50:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E46162880C; Fri, 21 Dec 2018 23:50:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 72A2E28807 for ; Fri, 21 Dec 2018 23:50:25 +0000 (UTC) Received: from localhost ([::1]:48926 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaUYi-0007Eh-Ma for patchwork-qemu-devel@patchwork.kernel.org; Fri, 21 Dec 2018 18:50:24 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47609) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaUWN-00059T-7W for qemu-devel@nongnu.org; Fri, 21 Dec 2018 18:47:59 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaUWM-0004pe-GJ for qemu-devel@nongnu.org; Fri, 21 Dec 2018 18:47:59 -0500 Received: from mx1.redhat.com ([209.132.183.28]:40836) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gaUWJ-0004mV-Fp; Fri, 21 Dec 2018 18:47:55 -0500 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C30427E9D7; Fri, 21 Dec 2018 23:47:54 +0000 (UTC) Received: from localhost (ovpn-116-185.ams2.redhat.com [10.36.116.185]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 596E76152F; Fri, 21 Dec 2018 23:47:54 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Sat, 22 Dec 2018 00:47:48 +0100 Message-Id: <20181221234750.23577-2-mreitz@redhat.com> In-Reply-To: <20181221234750.23577-1-mreitz@redhat.com> References: <20181221234750.23577-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Fri, 21 Dec 2018 23:47:54 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 1/3] iotests.py: Add qemu_nbd_pipe() X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP In some cases, we may want to deal with qemu-nbd errors (e.g. by launching it in a different configuration until it no longer throws any). In that case, we do not want its output ending up in the test output. It may still be useful for handling the error, though, so add a new function that works basically like qemu_nbd(), only that it returns the qemu-nbd output instead of making it end up in the log. In contrast to qemu_img_pipe(), it does still return the exit code as well, though, because that is even more important for error handling. Signed-off-by: Max Reitz Reviewed-by: Eric Blake --- tests/qemu-iotests/iotests.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index d537538ba0..9c3eb9e2f8 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -190,6 +190,20 @@ def qemu_nbd(*args): '''Run qemu-nbd in daemon mode and return the parent's exit code''' return subprocess.call(qemu_nbd_args + ['--fork'] + list(args)) +def qemu_nbd_pipe(*args): + '''Run qemu-nbd in daemon mode and return both the parent's exit code + and its output''' + subp = subprocess.Popen(qemu_nbd_args + ['--fork'] + list(args), + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + universal_newlines=True) + exitcode = subp.wait() + if exitcode < 0: + sys.stderr.write('qemu-nbd received signal %i: %s\n' % + (-exitcode, + ' '.join(qemu_nbd_args + ['--fork'] + list(args)))) + return exitcode, subp.communicate()[0] + def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt): '''Return True if two image files are identical''' return qemu_img('compare', '-f', fmt1, From patchwork Fri Dec 21 23:47:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 10741149 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3203814E2 for ; Fri, 21 Dec 2018 23:51:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 229B828807 for ; Fri, 21 Dec 2018 23:51:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 166612880C; Fri, 21 Dec 2018 23:51:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id CB3E128807 for ; Fri, 21 Dec 2018 23:51:32 +0000 (UTC) Received: from localhost ([::1]:48944 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaUZo-00080m-2n for patchwork-qemu-devel@patchwork.kernel.org; Fri, 21 Dec 2018 18:51:32 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47623) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaUWN-0005AM-Pg for qemu-devel@nongnu.org; Fri, 21 Dec 2018 18:48:01 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaUWN-0004qE-37 for qemu-devel@nongnu.org; Fri, 21 Dec 2018 18:47:59 -0500 Received: from mx1.redhat.com ([209.132.183.28]:37878) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gaUWL-0004o4-HT; Fri, 21 Dec 2018 18:47:57 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id DFC317FDE4; Fri, 21 Dec 2018 23:47:56 +0000 (UTC) Received: from localhost (ovpn-116-185.ams2.redhat.com [10.36.116.185]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 79AD7600CD; Fri, 21 Dec 2018 23:47:56 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Sat, 22 Dec 2018 00:47:49 +0100 Message-Id: <20181221234750.23577-3-mreitz@redhat.com> In-Reply-To: <20181221234750.23577-1-mreitz@redhat.com> References: <20181221234750.23577-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Fri, 21 Dec 2018 23:47:56 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 2/3] iotests: Bind qemu-nbd to localhost in 147 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP By default, qemu-nbd binds to 0.0.0.0. However, we then proceed to connect to "localhost". Usually, this works out fine; but if this test is run concurrently, some other test function may have bound a different server to ::1 (on the same port -- you can bind different serves to the same port, as long as one is on IPv4 and the other on IPv6). So running qemu-nbd works, it can bind to 0.0.0.0:NBD_PORT. But potentially a concurrent test has successfully taken [::1]:NBD_PORT. In this case, trying to connect to "localhost" will lead us to the IPv6 instance, where we do not want to end up. Fix this by just binding to "localhost". This will make qemu-nbd error out immediately and not give us cryptic errors later. (Also, it will allow us to just try a different port as of a future patch.) Signed-off-by: Max Reitz Reviewed-by: Eric Blake --- tests/qemu-iotests/147 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147 index 05b374b7d3..3e10a9969e 100755 --- a/tests/qemu-iotests/147 +++ b/tests/qemu-iotests/147 @@ -92,7 +92,7 @@ class QemuNBD(NBDBlockdevAddBase): self.assertEqual(qemu_nbd('-f', imgfmt, test_img, *args), 0) def test_inet(self): - self._server_up('-p', str(NBD_PORT)) + self._server_up('-b', 'localhost', '-p', str(NBD_PORT)) address = { 'type': 'inet', 'data': { 'host': 'localhost', From patchwork Fri Dec 21 23:47:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 10741143 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 50DF913B5 for ; Fri, 21 Dec 2018 23:49:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 34CE628803 for ; Fri, 21 Dec 2018 23:49:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 26C1F2880B; Fri, 21 Dec 2018 23:49:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7CA0B28803 for ; Fri, 21 Dec 2018 23:49:22 +0000 (UTC) Received: from localhost ([::1]:48914 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaUXg-0006Ol-S7 for patchwork-qemu-devel@patchwork.kernel.org; Fri, 21 Dec 2018 18:49:20 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47645) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaUWQ-0005GE-O6 for qemu-devel@nongnu.org; Fri, 21 Dec 2018 18:48:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaUWP-0004t4-Vz for qemu-devel@nongnu.org; Fri, 21 Dec 2018 18:48:02 -0500 Received: from mx1.redhat.com ([209.132.183.28]:39446) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gaUWN-0004qa-QZ; Fri, 21 Dec 2018 18:47:59 -0500 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2FD798E5A8; Fri, 21 Dec 2018 23:47:59 +0000 (UTC) Received: from localhost (ovpn-116-185.ams2.redhat.com [10.36.116.185]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 981F719C7C; Fri, 21 Dec 2018 23:47:58 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Sat, 22 Dec 2018 00:47:50 +0100 Message-Id: <20181221234750.23577-4-mreitz@redhat.com> In-Reply-To: <20181221234750.23577-1-mreitz@redhat.com> References: <20181221234750.23577-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Fri, 21 Dec 2018 23:47:59 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 3/3] iotests: Allow 147 to be run concurrently X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP To do this, we need to allow creating the NBD server on various ports instead of a single one (which may not even work if you run just one instance, because something entirely else might be using that port). So we just pick a random port in [32768, 32768 + 1024) and try to create a server there. If that fails, we just retry until something sticks. For the IPv6 test, we need a different range, though (just above that one). This is because "localhost" resolves to both 127.0.0.1 and ::1. This means that if you bind to it, it will bind to both, if possible, or just one if the other is already in use. Therefore, if the IPv6 test has already taken [::1]:some_port and we then try to take localhost:some_port, that will work -- only the second server will be bound to 127.0.0.1:some_port alone and not [::1]:some_port in addition. So we have two different servers on the same port, one for IPv4 and one for IPv6. But when we then try to connect to the server through localhost:some_port, we will always end up at the IPv6 one (as long as it is up), and this may not be the one we want. Thus, we must make sure not to create an IPv6-only NBD server on the same port as a normal "dual-stack" NBD server -- which is done by using distinct port ranges, as explained above. Signed-off-by: Max Reitz --- tests/qemu-iotests/147 | 98 +++++++++++++++++++++++++++++------------- 1 file changed, 68 insertions(+), 30 deletions(-) diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147 index 3e10a9969e..82513279b0 100755 --- a/tests/qemu-iotests/147 +++ b/tests/qemu-iotests/147 @@ -19,13 +19,17 @@ # import os +import random import socket import stat import time import iotests -from iotests import cachemode, imgfmt, qemu_img, qemu_nbd +from iotests import cachemode, imgfmt, qemu_img, qemu_nbd, qemu_nbd_pipe -NBD_PORT = 10811 +NBD_PORT_START = 32768 +NBD_PORT_END = NBD_PORT_START + 1024 +NBD_IPV6_PORT_START = NBD_PORT_END +NBD_IPV6_PORT_END = NBD_IPV6_PORT_START + 1024 test_img = os.path.join(iotests.test_dir, 'test.img') unix_socket = os.path.join(iotests.test_dir, 'nbd.socket') @@ -88,17 +92,29 @@ class QemuNBD(NBDBlockdevAddBase): except OSError: pass + def _try_server_up(self, *args): + status, msg = qemu_nbd_pipe('-f', imgfmt, test_img, *args) + if status == 0: + return True + if 'Address already in use' in msg: + return False + self.fail(msg) + def _server_up(self, *args): - self.assertEqual(qemu_nbd('-f', imgfmt, test_img, *args), 0) + self.assertTrue(self._try_server_up(*args)) def test_inet(self): - self._server_up('-b', 'localhost', '-p', str(NBD_PORT)) + while True: + nbd_port = random.randrange(NBD_PORT_START, NBD_PORT_END) + if self._try_server_up('-b', 'localhost', '-p', str(nbd_port)): + break + address = { 'type': 'inet', 'data': { 'host': 'localhost', - 'port': str(NBD_PORT) + 'port': str(nbd_port) } } - self.client_test('nbd://localhost:%i' % NBD_PORT, + self.client_test('nbd://localhost:%i' % nbd_port, flatten_sock_addr(address)) def test_unix(self): @@ -130,8 +146,13 @@ class BuiltinNBD(NBDBlockdevAddBase): except OSError: pass - def _server_up(self, address, export_name=None, export_name2=None): + # Returns False on EADDRINUSE; fails an assertion on other errors. + # Returns True on success. + def _try_server_up(self, address, export_name=None, export_name2=None): result = self.server.qmp('nbd-server-start', addr=address) + if 'error' in result and \ + 'Address already in use' in result['error']['desc']: + return False self.assert_qmp(result, 'return', {}) if export_name is None: @@ -146,20 +167,28 @@ class BuiltinNBD(NBDBlockdevAddBase): name=export_name2) self.assert_qmp(result, 'return', {}) + return True + + def _server_up(self, address, export_name=None, export_name2=None): + self.assertTrue(self._try_server_up(address, export_name, export_name2)) def _server_down(self): result = self.server.qmp('nbd-server-stop') self.assert_qmp(result, 'return', {}) def do_test_inet(self, export_name=None): - address = { 'type': 'inet', - 'data': { - 'host': 'localhost', - 'port': str(NBD_PORT) - } } - self._server_up(address, export_name) + while True: + nbd_port = random.randrange(NBD_PORT_START, NBD_PORT_END) + address = { 'type': 'inet', + 'data': { + 'host': 'localhost', + 'port': str(nbd_port) + } } + if self._try_server_up(address, export_name): + break + export_name = export_name or 'nbd-export' - self.client_test('nbd://localhost:%i/%s' % (NBD_PORT, export_name), + self.client_test('nbd://localhost:%i/%s' % (nbd_port, export_name), flatten_sock_addr(address), export_name) self._server_down() @@ -173,15 +202,19 @@ class BuiltinNBD(NBDBlockdevAddBase): self.do_test_inet('shadow') def test_inet_two_exports(self): - address = { 'type': 'inet', - 'data': { - 'host': 'localhost', - 'port': str(NBD_PORT) - } } - self._server_up(address, 'exp1', 'exp2') - self.client_test('nbd://localhost:%i/%s' % (NBD_PORT, 'exp1'), + while True: + nbd_port = random.randrange(NBD_PORT_START, NBD_PORT_END) + address = { 'type': 'inet', + 'data': { + 'host': 'localhost', + 'port': str(nbd_port) + } } + if self._try_server_up(address, 'exp1', 'exp2'): + break + + self.client_test('nbd://localhost:%i/%s' % (nbd_port, 'exp1'), flatten_sock_addr(address), 'exp1', 'node1', False) - self.client_test('nbd://localhost:%i/%s' % (NBD_PORT, 'exp2'), + self.client_test('nbd://localhost:%i/%s' % (nbd_port, 'exp2'), flatten_sock_addr(address), 'exp2', 'node2', False) result = self.vm.qmp('blockdev-del', node_name='node1') self.assert_qmp(result, 'return', {}) @@ -197,20 +230,25 @@ class BuiltinNBD(NBDBlockdevAddBase): except socket.gaierror: # IPv6 not available, skip return - address = { 'type': 'inet', - 'data': { - 'host': '::1', - 'port': str(NBD_PORT), - 'ipv4': False, - 'ipv6': True - } } + + while True: + nbd_port = random.randrange(NBD_IPV6_PORT_START, NBD_IPV6_PORT_END) + address = { 'type': 'inet', + 'data': { + 'host': '::1', + 'port': str(nbd_port), + 'ipv4': False, + 'ipv6': True + } } + if self._try_server_up(address): + break + filename = { 'driver': 'raw', 'file': { 'driver': 'nbd', 'export': 'nbd-export', 'server': flatten_sock_addr(address) } } - self._server_up(address) self.client_test(filename, flatten_sock_addr(address), 'nbd-export') self._server_down()