From patchwork Tue May 5 12:58:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11528991 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B0BDA912 for ; Tue, 5 May 2020 13:06:05 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 75A53206D7 for ; Tue, 5 May 2020 13:06:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fn4fTO2C" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 75A53206D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:53566 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxGu-0004f1-IY for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:06:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38698) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVx9u-0001Hg-HA for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:50 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:52148 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVx9s-0000mH-QP for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683528; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Qef1dmn4AtamnZwBgAi9HGsLrBW+ppcBl1pWHe8MRJ0=; b=fn4fTO2CDgrHBULgw1VQ1KeYnAIjkwcpvmSkFeyKCh6hyagOXrmasDUrCGpDWyBthiLRal DHB+AvqknY9WawS3/iDBYZ41TB4MRUa2pAG6SaWKpz5Qh8C3tQcu+ZGeyb44hUR+8zztJX QdlkJN2QLRF0P6A8N8rCmnxy0hkOJ/k= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-451-wOSZC0fuN8e83vY0m40FBw-1; Tue, 05 May 2020 08:58:31 -0400 X-MC-Unique: wOSZC0fuN8e83vY0m40FBw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A8E5880B713; Tue, 5 May 2020 12:58:30 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 235DD5C1D4; Tue, 5 May 2020 12:58:29 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 01/24] iotests: do a light delinting Date: Tue, 5 May 2020 14:58:03 +0200 Message-Id: <20200505125826.1001451-2-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.120; envelope-from=mreitz@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:38 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow This doesn't fix everything in here, but it does help clean up the pylint report considerably. This should be 100% style changes only; the intent is to make pylint more useful by working on establishing a baseline for iotests that we can gate against in the future. Signed-off-by: John Snow Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Max Reitz Message-Id: <20200331000014.11581-2-jsnow@redhat.com> Reviewed-by: Kevin Wolf Signed-off-by: Max Reitz --- tests/qemu-iotests/iotests.py | 83 ++++++++++++++++++----------------- 1 file changed, 43 insertions(+), 40 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 5f8c263d59..6f6363f3ec 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -16,11 +16,9 @@ # along with this program. If not, see . # -import errno import os import re import subprocess -import string import unittest import sys import struct @@ -35,7 +33,7 @@ import faulthandler sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python')) from qemu import qtest -assert sys.version_info >= (3,6) +assert sys.version_info >= (3, 6) faulthandler.enable() @@ -141,11 +139,11 @@ def qemu_img_log(*args): return result def img_info_log(filename, filter_path=None, imgopts=False, extra_args=[]): - args = [ 'info' ] + args = ['info'] if imgopts: args.append('--image-opts') else: - args += [ '-f', imgfmt ] + args += ['-f', imgfmt] args += extra_args args.append(filename) @@ -224,7 +222,7 @@ class QemuIoInteractive: # quit command is in close(), '\n' is added automatically assert '\n' not in cmd cmd = cmd.strip() - assert cmd != 'q' and cmd != 'quit' + assert cmd not in ('q', 'quit') self._p.stdin.write(cmd + '\n') self._p.stdin.flush() return self._read_output() @@ -246,10 +244,8 @@ def qemu_nbd_early_pipe(*args): sys.stderr.write('qemu-nbd received signal %i: %s\n' % (-exitcode, ' '.join(qemu_nbd_args + ['--fork'] + list(args)))) - if exitcode == 0: - return exitcode, '' - else: - return exitcode, subp.communicate()[0] + + return exitcode, subp.communicate()[0] if exitcode else '' def qemu_nbd_popen(*args): '''Run qemu-nbd in daemon mode and return the parent's exit code''' @@ -313,7 +309,7 @@ def filter_qmp(qmsg, filter_fn): items = qmsg.items() for k, v in items: - if isinstance(v, list) or isinstance(v, dict): + if isinstance(v, (dict, list)): qmsg[k] = filter_qmp(v, filter_fn) else: qmsg[k] = filter_fn(k, v) @@ -324,7 +320,7 @@ def filter_testfiles(msg): return msg.replace(prefix, 'TEST_DIR/PID-') def filter_qmp_testfiles(qmsg): - def _filter(key, value): + def _filter(_key, value): if is_str(value): return filter_testfiles(value) return value @@ -351,7 +347,7 @@ def filter_imgfmt(msg): return msg.replace(imgfmt, 'IMGFMT') def filter_qmp_imgfmt(qmsg): - def _filter(key, value): + def _filter(_key, value): if is_str(value): return filter_imgfmt(value) return value @@ -362,7 +358,7 @@ def log(msg, filters=[], indent=None): If indent is provided, JSON serializable messages are pretty-printed.''' for flt in filters: msg = flt(msg) - if isinstance(msg, dict) or isinstance(msg, list): + if isinstance(msg, (dict, list)): # Python < 3.4 needs to know not to add whitespace when pretty-printing: separators = (', ', ': ') if indent is None else (',', ': ') # Don't sort if it's already sorted @@ -373,14 +369,14 @@ def log(msg, filters=[], indent=None): print(msg) class Timeout: - def __init__(self, seconds, errmsg = "Timeout"): + def __init__(self, seconds, errmsg="Timeout"): self.seconds = seconds self.errmsg = errmsg def __enter__(self): signal.signal(signal.SIGALRM, self.timeout) signal.setitimer(signal.ITIMER_REAL, self.seconds) return self - def __exit__(self, type, value, traceback): + def __exit__(self, exc_type, value, traceback): signal.setitimer(signal.ITIMER_REAL, 0) return False def timeout(self, signum, frame): @@ -389,7 +385,7 @@ class Timeout: def file_pattern(name): return "{0}-{1}".format(os.getpid(), name) -class FilePaths(object): +class FilePaths: """ FilePaths is an auto-generated filename that cleans itself up. @@ -536,11 +532,11 @@ class VM(qtest.QEMUQtestMachine): self.pause_drive(drive, "write_aio") return self.qmp('human-monitor-command', - command_line='qemu-io %s "break %s bp_%s"' % (drive, event, drive)) + command_line='qemu-io %s "break %s bp_%s"' % (drive, event, drive)) def resume_drive(self, drive): self.qmp('human-monitor-command', - command_line='qemu-io %s "remove_break bp_%s"' % (drive, drive)) + command_line='qemu-io %s "remove_break bp_%s"' % (drive, drive)) def hmp_qemu_io(self, drive, cmd): '''Write to a given drive using an HMP command''' @@ -551,8 +547,8 @@ class VM(qtest.QEMUQtestMachine): if output is None: output = dict() if isinstance(obj, list): - for i in range(len(obj)): - self.flatten_qmp_object(obj[i], output, basestr + str(i) + '.') + for i, item in enumerate(obj): + self.flatten_qmp_object(item, output, basestr + str(i) + '.') elif isinstance(obj, dict): for key in obj: self.flatten_qmp_object(obj[key], output, basestr + key + '.') @@ -710,9 +706,7 @@ class VM(qtest.QEMUQtestMachine): for bitmap in bitmaps[node_name]: if bitmap.get('name', '') == bitmap_name: - if recording is None: - return bitmap - elif bitmap.get('recording') == recording: + if recording is None or bitmap.get('recording') == recording: return bitmap return None @@ -763,12 +757,13 @@ class VM(qtest.QEMUQtestMachine): assert node is not None, 'Cannot follow path %s%s' % (root, path) try: - node_id = next(edge['child'] for edge in graph['edges'] \ - if edge['parent'] == node['id'] and - edge['name'] == child_name) + node_id = next(edge['child'] for edge in graph['edges'] + if (edge['parent'] == node['id'] and + edge['name'] == child_name)) + + node = next(node for node in graph['nodes'] + if node['id'] == node_id) - node = next(node for node in graph['nodes'] \ - if node['id'] == node_id) except StopIteration: node = None @@ -786,6 +781,12 @@ index_re = re.compile(r'([^\[]+)\[([^\]]+)\]') class QMPTestCase(unittest.TestCase): '''Abstract base class for QMP test cases''' + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + # Many users of this class set a VM property we rely on heavily + # in the methods below. + self.vm = None + def dictpath(self, d, path): '''Traverse a path in a nested dict''' for component in path.split('/'): @@ -831,7 +832,7 @@ class QMPTestCase(unittest.TestCase): else: self.assertEqual(result, value, '"%s" is "%s", expected "%s"' - % (path, str(result), str(value))) + % (path, str(result), str(value))) def assert_no_active_block_jobs(self): result = self.vm.qmp('query-block-jobs') @@ -841,15 +842,15 @@ class QMPTestCase(unittest.TestCase): """Issue a query-named-block-nodes and assert node_name and/or file_name is present in the result""" def check_equal_or_none(a, b): - return a == None or b == None or a == b + return a is None or b is None or a == b assert node_name or file_name result = self.vm.qmp('query-named-block-nodes') for x in result["return"]: if check_equal_or_none(x.get("node-name"), node_name) and \ check_equal_or_none(x.get("file"), file_name): return - self.assertTrue(False, "Cannot find %s %s in result:\n%s" % \ - (node_name, file_name, result)) + self.fail("Cannot find %s %s in result:\n%s" % + (node_name, file_name, result)) def assert_json_filename_equal(self, json_filename, reference): '''Asserts that the given filename is a json: filename and that its @@ -898,13 +899,13 @@ class QMPTestCase(unittest.TestCase): self.assert_qmp(event, 'data/error', error) self.assert_no_active_block_jobs() return event - elif event['event'] == 'JOB_STATUS_CHANGE': + if event['event'] == 'JOB_STATUS_CHANGE': self.assert_qmp(event, 'data/id', drive) def wait_ready(self, drive='drive0'): - '''Wait until a block job BLOCK_JOB_READY event''' - f = {'data': {'type': 'mirror', 'device': drive } } - event = self.vm.event_wait(name='BLOCK_JOB_READY', match=f) + """Wait until a BLOCK_JOB_READY event, and return the event.""" + f = {'data': {'type': 'mirror', 'device': drive}} + return self.vm.event_wait(name='BLOCK_JOB_READY', match=f) def wait_ready_and_cancel(self, drive='drive0'): self.wait_ready(drive=drive) @@ -933,7 +934,7 @@ class QMPTestCase(unittest.TestCase): for job in result['return']: if job['device'] == job_id: found = True - if job['paused'] == True and job['busy'] == False: + if job['paused'] and not job['busy']: return job break assert found @@ -1030,8 +1031,8 @@ def qemu_pipe(*args): universal_newlines=True) exitcode = subp.wait() if exitcode < 0: - sys.stderr.write('qemu received signal %i: %s\n' % (-exitcode, - ' '.join(args))) + sys.stderr.write('qemu received signal %i: %s\n' % + (-exitcode, ' '.join(args))) return subp.communicate()[0] def supported_formats(read_only=False): @@ -1063,6 +1064,7 @@ def skip_if_unsupported(required_formats=[], read_only=False): if usf_list: test_case.case_skip('{}: formats {} are not whitelisted'.format( test_case, usf_list)) + return None else: return func(test_case, *args, **kwargs) return func_wrapper @@ -1074,6 +1076,7 @@ def skip_if_user_is_root(func): def func_wrapper(*args, **kwargs): if os.getuid() == 0: case_notrun('{}: cannot be run as root'.format(args[0])) + return None else: return func(*args, **kwargs) return func_wrapper From patchwork Tue May 5 12:58:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11528989 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2603C912 for ; Tue, 5 May 2020 13:05:48 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EFC34206B9 for ; Tue, 5 May 2020 13:05:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="NbIvNRHu" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EFC34206B9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:52254 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxGd-00048S-2B for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:05:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38634) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVx9k-0000qn-8x for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:40 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:33310 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVx9i-0000f7-Oz for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683515; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SPAfN00ZliTVI9TYbx640ixixPfDk00853mVgD7qnhU=; b=NbIvNRHuMjKd7PbSQiEuYcV+s+AazEPEIEZs3z+mamHZ4a+eU0vswbICNIzvA2SnhDqxkB t9i1aYt0nhor6jE1c14PNHcQP/fkhvDAJvb7VQOkpC+nXkhPOo/DodDLfWG435K+w+vHKW Cr1wDWSkjllzMqCFyGhgOnLnsYX0CF8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-223-7ZVPI7JUOCucE-9wv3M52w-1; Tue, 05 May 2020 08:58:33 -0400 X-MC-Unique: 7ZVPI7JUOCucE-9wv3M52w-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D962B8014C0; Tue, 5 May 2020 12:58:32 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7BE1B63F9E; Tue, 5 May 2020 12:58:32 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 02/24] iotests: don't use 'format' for drive_add Date: Tue, 5 May 2020 14:58:04 +0200 Message-Id: <20200505125826.1001451-3-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.120; envelope-from=mreitz@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:38 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow It shadows (with a different type) the built-in format. Use something else. Signed-off-by: John Snow Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Max Reitz Message-Id: <20200331000014.11581-3-jsnow@redhat.com> Reviewed-by: Kevin Wolf Signed-off-by: Max Reitz --- tests/qemu-iotests/055 | 3 ++- tests/qemu-iotests/iotests.py | 6 +++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/tests/qemu-iotests/055 b/tests/qemu-iotests/055 index 82b9f5f47d..4175fff5e4 100755 --- a/tests/qemu-iotests/055 +++ b/tests/qemu-iotests/055 @@ -469,7 +469,8 @@ class TestDriveCompression(iotests.QMPTestCase): qemu_img('create', '-f', fmt, blockdev_target_img, str(TestDriveCompression.image_len), *args) if attach_target: - self.vm.add_drive(blockdev_target_img, format=fmt, interface="none") + self.vm.add_drive(blockdev_target_img, + img_format=fmt, interface="none") self.vm.launch() diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 6f6363f3ec..aca7d50524 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -486,21 +486,21 @@ class VM(qtest.QEMUQtestMachine): self._args.append(opts) return self - def add_drive(self, path, opts='', interface='virtio', format=imgfmt): + def add_drive(self, path, opts='', interface='virtio', img_format=imgfmt): '''Add a virtio-blk drive to the VM''' options = ['if=%s' % interface, 'id=drive%d' % self._num_drives] if path is not None: options.append('file=%s' % path) - options.append('format=%s' % format) + options.append('format=%s' % img_format) options.append('cache=%s' % cachemode) options.append('aio=%s' % aiomode) if opts: options.append(opts) - if format == 'luks' and 'key-secret' not in opts: + if img_format == 'luks' and 'key-secret' not in opts: # default luks support if luks_default_secret_object not in self._args: self.add_object(luks_default_secret_object) From patchwork Tue May 5 12:58:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11528975 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 780DB912 for ; Tue, 5 May 2020 13:01:18 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4CDD4206B9 for ; Tue, 5 May 2020 13:01:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="OdGIjWNG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4CDD4206B9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:35600 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxCH-0005UW-7U for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:01:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38646) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVx9n-0000xz-0p for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:43 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:31989 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVx9m-0000k2-B8 for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683521; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PR8nXNvYiHfVBnPUWtf0oq1EhT2c7vsPZxtjO8nZxT8=; b=OdGIjWNGZ7mm7bpdQXYq/7kKyaqIBAV22XlYL3hbOvDQ/YYnvqglGe5DgqLAg+xls5mv8n l9svk3SlOPqx7S/bdpMaMyWnuO12t1JCxwyPZcPj6TqBhTGnUol50hIyCSZJLb53s/CtTV +sgTacT0UaWOc69gK+MmNdUoIZBjGvk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-83-RxBNsICvMIKw7anXBCuojw-1; Tue, 05 May 2020 08:58:36 -0400 X-MC-Unique: RxBNsICvMIKw7anXBCuojw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3E80E100CCC1; Tue, 5 May 2020 12:58:35 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 900CF6295D; Tue, 5 May 2020 12:58:34 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 03/24] iotests: ignore import warnings from pylint Date: Tue, 5 May 2020 14:58:05 +0200 Message-Id: <20200505125826.1001451-4-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.120; envelope-from=mreitz@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:38 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow The right way to solve this is to come up with a virtual environment infrastructure that sets all the paths correctly, and/or to create installable python modules that can be imported normally. That's hard, so just silence this error for now. Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Max Reitz Signed-off-by: John Snow Message-Id: <20200331000014.11581-4-jsnow@redhat.com> Reviewed-by: Kevin Wolf Signed-off-by: Max Reitz --- tests/qemu-iotests/iotests.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index aca7d50524..37a32c7461 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -30,6 +30,7 @@ import io from collections import OrderedDict import faulthandler +# pylint: disable=import-error, wrong-import-position sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python')) from qemu import qtest From patchwork Tue May 5 12:58:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11528977 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 767841392 for ; Tue, 5 May 2020 13:01:37 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4C2A3206B9 for ; Tue, 5 May 2020 13:01:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Da440sEI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4C2A3206B9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:36632 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxCa-0005uB-C3 for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:01:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38658) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVx9q-00017e-Ky for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:46 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:45963 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVx9p-0000kP-Pl for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683525; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YnyBd8ScGKasPY9cTIUbuFql1oAw7KI2fvvcNpn2tYQ=; b=Da440sEIkMfq3hM0kDps0IqCZRTHLzp7uhmAZyv6RlYn6W+c92dCE3Z+DedWP4hpWvakv3 /0yjIdjLtNsy9qddWoCQPYOlYyiebWoeUMexjHukVPe2t+AbMWKLtJaFywB2zQWUgo+OWz U64Weyq8Fu6BvV5JXEx4o0O77cXk8c0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-29-aMrE8pBlP92UK-S9yIKLRw-1; Tue, 05 May 2020 08:58:38 -0400 X-MC-Unique: aMrE8pBlP92UK-S9yIKLRw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5B7191899536; Tue, 5 May 2020 12:58:37 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F07C960CD1; Tue, 5 May 2020 12:58:36 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 04/24] iotests: replace mutable list default args Date: Tue, 5 May 2020 14:58:06 +0200 Message-Id: <20200505125826.1001451-5-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.120; envelope-from=mreitz@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:38 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow It's bad hygiene: if we modify this list, it will be modified across all invocations. (Remaining bad usages are fixed in a subsequent patch which changes the function signature anyway.) Signed-off-by: John Snow Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Max Reitz Message-Id: <20200331000014.11581-5-jsnow@redhat.com> Reviewed-by: Kevin Wolf Signed-off-by: Max Reitz --- tests/qemu-iotests/iotests.py | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 37a32c7461..d8dc60275b 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -139,7 +139,7 @@ def qemu_img_log(*args): log(result, filters=[filter_testfiles]) return result -def img_info_log(filename, filter_path=None, imgopts=False, extra_args=[]): +def img_info_log(filename, filter_path=None, imgopts=False, extra_args=()): args = ['info'] if imgopts: args.append('--image-opts') @@ -354,7 +354,7 @@ def filter_qmp_imgfmt(qmsg): return value return filter_qmp(qmsg, _filter) -def log(msg, filters=[], indent=None): +def log(msg, filters=(), indent=None): '''Logs either a string message or a JSON serializable message (like QMP). If indent is provided, JSON serializable messages are pretty-printed.''' for flt in filters: @@ -570,7 +570,7 @@ class VM(qtest.QEMUQtestMachine): result.append(filter_qmp_event(ev)) return result - def qmp_log(self, cmd, filters=[], indent=None, **kwargs): + def qmp_log(self, cmd, filters=(), indent=None, **kwargs): full_cmd = OrderedDict(( ("execute", cmd), ("arguments", ordered_qmp(kwargs)) @@ -974,7 +974,7 @@ def case_notrun(reason): open('%s/%s.casenotrun' % (output_dir, seq), 'a').write( ' [case not run] ' + reason + '\n') -def verify_image_format(supported_fmts=[], unsupported_fmts=[]): +def verify_image_format(supported_fmts=(), unsupported_fmts=()): assert not (supported_fmts and unsupported_fmts) if 'generic' in supported_fmts and \ @@ -988,7 +988,7 @@ def verify_image_format(supported_fmts=[], unsupported_fmts=[]): if not_sup or (imgfmt in unsupported_fmts): notrun('not suitable for this image format: %s' % imgfmt) -def verify_protocol(supported=[], unsupported=[]): +def verify_protocol(supported=(), unsupported=()): assert not (supported and unsupported) if 'generic' in supported: @@ -1007,11 +1007,11 @@ def verify_platform(supported=None, unsupported=None): if not any((sys.platform.startswith(x) for x in supported)): notrun('not suitable for this OS: %s' % sys.platform) -def verify_cache_mode(supported_cache_modes=[]): +def verify_cache_mode(supported_cache_modes=()): if supported_cache_modes and (cachemode not in supported_cache_modes): notrun('not suitable for this cache mode: %s' % cachemode) -def verify_aio_mode(supported_aio_modes=[]): +def verify_aio_mode(supported_aio_modes=()): if supported_aio_modes and (aiomode not in supported_aio_modes): notrun('not suitable for this aio mode: %s' % aiomode) @@ -1051,7 +1051,7 @@ def supported_formats(read_only=False): return supported_formats.formats[read_only] -def skip_if_unsupported(required_formats=[], read_only=False): +def skip_if_unsupported(required_formats=(), read_only=False): '''Skip Test Decorator Runs the test if all the required formats are whitelisted''' def skip_test_decorator(func): @@ -1102,11 +1102,11 @@ def execute_unittest(output, verbosity, debug): sys.stderr.write(out) def execute_test(test_function=None, - supported_fmts=[], + supported_fmts=(), supported_platforms=None, - supported_cache_modes=[], supported_aio_modes={}, - unsupported_fmts=[], supported_protocols=[], - unsupported_protocols=[]): + supported_cache_modes=(), supported_aio_modes=(), + unsupported_fmts=(), supported_protocols=(), + unsupported_protocols=()): """Run either unittest or script-style tests.""" # We are using TEST_DIR and QEMU_DEFAULT_MACHINE as proxies to From patchwork Tue May 5 12:58:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11528979 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B3A87912 for ; Tue, 5 May 2020 13:01:59 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8904F206B9 for ; Tue, 5 May 2020 13:01:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Vz5nR6PF" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8904F206B9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:38600 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxCw-0006h4-Ku for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:01:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38690) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVx9t-0001Et-Ir for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:49 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:38263 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVx9s-0000m4-EZ for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683527; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fm8XXGHUuXORUfzHemUsdHhurQve0/uqxYuruDz3q3E=; b=Vz5nR6PFFem9z22fIg7+v1IbG75J1bB+QklDoYyGQMVBIvGCbOVchrcxM+E/J5kqzSSUC6 kHYTm84w9jkc00aGSt6VLfhuCsVweTF7sgO9ZI3Hbl7PVTTC+qsO+L97Fk+YZjYjWFxPkI DtS8o2o8fRKwezZNyKGWumRCef6wIww= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-20-5-ksvQ0TPoa2HUrMHW4o2A-1; Tue, 05 May 2020 08:58:40 -0400 X-MC-Unique: 5-ksvQ0TPoa2HUrMHW4o2A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7F5CA835B42; Tue, 5 May 2020 12:58:39 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 24A787054F; Tue, 5 May 2020 12:58:38 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 05/24] iotests: add pylintrc file Date: Tue, 5 May 2020 14:58:07 +0200 Message-Id: <20200505125826.1001451-6-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=mreitz@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 03:48:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow This allows others to get repeatable results with pylint. If you run `pylint iotests.py`, you should see a 100% pass. Signed-off-by: John Snow Reviewed-by: Max Reitz Message-Id: <20200331000014.11581-6-jsnow@redhat.com> Reviewed-by: Kevin Wolf Signed-off-by: Max Reitz --- tests/qemu-iotests/pylintrc | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 tests/qemu-iotests/pylintrc diff --git a/tests/qemu-iotests/pylintrc b/tests/qemu-iotests/pylintrc new file mode 100644 index 0000000000..daec2c4942 --- /dev/null +++ b/tests/qemu-iotests/pylintrc @@ -0,0 +1,22 @@ +[MESSAGES CONTROL] + +# Disable the message, report, category or checker with the given id(s). You +# can either give multiple identifiers separated by comma (,) or put this +# option multiple times (only on the command line, not in the configuration +# file where it should appear only once). You can also use "--disable=all" to +# disable everything first and then reenable specific checks. For example, if +# you want to run only the similarities checker, you can use "--disable=all +# --enable=similarities". If you want to run only the classes checker, but have +# no Warning level messages displayed, use "--disable=all --enable=classes +# --disable=W". +disable=invalid-name, + no-else-return, + too-few-public-methods, + too-many-arguments, + too-many-branches, + too-many-lines, + too-many-locals, + too-many-public-methods, + # These are temporary, and should be removed: + line-too-long, + missing-docstring, From patchwork Tue May 5 12:58:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11528993 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7843E1392 for ; Tue, 5 May 2020 13:06:09 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4E9A9206B9 for ; Tue, 5 May 2020 13:06:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="WWpkhlsa" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4E9A9206B9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:53886 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxGy-0004n5-9r for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:06:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38672) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVx9s-0001Bx-AI for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:48 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:52899 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVx9r-0000lj-GJ for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683526; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sQB5k6xN6HD62SbaGywdMB2xlMYvChcdkicnAIwVHzY=; b=WWpkhlsaTurP5SDum4RcYNRylpqRrfzKPGWtmF8v+HfhfGQY66AGnapEWw2EhHUp3gi8PX froyvjeLTH0EkMznq6pELUzYaXgXEu8S0sqfjhMlhFCPf4NvUWWbDlEzX1+w6IRRbOrS/y o4RSkZFyz3Uf4PapVq6Endl2rUjCfpY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-198-UBA-xrIHNSi2Y3aON6Nu9A-1; Tue, 05 May 2020 08:58:42 -0400 X-MC-Unique: UBA-xrIHNSi2Y3aON6Nu9A-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8C4D21005510; Tue, 5 May 2020 12:58:41 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 320F560C18; Tue, 5 May 2020 12:58:41 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 06/24] iotests: alphabetize standard imports Date: Tue, 5 May 2020 14:58:08 +0200 Message-Id: <20200505125826.1001451-7-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=mreitz@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 03:48:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow I had to fix a merge conflict, so do this tiny harmless thing while I'm here. Signed-off-by: John Snow Reviewed-by: Max Reitz Message-Id: <20200331000014.11581-7-jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Kevin Wolf Signed-off-by: Max Reitz --- tests/qemu-iotests/iotests.py | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index d8dc60275b..dae124872e 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -16,19 +16,19 @@ # along with this program. If not, see . # +import atexit +from collections import OrderedDict +import faulthandler +import io +import json +import logging import os import re +import signal +import struct import subprocess -import unittest import sys -import struct -import json -import signal -import logging -import atexit -import io -from collections import OrderedDict -import faulthandler +import unittest # pylint: disable=import-error, wrong-import-position sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python')) From patchwork Tue May 5 12:58:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11528983 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E44F11392 for ; Tue, 5 May 2020 13:03:54 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BB1CF206D7 for ; Tue, 5 May 2020 13:03:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bTDH14Zc" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BB1CF206D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:44912 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxEn-0000yO-OS for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:03:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38694) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVx9t-0001Fl-Py for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:49 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:27735 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVx9s-0000m0-Ev for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683527; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=69sn+hnFqrL5p0Pz7Z1IzCwG6ftghBseFxblIzKJAQ8=; b=bTDH14ZcNCFhhEwPSEl7T3qDSEFCV20i7yYzMSdU/IDt/ivFj1aNDYJJuKDWYhM6ui27LF 2BfhgQHV/rbSu3gCHN2xYWz59hQU+/zjd3uZ8byPlQ5b4mcNB1MPe08w+ZY5c0EFZne0GS pXcf8QyQOFKEKOLdHsTuY59UGHGWq+s= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-312-Pt6-P0E6MCuV1Q80vgyQVg-1; Tue, 05 May 2020 08:58:45 -0400 X-MC-Unique: Pt6-P0E6MCuV1Q80vgyQVg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9F378A0BF6; Tue, 5 May 2020 12:58:43 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4162E1053B00; Tue, 5 May 2020 12:58:43 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 07/24] iotests: drop pre-Python 3.4 compatibility code Date: Tue, 5 May 2020 14:58:09 +0200 Message-Id: <20200505125826.1001451-8-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=mreitz@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 03:48:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow We no longer need to accommodate <3.4, drop this code. (The lines were > 79 chars and it stood out.) Signed-off-by: John Snow Reviewed-by: Max Reitz Message-Id: <20200331000014.11581-8-jsnow@redhat.com> Reviewed-by: Kevin Wolf Signed-off-by: Max Reitz --- tests/qemu-iotests/iotests.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index dae124872e..374a8f6077 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -360,12 +360,9 @@ def log(msg, filters=(), indent=None): for flt in filters: msg = flt(msg) if isinstance(msg, (dict, list)): - # Python < 3.4 needs to know not to add whitespace when pretty-printing: - separators = (', ', ': ') if indent is None else (',', ': ') # Don't sort if it's already sorted do_sort = not isinstance(msg, OrderedDict) - print(json.dumps(msg, sort_keys=do_sort, - indent=indent, separators=separators)) + print(json.dumps(msg, sort_keys=do_sort, indent=indent)) else: print(msg) From patchwork Tue May 5 12:58:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529017 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3B183912 for ; Tue, 5 May 2020 13:08:19 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0F6C22073B for ; Tue, 5 May 2020 13:08:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Sqw5BT5O" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0F6C22073B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:35574 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxJ4-0000ST-6l for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:08:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38706) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVx9w-0001ND-C8 for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:52 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:24655 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVx9v-0000mu-Kk for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683530; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cQR0M9fWCh4ng08cJMMLE0yhSJzmSpgm/cT233yhunA=; b=Sqw5BT5OPuO5oVsGefNBKN7Fctc8+MoLOXmp3wHPnZkVvVlWPx/lzqgVk4IiGLVB7xbm1X cOzA2rbmm95mj9XkAL0SbQxOnO6+nU+lb2AZri+kvdYTcobJXSJgjJqcpde+JU7yMtY/EY 2V5XWNZIn26UD93h42464obUbeiGzDM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-374-voh5NsUNPQeQMHMLUaInxg-1; Tue, 05 May 2020 08:58:46 -0400 X-MC-Unique: voh5NsUNPQeQMHMLUaInxg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E11971005510; Tue, 5 May 2020 12:58:45 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 81A375C1D3; Tue, 5 May 2020 12:58:45 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 08/24] iotests: touch up log function signature Date: Tue, 5 May 2020 14:58:10 +0200 Message-Id: <20200505125826.1001451-9-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=mreitz@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 03:48:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow Representing nested, recursive data structures in mypy is notoriously difficult; the best we can reliably do right now is denote the leaf types as "Any" while describing the general shape of the data. Regardless, this fully annotates the log() function. Typing notes: TypeVar is a Type variable that can optionally be constrained by a sequence of possible types. This variable is bound to a specific type per-invocation, like a Generic. log() behaves as log() now, where the incoming type informs the signature it expects for any filter arguments passed in. If Msg is a str, then filter should take and return a str. Signed-off-by: John Snow Reviewed-by: Max Reitz Message-Id: <20200331000014.11581-9-jsnow@redhat.com> Reviewed-by: Kevin Wolf Signed-off-by: Max Reitz --- tests/qemu-iotests/iotests.py | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 374a8f6077..69f24223d2 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -28,6 +28,7 @@ import signal import struct import subprocess import sys +from typing import (Any, Callable, Dict, Iterable, List, Optional, TypeVar) import unittest # pylint: disable=import-error, wrong-import-position @@ -354,9 +355,16 @@ def filter_qmp_imgfmt(qmsg): return value return filter_qmp(qmsg, _filter) -def log(msg, filters=(), indent=None): - '''Logs either a string message or a JSON serializable message (like QMP). - If indent is provided, JSON serializable messages are pretty-printed.''' + +Msg = TypeVar('Msg', Dict[str, Any], List[Any], str) + +def log(msg: Msg, + filters: Iterable[Callable[[Msg], Msg]] = (), + indent: Optional[int] = None) -> None: + """ + Logs either a string message or a JSON serializable message (like QMP). + If indent is provided, JSON serializable messages are pretty-printed. + """ for flt in filters: msg = flt(msg) if isinstance(msg, (dict, list)): From patchwork Tue May 5 12:58:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529027 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 68406912 for ; Tue, 5 May 2020 13:10:09 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3CEF02078C for ; Tue, 5 May 2020 13:10:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="c/H1WK+x" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3CEF02078C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:45410 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxKq-0004sE-Du for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:10:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38770) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxA6-0001oN-UB for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:02 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:48896 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxA5-000171-2x for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683540; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0gr15OrJS9tsXYgJJWW/K6LyEGhoZx2WchC4Ory6oOA=; b=c/H1WK+xdeWIWh1RnsDS/oCW6mBUSzpcjpnP5KoZ9LwVnbgvYhyESedcnsyq+lcVRj+z13 6etM0JaJ2w1leFq6wb9lcbn+hKfODbxZBPpDfbNFggnPzZFJKXQ69sQBi5fjA1uqQoWb9v zcTr2kwoiittW1w3HjCIb8IoXhO38Y8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-441-zab3n-jLN0KNVViENfGLkA-1; Tue, 05 May 2020 08:58:49 -0400 X-MC-Unique: zab3n-jLN0KNVViENfGLkA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 31D2A1899526; Tue, 5 May 2020 12:58:48 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A09305D9DA; Tue, 5 May 2020 12:58:47 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 09/24] iotests: limit line length to 79 chars Date: Tue, 5 May 2020 14:58:11 +0200 Message-Id: <20200505125826.1001451-10-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.81; envelope-from=mreitz@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:19 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow 79 is the PEP8 recommendation. This recommendation works well for reading patch diffs in TUI email clients. Signed-off-by: John Snow Reviewed-by: Max Reitz Message-Id: <20200331000014.11581-10-jsnow@redhat.com> Reviewed-by: Kevin Wolf Signed-off-by: Max Reitz --- tests/qemu-iotests/iotests.py | 64 +++++++++++++++++++++++------------ tests/qemu-iotests/pylintrc | 6 +++- 2 files changed, 47 insertions(+), 23 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 69f24223d2..9f5da32dae 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -80,9 +80,11 @@ luks_default_key_secret_opt = 'key-secret=keysec0' def qemu_img(*args): '''Run qemu-img and return the exit code''' devnull = open('/dev/null', 'r+') - exitcode = subprocess.call(qemu_img_args + list(args), stdin=devnull, stdout=devnull) + exitcode = subprocess.call(qemu_img_args + list(args), + stdin=devnull, stdout=devnull) if exitcode < 0: - sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args)))) + sys.stderr.write('qemu-img received signal %i: %s\n' + % (-exitcode, ' '.join(qemu_img_args + list(args)))) return exitcode def ordered_qmp(qmsg, conv_keys=True): @@ -121,7 +123,8 @@ def qemu_img_verbose(*args): '''Run qemu-img without suppressing its output and return the exit code''' exitcode = subprocess.call(qemu_img_args + list(args)) if exitcode < 0: - sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args)))) + sys.stderr.write('qemu-img received signal %i: %s\n' + % (-exitcode, ' '.join(qemu_img_args + list(args)))) return exitcode def qemu_img_pipe(*args): @@ -132,7 +135,8 @@ def qemu_img_pipe(*args): universal_newlines=True) exitcode = subp.wait() if exitcode < 0: - sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args)))) + sys.stderr.write('qemu-img received signal %i: %s\n' + % (-exitcode, ' '.join(qemu_img_args + list(args)))) return subp.communicate()[0] def qemu_img_log(*args): @@ -162,7 +166,8 @@ def qemu_io(*args): universal_newlines=True) exitcode = subp.wait() if exitcode < 0: - sys.stderr.write('qemu-io received signal %i: %s\n' % (-exitcode, ' '.join(args))) + sys.stderr.write('qemu-io received signal %i: %s\n' + % (-exitcode, ' '.join(args))) return subp.communicate()[0] def qemu_io_log(*args): @@ -284,10 +289,13 @@ win32_re = re.compile(r"\r") def filter_win32(msg): return win32_re.sub("", msg) -qemu_io_re = re.compile(r"[0-9]* ops; [0-9\/:. sec]* \([0-9\/.inf]* [EPTGMKiBbytes]*\/sec and [0-9\/.inf]* ops\/sec\)") +qemu_io_re = re.compile(r"[0-9]* ops; [0-9\/:. sec]* " + r"\([0-9\/.inf]* [EPTGMKiBbytes]*\/sec " + r"and [0-9\/.inf]* ops\/sec\)") def filter_qemu_io(msg): msg = filter_win32(msg) - return qemu_io_re.sub("X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)", msg) + return qemu_io_re.sub("X ops; XX:XX:XX.X " + "(XXX YYY/sec and XXX ops/sec)", msg) chown_re = re.compile(r"chown [0-9]+:[0-9]+") def filter_chown(msg): @@ -340,7 +348,9 @@ def filter_img_info(output, filename): line = filter_testfiles(line) line = line.replace(imgfmt, 'IMGFMT') line = re.sub('iters: [0-9]+', 'iters: XXX', line) - line = re.sub('uuid: [-a-f0-9]+', 'uuid: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX', line) + line = re.sub('uuid: [-a-f0-9]+', + 'uuid: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX', + line) line = re.sub('cid: [0-9]+', 'cid: XXXXXXXXXX', line) lines.append(line) return '\n'.join(lines) @@ -538,11 +548,13 @@ class VM(qtest.QEMUQtestMachine): self.pause_drive(drive, "write_aio") return self.qmp('human-monitor-command', - command_line='qemu-io %s "break %s bp_%s"' % (drive, event, drive)) + command_line='qemu-io %s "break %s bp_%s"' + % (drive, event, drive)) def resume_drive(self, drive): self.qmp('human-monitor-command', - command_line='qemu-io %s "remove_break bp_%s"' % (drive, drive)) + command_line='qemu-io %s "remove_break bp_%s"' + % (drive, drive)) def hmp_qemu_io(self, drive, cmd): '''Write to a given drive using an HMP command''' @@ -802,16 +814,18 @@ class QMPTestCase(unittest.TestCase): idx = int(idx) if not isinstance(d, dict) or component not in d: - self.fail('failed path traversal for "%s" in "%s"' % (path, str(d))) + self.fail(f'failed path traversal for "{path}" in "{d}"') d = d[component] if m: if not isinstance(d, list): - self.fail('path component "%s" in "%s" is not a list in "%s"' % (component, path, str(d))) + self.fail(f'path component "{component}" in "{path}" ' + f'is not a list in "{d}"') try: d = d[idx] except IndexError: - self.fail('invalid index "%s" in path "%s" in "%s"' % (idx, path, str(d))) + self.fail(f'invalid index "{idx}" in path "{path}" ' + f'in "{d}"') return d def assert_qmp_absent(self, d, path): @@ -862,10 +876,13 @@ class QMPTestCase(unittest.TestCase): '''Asserts that the given filename is a json: filename and that its content is equal to the given reference object''' self.assertEqual(json_filename[:5], 'json:') - self.assertEqual(self.vm.flatten_qmp_object(json.loads(json_filename[5:])), - self.vm.flatten_qmp_object(reference)) + self.assertEqual( + self.vm.flatten_qmp_object(json.loads(json_filename[5:])), + self.vm.flatten_qmp_object(reference) + ) - def cancel_and_wait(self, drive='drive0', force=False, resume=False, wait=60.0): + def cancel_and_wait(self, drive='drive0', force=False, + resume=False, wait=60.0): '''Cancel a block job and wait for it to finish, returning the event''' result = self.vm.qmp('block-job-cancel', device=drive, force=force) self.assert_qmp(result, 'return', {}) @@ -889,8 +906,8 @@ class QMPTestCase(unittest.TestCase): self.assert_no_active_block_jobs() return result - def wait_until_completed(self, drive='drive0', check_offset=True, wait=60.0, - error=None): + def wait_until_completed(self, drive='drive0', check_offset=True, + wait=60.0, error=None): '''Wait for a block job to finish, returning the event''' while True: for event in self.vm.get_qmp_events(wait=wait): @@ -1029,8 +1046,11 @@ def verify_quorum(): notrun('quorum support missing') def qemu_pipe(*args): - '''Run qemu with an option to print something and exit (e.g. a help option), - and return its output''' + """ + Run qemu with an option to print something and exit (e.g. a help option). + + :return: QEMU's stdout output. + """ args = [qemu_prog] + qemu_opts + list(args) subp = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, @@ -1068,8 +1088,8 @@ def skip_if_unsupported(required_formats=(), read_only=False): usf_list = list(set(fmts) - set(supported_formats(read_only))) if usf_list: - test_case.case_skip('{}: formats {} are not whitelisted'.format( - test_case, usf_list)) + msg = f'{test_case}: formats {usf_list} are not whitelisted' + test_case.case_skip(msg) return None else: return func(test_case, *args, **kwargs) diff --git a/tests/qemu-iotests/pylintrc b/tests/qemu-iotests/pylintrc index daec2c4942..5481afe528 100644 --- a/tests/qemu-iotests/pylintrc +++ b/tests/qemu-iotests/pylintrc @@ -18,5 +18,9 @@ disable=invalid-name, too-many-locals, too-many-public-methods, # These are temporary, and should be removed: - line-too-long, missing-docstring, + +[FORMAT] + +# Maximum number of characters on a single line. +max-line-length=79 From patchwork Tue May 5 12:58:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529015 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1F3A41392 for ; Tue, 5 May 2020 13:07:54 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E96D92073B for ; Tue, 5 May 2020 13:07:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="iOZejpNz" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E96D92073B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:34592 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxIf-0008Uk-1Z for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:07:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38716) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxA0-0001Ww-3S for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:56 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:59404 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVx9z-0000vH-7b for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683534; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mb9m9GfCxKbgsPWdefIelFbw1RfASfl7wVa2Lb2r6UI=; b=iOZejpNzx9bJYW20f4EQ5pufQjGK8sN+UuQZrpWue8A0jj0Wpr9JuUVICOmqFBOwUhCK1H VN01mI1gKy8f8h8iKHXCMVFQEfi/0ycziTlei2oxbgOb7c6rdXktXqbjCPK66+kZzXkM6H eE2qoRhtbKrVQHbU8umrNxmXn9q7QgA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-398-7D9XcggRMBaS-JaoKdoVgA-1; Tue, 05 May 2020 08:58:51 -0400 X-MC-Unique: 7D9XcggRMBaS-JaoKdoVgA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4F2071899521; Tue, 5 May 2020 12:58:50 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E50345C1D3; Tue, 5 May 2020 12:58:49 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 10/24] iotests: add hmp helper with logging Date: Tue, 5 May 2020 14:58:12 +0200 Message-Id: <20200505125826.1001451-11-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.61; envelope-from=mreitz@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:40 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow Minor cleanup for HMP functions; helps with line length and consolidates HMP helpers through one implementation function. Although we are adding a universal toggle to turn QMP logging on or off, many existing callers to hmp functions don't expect that output to be logged, which causes quite a few changes in the test output. For now, offer a use_log parameter. Typing notes: QMPResponse is just an alias for Dict[str, Any]. It holds no special meanings and it is not a formal subtype of Dict[str, Any]. It is best thought of as a lexical synonym. We may well wish to add stricter subtypes in the future for certain shapes of data that are not formalized as Python objects, at which point we can simply retire the alias and allow mypy to more strictly check usages of the name. Signed-off-by: John Snow Message-Id: <20200331000014.11581-11-jsnow@redhat.com> Reviewed-by: Max Reitz Reviewed-by: Kevin Wolf Signed-off-by: Max Reitz --- tests/qemu-iotests/iotests.py | 39 +++++++++++++++++++++-------------- 1 file changed, 24 insertions(+), 15 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 9f5da32dae..cf10c428b5 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -37,6 +37,10 @@ from qemu import qtest assert sys.version_info >= (3, 6) +# Type Aliases +QMPResponse = Dict[str, Any] + + faulthandler.enable() # This will not work if arguments contain spaces but is necessary if we @@ -541,25 +545,30 @@ class VM(qtest.QEMUQtestMachine): self._args.append(addr) return self - def pause_drive(self, drive, event=None): - '''Pause drive r/w operations''' + def hmp(self, command_line: str, use_log: bool = False) -> QMPResponse: + cmd = 'human-monitor-command' + kwargs = {'command-line': command_line} + if use_log: + return self.qmp_log(cmd, **kwargs) + else: + return self.qmp(cmd, **kwargs) + + def pause_drive(self, drive: str, event: Optional[str] = None) -> None: + """Pause drive r/w operations""" if not event: self.pause_drive(drive, "read_aio") self.pause_drive(drive, "write_aio") return - self.qmp('human-monitor-command', - command_line='qemu-io %s "break %s bp_%s"' - % (drive, event, drive)) - - def resume_drive(self, drive): - self.qmp('human-monitor-command', - command_line='qemu-io %s "remove_break bp_%s"' - % (drive, drive)) - - def hmp_qemu_io(self, drive, cmd): - '''Write to a given drive using an HMP command''' - return self.qmp('human-monitor-command', - command_line='qemu-io %s "%s"' % (drive, cmd)) + self.hmp(f'qemu-io {drive} "break {event} bp_{drive}"') + + def resume_drive(self, drive: str) -> None: + """Resume drive r/w operations""" + self.hmp(f'qemu-io {drive} "remove_break bp_{drive}"') + + def hmp_qemu_io(self, drive: str, cmd: str, + use_log: bool = False) -> QMPResponse: + """Write to a given drive using an HMP command""" + return self.hmp(f'qemu-io {drive} "{cmd}"', use_log=use_log) def flatten_qmp_object(self, obj, output=None, basestr=''): if output is None: From patchwork Tue May 5 12:58:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529025 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 66F371392 for ; Tue, 5 May 2020 13:10:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2BD8A2078C for ; Tue, 5 May 2020 13:10:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="IoaIUpun" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2BD8A2078C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:44816 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxKk-0004d6-A7 for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:10:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38730) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxA2-0001dr-PB for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:58 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:28392 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxA0-00010Y-P0 for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683536; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I8edth6vbBVd4BR6OpM7IDKLx3IWwjfZCr6aJr/dPlQ=; b=IoaIUpunRybnC5787etKnH2uUcMzw85AJd5khcaFjIuV6klQmLOO8TfHhC6bvLtEEVzzw3 zgx0yPob7tOcNHc2DRTu+4Lj4/lJl/LSrAaGIAh3fLQW2lqZ4GBBeWr1sCLldzMb89j7QO ktFe67xprJqbSynzCEISB7E643M1L44= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-209-7mAwccpHPy-yZ_tvxBmA4A-1; Tue, 05 May 2020 08:58:53 -0400 X-MC-Unique: 7mAwccpHPy-yZ_tvxBmA4A-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D262F1899521; Tue, 5 May 2020 12:58:52 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1E39B60BEC; Tue, 5 May 2020 12:58:51 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 11/24] iotests: add script_initialize Date: Tue, 5 May 2020 14:58:13 +0200 Message-Id: <20200505125826.1001451-12-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.81; envelope-from=mreitz@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:19 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow Like script_main, but doesn't require a single point of entry. Replace all existing initialization sections with this drop-in replacement. This brings debug support to all existing script-style iotests. Signed-off-by: John Snow Reviewed-by: Max Reitz Message-Id: <20200331000014.11581-12-jsnow@redhat.com> Reviewed-by: Kevin Wolf [mreitz: Give 274 the same treatment] Signed-off-by: Max Reitz --- tests/qemu-iotests/149 | 3 +- tests/qemu-iotests/194 | 4 +- tests/qemu-iotests/202 | 4 +- tests/qemu-iotests/203 | 4 +- tests/qemu-iotests/206 | 2 +- tests/qemu-iotests/207 | 6 ++- tests/qemu-iotests/208 | 2 +- tests/qemu-iotests/209 | 2 +- tests/qemu-iotests/210 | 6 ++- tests/qemu-iotests/211 | 6 ++- tests/qemu-iotests/212 | 6 ++- tests/qemu-iotests/213 | 6 ++- tests/qemu-iotests/216 | 4 +- tests/qemu-iotests/218 | 2 +- tests/qemu-iotests/219 | 2 +- tests/qemu-iotests/222 | 7 ++-- tests/qemu-iotests/224 | 4 +- tests/qemu-iotests/228 | 6 ++- tests/qemu-iotests/234 | 4 +- tests/qemu-iotests/235 | 4 +- tests/qemu-iotests/236 | 2 +- tests/qemu-iotests/237 | 2 +- tests/qemu-iotests/238 | 2 + tests/qemu-iotests/242 | 2 +- tests/qemu-iotests/246 | 2 +- tests/qemu-iotests/248 | 2 +- tests/qemu-iotests/254 | 2 +- tests/qemu-iotests/255 | 2 +- tests/qemu-iotests/256 | 2 +- tests/qemu-iotests/258 | 7 ++-- tests/qemu-iotests/260 | 4 +- tests/qemu-iotests/262 | 4 +- tests/qemu-iotests/264 | 4 +- tests/qemu-iotests/274 | 4 +- tests/qemu-iotests/277 | 2 + tests/qemu-iotests/280 | 8 ++-- tests/qemu-iotests/283 | 4 +- tests/qemu-iotests/iotests.py | 76 +++++++++++++++++++++++------------ 38 files changed, 132 insertions(+), 83 deletions(-) diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149 index b4a21bf7b7..852768f80a 100755 --- a/tests/qemu-iotests/149 +++ b/tests/qemu-iotests/149 @@ -382,8 +382,7 @@ def test_once(config, qemu_img=False): # Obviously we only work with the luks image format -iotests.verify_image_format(supported_fmts=['luks']) -iotests.verify_platform() +iotests.script_initialize(supported_fmts=['luks']) # We need sudo in order to run cryptsetup to create # dm-crypt devices. This is safe to use on any diff --git a/tests/qemu-iotests/194 b/tests/qemu-iotests/194 index 9dc1bd3510..8b1f720af4 100755 --- a/tests/qemu-iotests/194 +++ b/tests/qemu-iotests/194 @@ -21,8 +21,8 @@ import iotests -iotests.verify_image_format(supported_fmts=['qcow2', 'qed', 'raw']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2', 'qed', 'raw'], + supported_platforms=['linux']) with iotests.FilePath('source.img') as source_img_path, \ iotests.FilePath('dest.img') as dest_img_path, \ diff --git a/tests/qemu-iotests/202 b/tests/qemu-iotests/202 index 920a8683ef..e3900a44d1 100755 --- a/tests/qemu-iotests/202 +++ b/tests/qemu-iotests/202 @@ -24,8 +24,8 @@ import iotests -iotests.verify_image_format(supported_fmts=['qcow2']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2'], + supported_platforms=['linux']) with iotests.FilePath('disk0.img') as disk0_img_path, \ iotests.FilePath('disk1.img') as disk1_img_path, \ diff --git a/tests/qemu-iotests/203 b/tests/qemu-iotests/203 index 49eff5d405..4b4bd3307d 100755 --- a/tests/qemu-iotests/203 +++ b/tests/qemu-iotests/203 @@ -24,8 +24,8 @@ import iotests -iotests.verify_image_format(supported_fmts=['qcow2']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2'], + supported_platforms=['linux']) with iotests.FilePath('disk0.img') as disk0_img_path, \ iotests.FilePath('disk1.img') as disk1_img_path, \ diff --git a/tests/qemu-iotests/206 b/tests/qemu-iotests/206 index e2b50ae24d..f42432a838 100755 --- a/tests/qemu-iotests/206 +++ b/tests/qemu-iotests/206 @@ -23,7 +23,7 @@ import iotests from iotests import imgfmt -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) with iotests.FilePath('t.qcow2') as disk_path, \ iotests.FilePath('t.qcow2.base') as backing_path, \ diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207 index 3d9c1208ca..a6621410da 100755 --- a/tests/qemu-iotests/207 +++ b/tests/qemu-iotests/207 @@ -24,8 +24,10 @@ import iotests import subprocess import re -iotests.verify_image_format(supported_fmts=['raw']) -iotests.verify_protocol(supported=['ssh']) +iotests.script_initialize( + supported_fmts=['raw'], + supported_protocols=['ssh'], +) def filter_hash(qmsg): def _filter(key, value): diff --git a/tests/qemu-iotests/208 b/tests/qemu-iotests/208 index 1c3fc8c7fd..6cb642f821 100755 --- a/tests/qemu-iotests/208 +++ b/tests/qemu-iotests/208 @@ -22,7 +22,7 @@ import iotests -iotests.verify_image_format(supported_fmts=['generic']) +iotests.script_initialize(supported_fmts=['generic']) with iotests.FilePath('disk.img') as disk_img_path, \ iotests.FilePath('disk-snapshot.img') as disk_snapshot_img_path, \ diff --git a/tests/qemu-iotests/209 b/tests/qemu-iotests/209 index 65c1a1e70a..8c804f4a30 100755 --- a/tests/qemu-iotests/209 +++ b/tests/qemu-iotests/209 @@ -22,7 +22,7 @@ import iotests from iotests import qemu_img_create, qemu_io, qemu_img_verbose, qemu_nbd, \ file_path -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) disk = file_path('disk') nbd_sock = file_path('nbd-sock', base_dir=iotests.sock_dir) diff --git a/tests/qemu-iotests/210 b/tests/qemu-iotests/210 index e49896e23d..7bf591f313 100755 --- a/tests/qemu-iotests/210 +++ b/tests/qemu-iotests/210 @@ -23,8 +23,10 @@ import iotests from iotests import imgfmt -iotests.verify_image_format(supported_fmts=['luks']) -iotests.verify_protocol(supported=['file']) +iotests.script_initialize( + supported_fmts=['luks'], + supported_protocols=['file'], +) with iotests.FilePath('t.luks') as disk_path, \ iotests.VM() as vm: diff --git a/tests/qemu-iotests/211 b/tests/qemu-iotests/211 index 163994d559..4969edb00c 100755 --- a/tests/qemu-iotests/211 +++ b/tests/qemu-iotests/211 @@ -23,8 +23,10 @@ import iotests from iotests import imgfmt -iotests.verify_image_format(supported_fmts=['vdi']) -iotests.verify_protocol(supported=['file']) +iotests.script_initialize( + supported_fmts=['vdi'], + supported_protocols=['file'], +) def blockdev_create(vm, options): error = vm.blockdev_create(options) diff --git a/tests/qemu-iotests/212 b/tests/qemu-iotests/212 index 800f92dd84..45d08842bb 100755 --- a/tests/qemu-iotests/212 +++ b/tests/qemu-iotests/212 @@ -23,8 +23,10 @@ import iotests from iotests import imgfmt -iotests.verify_image_format(supported_fmts=['parallels']) -iotests.verify_protocol(supported=['file']) +iotests.script_initialize( + supported_fmts=['parallels'], + supported_protocols=['file'], +) with iotests.FilePath('t.parallels') as disk_path, \ iotests.VM() as vm: diff --git a/tests/qemu-iotests/213 b/tests/qemu-iotests/213 index 1eee45276a..cf638eb927 100755 --- a/tests/qemu-iotests/213 +++ b/tests/qemu-iotests/213 @@ -23,8 +23,10 @@ import iotests from iotests import imgfmt -iotests.verify_image_format(supported_fmts=['vhdx']) -iotests.verify_protocol(supported=['file']) +iotests.script_initialize( + supported_fmts=['vhdx'], + supported_protocols=['file'], +) with iotests.FilePath('t.vhdx') as disk_path, \ iotests.VM() as vm: diff --git a/tests/qemu-iotests/216 b/tests/qemu-iotests/216 index 372f042d3e..de11d85b5d 100755 --- a/tests/qemu-iotests/216 +++ b/tests/qemu-iotests/216 @@ -23,8 +23,8 @@ import iotests from iotests import log, qemu_img, qemu_io_silent # Need backing file support -iotests.verify_image_format(supported_fmts=['qcow2', 'qcow', 'qed', 'vmdk']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2', 'qcow', 'qed', 'vmdk'], + supported_platforms=['linux']) log('') log('=== Copy-on-read across nodes ===') diff --git a/tests/qemu-iotests/218 b/tests/qemu-iotests/218 index 1325ba9eaa..5586870456 100755 --- a/tests/qemu-iotests/218 +++ b/tests/qemu-iotests/218 @@ -29,7 +29,7 @@ import iotests from iotests import log, qemu_img, qemu_io_silent -iotests.verify_image_format(supported_fmts=['qcow2', 'raw']) +iotests.script_initialize(supported_fmts=['qcow2', 'raw']) # Launches the VM, adds two null-co nodes (source and target), and diff --git a/tests/qemu-iotests/219 b/tests/qemu-iotests/219 index b8774770c4..db272c5249 100755 --- a/tests/qemu-iotests/219 +++ b/tests/qemu-iotests/219 @@ -21,7 +21,7 @@ import iotests -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) img_size = 4 * 1024 * 1024 diff --git a/tests/qemu-iotests/222 b/tests/qemu-iotests/222 index bf1718e179..6602f6b4ba 100755 --- a/tests/qemu-iotests/222 +++ b/tests/qemu-iotests/222 @@ -24,9 +24,10 @@ import iotests from iotests import log, qemu_img, qemu_io, qemu_io_silent -iotests.verify_platform(['linux']) -iotests.verify_image_format(supported_fmts=['qcow2', 'qcow', 'qed', 'vmdk', - 'vhdx', 'raw']) +iotests.script_initialize( + supported_fmts=['qcow2', 'qcow', 'qed', 'vmdk', 'vhdx', 'raw'], + supported_platforms=['linux'], +) patterns = [("0x5d", "0", "64k"), ("0xd5", "1M", "64k"), diff --git a/tests/qemu-iotests/224 b/tests/qemu-iotests/224 index e91fb26fd8..81ca1e4898 100755 --- a/tests/qemu-iotests/224 +++ b/tests/qemu-iotests/224 @@ -26,8 +26,8 @@ from iotests import log, qemu_img, qemu_io_silent, filter_qmp_testfiles, \ import json # Need backing file support (for arbitrary backing formats) -iotests.verify_image_format(supported_fmts=['qcow2', 'qcow', 'qed']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2', 'qcow', 'qed'], + supported_platforms=['linux']) # There are two variations of this test: diff --git a/tests/qemu-iotests/228 b/tests/qemu-iotests/228 index 64bc82ee23..da0900fb82 100755 --- a/tests/qemu-iotests/228 +++ b/tests/qemu-iotests/228 @@ -25,8 +25,10 @@ from iotests import log, qemu_img, filter_testfiles, filter_imgfmt, \ filter_qmp_testfiles, filter_qmp_imgfmt # Need backing file and change-backing-file support -iotests.verify_image_format(supported_fmts=['qcow2', 'qed']) -iotests.verify_platform(['linux']) +iotests.script_initialize( + supported_fmts=['qcow2', 'qed'], + supported_platforms=['linux'], +) def log_node_info(node): diff --git a/tests/qemu-iotests/234 b/tests/qemu-iotests/234 index 324c1549fd..73c899ddae 100755 --- a/tests/qemu-iotests/234 +++ b/tests/qemu-iotests/234 @@ -23,8 +23,8 @@ import iotests import os -iotests.verify_image_format(supported_fmts=['qcow2']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2'], + supported_platforms=['linux']) with iotests.FilePath('img') as img_path, \ iotests.FilePath('backing') as backing_path, \ diff --git a/tests/qemu-iotests/235 b/tests/qemu-iotests/235 index 760826128e..d1b10ac36b 100755 --- a/tests/qemu-iotests/235 +++ b/tests/qemu-iotests/235 @@ -27,6 +27,8 @@ sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python')) from qemu.machine import QEMUMachine +iotests.script_initialize(supported_fmts=['qcow2']) + # Note: # This test was added to check that mirror dead-lock was fixed (see previous # commit before this test addition). @@ -40,8 +42,6 @@ from qemu.machine import QEMUMachine size = 1 * 1024 * 1024 * 1024 -iotests.verify_image_format(supported_fmts=['qcow2']) - disk = file_path('disk') # prepare source image diff --git a/tests/qemu-iotests/236 b/tests/qemu-iotests/236 index 8ce927a16c..6f5cee2444 100755 --- a/tests/qemu-iotests/236 +++ b/tests/qemu-iotests/236 @@ -22,7 +22,7 @@ import iotests from iotests import log -iotests.verify_image_format(supported_fmts=['generic']) +iotests.script_initialize(supported_fmts=['generic']) size = 64 * 1024 * 1024 granularity = 64 * 1024 diff --git a/tests/qemu-iotests/237 b/tests/qemu-iotests/237 index 50ba364a3e..5b21ad3509 100755 --- a/tests/qemu-iotests/237 +++ b/tests/qemu-iotests/237 @@ -24,7 +24,7 @@ import math import iotests from iotests import imgfmt -iotests.verify_image_format(supported_fmts=['vmdk']) +iotests.script_initialize(supported_fmts=['vmdk']) with iotests.FilePath('t.vmdk') as disk_path, \ iotests.FilePath('t.vmdk.1') as extent1_path, \ diff --git a/tests/qemu-iotests/238 b/tests/qemu-iotests/238 index d4e060228c..b8fcf15a1f 100755 --- a/tests/qemu-iotests/238 +++ b/tests/qemu-iotests/238 @@ -23,6 +23,8 @@ import os import iotests from iotests import log +iotests.script_initialize() + virtio_scsi_device = iotests.get_virtio_scsi_device() vm = iotests.VM() diff --git a/tests/qemu-iotests/242 b/tests/qemu-iotests/242 index 97617876bc..64f1bd95e4 100755 --- a/tests/qemu-iotests/242 +++ b/tests/qemu-iotests/242 @@ -24,7 +24,7 @@ import struct from iotests import qemu_img_create, qemu_io, qemu_img_pipe, \ file_path, img_info_log, log, filter_qemu_io -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) disk = file_path('disk') chunk = 256 * 1024 diff --git a/tests/qemu-iotests/246 b/tests/qemu-iotests/246 index 59a216a839..58a479cc1f 100755 --- a/tests/qemu-iotests/246 +++ b/tests/qemu-iotests/246 @@ -22,7 +22,7 @@ import iotests from iotests import log -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) size = 64 * 1024 * 1024 * 1024 gran_small = 32 * 1024 gran_large = 128 * 1024 diff --git a/tests/qemu-iotests/248 b/tests/qemu-iotests/248 index 68c374692e..18ba03467e 100755 --- a/tests/qemu-iotests/248 +++ b/tests/qemu-iotests/248 @@ -21,7 +21,7 @@ import iotests from iotests import qemu_img_create, qemu_io, file_path, filter_qmp_testfiles -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) source, target = file_path('source', 'target') size = 5 * 1024 * 1024 diff --git a/tests/qemu-iotests/254 b/tests/qemu-iotests/254 index ee66c986db..150e58be8e 100755 --- a/tests/qemu-iotests/254 +++ b/tests/qemu-iotests/254 @@ -21,7 +21,7 @@ import iotests from iotests import qemu_img_create, file_path, log -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) disk, top = file_path('disk', 'top') size = 1024 * 1024 diff --git a/tests/qemu-iotests/255 b/tests/qemu-iotests/255 index 4a4818bafb..8f08f741da 100755 --- a/tests/qemu-iotests/255 +++ b/tests/qemu-iotests/255 @@ -23,7 +23,7 @@ import iotests from iotests import imgfmt -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) iotests.log('Finishing a commit job with background reads') iotests.log('============================================') diff --git a/tests/qemu-iotests/256 b/tests/qemu-iotests/256 index e34074c83e..db8d6f31cf 100755 --- a/tests/qemu-iotests/256 +++ b/tests/qemu-iotests/256 @@ -23,7 +23,7 @@ import os import iotests from iotests import log -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) size = 64 * 1024 * 1024 with iotests.FilePath('img0') as img0_path, \ diff --git a/tests/qemu-iotests/258 b/tests/qemu-iotests/258 index 091755a45c..a65151dda6 100755 --- a/tests/qemu-iotests/258 +++ b/tests/qemu-iotests/258 @@ -24,9 +24,10 @@ from iotests import log, qemu_img, qemu_io_silent, \ filter_qmp_testfiles, filter_qmp_imgfmt # Need backing file and change-backing-file support -iotests.verify_image_format(supported_fmts=['qcow2', 'qed']) -iotests.verify_platform(['linux']) - +iotests.script_initialize( + supported_fmts=['qcow2', 'qed'], + supported_platforms=['linux'], +) # Returns a node for blockdev-add def node(node_name, path, backing=None, fmt=None, throttle=None): diff --git a/tests/qemu-iotests/260 b/tests/qemu-iotests/260 index 30c0de380d..804a7addb9 100755 --- a/tests/qemu-iotests/260 +++ b/tests/qemu-iotests/260 @@ -21,7 +21,9 @@ import iotests from iotests import qemu_img_create, file_path, log, filter_qmp_event -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize( + supported_fmts=['qcow2'] +) base, top = file_path('base', 'top') size = 64 * 1024 * 3 diff --git a/tests/qemu-iotests/262 b/tests/qemu-iotests/262 index 8835dce7be..03af061f94 100755 --- a/tests/qemu-iotests/262 +++ b/tests/qemu-iotests/262 @@ -23,8 +23,8 @@ import iotests import os -iotests.verify_image_format(supported_fmts=['qcow2']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2'], + supported_platforms=['linux']) with iotests.FilePath('img') as img_path, \ iotests.FilePath('mig_fifo') as fifo, \ diff --git a/tests/qemu-iotests/264 b/tests/qemu-iotests/264 index 879123a343..304a7443d7 100755 --- a/tests/qemu-iotests/264 +++ b/tests/qemu-iotests/264 @@ -24,7 +24,9 @@ import iotests from iotests import qemu_img_create, qemu_io_silent_check, file_path, \ qemu_nbd_popen, log -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize( + supported_fmts=['qcow2'], +) disk_a, disk_b, nbd_sock = file_path('disk_a', 'disk_b', 'nbd-sock') nbd_uri = 'nbd+unix:///?socket=' + nbd_sock diff --git a/tests/qemu-iotests/274 b/tests/qemu-iotests/274 index e951f723b8..5d1bf34dff 100755 --- a/tests/qemu-iotests/274 +++ b/tests/qemu-iotests/274 @@ -21,8 +21,8 @@ import iotests -iotests.verify_image_format(supported_fmts=['qcow2']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2'], + supported_platforms=['linux']) size_short = 1 * 1024 * 1024 size_long = 2 * 1024 * 1024 diff --git a/tests/qemu-iotests/277 b/tests/qemu-iotests/277 index 04aa15a3d5..d34f87021f 100755 --- a/tests/qemu-iotests/277 +++ b/tests/qemu-iotests/277 @@ -23,6 +23,8 @@ import subprocess import iotests from iotests import file_path, log +iotests.script_initialize() + nbd_sock, conf_file = file_path('nbd-sock', 'nbd-fault-injector.conf') diff --git a/tests/qemu-iotests/280 b/tests/qemu-iotests/280 index 69288fdd0e..f594bb9ed2 100755 --- a/tests/qemu-iotests/280 +++ b/tests/qemu-iotests/280 @@ -22,9 +22,11 @@ import iotests import os -iotests.verify_image_format(supported_fmts=['qcow2']) -iotests.verify_protocol(supported=['file']) -iotests.verify_platform(['linux']) +iotests.script_initialize( + supported_fmts=['qcow2'], + supported_protocols=['file'], + supported_platforms=['linux'], +) with iotests.FilePath('base') as base_path , \ iotests.FilePath('top') as top_path, \ diff --git a/tests/qemu-iotests/283 b/tests/qemu-iotests/283 index 55b7cff953..e17b953333 100644 --- a/tests/qemu-iotests/283 +++ b/tests/qemu-iotests/283 @@ -21,7 +21,9 @@ import iotests # The test is unrelated to formats, restrict it to qcow2 to avoid extra runs -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize( + supported_fmts=['qcow2'], +) size = 1024 * 1024 diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index cf10c428b5..9f85e1fba3 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -28,7 +28,8 @@ import signal import struct import subprocess import sys -from typing import (Any, Callable, Dict, Iterable, List, Optional, TypeVar) +from typing import (Any, Callable, Dict, Iterable, + List, Optional, Sequence, TypeVar) import unittest # pylint: disable=import-error, wrong-import-position @@ -1029,12 +1030,11 @@ def verify_protocol(supported=(), unsupported=()): if not_sup or (imgproto in unsupported): notrun('not suitable for this protocol: %s' % imgproto) -def verify_platform(supported=None, unsupported=None): - if unsupported is not None: - if any((sys.platform.startswith(x) for x in unsupported)): - notrun('not suitable for this OS: %s' % sys.platform) +def verify_platform(supported=(), unsupported=()): + if any((sys.platform.startswith(x) for x in unsupported)): + notrun('not suitable for this OS: %s' % sys.platform) - if supported is not None: + if supported: if not any((sys.platform.startswith(x) for x in supported)): notrun('not suitable for this OS: %s' % sys.platform) @@ -1116,7 +1116,18 @@ def skip_if_user_is_root(func): return func(*args, **kwargs) return func_wrapper -def execute_unittest(output, verbosity, debug): +def execute_unittest(debug=False): + """Executes unittests within the calling module.""" + + verbosity = 2 if debug else 1 + + if debug: + output = sys.stdout + else: + # We need to filter out the time taken from the output so that + # qemu-iotest can reliably diff the results against master output. + output = io.StringIO() + runner = unittest.TextTestRunner(stream=output, descriptions=True, verbosity=verbosity) try: @@ -1124,6 +1135,8 @@ def execute_unittest(output, verbosity, debug): # exception unittest.main(testRunner=runner) finally: + # We need to filter out the time taken from the output so that + # qemu-iotest can reliably diff the results against master output. if not debug: out = output.getvalue() out = re.sub(r'Ran (\d+) tests? in [\d.]+s', r'Ran \1 tests', out) @@ -1135,13 +1148,19 @@ def execute_unittest(output, verbosity, debug): sys.stderr.write(out) -def execute_test(test_function=None, - supported_fmts=(), - supported_platforms=None, - supported_cache_modes=(), supported_aio_modes=(), - unsupported_fmts=(), supported_protocols=(), - unsupported_protocols=()): - """Run either unittest or script-style tests.""" +def execute_setup_common(supported_fmts: Sequence[str] = (), + supported_platforms: Sequence[str] = (), + supported_cache_modes: Sequence[str] = (), + supported_aio_modes: Sequence[str] = (), + unsupported_fmts: Sequence[str] = (), + supported_protocols: Sequence[str] = (), + unsupported_protocols: Sequence[str] = ()) -> bool: + """ + Perform necessary setup for either script-style or unittest-style tests. + + :return: Bool; Whether or not debug mode has been requested via the CLI. + """ + # Note: Python 3.6 and pylint do not like 'Collection' so use 'Sequence'. # We are using TEST_DIR and QEMU_DEFAULT_MACHINE as proxies to # indicate that we're not being run via "check". There may be @@ -1151,34 +1170,39 @@ def execute_test(test_function=None, sys.stderr.write('Please run this test via the "check" script\n') sys.exit(os.EX_USAGE) - debug = '-d' in sys.argv - verbosity = 1 verify_image_format(supported_fmts, unsupported_fmts) verify_protocol(supported_protocols, unsupported_protocols) verify_platform(supported=supported_platforms) verify_cache_mode(supported_cache_modes) verify_aio_mode(supported_aio_modes) + debug = '-d' in sys.argv if debug: - output = sys.stdout - verbosity = 2 sys.argv.remove('-d') - else: - # We need to filter out the time taken from the output so that - # qemu-iotest can reliably diff the results against master output. - output = io.StringIO() - logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN)) + return debug + +def execute_test(*args, test_function=None, **kwargs): + """Run either unittest or script-style tests.""" + + debug = execute_setup_common(*args, **kwargs) if not test_function: - execute_unittest(output, verbosity, debug) + execute_unittest(debug) else: test_function() +# This is called from script-style iotests without a single point of entry +def script_initialize(*args, **kwargs): + """Initialize script-style tests without running any tests.""" + execute_setup_common(*args, **kwargs) + +# This is called from script-style iotests with a single point of entry def script_main(test_function, *args, **kwargs): """Run script-style tests outside of the unittest framework""" - execute_test(test_function, *args, **kwargs) + execute_test(*args, test_function=test_function, **kwargs) +# This is called from unittest style iotests def main(*args, **kwargs): """Run tests using the unittest framework""" - execute_test(None, *args, **kwargs) + execute_test(*args, **kwargs) From patchwork Tue May 5 12:58:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529033 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 22CF6912 for ; Tue, 5 May 2020 13:11:57 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id ED79E2078C for ; Tue, 5 May 2020 13:11:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Kduo2br5" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ED79E2078C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:53944 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxMa-0008WF-2m for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:11:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38748) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxA4-0001hR-9j for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:00 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:31290 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxA3-00014E-34 for qemu-devel@nongnu.org; Tue, 05 May 2020 08:58:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683538; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vQ7OgkzblVUfXZhWwzaNGGwiSkdBR0Ztb479HveURPo=; b=Kduo2br5zChW+OTA19AGSP94rL1OnmYStTywm5U9xxdJ8VLQPWgE/v4REgYv157R6552no +/UalRzcB7OC49eKpqYJ7+0iCx5blUi9d+oy8EAG3bz+tJ3Y+hWfY84uZdgtXitX264OlM Yzdd+nka59duiX1Mq/jxQbULU1nFBVc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-175-nDDJRC4UOcy_TjbSapWofQ-1; Tue, 05 May 2020 08:58:56 -0400 X-MC-Unique: nDDJRC4UOcy_TjbSapWofQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 35705800687; Tue, 5 May 2020 12:58:55 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C6E2560C18; Tue, 5 May 2020 12:58:54 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 12/24] iotest 258: use script_main Date: Tue, 5 May 2020 14:58:14 +0200 Message-Id: <20200505125826.1001451-13-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.81; envelope-from=mreitz@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:19 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow Since this one is nicely factored to use a single entry point, use script_main to run the tests. Signed-off-by: John Snow Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Max Reitz Message-Id: <20200331000014.11581-13-jsnow@redhat.com> Reviewed-by: Kevin Wolf Signed-off-by: Max Reitz --- tests/qemu-iotests/258 | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/tests/qemu-iotests/258 b/tests/qemu-iotests/258 index a65151dda6..e305a1502f 100755 --- a/tests/qemu-iotests/258 +++ b/tests/qemu-iotests/258 @@ -23,12 +23,6 @@ import iotests from iotests import log, qemu_img, qemu_io_silent, \ filter_qmp_testfiles, filter_qmp_imgfmt -# Need backing file and change-backing-file support -iotests.script_initialize( - supported_fmts=['qcow2', 'qed'], - supported_platforms=['linux'], -) - # Returns a node for blockdev-add def node(node_name, path, backing=None, fmt=None, throttle=None): if fmt is None: @@ -161,4 +155,7 @@ def main(): test_concurrent_finish(False) if __name__ == '__main__': - main() + # Need backing file and change-backing-file support + iotests.script_main(main, + supported_fmts=['qcow2', 'qed'], + supported_platforms=['linux']) From patchwork Tue May 5 12:58:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529019 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 69C451392 for ; Tue, 5 May 2020 13:08:31 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3FF9E2073B for ; Tue, 5 May 2020 13:08:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="XCJCdH6P" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3FF9E2073B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:36924 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxJG-00013X-Bl for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:08:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38768) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxA6-0001nF-Hc for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:02 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:45783 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxA4-00016x-UX for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683540; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eK05NuIwz2Yp0URQsMdJJWvulUGEqKMDdJ/JSnMJrSM=; b=XCJCdH6PfRZVtRqtBPM/9SldCHL3r1Bh9wmE0EjuP9Jf8ZULqdMgf0ZNr5C9aY7BUHnRvv FSZeA4oJwTbZXR2byJ6IaKevugEjOnupUiIKiLjR8up9Y6BlhyoI59X0xgF/axUlt8dT9S FRBcdWl/WUkhwTo8ZJjhjWXbRbRd0MA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-275-cp1snBgVMnWygZAZuZPi_A-1; Tue, 05 May 2020 08:58:58 -0400 X-MC-Unique: cp1snBgVMnWygZAZuZPi_A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3DC12A0C14; Tue, 5 May 2020 12:58:57 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D58095C1D4; Tue, 5 May 2020 12:58:56 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 13/24] iotests: Mark verify functions as private Date: Tue, 5 May 2020 14:58:15 +0200 Message-Id: <20200505125826.1001451-14-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.120; envelope-from=mreitz@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:38 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow Mark the verify functions as "private" with a leading underscore, to discourage their use. Update type signatures while we're here. (Also, make pending patches not yet using the new entry points fail in a very obvious way.) Signed-off-by: John Snow Message-Id: <20200331000014.11581-14-jsnow@redhat.com> Reviewed-by: Max Reitz Reviewed-by: Kevin Wolf Signed-off-by: Max Reitz --- tests/qemu-iotests/iotests.py | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 9f85e1fba3..35d8cae997 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -1006,7 +1006,8 @@ def case_notrun(reason): open('%s/%s.casenotrun' % (output_dir, seq), 'a').write( ' [case not run] ' + reason + '\n') -def verify_image_format(supported_fmts=(), unsupported_fmts=()): +def _verify_image_format(supported_fmts: Sequence[str] = (), + unsupported_fmts: Sequence[str] = ()) -> None: assert not (supported_fmts and unsupported_fmts) if 'generic' in supported_fmts and \ @@ -1020,7 +1021,8 @@ def verify_image_format(supported_fmts=(), unsupported_fmts=()): if not_sup or (imgfmt in unsupported_fmts): notrun('not suitable for this image format: %s' % imgfmt) -def verify_protocol(supported=(), unsupported=()): +def _verify_protocol(supported: Sequence[str] = (), + unsupported: Sequence[str] = ()) -> None: assert not (supported and unsupported) if 'generic' in supported: @@ -1030,7 +1032,8 @@ def verify_protocol(supported=(), unsupported=()): if not_sup or (imgproto in unsupported): notrun('not suitable for this protocol: %s' % imgproto) -def verify_platform(supported=(), unsupported=()): +def _verify_platform(supported: Sequence[str] = (), + unsupported: Sequence[str] = ()) -> None: if any((sys.platform.startswith(x) for x in unsupported)): notrun('not suitable for this OS: %s' % sys.platform) @@ -1038,11 +1041,11 @@ def verify_platform(supported=(), unsupported=()): if not any((sys.platform.startswith(x) for x in supported)): notrun('not suitable for this OS: %s' % sys.platform) -def verify_cache_mode(supported_cache_modes=()): +def _verify_cache_mode(supported_cache_modes: Sequence[str] = ()) -> None: if supported_cache_modes and (cachemode not in supported_cache_modes): notrun('not suitable for this cache mode: %s' % cachemode) -def verify_aio_mode(supported_aio_modes=()): +def _verify_aio_mode(supported_aio_modes: Sequence[str] = ()): if supported_aio_modes and (aiomode not in supported_aio_modes): notrun('not suitable for this aio mode: %s' % aiomode) @@ -1170,11 +1173,11 @@ def execute_setup_common(supported_fmts: Sequence[str] = (), sys.stderr.write('Please run this test via the "check" script\n') sys.exit(os.EX_USAGE) - verify_image_format(supported_fmts, unsupported_fmts) - verify_protocol(supported_protocols, unsupported_protocols) - verify_platform(supported=supported_platforms) - verify_cache_mode(supported_cache_modes) - verify_aio_mode(supported_aio_modes) + _verify_image_format(supported_fmts, unsupported_fmts) + _verify_protocol(supported_protocols, unsupported_protocols) + _verify_platform(supported=supported_platforms) + _verify_cache_mode(supported_cache_modes) + _verify_aio_mode(supported_aio_modes) debug = '-d' in sys.argv if debug: From patchwork Tue May 5 12:58:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11528987 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0F4B91392 for ; Tue, 5 May 2020 13:05:11 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C7E85206B9 for ; Tue, 5 May 2020 13:05:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Arxz+Y9S" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C7E85206B9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:49226 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxG1-0002pd-Pk for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:05:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38866) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxAJ-0001yz-Pk for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:15 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:28912 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxAG-0001G3-OO for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683552; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pbXi/6hdtFIwWPOcv1q8IAUrdGfW8a8cRmcfOXUeG0A=; b=Arxz+Y9SiSQkJuzPgZg90V0ZbhB9O2pTusXeDPP8ml9EzHt+o1NWr3fBdj1MHl4U1tewiL nIwYsJfV0xoMhAz0SXwnwH4USS5jcB+6qYKdfqrmtydVcWm96w8Lrx75QlUFUd+RMS1SEK Vnx5y4BLStao5NYVxlXaQwGEOq1y2l8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-280-WVjr-pGnNmGBczsYJ2pl9g-1; Tue, 05 May 2020 08:59:00 -0400 X-MC-Unique: WVjr-pGnNmGBczsYJ2pl9g-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A6B81A0C01; Tue, 5 May 2020 12:58:59 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0B55060E3E; Tue, 5 May 2020 12:58:58 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 14/24] iotests: use python logging for iotests.log() Date: Tue, 5 May 2020 14:58:16 +0200 Message-Id: <20200505125826.1001451-15-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.61; envelope-from=mreitz@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:40 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow We can turn logging on/off globally instead of per-function. Remove use_log from run_job, and use python logging to turn on diffable output when we run through a script entry point. iotest 245 changes output order due to buffering reasons. An extended note on python logging: A NullHandler is added to `qemu.iotests` to stop output from being generated if this code is used as a library without configuring logging. A NullHandler is only needed at the root, so a duplicate handler is not needed for `qemu.iotests.diff_io`. When logging is not configured, messages at the 'WARNING' levels or above are printed with default settings. The NullHandler stops this from occurring, which is considered good hygiene for code used as a library. See https://docs.python.org/3/howto/logging.html#library-config When logging is actually enabled (always at the behest of an explicit call by a client script), a root logger is implicitly created at the root, which allows messages to propagate upwards and be handled/emitted from the root logger with default settings. When we want iotest logging, we attach a handler to the qemu.iotests.diff_io logger and disable propagation to avoid possible double-printing. For more information on python logging infrastructure, I highly recommend downloading the pip package `logging_tree`, which provides convenient visualizations of the hierarchical logging configuration under different circumstances. See https://pypi.org/project/logging_tree/ for more information. Signed-off-by: John Snow Reviewed-by: Max Reitz Message-Id: <20200331000014.11581-15-jsnow@redhat.com> Reviewed-by: Kevin Wolf Signed-off-by: Max Reitz --- tests/qemu-iotests/030 | 4 +-- tests/qemu-iotests/155 | 2 +- tests/qemu-iotests/245 | 1 + tests/qemu-iotests/245.out | 10 +++---- tests/qemu-iotests/iotests.py | 53 ++++++++++++++++++++--------------- 5 files changed, 39 insertions(+), 31 deletions(-) diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030 index aa911d266a..104e3cee1b 100755 --- a/tests/qemu-iotests/030 +++ b/tests/qemu-iotests/030 @@ -411,8 +411,8 @@ class TestParallelOps(iotests.QMPTestCase): result = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) self.assert_qmp(result, 'return', {}) - self.vm.run_job(job='drive0', auto_dismiss=True, use_log=False) - self.vm.run_job(job='node4', auto_dismiss=True, use_log=False) + self.vm.run_job(job='drive0', auto_dismiss=True) + self.vm.run_job(job='node4', auto_dismiss=True) self.assert_no_active_block_jobs() # Test a block-stream and a block-commit job in parallel diff --git a/tests/qemu-iotests/155 b/tests/qemu-iotests/155 index 571bce9de4..cb371d4649 100755 --- a/tests/qemu-iotests/155 +++ b/tests/qemu-iotests/155 @@ -188,7 +188,7 @@ class MirrorBaseClass(BaseClass): self.assert_qmp(result, 'return', {}) - self.vm.run_job('mirror-job', use_log=False, auto_finalize=False, + self.vm.run_job('mirror-job', auto_finalize=False, pre_finalize=self.openBacking, auto_dismiss=True) def testFull(self): diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245 index 1001275a44..4f5f0bb901 100755 --- a/tests/qemu-iotests/245 +++ b/tests/qemu-iotests/245 @@ -1027,5 +1027,6 @@ class TestBlockdevReopen(iotests.QMPTestCase): self.run_test_iothreads(None, 'iothread0') if __name__ == '__main__': + iotests.activate_logging() iotests.main(supported_fmts=["qcow2"], supported_protocols=["file"]) diff --git a/tests/qemu-iotests/245.out b/tests/qemu-iotests/245.out index 682b93394d..4b33dcaf5c 100644 --- a/tests/qemu-iotests/245.out +++ b/tests/qemu-iotests/245.out @@ -1,8 +1,3 @@ -..................... ----------------------------------------------------------------------- -Ran 21 tests - -OK {"execute": "job-finalize", "arguments": {"id": "commit0"}} {"return": {}} {"data": {"id": "commit0", "type": "commit"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -15,3 +10,8 @@ OK {"return": {}} {"data": {"id": "stream0", "type": "stream"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} {"data": {"device": "stream0", "len": 3145728, "offset": 3145728, "speed": 0, "type": "stream"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} +..................... +---------------------------------------------------------------------- +Ran 21 tests + +OK diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 35d8cae997..6c0e781af7 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -42,6 +42,14 @@ assert sys.version_info >= (3, 6) QMPResponse = Dict[str, Any] +# Use this logger for logging messages directly from the iotests module +logger = logging.getLogger('qemu.iotests') +logger.addHandler(logging.NullHandler()) + +# Use this logger for messages that ought to be used for diff output. +test_logger = logging.getLogger('qemu.iotests.diff_io') + + faulthandler.enable() # This will not work if arguments contain spaces but is necessary if we @@ -385,9 +393,9 @@ def log(msg: Msg, if isinstance(msg, (dict, list)): # Don't sort if it's already sorted do_sort = not isinstance(msg, OrderedDict) - print(json.dumps(msg, sort_keys=do_sort, indent=indent)) + test_logger.info(json.dumps(msg, sort_keys=do_sort, indent=indent)) else: - print(msg) + test_logger.info(msg) class Timeout: def __init__(self, seconds, errmsg="Timeout"): @@ -609,7 +617,7 @@ class VM(qtest.QEMUQtestMachine): # Returns None on success, and an error string on failure def run_job(self, job, auto_finalize=True, auto_dismiss=False, - pre_finalize=None, cancel=False, use_log=True, wait=60.0): + pre_finalize=None, cancel=False, wait=60.0): """ run_job moves a job from creation through to dismissal. @@ -622,7 +630,6 @@ class VM(qtest.QEMUQtestMachine): invoked prior to issuing job-finalize, if any. :param cancel: Bool. When true, cancels the job after the pre_finalize callback. - :param use_log: Bool. When false, does not log QMP messages. :param wait: Float. Timeout value specifying how long to wait for any event, in seconds. Defaults to 60.0. """ @@ -640,8 +647,7 @@ class VM(qtest.QEMUQtestMachine): while True: ev = filter_qmp_event(self.events_wait(events, timeout=wait)) if ev['event'] != 'JOB_STATUS_CHANGE': - if use_log: - log(ev) + log(ev) continue status = ev['data']['status'] if status == 'aborting': @@ -649,29 +655,18 @@ class VM(qtest.QEMUQtestMachine): for j in result['return']: if j['id'] == job: error = j['error'] - if use_log: - log('Job failed: %s' % (j['error'])) + log('Job failed: %s' % (j['error'])) elif status == 'ready': - if use_log: - self.qmp_log('job-complete', id=job) - else: - self.qmp('job-complete', id=job) + self.qmp_log('job-complete', id=job) elif status == 'pending' and not auto_finalize: if pre_finalize: pre_finalize() - if cancel and use_log: + if cancel: self.qmp_log('job-cancel', id=job) - elif cancel: - self.qmp('job-cancel', id=job) - elif use_log: - self.qmp_log('job-finalize', id=job) else: - self.qmp('job-finalize', id=job) + self.qmp_log('job-finalize', id=job) elif status == 'concluded' and not auto_dismiss: - if use_log: - self.qmp_log('job-dismiss', id=job) - else: - self.qmp('job-dismiss', id=job) + self.qmp_log('job-dismiss', id=job) elif status == 'null': return error @@ -991,7 +986,7 @@ def notrun(reason): seq = os.path.basename(sys.argv[0]) open('%s/%s.notrun' % (output_dir, seq), 'w').write(reason + '\n') - print('%s not run: %s' % (seq, reason)) + logger.warning("%s not run: %s", seq, reason) sys.exit(0) def case_notrun(reason): @@ -1183,6 +1178,7 @@ def execute_setup_common(supported_fmts: Sequence[str] = (), if debug: sys.argv.remove('-d') logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN)) + logger.debug("iotests debugging messages active") return debug @@ -1195,14 +1191,25 @@ def execute_test(*args, test_function=None, **kwargs): else: test_function() +def activate_logging(): + """Activate iotests.log() output to stdout for script-style tests.""" + handler = logging.StreamHandler(stream=sys.stdout) + formatter = logging.Formatter('%(message)s') + handler.setFormatter(formatter) + test_logger.addHandler(handler) + test_logger.setLevel(logging.INFO) + test_logger.propagate = False + # This is called from script-style iotests without a single point of entry def script_initialize(*args, **kwargs): """Initialize script-style tests without running any tests.""" + activate_logging() execute_setup_common(*args, **kwargs) # This is called from script-style iotests with a single point of entry def script_main(test_function, *args, **kwargs): """Run script-style tests outside of the unittest framework""" + activate_logging() execute_test(*args, test_function=test_function, **kwargs) # This is called from unittest style iotests From patchwork Tue May 5 12:58:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529037 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 38E27912 for ; Tue, 5 May 2020 13:13:35 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F171C20658 for ; Tue, 5 May 2020 13:13:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="EWH1UU7Z" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F171C20658 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:60394 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxOA-0003Eh-2e for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:13:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38916) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxAO-00023Y-63 for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:20 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:39101 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxAL-0001Jw-Ki for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683556; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CKQs8FyU5NuN+EAs4LH/Hi2t6c1XAyYoEebmGWSoEKY=; b=EWH1UU7ZQnEJ1091dxx5EFS3MJVfsXDZpxjEgJBpvdsR0OyzSX42GUAJgHxmtD2Ybd5Ixg tVnX+LxiBCZyMbHMml9loATV23+bGotKkpCqCcx++Prj9m+LQSXZA4g92nPjpTsZt6HN+r CReYnLVNlo+Uk1pqC1ZqyffqfyBB4xk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-286-kGqTsKFsM9mY4YjEfWgCZg-1; Tue, 05 May 2020 08:59:03 -0400 X-MC-Unique: kGqTsKFsM9mY4YjEfWgCZg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 24172801504; Tue, 5 May 2020 12:59:02 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 92AFA63F95; Tue, 5 May 2020 12:59:01 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 15/24] block: Add blk_new_with_bs() helper Date: Tue, 5 May 2020 14:58:17 +0200 Message-Id: <20200505125826.1001451-16-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.61; envelope-from=mreitz@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:40 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: Eric Blake There are several callers that need to create a new block backend from an existing BDS; make the task slightly easier with a common helper routine. Suggested-by: Max Reitz Signed-off-by: Eric Blake Message-Id: <20200424190903.522087-2-eblake@redhat.com> [mreitz: Set @ret only in error paths, see https://lists.nongnu.org/archive/html/qemu-block/2020-04/msg01216.html] Signed-off-by: Max Reitz Message-Id: <20200428192648.749066-2-eblake@redhat.com> Signed-off-by: Max Reitz --- include/sysemu/block-backend.h | 2 ++ block/block-backend.c | 23 +++++++++++++++++++++++ block/crypto.c | 9 ++++----- block/parallels.c | 8 ++++---- block/qcow.c | 8 ++++---- block/qcow2.c | 18 ++++++++---------- block/qed.c | 8 ++++---- block/sheepdog.c | 10 +++++----- block/vdi.c | 8 ++++---- block/vhdx.c | 8 ++++---- block/vmdk.c | 9 ++++----- block/vpc.c | 8 ++++---- blockdev.c | 8 +++----- blockjob.c | 7 ++----- 14 files changed, 75 insertions(+), 59 deletions(-) diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h index 34de7faa81..0917663d89 100644 --- a/include/sysemu/block-backend.h +++ b/include/sysemu/block-backend.h @@ -77,6 +77,8 @@ typedef struct BlockBackendPublic { } BlockBackendPublic; BlockBackend *blk_new(AioContext *ctx, uint64_t perm, uint64_t shared_perm); +BlockBackend *blk_new_with_bs(BlockDriverState *bs, uint64_t perm, + uint64_t shared_perm, Error **errp); BlockBackend *blk_new_open(const char *filename, const char *reference, QDict *options, int flags, Error **errp); int blk_get_refcnt(BlockBackend *blk); diff --git a/block/block-backend.c b/block/block-backend.c index 17ed6d8c5b..f4944861fa 100644 --- a/block/block-backend.c +++ b/block/block-backend.c @@ -355,6 +355,29 @@ BlockBackend *blk_new(AioContext *ctx, uint64_t perm, uint64_t shared_perm) return blk; } +/* + * Create a new BlockBackend connected to an existing BlockDriverState. + * + * @perm is a bitmasks of BLK_PERM_* constants which describes the + * permissions to request for @bs that is attached to this + * BlockBackend. @shared_perm is a bitmask which describes which + * permissions may be granted to other users of the attached node. + * Both sets of permissions can be changed later using blk_set_perm(). + * + * Return the new BlockBackend on success, null on failure. + */ +BlockBackend *blk_new_with_bs(BlockDriverState *bs, uint64_t perm, + uint64_t shared_perm, Error **errp) +{ + BlockBackend *blk = blk_new(bdrv_get_aio_context(bs), perm, shared_perm); + + if (blk_insert_bs(blk, bs, errp) < 0) { + blk_unref(blk); + return NULL; + } + return blk; +} + /* * Creates a new BlockBackend, opens a new BlockDriverState, and connects both. * The new BlockBackend is in the main AioContext. diff --git a/block/crypto.c b/block/crypto.c index e02f343590..ca44dae4f7 100644 --- a/block/crypto.c +++ b/block/crypto.c @@ -261,11 +261,10 @@ static int block_crypto_co_create_generic(BlockDriverState *bs, QCryptoBlock *crypto = NULL; struct BlockCryptoCreateData data; - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs, BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL, + errp); + if (!blk) { + ret = -EPERM; goto cleanup; } diff --git a/block/parallels.c b/block/parallels.c index 2be92cf417..8db64a55e3 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -559,10 +559,10 @@ static int coroutine_fn parallels_co_create(BlockdevCreateOptions* opts, return -EIO; } - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs, BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL, + errp); + if (!blk) { + ret = -EPERM; goto out; } blk_set_allow_write_beyond_eof(blk, true); diff --git a/block/qcow.c b/block/qcow.c index 6b5f2269f0..b0475b73a5 100644 --- a/block/qcow.c +++ b/block/qcow.c @@ -849,10 +849,10 @@ static int coroutine_fn qcow_co_create(BlockdevCreateOptions *opts, return -EIO; } - qcow_blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(qcow_blk, bs, errp); - if (ret < 0) { + qcow_blk = blk_new_with_bs(bs, BLK_PERM_WRITE | BLK_PERM_RESIZE, + BLK_PERM_ALL, errp); + if (!qcow_blk) { + ret = -EPERM; goto exit; } blk_set_allow_write_beyond_eof(qcow_blk, true); diff --git a/block/qcow2.c b/block/qcow2.c index 2ba0b17c39..0edc7f4643 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -3405,10 +3405,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) } /* Create BlockBackend to write to the image */ - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs, BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL, + errp); + if (!blk) { + ret = -EPERM; goto out; } blk_set_allow_write_beyond_eof(blk, true); @@ -5412,12 +5412,10 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, } if (new_size) { - BlockBackend *blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { - blk_unref(blk); - return ret; + BlockBackend *blk = blk_new_with_bs(bs, BLK_PERM_RESIZE, BLK_PERM_ALL, + errp); + if (!blk) { + return -EPERM; } /* diff --git a/block/qed.c b/block/qed.c index b0fdb8f565..fb609cfba1 100644 --- a/block/qed.c +++ b/block/qed.c @@ -651,10 +651,10 @@ static int coroutine_fn bdrv_qed_co_create(BlockdevCreateOptions *opts, return -EIO; } - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs, BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL, + errp); + if (!blk) { + ret = -EPERM; goto out; } blk_set_allow_write_beyond_eof(blk, true); diff --git a/block/sheepdog.c b/block/sheepdog.c index 76729f40a4..2eb61938ff 100644 --- a/block/sheepdog.c +++ b/block/sheepdog.c @@ -1803,12 +1803,12 @@ static int sd_prealloc(BlockDriverState *bs, int64_t old_size, int64_t new_size, void *buf = NULL; int ret; - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE | BLK_PERM_RESIZE, - BLK_PERM_ALL); + blk = blk_new_with_bs(bs, + BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE | BLK_PERM_RESIZE, + BLK_PERM_ALL, errp); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + if (!blk) { + ret = -EPERM; goto out_with_err_set; } diff --git a/block/vdi.c b/block/vdi.c index 0c7835ae70..2d28046615 100644 --- a/block/vdi.c +++ b/block/vdi.c @@ -804,10 +804,10 @@ static int coroutine_fn vdi_co_do_create(BlockdevCreateOptions *create_options, goto exit; } - blk = blk_new(bdrv_get_aio_context(bs_file), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs_file, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs_file, BLK_PERM_WRITE | BLK_PERM_RESIZE, + BLK_PERM_ALL, errp); + if (!blk) { + ret = -EPERM; goto exit; } diff --git a/block/vhdx.c b/block/vhdx.c index 45be0a4321..9d3951f4ee 100644 --- a/block/vhdx.c +++ b/block/vhdx.c @@ -1984,10 +1984,10 @@ static int coroutine_fn vhdx_co_create(BlockdevCreateOptions *opts, return -EIO; } - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs, BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL, + errp); + if (!blk) { + ret = -EPERM; goto delete_and_exit; } blk_set_allow_write_beyond_eof(blk, true); diff --git a/block/vmdk.c b/block/vmdk.c index 8ec18f35a5..b02fdd14b2 100644 --- a/block/vmdk.c +++ b/block/vmdk.c @@ -2717,11 +2717,10 @@ static BlockBackend *vmdk_co_create_cb(int64_t size, int idx, if (!bs) { return NULL; } - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE | BLK_PERM_RESIZE, - BLK_PERM_ALL); - if (blk_insert_bs(blk, bs, errp)) { - bdrv_unref(bs); + blk = blk_new_with_bs(bs, + BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE | BLK_PERM_RESIZE, + BLK_PERM_ALL, errp); + if (!blk) { return NULL; } blk_set_allow_write_beyond_eof(blk, true); diff --git a/block/vpc.c b/block/vpc.c index 2d1eade146..5e31dd1e47 100644 --- a/block/vpc.c +++ b/block/vpc.c @@ -1012,10 +1012,10 @@ static int coroutine_fn vpc_co_create(BlockdevCreateOptions *opts, return -EIO; } - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs, BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL, + errp); + if (!blk) { + ret = -EPERM; goto out; } blk_set_allow_write_beyond_eof(blk, true); diff --git a/blockdev.c b/blockdev.c index dc1a0c7c2f..b3c840ec03 100644 --- a/blockdev.c +++ b/blockdev.c @@ -2711,7 +2711,6 @@ void qmp_block_resize(bool has_device, const char *device, BlockBackend *blk = NULL; BlockDriverState *bs; AioContext *aio_context; - int ret; bs = bdrv_lookup_bs(has_device ? device : NULL, has_node_name ? node_name : NULL, @@ -2734,14 +2733,13 @@ void qmp_block_resize(bool has_device, const char *device, goto out; } - blk = blk_new(bdrv_get_aio_context(bs), BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs, BLK_PERM_RESIZE, BLK_PERM_ALL, errp); + if (!blk) { goto out; } bdrv_drained_begin(bs); - ret = blk_truncate(blk, size, false, PREALLOC_MODE_OFF, 0, errp); + blk_truncate(blk, size, false, PREALLOC_MODE_OFF, 0, errp); bdrv_drained_end(bs); out: diff --git a/blockjob.c b/blockjob.c index fc850312c1..2affa1844d 100644 --- a/blockjob.c +++ b/blockjob.c @@ -397,16 +397,13 @@ void *block_job_create(const char *job_id, const BlockJobDriver *driver, { BlockBackend *blk; BlockJob *job; - int ret; if (job_id == NULL && !(flags & JOB_INTERNAL)) { job_id = bdrv_get_device_name(bs); } - blk = blk_new(bdrv_get_aio_context(bs), perm, shared_perm); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { - blk_unref(blk); + blk = blk_new_with_bs(bs, perm, shared_perm, errp); + if (!blk) { return NULL; } From patchwork Tue May 5 12:58:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529053 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B69A814C0 for ; Tue, 5 May 2020 13:17:48 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8CBDF2078C for ; Tue, 5 May 2020 13:17:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="jCen9nAq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8CBDF2078C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:48984 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxSF-00021n-MV for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:17:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38992) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxAT-0002Ij-7U for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:25 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:59216 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxAR-0001OR-Vr for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683563; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ODHBwiQG7zLsgD+hURCtxe6U8MhrFsHu+crICg+y2ok=; b=jCen9nAqKu+vchPCLQhKy38jMdP35AnYxhV3UP1O7GJUcpRsuwNnhnv5MbWt2lkuPOlHo+ r5K/CgupDvKfCYgcT4BtrnaVWrHSTCrbGyn0GK/kiyi2e1siDL53qjh40iXCl20Y78zTjb p/KiFe7Zt+P5y5vkrPHx1ePfPXSWu4o= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-396-S1IrA6y_PaiyBVa4ND_fIg-1; Tue, 05 May 2020 08:59:05 -0400 X-MC-Unique: S1IrA6y_PaiyBVa4ND_fIg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5BED1A0C01; Tue, 5 May 2020 12:59:04 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id CAD455D9DD; Tue, 5 May 2020 12:59:03 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 16/24] qcow2: Allow resize of images with internal snapshots Date: Tue, 5 May 2020 14:58:18 +0200 Message-Id: <20200505125826.1001451-17-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.61; envelope-from=mreitz@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:40 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: Eric Blake We originally refused to allow resize of images with internal snapshots because the v2 image format did not require the tracking of snapshot size, making it impossible to safely revert to a snapshot with a different size than the current view of the image. But the snapshot size tracking was rectified in v3, and our recent fixes to qemu-img amend (see 0a85af35) guarantee that we always have a valid snapshot size. Thus, we no longer need to artificially limit image resizes, but it does become one more thing that would prevent a downgrade back to v2. And now that we support different-sized snapshots, it's also easy to fix reverting to a snapshot to apply the new size. Upgrade iotest 61 to cover this (we previously had NO coverage of refusal to resize while snapshots exist). Note that the amend process can fail but still have effects: in particular, since we break things into upgrade, resize, downgrade, a failure during resize does not roll back changes made during upgrade, nor does failure in downgrade roll back a resize. But this situation is pre-existing even without this patch; and without journaling, the best we could do is minimize the chance of partial failure by collecting all changes prior to doing any writes - which adds a lot of complexity but could still fail with EIO. On the other hand, we are careful that even if we have partial modification but then fail, the image is left viable (that is, we are careful to sequence things so that after each successful cluster write, there may be transient leaked clusters but no corrupt metadata). And complicating the code to make it more transaction-like is not worth the effort: a user can always request multiple 'qemu-img amend' changing one thing each, if they need finer-grained control over detecting the first failure than what they get by letting qemu decide how to sequence multiple changes. Signed-off-by: Eric Blake Reviewed-by: Max Reitz Message-Id: <20200428192648.749066-3-eblake@redhat.com> Signed-off-by: Max Reitz --- block/qcow2-snapshot.c | 20 ++++++++++++++++---- block/qcow2.c | 25 ++++++++++++++++++++++--- tests/qemu-iotests/061 | 35 +++++++++++++++++++++++++++++++++++ tests/qemu-iotests/061.out | 28 ++++++++++++++++++++++++++++ 4 files changed, 101 insertions(+), 7 deletions(-) diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c index 82c32d4c9b..2756b37d24 100644 --- a/block/qcow2-snapshot.c +++ b/block/qcow2-snapshot.c @@ -23,6 +23,7 @@ */ #include "qemu/osdep.h" +#include "sysemu/block-backend.h" #include "qapi/error.h" #include "qcow2.h" #include "qemu/bswap.h" @@ -775,10 +776,21 @@ int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id) } if (sn->disk_size != bs->total_sectors * BDRV_SECTOR_SIZE) { - error_report("qcow2: Loading snapshots with different disk " - "size is not implemented"); - ret = -ENOTSUP; - goto fail; + BlockBackend *blk = blk_new_with_bs(bs, BLK_PERM_RESIZE, BLK_PERM_ALL, + &local_err); + if (!blk) { + error_report_err(local_err); + ret = -ENOTSUP; + goto fail; + } + + ret = blk_truncate(blk, sn->disk_size, true, PREALLOC_MODE_OFF, 0, + &local_err); + blk_unref(blk); + if (ret < 0) { + error_report_err(local_err); + goto fail; + } } /* diff --git a/block/qcow2.c b/block/qcow2.c index 0edc7f4643..3e8b3d022b 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -3989,9 +3989,12 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset, qemu_co_mutex_lock(&s->lock); - /* cannot proceed if image has snapshots */ - if (s->nb_snapshots) { - error_setg(errp, "Can't resize an image which has snapshots"); + /* + * Even though we store snapshot size for all images, it was not + * required until v3, so it is not safe to proceed for v2. + */ + if (s->nb_snapshots && s->qcow_version < 3) { + error_setg(errp, "Can't resize a v2 image which has snapshots"); ret = -ENOTSUP; goto fail; } @@ -5005,6 +5008,7 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version, BDRVQcow2State *s = bs->opaque; int current_version = s->qcow_version; int ret; + int i; /* This is qcow2_downgrade(), not qcow2_upgrade() */ assert(target_version < current_version); @@ -5022,6 +5026,21 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version, return -ENOTSUP; } + /* + * If any internal snapshot has a different size than the current + * image size, or VM state size that exceeds 32 bits, downgrading + * is unsafe. Even though we would still use v3-compliant output + * to preserve that data, other v2 programs might not realize + * those optional fields are important. + */ + for (i = 0; i < s->nb_snapshots; i++) { + if (s->snapshots[i].vm_state_size > UINT32_MAX || + s->snapshots[i].disk_size != bs->total_sectors * BDRV_SECTOR_SIZE) { + error_setg(errp, "Internal snapshots prevent downgrade of image"); + return -ENOTSUP; + } + } + /* clear incompatible features */ if (s->incompatible_features & QCOW2_INCOMPAT_DIRTY) { ret = qcow2_mark_clean(bs); diff --git a/tests/qemu-iotests/061 b/tests/qemu-iotests/061 index ce285d3084..10eb243164 100755 --- a/tests/qemu-iotests/061 +++ b/tests/qemu-iotests/061 @@ -111,6 +111,41 @@ $PYTHON qcow2.py "$TEST_IMG" dump-header $QEMU_IO -c "read -P 0x2a 42M 64k" "$TEST_IMG" | _filter_qemu_io _check_test_img +echo +echo "=== Testing resize with snapshots ===" +echo +_make_test_img -o "compat=0.10" 32M +$QEMU_IO -c "write -P 0x2a 24M 64k" "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG snapshot -c foo "$TEST_IMG" +$QEMU_IMG resize "$TEST_IMG" 64M && + echo "unexpected pass" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG amend -o "compat=1.1,size=128M" "$TEST_IMG" || + echo "unexpected fail" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG snapshot -c bar "$TEST_IMG" +$QEMU_IMG resize --shrink "$TEST_IMG" 64M || + echo "unexpected fail" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG amend -o "compat=0.10,size=32M" "$TEST_IMG" && + echo "unexpected pass" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG snapshot -a bar "$TEST_IMG" || + echo "unexpected fail" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG snapshot -d bar "$TEST_IMG" +$QEMU_IMG amend -o "compat=0.10,size=32M" "$TEST_IMG" || + echo "unexpected fail" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +_check_test_img + + echo echo "=== Testing dirty lazy_refcounts=off ===" echo diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out index 413cc4e0f4..5a8d36d005 100644 --- a/tests/qemu-iotests/061.out +++ b/tests/qemu-iotests/061.out @@ -271,6 +271,34 @@ read 65536/65536 bytes at offset 44040192 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) No errors were found on the image. +=== Testing resize with snapshots === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 +wrote 65536/65536 bytes at offset 25165824 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +qemu-img: Can't resize a v2 image which has snapshots +version 2 +size 33554432 +nb_snapshots 1 +version 3 +size 134217728 +nb_snapshots 1 +Image resized. +version 3 +size 67108864 +nb_snapshots 2 +qemu-img: Internal snapshots prevent downgrade of image +version 3 +size 33554432 +nb_snapshots 2 +version 3 +size 134217728 +nb_snapshots 2 +version 2 +size 33554432 +nb_snapshots 1 +No errors were found on the image. + === Testing dirty lazy_refcounts=off === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 From patchwork Tue May 5 12:58:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529039 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4C402912 for ; Tue, 5 May 2020 13:13:55 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 223FD20658 for ; Tue, 5 May 2020 13:13:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Ffvheymo" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 223FD20658 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:34034 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxOU-00040a-AQ for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:13:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38830) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxAG-0001v4-MP for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:12 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:23699 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxAE-0001Fo-Lf for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683550; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iBJc/5fhV3AkeEoXdifV7nDHIfYCiL4PhzT8qrtsjdQ=; b=Ffvheymo9gDtHnl15u8iAzc5VkKsI1yMs5gelXftZ+w51gWZsc/8tC7R0WMrJyGulYYcp5 3CnZdCpO7DBlul/rui3JZIuAaUb6p/q3UiYj5GKNW3MccvK4XYRdHjkXq/5oMVwjimOemn +kwqo6MM5GDom4gsPvPgF1K9YBSWrAc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-68-byxQWoV6OGSpL3iLnDdU_A-1; Tue, 05 May 2020 08:59:07 -0400 X-MC-Unique: byxQWoV6OGSpL3iLnDdU_A-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 82654107ACF4; Tue, 5 May 2020 12:59:06 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 254662DE9C; Tue, 5 May 2020 12:59:05 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 17/24] qcow2: Tweak comment about bitmaps vs. resize Date: Tue, 5 May 2020 14:58:19 +0200 Message-Id: <20200505125826.1001451-18-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=mreitz@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 03:48:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: Eric Blake Our comment did not actually match the code. Rewrite the comment to be less sensitive to any future changes to qcow2-bitmap.c that might implement scenarios that we currently reject. Signed-off-by: Eric Blake Reviewed-by: Max Reitz Message-Id: <20200428192648.749066-4-eblake@redhat.com> Signed-off-by: Max Reitz --- block/qcow2.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/qcow2.c b/block/qcow2.c index 3e8b3d022b..ad934109a8 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -3999,7 +3999,7 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset, goto fail; } - /* cannot proceed if image has bitmaps */ + /* See qcow2-bitmap.c for which bitmap scenarios prevent a resize. */ if (qcow2_truncate_bitmaps_check(bs, errp)) { ret = -ENOTSUP; goto fail; From patchwork Tue May 5 12:58:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529023 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4ADDB1392 for ; Tue, 5 May 2020 13:09:39 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1ECC02078C for ; Tue, 5 May 2020 13:09:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="N4odJrJa" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1ECC02078C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:43014 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxKM-0003tC-Az for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:09:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38872) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxAK-0001zY-3p for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:16 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:57105 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxAI-0001IP-UV for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683553; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YPVPkTBT7gtmChDFl5iKp++k8KNrrXiv+H5bePQX/1k=; b=N4odJrJaeQTWNJYjiua5Uqh2YQQz25Liv3HMRguYE0kTLG39sJ7lEOa98xO3d1tnUolFqh DkMRX2BImFksvaH1RviPqIbJT1Y/zIOmxYDhshretEBFAeRfg9TVYxXDoNllYs1YF9gdU+ IAWsnmei2vphcKWu51vcsejUSZFowas= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-405--3Hf_VyIP9aVqi5HTSP8nQ-1; Tue, 05 May 2020 08:59:09 -0400 X-MC-Unique: -3Hf_VyIP9aVqi5HTSP8nQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D5944835B43; Tue, 5 May 2020 12:59:08 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7667E1002387; Tue, 5 May 2020 12:59:08 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 18/24] block: Comment cleanups Date: Tue, 5 May 2020 14:58:20 +0200 Message-Id: <20200505125826.1001451-19-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.81; envelope-from=mreitz@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:19 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: Eric Blake It's been a while since we got rid of the sector-based bdrv_read and bdrv_write (commit 2e11d756); let's finish the job on a few remaining comments. Signed-off-by: Eric Blake Message-Id: <20200428213807.776655-1-eblake@redhat.com> Reviewed-by: Alberto Garcia Signed-off-by: Max Reitz --- block/io.c | 3 ++- block/qcow2-refcount.c | 2 +- block/vvfat.c | 10 +++++----- tests/qemu-iotests/001 | 2 +- tests/qemu-iotests/052 | 2 +- tests/qemu-iotests/134 | 2 +- tests/qemu-iotests/188 | 2 +- 7 files changed, 12 insertions(+), 11 deletions(-) diff --git a/block/io.c b/block/io.c index a4f9714230..7d30e61edc 100644 --- a/block/io.c +++ b/block/io.c @@ -960,7 +960,7 @@ int bdrv_pwrite_zeroes(BdrvChild *child, int64_t offset, * flags are passed through to bdrv_pwrite_zeroes (e.g. BDRV_REQ_MAY_UNMAP, * BDRV_REQ_FUA). * - * Returns < 0 on error, 0 on success. For error codes see bdrv_write(). + * Returns < 0 on error, 0 on success. For error codes see bdrv_pwrite(). */ int bdrv_make_zero(BdrvChild *child, BdrvRequestFlags flags) { @@ -994,6 +994,7 @@ int bdrv_make_zero(BdrvChild *child, BdrvRequestFlags flags) } } +/* return < 0 if error. See bdrv_pwrite() for the return codes */ int bdrv_preadv(BdrvChild *child, int64_t offset, QEMUIOVector *qiov) { int ret; diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index d9650b9b6c..0457a6060d 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -2660,7 +2660,7 @@ fail: * - 0 if writing to this offset will not affect the mentioned metadata * - a positive QCow2MetadataOverlap value indicating one overlapping section * - a negative value (-errno) indicating an error while performing a check, - * e.g. when bdrv_read failed on QCOW2_OL_INACTIVE_L2 + * e.g. when bdrv_pread failed on QCOW2_OL_INACTIVE_L2 */ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset, int64_t size) diff --git a/block/vvfat.c b/block/vvfat.c index ab800c4887..6d5c090dec 100644 --- a/block/vvfat.c +++ b/block/vvfat.c @@ -2148,7 +2148,7 @@ DLOG(checkpoint()); * - get modified FAT * - compare the two FATs (TODO) * - get buffer for marking used clusters - * - recurse direntries from root (using bs->bdrv_read to make + * - recurse direntries from root (using bs->bdrv_pread to make * sure to get the new data) * - check that the FAT agrees with the size * - count the number of clusters occupied by this directory and @@ -2913,9 +2913,9 @@ static int handle_deletes(BDRVVVFATState* s) /* * synchronize mapping with new state: * - * - copy FAT (with bdrv_read) + * - copy FAT (with bdrv_pread) * - mark all filenames corresponding to mappings as deleted - * - recurse direntries from root (using bs->bdrv_read) + * - recurse direntries from root (using bs->bdrv_pread) * - delete files corresponding to mappings marked as deleted */ static int do_commit(BDRVVVFATState* s) @@ -2935,10 +2935,10 @@ static int do_commit(BDRVVVFATState* s) return ret; } - /* copy FAT (with bdrv_read) */ + /* copy FAT (with bdrv_pread) */ memcpy(s->fat.pointer, s->fat2, 0x200 * s->sectors_per_fat); - /* recurse direntries from root (using bs->bdrv_read) */ + /* recurse direntries from root (using bs->bdrv_pread) */ ret = commit_direntries(s, 0, -1); if (ret) { fprintf(stderr, "Fatal: error while committing (%d)\n", ret); diff --git a/tests/qemu-iotests/001 b/tests/qemu-iotests/001 index d87a535c33..696726e45f 100755 --- a/tests/qemu-iotests/001 +++ b/tests/qemu-iotests/001 @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Test simple read/write using plain bdrv_read/bdrv_write +# Test simple read/write using plain bdrv_pread/bdrv_pwrite # # Copyright (C) 2009 Red Hat, Inc. # diff --git a/tests/qemu-iotests/052 b/tests/qemu-iotests/052 index 45a140910d..8d5c10601f 100755 --- a/tests/qemu-iotests/052 +++ b/tests/qemu-iotests/052 @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Test bdrv_read/bdrv_write using BDRV_O_SNAPSHOT +# Test bdrv_pread/bdrv_pwrite using BDRV_O_SNAPSHOT # # Copyright (C) 2013 Red Hat, Inc. # diff --git a/tests/qemu-iotests/134 b/tests/qemu-iotests/134 index 5f0fb86211..5162d21662 100755 --- a/tests/qemu-iotests/134 +++ b/tests/qemu-iotests/134 @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Test encrypted read/write using plain bdrv_read/bdrv_write +# Test encrypted read/write using plain bdrv_pread/bdrv_pwrite # # Copyright (C) 2015 Red Hat, Inc. # diff --git a/tests/qemu-iotests/188 b/tests/qemu-iotests/188 index afca44df54..09b9b6083a 100755 --- a/tests/qemu-iotests/188 +++ b/tests/qemu-iotests/188 @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Test encrypted read/write using plain bdrv_read/bdrv_write +# Test encrypted read/write using plain bdrv_pread/bdrv_pwrite # # Copyright (C) 2017 Red Hat, Inc. # From patchwork Tue May 5 12:58:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529043 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 496F614B4 for ; Tue, 5 May 2020 13:15:35 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CD7F4206A5 for ; Tue, 5 May 2020 13:15:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="crNUD2U+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CD7F4206A5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:40474 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxQ5-0006nQ-M7 for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:15:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38930) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxAO-00024t-9K for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:20 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:48208 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxAM-0001KQ-KM for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683557; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wAOqv4arudtz951dWEWJ85qchv6ghkaQN0paVNWP8EA=; b=crNUD2U+DPZqDpUkgD/lSb3cBhwG9zI7V/MloKCS1Yb0Vs1FT4iH1D3ormm6WHm8rfiU2g r3keA69FrC/URAYR0IigmhXsO5+Q3jBPfjIhO/Fkwwm9znit1kndaqw/Psp2E9ahZADGFK jAKK6lPrrMlh8UUO5tbo5nPkgIL/vt0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-288-QP-Ps5cmPU20JulTn63-Vg-1; Tue, 05 May 2020 08:59:11 -0400 X-MC-Unique: QP-Ps5cmPU20JulTn63-Vg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E3D60100CCD0; Tue, 5 May 2020 12:59:10 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 88204610AB; Tue, 5 May 2020 12:59:10 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 19/24] Fix iotest 153 Date: Tue, 5 May 2020 14:58:21 +0200 Message-Id: <20200505125826.1001451-20-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.61; envelope-from=mreitz@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:40 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: Maxim Levitsky Commit f62514b3def5fb2acbef64d0e053c0c31fa45aff made qemu-img reject -o "" but this test uses it. Since this test only tries to do a dry-run run of qemu-img amend, replace the -o "" with dummy -o "size=$size". Fixes: f62514b3def5fb2acbef64d0e053c0c31fa45aff Signed-off-by: Maxim Levitsky Message-Id: <20200504131959.9533-1-mlevitsk@redhat.com> Signed-off-by: Max Reitz --- tests/qemu-iotests/153 | 2 +- tests/qemu-iotests/153.out | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/qemu-iotests/153 b/tests/qemu-iotests/153 index 2b13111768..cf961d3609 100755 --- a/tests/qemu-iotests/153 +++ b/tests/qemu-iotests/153 @@ -122,7 +122,7 @@ for opts1 in "" "read-only=on" "read-only=on,force-share=on"; do _run_cmd $QEMU_IMG check $L "${TEST_IMG}" _run_cmd $QEMU_IMG compare $L "${TEST_IMG}" "${TEST_IMG}" _run_cmd $QEMU_IMG map $L "${TEST_IMG}" - _run_cmd $QEMU_IMG amend -o "" $L "${TEST_IMG}" + _run_cmd $QEMU_IMG amend -o "size=$size" $L "${TEST_IMG}" _run_cmd $QEMU_IMG commit $L "${TEST_IMG}" _run_cmd $QEMU_IMG resize $L "${TEST_IMG}" $size _run_cmd $QEMU_IMG rebase $L "${TEST_IMG}" -b "${TEST_IMG}.base" diff --git a/tests/qemu-iotests/153.out b/tests/qemu-iotests/153.out index f7464dd8d3..b2a90caa6b 100644 --- a/tests/qemu-iotests/153.out +++ b/tests/qemu-iotests/153.out @@ -56,7 +56,7 @@ _qemu_img_wrapper map TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get shared "write" lock Is another process using the image [TEST_DIR/t.qcow2]? -_qemu_img_wrapper amend -o TEST_DIR/t.qcow2 +_qemu_img_wrapper amend -o size=32M TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock Is another process using the image [TEST_DIR/t.qcow2]? @@ -118,7 +118,7 @@ _qemu_img_wrapper compare -U TEST_DIR/t.qcow2 TEST_DIR/t.qcow2 _qemu_img_wrapper map -U TEST_DIR/t.qcow2 -_qemu_img_wrapper amend -o -U TEST_DIR/t.qcow2 +_qemu_img_wrapper amend -o size=32M -U TEST_DIR/t.qcow2 qemu-img: unrecognized option '-U' Try 'qemu-img --help' for more information @@ -187,7 +187,7 @@ _qemu_img_wrapper compare TEST_DIR/t.qcow2 TEST_DIR/t.qcow2 _qemu_img_wrapper map TEST_DIR/t.qcow2 -_qemu_img_wrapper amend -o TEST_DIR/t.qcow2 +_qemu_img_wrapper amend -o size=32M TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock Is another process using the image [TEST_DIR/t.qcow2]? @@ -241,7 +241,7 @@ _qemu_img_wrapper compare -U TEST_DIR/t.qcow2 TEST_DIR/t.qcow2 _qemu_img_wrapper map -U TEST_DIR/t.qcow2 -_qemu_img_wrapper amend -o -U TEST_DIR/t.qcow2 +_qemu_img_wrapper amend -o size=32M -U TEST_DIR/t.qcow2 qemu-img: unrecognized option '-U' Try 'qemu-img --help' for more information @@ -303,7 +303,7 @@ _qemu_img_wrapper compare TEST_DIR/t.qcow2 TEST_DIR/t.qcow2 _qemu_img_wrapper map TEST_DIR/t.qcow2 -_qemu_img_wrapper amend -o TEST_DIR/t.qcow2 +_qemu_img_wrapper amend -o size=32M TEST_DIR/t.qcow2 _qemu_img_wrapper commit TEST_DIR/t.qcow2 @@ -345,7 +345,7 @@ _qemu_img_wrapper compare -U TEST_DIR/t.qcow2 TEST_DIR/t.qcow2 _qemu_img_wrapper map -U TEST_DIR/t.qcow2 -_qemu_img_wrapper amend -o -U TEST_DIR/t.qcow2 +_qemu_img_wrapper amend -o size=32M -U TEST_DIR/t.qcow2 qemu-img: unrecognized option '-U' Try 'qemu-img --help' for more information From patchwork Tue May 5 12:58:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11528997 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E0D93912 for ; Tue, 5 May 2020 13:07:07 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B6A0920735 for ; Tue, 5 May 2020 13:07:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Maiyb5Iq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B6A0920735 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:59344 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxHu-000751-Pe for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:07:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38912) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxAM-00022o-RP for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:18 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:37259 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxAL-0001Jp-KM for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683556; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hsrwIJ/Flr1sv6EY55lTdvGYeHZHKszAYF5KUsMQUjI=; b=Maiyb5IqZSH6QQ9OFuS0MIMb/l2s5Ei/Wd2FASUvg5gwBHqrjlkRcphRpkexqXAQWWXttf v0X8g8LwLWbqFE/YeQuzypyXrK85iqbAVPHZ0UJ4Qcq3Wv05uS5IDO8ZH3fUkoSPb0fv6r 9H3r6SJez9PCmGifRm3Df3Ly9KP+IGk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-384-4YIJ62UQN8efgzfsB6SlLA-1; Tue, 05 May 2020 08:59:14 -0400 X-MC-Unique: 4YIJ62UQN8efgzfsB6SlLA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 42B4B800687; Tue, 5 May 2020 12:59:13 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B13AD5C1D4; Tue, 5 May 2020 12:59:12 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 20/24] block/block-copy: rename in-flight requests to tasks Date: Tue, 5 May 2020 14:58:22 +0200 Message-Id: <20200505125826.1001451-21-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=mreitz@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 03:48:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy We are going to use aio-task-pool API and extend in-flight request structure to be a successor of AioTask, so rename things appropriately. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20200429130847.28124-2-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz --- block/block-copy.c | 98 +++++++++++++++++++++++----------------------- 1 file changed, 48 insertions(+), 50 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index 05227e18bf..bbb29366dc 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -24,12 +24,12 @@ #define BLOCK_COPY_MAX_BUFFER (1 * MiB) #define BLOCK_COPY_MAX_MEM (128 * MiB) -typedef struct BlockCopyInFlightReq { +typedef struct BlockCopyTask { int64_t offset; int64_t bytes; - QLIST_ENTRY(BlockCopyInFlightReq) list; - CoQueue wait_queue; /* coroutines blocked on this request */ -} BlockCopyInFlightReq; + QLIST_ENTRY(BlockCopyTask) list; + CoQueue wait_queue; /* coroutines blocked on this task */ +} BlockCopyTask; typedef struct BlockCopyState { /* @@ -45,7 +45,7 @@ typedef struct BlockCopyState { bool use_copy_range; int64_t copy_size; uint64_t len; - QLIST_HEAD(, BlockCopyInFlightReq) inflight_reqs; + QLIST_HEAD(, BlockCopyTask) tasks; BdrvRequestFlags write_flags; @@ -73,15 +73,14 @@ typedef struct BlockCopyState { SharedResource *mem; } BlockCopyState; -static BlockCopyInFlightReq *find_conflicting_inflight_req(BlockCopyState *s, - int64_t offset, - int64_t bytes) +static BlockCopyTask *find_conflicting_task(BlockCopyState *s, + int64_t offset, int64_t bytes) { - BlockCopyInFlightReq *req; + BlockCopyTask *t; - QLIST_FOREACH(req, &s->inflight_reqs, list) { - if (offset + bytes > req->offset && offset < req->offset + req->bytes) { - return req; + QLIST_FOREACH(t, &s->tasks, list) { + if (offset + bytes > t->offset && offset < t->offset + t->bytes) { + return t; } } @@ -89,73 +88,72 @@ static BlockCopyInFlightReq *find_conflicting_inflight_req(BlockCopyState *s, } /* - * If there are no intersecting requests return false. Otherwise, wait for the - * first found intersecting request to finish and return true. + * If there are no intersecting tasks return false. Otherwise, wait for the + * first found intersecting tasks to finish and return true. */ static bool coroutine_fn block_copy_wait_one(BlockCopyState *s, int64_t offset, int64_t bytes) { - BlockCopyInFlightReq *req = find_conflicting_inflight_req(s, offset, bytes); + BlockCopyTask *task = find_conflicting_task(s, offset, bytes); - if (!req) { + if (!task) { return false; } - qemu_co_queue_wait(&req->wait_queue, NULL); + qemu_co_queue_wait(&task->wait_queue, NULL); return true; } /* Called only on full-dirty region */ -static void block_copy_inflight_req_begin(BlockCopyState *s, - BlockCopyInFlightReq *req, - int64_t offset, int64_t bytes) +static void block_copy_task_begin(BlockCopyState *s, BlockCopyTask *task, + int64_t offset, int64_t bytes) { - assert(!find_conflicting_inflight_req(s, offset, bytes)); + assert(!find_conflicting_task(s, offset, bytes)); bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes); s->in_flight_bytes += bytes; - req->offset = offset; - req->bytes = bytes; - qemu_co_queue_init(&req->wait_queue); - QLIST_INSERT_HEAD(&s->inflight_reqs, req, list); + task->offset = offset; + task->bytes = bytes; + qemu_co_queue_init(&task->wait_queue); + QLIST_INSERT_HEAD(&s->tasks, task, list); } /* - * block_copy_inflight_req_shrink + * block_copy_task_shrink * - * Drop the tail of the request to be handled later. Set dirty bits back and - * wake up all requests waiting for us (may be some of them are not intersecting - * with shrunk request) + * Drop the tail of the task to be handled later. Set dirty bits back and + * wake up all tasks waiting for us (may be some of them are not intersecting + * with shrunk task) */ -static void coroutine_fn block_copy_inflight_req_shrink(BlockCopyState *s, - BlockCopyInFlightReq *req, int64_t new_bytes) +static void coroutine_fn block_copy_task_shrink(BlockCopyState *s, + BlockCopyTask *task, + int64_t new_bytes) { - if (new_bytes == req->bytes) { + if (new_bytes == task->bytes) { return; } - assert(new_bytes > 0 && new_bytes < req->bytes); + assert(new_bytes > 0 && new_bytes < task->bytes); - s->in_flight_bytes -= req->bytes - new_bytes; + s->in_flight_bytes -= task->bytes - new_bytes; bdrv_set_dirty_bitmap(s->copy_bitmap, - req->offset + new_bytes, req->bytes - new_bytes); + task->offset + new_bytes, task->bytes - new_bytes); - req->bytes = new_bytes; - qemu_co_queue_restart_all(&req->wait_queue); + task->bytes = new_bytes; + qemu_co_queue_restart_all(&task->wait_queue); } -static void coroutine_fn block_copy_inflight_req_end(BlockCopyState *s, - BlockCopyInFlightReq *req, - int ret) +static void coroutine_fn block_copy_task_end(BlockCopyState *s, + BlockCopyTask *task, int ret) { - s->in_flight_bytes -= req->bytes; + s->in_flight_bytes -= task->bytes; if (ret < 0) { - bdrv_set_dirty_bitmap(s->copy_bitmap, req->offset, req->bytes); + bdrv_set_dirty_bitmap(s->copy_bitmap, task->offset, task->bytes); } - QLIST_REMOVE(req, list); - qemu_co_queue_restart_all(&req->wait_queue); + QLIST_REMOVE(task, list); + qemu_co_queue_restart_all(&task->wait_queue); } void block_copy_state_free(BlockCopyState *s) @@ -223,7 +221,7 @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, s->copy_size = MAX(s->cluster_size, BLOCK_COPY_MAX_BUFFER); } - QLIST_INIT(&s->inflight_reqs); + QLIST_INIT(&s->tasks); return s; } @@ -474,7 +472,7 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, assert(QEMU_IS_ALIGNED(bytes, s->cluster_size)); while (bytes) { - BlockCopyInFlightReq req; + BlockCopyTask task; int64_t next_zero, cur_bytes, status_bytes; if (!bdrv_dirty_bitmap_get(s->copy_bitmap, offset)) { @@ -495,14 +493,14 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, assert(next_zero < offset + cur_bytes); /* no need to do MIN() */ cur_bytes = next_zero - offset; } - block_copy_inflight_req_begin(s, &req, offset, cur_bytes); + block_copy_task_begin(s, &task, offset, cur_bytes); ret = block_copy_block_status(s, offset, cur_bytes, &status_bytes); assert(ret >= 0); /* never fail */ cur_bytes = MIN(cur_bytes, status_bytes); - block_copy_inflight_req_shrink(s, &req, cur_bytes); + block_copy_task_shrink(s, &task, cur_bytes); if (s->skip_unallocated && !(ret & BDRV_BLOCK_ALLOCATED)) { - block_copy_inflight_req_end(s, &req, 0); + block_copy_task_end(s, &task, 0); progress_set_remaining(s->progress, bdrv_get_dirty_count(s->copy_bitmap) + s->in_flight_bytes); @@ -518,7 +516,7 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, ret = block_copy_do_copy(s, offset, cur_bytes, ret & BDRV_BLOCK_ZERO, error_is_read); co_put_to_shres(s->mem, cur_bytes); - block_copy_inflight_req_end(s, &req, ret); + block_copy_task_end(s, &task, ret); if (ret < 0) { return ret; } From patchwork Tue May 5 12:58:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529021 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1C966912 for ; Tue, 5 May 2020 13:09:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E6A482073B for ; Tue, 5 May 2020 13:09:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LlxYbydW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E6A482073B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:39640 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxJl-0002Px-TZ for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:09:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38926) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxAO-00024q-6c for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:20 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:25191 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxAN-0001Ke-Bd for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683558; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X+Sj31gR/RLHmlc21FjoAOJRNiclOU1V69F/XyE11vs=; b=LlxYbydWd+s0AcwGCo8Fm4dN5bNOrtQcUL8Qkce/EFBmaFZmzEAvPa58eDdQqFijmKLlQl 3OB7R0Ed+OgQQ19fGv/kCoT/3VwAIz0xFED95Ha9NEtPnwjVfP8GhKUqFqQ6sUtSh/sAKg 5GbivwLjM1mqMSZmHvpo+yS1KJ2mzKo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-189-UnLzuQC4MOuhK-Xq_2VQCQ-1; Tue, 05 May 2020 08:59:16 -0400 X-MC-Unique: UnLzuQC4MOuhK-Xq_2VQCQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6511981CBE3; Tue, 5 May 2020 12:59:15 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 07E0F7053E; Tue, 5 May 2020 12:59:14 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 21/24] block/block-copy: alloc task on each iteration Date: Tue, 5 May 2020 14:58:23 +0200 Message-Id: <20200505125826.1001451-22-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.120; envelope-from=mreitz@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:38 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy We are going to use aio-task-pool API, so tasks will be handled in parallel. We need therefore separate allocated task on each iteration. Introduce this logic now. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20200429130847.28124-3-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz --- block/block-copy.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index bbb29366dc..8d1b9ab9f0 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -106,9 +106,11 @@ static bool coroutine_fn block_copy_wait_one(BlockCopyState *s, int64_t offset, } /* Called only on full-dirty region */ -static void block_copy_task_begin(BlockCopyState *s, BlockCopyTask *task, - int64_t offset, int64_t bytes) +static BlockCopyTask *block_copy_task_create(BlockCopyState *s, + int64_t offset, int64_t bytes) { + BlockCopyTask *task = g_new(BlockCopyTask, 1); + assert(!find_conflicting_task(s, offset, bytes)); bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes); @@ -118,6 +120,8 @@ static void block_copy_task_begin(BlockCopyState *s, BlockCopyTask *task, task->bytes = bytes; qemu_co_queue_init(&task->wait_queue); QLIST_INSERT_HEAD(&s->tasks, task, list); + + return task; } /* @@ -472,7 +476,7 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, assert(QEMU_IS_ALIGNED(bytes, s->cluster_size)); while (bytes) { - BlockCopyTask task; + g_autofree BlockCopyTask *task = NULL; int64_t next_zero, cur_bytes, status_bytes; if (!bdrv_dirty_bitmap_get(s->copy_bitmap, offset)) { @@ -493,14 +497,14 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, assert(next_zero < offset + cur_bytes); /* no need to do MIN() */ cur_bytes = next_zero - offset; } - block_copy_task_begin(s, &task, offset, cur_bytes); + task = block_copy_task_create(s, offset, cur_bytes); ret = block_copy_block_status(s, offset, cur_bytes, &status_bytes); assert(ret >= 0); /* never fail */ cur_bytes = MIN(cur_bytes, status_bytes); - block_copy_task_shrink(s, &task, cur_bytes); + block_copy_task_shrink(s, task, cur_bytes); if (s->skip_unallocated && !(ret & BDRV_BLOCK_ALLOCATED)) { - block_copy_task_end(s, &task, 0); + block_copy_task_end(s, task, 0); progress_set_remaining(s->progress, bdrv_get_dirty_count(s->copy_bitmap) + s->in_flight_bytes); @@ -516,7 +520,7 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, ret = block_copy_do_copy(s, offset, cur_bytes, ret & BDRV_BLOCK_ZERO, error_is_read); co_put_to_shres(s->mem, cur_bytes); - block_copy_task_end(s, &task, ret); + block_copy_task_end(s, task, ret); if (ret < 0) { return ret; } From patchwork Tue May 5 12:58:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529045 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 83F2414B4 for ; Tue, 5 May 2020 13:15:38 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 59D8D206A5 for ; Tue, 5 May 2020 13:15:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="TaDgt1ZN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 59D8D206A5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:40730 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxQ9-0006tx-H1 for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:15:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38978) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxAS-0002GA-9l for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:24 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:57856 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxAR-0001N8-Fm for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683562; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FiYlbID0TC7g/mAytPhqN/bsf8MIyh0E9vM2SOuLYJ0=; b=TaDgt1ZNZisOvXx6WCpa1Bxhoc7oqH/WQblkyw2iyq/TCIxUqBhjzJ4CRx5y+3bJPboB9S UPFtNO36GJnZ7SdXGhbutacBhqr4myL+wNRUewHx6ZwjxXEldziLpYSiRRwNiKM+tAG+D+ 4PJ6UNFsWx/8tQKk8WZucNs+T55FfeI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-126-aKVhQyHlNAyfRGnWJfxa1Q-1; Tue, 05 May 2020 08:59:18 -0400 X-MC-Unique: aKVhQyHlNAyfRGnWJfxa1Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7C2C8461; Tue, 5 May 2020 12:59:17 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1C7E26918D; Tue, 5 May 2020 12:59:16 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 22/24] block/block-copy: add state pointer to BlockCopyTask Date: Tue, 5 May 2020 14:58:24 +0200 Message-Id: <20200505125826.1001451-23-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.61; envelope-from=mreitz@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:40 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy We are going to use aio-task-pool API, so we'll need state pointer in BlockCopyTask anyway. Add it now and use where possible. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20200429130847.28124-4-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz --- block/block-copy.c | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index 8d1b9ab9f0..35ff9cc3ef 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -25,6 +25,7 @@ #define BLOCK_COPY_MAX_MEM (128 * MiB) typedef struct BlockCopyTask { + BlockCopyState *s; int64_t offset; int64_t bytes; QLIST_ENTRY(BlockCopyTask) list; @@ -116,8 +117,11 @@ static BlockCopyTask *block_copy_task_create(BlockCopyState *s, bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes); s->in_flight_bytes += bytes; - task->offset = offset; - task->bytes = bytes; + *task = (BlockCopyTask) { + .s = s, + .offset = offset, + .bytes = bytes, + }; qemu_co_queue_init(&task->wait_queue); QLIST_INSERT_HEAD(&s->tasks, task, list); @@ -131,8 +135,7 @@ static BlockCopyTask *block_copy_task_create(BlockCopyState *s, * wake up all tasks waiting for us (may be some of them are not intersecting * with shrunk task) */ -static void coroutine_fn block_copy_task_shrink(BlockCopyState *s, - BlockCopyTask *task, +static void coroutine_fn block_copy_task_shrink(BlockCopyTask *task, int64_t new_bytes) { if (new_bytes == task->bytes) { @@ -141,20 +144,19 @@ static void coroutine_fn block_copy_task_shrink(BlockCopyState *s, assert(new_bytes > 0 && new_bytes < task->bytes); - s->in_flight_bytes -= task->bytes - new_bytes; - bdrv_set_dirty_bitmap(s->copy_bitmap, + task->s->in_flight_bytes -= task->bytes - new_bytes; + bdrv_set_dirty_bitmap(task->s->copy_bitmap, task->offset + new_bytes, task->bytes - new_bytes); task->bytes = new_bytes; qemu_co_queue_restart_all(&task->wait_queue); } -static void coroutine_fn block_copy_task_end(BlockCopyState *s, - BlockCopyTask *task, int ret) +static void coroutine_fn block_copy_task_end(BlockCopyTask *task, int ret) { - s->in_flight_bytes -= task->bytes; + task->s->in_flight_bytes -= task->bytes; if (ret < 0) { - bdrv_set_dirty_bitmap(s->copy_bitmap, task->offset, task->bytes); + bdrv_set_dirty_bitmap(task->s->copy_bitmap, task->offset, task->bytes); } QLIST_REMOVE(task, list); qemu_co_queue_restart_all(&task->wait_queue); @@ -502,9 +504,9 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, ret = block_copy_block_status(s, offset, cur_bytes, &status_bytes); assert(ret >= 0); /* never fail */ cur_bytes = MIN(cur_bytes, status_bytes); - block_copy_task_shrink(s, task, cur_bytes); + block_copy_task_shrink(task, cur_bytes); if (s->skip_unallocated && !(ret & BDRV_BLOCK_ALLOCATED)) { - block_copy_task_end(s, task, 0); + block_copy_task_end(task, 0); progress_set_remaining(s->progress, bdrv_get_dirty_count(s->copy_bitmap) + s->in_flight_bytes); @@ -520,7 +522,7 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, ret = block_copy_do_copy(s, offset, cur_bytes, ret & BDRV_BLOCK_ZERO, error_is_read); co_put_to_shres(s->mem, cur_bytes); - block_copy_task_end(s, task, ret); + block_copy_task_end(task, ret); if (ret < 0) { return ret; } From patchwork Tue May 5 12:58:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529035 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B228C912 for ; Tue, 5 May 2020 13:12:35 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 879212078C for ; Tue, 5 May 2020 13:12:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="X6hi3LrH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 879212078C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:56802 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxNC-0001Vl-KO for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:12:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38984) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxAS-0002Gv-IJ for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:24 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:52196 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxAR-0001Nk-HG for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683562; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EeuWiltkI1rpgdxq90Zxb5Zl+jCv5+GW6Q5mC7k1S+k=; b=X6hi3LrHEASpzY8W54TlFQg10PXvRQSWYe0ojOmUaKz7nDfvz/QqARVXnmllzVPJlhUP84 WcTV9K7GbFwOG3XP2fqca9PSaDq7bgcW7muNnU8o+1NnwZ6JMXFoGP6CwZoqB7cVbDHlTR w8TDVVC5B5FFPMwbtNNS9Hw6niu8EAI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-421-nlY7tz6vN3ixPNE9C5vMXQ-1; Tue, 05 May 2020 08:59:20 -0400 X-MC-Unique: nlY7tz6vN3ixPNE9C5vMXQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C05AD800687; Tue, 5 May 2020 12:59:19 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5B73410021B3; Tue, 5 May 2020 12:59:19 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 23/24] block/block-copy: refactor task creation Date: Tue, 5 May 2020 14:58:25 +0200 Message-Id: <20200505125826.1001451-24-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.81; envelope-from=mreitz@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 00:37:19 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Instead of just relying on the comment "Called only on full-dirty region" in block_copy_task_create() let's move initial dirty area search directly to block_copy_task_create(). Let's also use effective bdrv_dirty_bitmap_next_dirty_area instead of looping through all non-dirty clusters. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20200429130847.28124-5-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz --- block/block-copy.c | 80 ++++++++++++++++++++++++++-------------------- 1 file changed, 46 insertions(+), 34 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index 35ff9cc3ef..f560338647 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -32,6 +32,11 @@ typedef struct BlockCopyTask { CoQueue wait_queue; /* coroutines blocked on this task */ } BlockCopyTask; +static int64_t task_end(BlockCopyTask *task) +{ + return task->offset + task->bytes; +} + typedef struct BlockCopyState { /* * BdrvChild objects are not owned or managed by block-copy. They are @@ -106,17 +111,29 @@ static bool coroutine_fn block_copy_wait_one(BlockCopyState *s, int64_t offset, return true; } -/* Called only on full-dirty region */ +/* + * Search for the first dirty area in offset/bytes range and create task at + * the beginning of it. + */ static BlockCopyTask *block_copy_task_create(BlockCopyState *s, int64_t offset, int64_t bytes) { - BlockCopyTask *task = g_new(BlockCopyTask, 1); + BlockCopyTask *task; + if (!bdrv_dirty_bitmap_next_dirty_area(s->copy_bitmap, + offset, offset + bytes, + s->copy_size, &offset, &bytes)) + { + return NULL; + } + + /* region is dirty, so no existent tasks possible in it */ assert(!find_conflicting_task(s, offset, bytes)); bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes); s->in_flight_bytes += bytes; + task = g_new(BlockCopyTask, 1); *task = (BlockCopyTask) { .s = s, .offset = offset, @@ -466,6 +483,7 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, { int ret = 0; bool found_dirty = false; + int64_t end = offset + bytes; /* * block_copy() user is responsible for keeping source and target in same @@ -479,58 +497,52 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, while (bytes) { g_autofree BlockCopyTask *task = NULL; - int64_t next_zero, cur_bytes, status_bytes; + int64_t status_bytes; - if (!bdrv_dirty_bitmap_get(s->copy_bitmap, offset)) { - trace_block_copy_skip(s, offset); - offset += s->cluster_size; - bytes -= s->cluster_size; - continue; /* already copied */ + task = block_copy_task_create(s, offset, bytes); + if (!task) { + /* No more dirty bits in the bitmap */ + trace_block_copy_skip_range(s, offset, bytes); + break; + } + if (task->offset > offset) { + trace_block_copy_skip_range(s, offset, task->offset - offset); } found_dirty = true; - cur_bytes = MIN(bytes, s->copy_size); - - next_zero = bdrv_dirty_bitmap_next_zero(s->copy_bitmap, offset, - cur_bytes); - if (next_zero >= 0) { - assert(next_zero > offset); /* offset is dirty */ - assert(next_zero < offset + cur_bytes); /* no need to do MIN() */ - cur_bytes = next_zero - offset; - } - task = block_copy_task_create(s, offset, cur_bytes); - - ret = block_copy_block_status(s, offset, cur_bytes, &status_bytes); + ret = block_copy_block_status(s, task->offset, task->bytes, + &status_bytes); assert(ret >= 0); /* never fail */ - cur_bytes = MIN(cur_bytes, status_bytes); - block_copy_task_shrink(task, cur_bytes); + if (status_bytes < task->bytes) { + block_copy_task_shrink(task, status_bytes); + } if (s->skip_unallocated && !(ret & BDRV_BLOCK_ALLOCATED)) { block_copy_task_end(task, 0); progress_set_remaining(s->progress, bdrv_get_dirty_count(s->copy_bitmap) + s->in_flight_bytes); - trace_block_copy_skip_range(s, offset, status_bytes); - offset += status_bytes; - bytes -= status_bytes; + trace_block_copy_skip_range(s, task->offset, task->bytes); + offset = task_end(task); + bytes = end - offset; continue; } - trace_block_copy_process(s, offset); + trace_block_copy_process(s, task->offset); - co_get_from_shres(s->mem, cur_bytes); - ret = block_copy_do_copy(s, offset, cur_bytes, ret & BDRV_BLOCK_ZERO, - error_is_read); - co_put_to_shres(s->mem, cur_bytes); + co_get_from_shres(s->mem, task->bytes); + ret = block_copy_do_copy(s, task->offset, task->bytes, + ret & BDRV_BLOCK_ZERO, error_is_read); + co_put_to_shres(s->mem, task->bytes); block_copy_task_end(task, ret); if (ret < 0) { return ret; } - progress_work_done(s->progress, cur_bytes); - s->progress_bytes_callback(cur_bytes, s->progress_opaque); - offset += cur_bytes; - bytes -= cur_bytes; + progress_work_done(s->progress, task->bytes); + s->progress_bytes_callback(task->bytes, s->progress_opaque); + offset = task_end(task); + bytes = end - offset; } return found_dirty; From patchwork Tue May 5 12:58:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Reitz X-Patchwork-Id: 11529041 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A9E7C912 for ; Tue, 5 May 2020 13:14:23 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8034F206A5 for ; Tue, 5 May 2020 13:14:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="NhE4R+5I" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8034F206A5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:36452 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jVxOw-00051l-Kv for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 May 2020 09:14:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39018) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jVxAW-0002Sl-MZ for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:28 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:43268 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jVxAV-0001To-PB for qemu-devel@nongnu.org; Tue, 05 May 2020 08:59:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1588683567; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Sqx+AKD2vYUF8SPf8E3CW3TO8V+h79wEyvIM7fcPfUY=; b=NhE4R+5I2LWtd0psYicQrxnb9hpZ2qEkwGPlqPwlS3wkUwEBLowzB0DKZLhBZ1zhQGTZnX 2kd2bsinRM/FwF5sV24OqeCU2iu5Dy/Rmdcm+pPBLO5d8VydNalQTyWFnRYhZhIJsr21/u mcVHLFWoaMsE1amLmaPNFUuk0RWt/rc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-304-FuAYeasDMDKI3A6WS5eCTA-1; Tue, 05 May 2020 08:59:23 -0400 X-MC-Unique: FuAYeasDMDKI3A6WS5eCTA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 176AE1899520; Tue, 5 May 2020 12:59:22 +0000 (UTC) Received: from localhost (ovpn-113-206.ams2.redhat.com [10.36.113.206]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 82CF360CC0; Tue, 5 May 2020 12:59:21 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 24/24] block/block-copy: use aio-task-pool API Date: Tue, 5 May 2020 14:58:26 +0200 Message-Id: <20200505125826.1001451-25-mreitz@redhat.com> In-Reply-To: <20200505125826.1001451-1-mreitz@redhat.com> References: <20200505125826.1001451-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=mreitz@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/05 03:48:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: Vladimir Sementsov-Ogievskiy Run block_copy iterations in parallel in aio tasks. Changes: - BlockCopyTask becomes aio task structure. Add zeroes field to pass it to block_copy_do_copy - add call state - it's a state of one call of block_copy(), shared between parallel tasks. For now used only to keep information about first error: is it read or not. - convert block_copy_dirty_clusters to aio-task loop. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20200429130847.28124-6-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz --- block/block-copy.c | 119 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 106 insertions(+), 13 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index f560338647..03500680f7 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -19,15 +19,29 @@ #include "block/block-copy.h" #include "sysemu/block-backend.h" #include "qemu/units.h" +#include "qemu/coroutine.h" +#include "block/aio_task.h" #define BLOCK_COPY_MAX_COPY_RANGE (16 * MiB) #define BLOCK_COPY_MAX_BUFFER (1 * MiB) #define BLOCK_COPY_MAX_MEM (128 * MiB) +#define BLOCK_COPY_MAX_WORKERS 64 + +static coroutine_fn int block_copy_task_entry(AioTask *task); + +typedef struct BlockCopyCallState { + bool failed; + bool error_is_read; +} BlockCopyCallState; typedef struct BlockCopyTask { + AioTask task; + BlockCopyState *s; + BlockCopyCallState *call_state; int64_t offset; int64_t bytes; + bool zeroes; QLIST_ENTRY(BlockCopyTask) list; CoQueue wait_queue; /* coroutines blocked on this task */ } BlockCopyTask; @@ -116,6 +130,7 @@ static bool coroutine_fn block_copy_wait_one(BlockCopyState *s, int64_t offset, * the beginning of it. */ static BlockCopyTask *block_copy_task_create(BlockCopyState *s, + BlockCopyCallState *call_state, int64_t offset, int64_t bytes) { BlockCopyTask *task; @@ -135,7 +150,9 @@ static BlockCopyTask *block_copy_task_create(BlockCopyState *s, task = g_new(BlockCopyTask, 1); *task = (BlockCopyTask) { + .task.func = block_copy_task_entry, .s = s, + .call_state = call_state, .offset = offset, .bytes = bytes, }; @@ -263,6 +280,38 @@ void block_copy_set_progress_meter(BlockCopyState *s, ProgressMeter *pm) s->progress = pm; } +/* + * Takes ownership of @task + * + * If pool is NULL directly run the task, otherwise schedule it into the pool. + * + * Returns: task.func return code if pool is NULL + * otherwise -ECANCELED if pool status is bad + * otherwise 0 (successfully scheduled) + */ +static coroutine_fn int block_copy_task_run(AioTaskPool *pool, + BlockCopyTask *task) +{ + if (!pool) { + int ret = task->task.func(&task->task); + + g_free(task); + return ret; + } + + aio_task_pool_wait_slot(pool); + if (aio_task_pool_status(pool) < 0) { + co_put_to_shres(task->s->mem, task->bytes); + block_copy_task_end(task, -ECANCELED); + g_free(task); + return -ECANCELED; + } + + aio_task_pool_start_task(pool, &task->task); + + return 0; +} + /* * block_copy_do_copy * @@ -366,6 +415,27 @@ out: return ret; } +static coroutine_fn int block_copy_task_entry(AioTask *task) +{ + BlockCopyTask *t = container_of(task, BlockCopyTask, task); + bool error_is_read; + int ret; + + ret = block_copy_do_copy(t->s, t->offset, t->bytes, t->zeroes, + &error_is_read); + if (ret < 0 && !t->call_state->failed) { + t->call_state->failed = true; + t->call_state->error_is_read = error_is_read; + } else { + progress_work_done(t->s->progress, t->bytes); + t->s->progress_bytes_callback(t->bytes, t->s->progress_opaque); + } + co_put_to_shres(t->s->mem, t->bytes); + block_copy_task_end(t, ret); + + return ret; +} + static int block_copy_block_status(BlockCopyState *s, int64_t offset, int64_t bytes, int64_t *pnum) { @@ -484,6 +554,8 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, int ret = 0; bool found_dirty = false; int64_t end = offset + bytes; + AioTaskPool *aio = NULL; + BlockCopyCallState call_state = {false, false}; /* * block_copy() user is responsible for keeping source and target in same @@ -495,11 +567,11 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, assert(QEMU_IS_ALIGNED(offset, s->cluster_size)); assert(QEMU_IS_ALIGNED(bytes, s->cluster_size)); - while (bytes) { - g_autofree BlockCopyTask *task = NULL; + while (bytes && aio_task_pool_status(aio) == 0) { + BlockCopyTask *task; int64_t status_bytes; - task = block_copy_task_create(s, offset, bytes); + task = block_copy_task_create(s, &call_state, offset, bytes); if (!task) { /* No more dirty bits in the bitmap */ trace_block_copy_skip_range(s, offset, bytes); @@ -519,6 +591,7 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, } if (s->skip_unallocated && !(ret & BDRV_BLOCK_ALLOCATED)) { block_copy_task_end(task, 0); + g_free(task); progress_set_remaining(s->progress, bdrv_get_dirty_count(s->copy_bitmap) + s->in_flight_bytes); @@ -527,25 +600,45 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, bytes = end - offset; continue; } + task->zeroes = ret & BDRV_BLOCK_ZERO; trace_block_copy_process(s, task->offset); co_get_from_shres(s->mem, task->bytes); - ret = block_copy_do_copy(s, task->offset, task->bytes, - ret & BDRV_BLOCK_ZERO, error_is_read); - co_put_to_shres(s->mem, task->bytes); - block_copy_task_end(task, ret); - if (ret < 0) { - return ret; - } - progress_work_done(s->progress, task->bytes); - s->progress_bytes_callback(task->bytes, s->progress_opaque); offset = task_end(task); bytes = end - offset; + + if (!aio && bytes) { + aio = aio_task_pool_new(BLOCK_COPY_MAX_WORKERS); + } + + ret = block_copy_task_run(aio, task); + if (ret < 0) { + goto out; + } + } + +out: + if (aio) { + aio_task_pool_wait_all(aio); + + /* + * We are not really interested in -ECANCELED returned from + * block_copy_task_run. If it fails, it means some task already failed + * for real reason, let's return first failure. + * Still, assert that we don't rewrite failure by success. + */ + assert(ret == 0 || aio_task_pool_status(aio) < 0); + ret = aio_task_pool_status(aio); + + aio_task_pool_free(aio); + } + if (error_is_read && ret < 0) { + *error_is_read = call_state.error_is_read; } - return found_dirty; + return ret < 0 ? ret : found_dirty; } /*