From patchwork Tue Jul 16 00:01:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 11045109 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4D9DB6C5 for ; Tue, 16 Jul 2019 00:01:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3BB7728516 for ; Tue, 16 Jul 2019 00:01:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2EFEA2853C; Tue, 16 Jul 2019 00:01:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9A96128516 for ; Tue, 16 Jul 2019 00:01:54 +0000 (UTC) Received: from localhost ([::1]:44414 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hnAul-0008Mu-OO for patchwork-qemu-devel@patchwork.kernel.org; Mon, 15 Jul 2019 20:01:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44558) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hnAuP-0006nl-3W for qemu-devel@nongnu.org; Mon, 15 Jul 2019 20:01:30 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hnAuN-0007Ta-Kn for qemu-devel@nongnu.org; Mon, 15 Jul 2019 20:01:29 -0400 Received: from mx1.redhat.com ([209.132.183.28]:39678) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hnAuK-0007Pr-JS; Mon, 15 Jul 2019 20:01:24 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C866459440; Tue, 16 Jul 2019 00:01:23 +0000 (UTC) Received: from probe.bos.redhat.com (dhcp-17-130.bos.redhat.com [10.18.17.130]) by smtp.corp.redhat.com (Postfix) with ESMTP id BE4AD6085B; Tue, 16 Jul 2019 00:01:22 +0000 (UTC) From: John Snow To: qemu-block@nongnu.org, qemu-devel@nongnu.org Date: Mon, 15 Jul 2019 20:01:08 -0400 Message-Id: <20190716000117.25219-3-jsnow@redhat.com> In-Reply-To: <20190716000117.25219-1-jsnow@redhat.com> References: <20190716000117.25219-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Tue, 16 Jul 2019 00:01:23 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v2 02/11] iotests/257: add EmulatedBitmap class X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Represent a bitmap with an object that we can mark and clear bits in. This makes it easier to manage partial writes when we don't write a full group's worth of patterns before an error. Signed-off-by: John Snow Reviewed-by: Max Reitz --- tests/qemu-iotests/257 | 124 +++++++++++++++++++++++++---------------- 1 file changed, 75 insertions(+), 49 deletions(-) diff --git a/tests/qemu-iotests/257 b/tests/qemu-iotests/257 index 02f9ae0649..bc66ea03b2 100755 --- a/tests/qemu-iotests/257 +++ b/tests/qemu-iotests/257 @@ -85,6 +85,59 @@ GROUPS = [ Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right ] + +class EmulatedBitmap: + def __init__(self, granularity=GRANULARITY): + self._bits = set() + self.granularity = granularity + + def dirty_bits(self, bits): + self._bits |= set(bits) + + def dirty_group(self, n): + self.dirty_bits(GROUPS[n].bits(self.granularity)) + + def clear(self): + self._bits = set() + + def clear_bits(self, bits): + self._bits -= set(bits) + + def clear_bit(self, bit): + self.clear_bits({bit}) + + def clear_group(self, n): + self.clear_bits(GROUPS[n].bits(self.granularity)) + + @property + def first_bit(self): + return sorted(self.bits)[0] + + @property + def bits(self): + return self._bits + + @property + def count(self): + return len(self.bits) + + def compare(self, qmp_bitmap): + """ + Print a nice human-readable message checking that a bitmap as reported + by the QMP interface has as many bits set as we expect it to. + """ + + name = qmp_bitmap.get('name', '(anonymous)') + log("= Checking Bitmap {:s} =".format(name)) + + want = self.count + have = qmp_bitmap['count'] // qmp_bitmap['granularity'] + + log("expecting {:d} dirty sectors; have {:d}. {:s}".format( + want, have, "OK!" if want == have else "ERROR!")) + log('') + + class Drive: """Represents, vaguely, a drive attached to a VM. Includes format, graph, and device information.""" @@ -195,27 +248,6 @@ def perform_writes(drive, n): log('') return bitmaps -def calculate_bits(groups=None): - """Calculate how many bits we expect to see dirtied.""" - if groups: - bits = set.union(*(GROUPS[group].bits(GRANULARITY) for group in groups)) - return len(bits) - return 0 - -def bitmap_comparison(bitmap, groups=None, want=0): - """ - Print a nice human-readable message checking that this bitmap has as - many bits set as we expect it to. - """ - log("= Checking Bitmap {:s} =".format(bitmap.get('name', '(anonymous)'))) - - if groups: - want = calculate_bits(groups) - have = bitmap['count'] // bitmap['granularity'] - - log("expecting {:d} dirty sectors; have {:d}. {:s}".format( - want, have, "OK!" if want == have else "ERROR!")) - log('') def compare_images(image, reference, baseimg=None, expected_match=True): """ @@ -321,12 +353,13 @@ def test_bitmap_sync(bsync_mode, failure=None): vm.qmp_log("block-dirty-bitmap-add", node=drive0.name, name="bitmap0", granularity=GRANULARITY) log('') + ebitmap = EmulatedBitmap() # 1 - Writes and Reference Backup bitmaps = perform_writes(drive0, 1) - dirty_groups = {1} + ebitmap.dirty_group(1) bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0') - bitmap_comparison(bitmap, groups=dirty_groups) + ebitmap.compare(bitmap) reference_backup(drive0, 1, fbackup1) # 1 - Bitmap Backup (Optional induced failure) @@ -342,54 +375,47 @@ def test_bitmap_sync(bsync_mode, failure=None): log('') bitmaps = perform_writes(drive0, 2) # Named bitmap (static, should be unchanged) - bitmap_comparison(get_bitmap(bitmaps, drive0.device, 'bitmap0'), - groups=dirty_groups) + ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0')) # Anonymous bitmap (dynamic, shows new writes) - bitmap_comparison(get_bitmap(bitmaps, drive0.device, '', - recording=True), groups={2}) - dirty_groups.add(2) + anonymous = EmulatedBitmap() + anonymous.dirty_group(2) + anonymous.compare(get_bitmap(bitmaps, drive0.device, '', + recording=True)) + + # Simulate the order in which this will happen: + # group 1 gets cleared first, then group two gets written. + if ((bsync_mode == 'on-success' and not failure) or + (bsync_mode == 'always')): + ebitmap.clear_group(1) + ebitmap.dirty_group(2) vm.run_job(job, auto_dismiss=True, auto_finalize=False, pre_finalize=_callback, cancel=(failure == 'simulated')) bitmaps = query_bitmaps(vm) - bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0') log(bitmaps, indent=2) log('') - if ((bsync_mode == 'on-success' and not failure) or - (bsync_mode == 'always' and failure != 'intermediate')): - dirty_groups.remove(1) - if bsync_mode == 'always' and failure == 'intermediate': # We manage to copy one sector (one bit) before the error. - bitmap_comparison(bitmap, - want=calculate_bits(groups=dirty_groups) - 1) - else: - bitmap_comparison(bitmap, groups=dirty_groups) + ebitmap.clear_bit(ebitmap.first_bit) + ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0')) # 2 - Writes and Reference Backup bitmaps = perform_writes(drive0, 3) - dirty_groups.add(3) - bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0') - if bsync_mode == 'always' and failure == 'intermediate': - # We're one bit short, still. - bitmap_comparison(bitmap, - want=calculate_bits(groups=dirty_groups) - 1) - else: - bitmap_comparison(bitmap, groups=dirty_groups) + ebitmap.dirty_group(3) + ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0')) reference_backup(drive0, 2, fbackup2) # 2 - Bitmap Backup (In failure modes, this is a recovery.) job = bitmap_backup(drive0, 2, bsync2, "bitmap0", bsync_mode) vm.run_job(job, auto_dismiss=True, auto_finalize=False) bitmaps = query_bitmaps(vm) - bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0') log(bitmaps, indent=2) log('') - bitmap_comparison(bitmap, groups={} - if bsync_mode != 'never' - else dirty_groups) + if bsync_mode != 'never': + ebitmap.clear() + ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0')) log('--- Cleanup ---\n') vm.qmp_log("block-dirty-bitmap-remove",