From patchwork Fri Jul 17 08:14:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Shinkevich X-Patchwork-Id: 11669497 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DF179913 for ; Fri, 17 Jul 2020 08:18:44 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BF35920737 for ; Fri, 17 Jul 2020 08:18:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BF35920737 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=virtuozzo.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:53526 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jwLZs-0008SQ-1Z for patchwork-qemu-devel@patchwork.kernel.org; Fri, 17 Jul 2020 04:18:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60480) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWT-0001tU-Eo; Fri, 17 Jul 2020 04:15:14 -0400 Received: from relay.sw.ru ([185.231.240.75]:49982 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWK-0007JJ-HJ; Fri, 17 Jul 2020 04:15:13 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jwLWC-0003cP-Gn; Fri, 17 Jul 2020 11:14:56 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v11 01/11] qcow2: Fix capitalization of header extension constant. Date: Fri, 17 Jul 2020 11:14:49 +0300 Message-Id: <1594973699-781898-2-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/17 04:15:01 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Make the capitalization of the hexadecimal numbers consistent for the QCOW2 header extension constants in docs/interop/qcow2.txt. Suggested-by: Eric Blake Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- block/qcow2.c | 2 +- docs/interop/qcow2.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index fadf342..6ad6bdc 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -66,7 +66,7 @@ typedef struct { } QEMU_PACKED QCowExtension; #define QCOW2_EXT_MAGIC_END 0 -#define QCOW2_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA +#define QCOW2_EXT_MAGIC_BACKING_FORMAT 0xe2792aca #define QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857 #define QCOW2_EXT_MAGIC_CRYPTO_HEADER 0x0537be77 #define QCOW2_EXT_MAGIC_BITMAPS 0x23852875 diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index cb72346..f072e27 100644 --- a/docs/interop/qcow2.txt +++ b/docs/interop/qcow2.txt @@ -231,7 +231,7 @@ be stored. Each extension has a structure like the following: Byte 0 - 3: Header extension type: 0x00000000 - End of the header extension area - 0xE2792ACA - Backing file format name string + 0xe2792aca - Backing file format name string 0x6803f857 - Feature name table 0x23852875 - Bitmaps extension 0x0537be77 - Full disk encryption header pointer From patchwork Fri Jul 17 08:14:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Shinkevich X-Patchwork-Id: 11669491 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 21F08913 for ; Fri, 17 Jul 2020 08:18:04 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 024D920737 for ; Fri, 17 Jul 2020 08:18:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 024D920737 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=virtuozzo.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:50346 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jwLZD-00079t-9N for patchwork-qemu-devel@patchwork.kernel.org; Fri, 17 Jul 2020 04:18:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60420) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWN-0001qp-Qt; Fri, 17 Jul 2020 04:15:09 -0400 Received: from relay.sw.ru ([185.231.240.75]:49994 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWK-0007JP-Cs; Fri, 17 Jul 2020 04:15:07 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jwLWC-0003cP-Hu; Fri, 17 Jul 2020 11:14:56 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v11 02/11] qcow2_format.py: make printable data an extension class member Date: Fri, 17 Jul 2020 11:14:50 +0300 Message-Id: <1594973699-781898-3-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/17 04:15:01 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Let us differ binary data type from string one for the extension data variable and keep the string as the QcowHeaderExtension class member. Signed-off-by: Andrey Shinkevich Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index cc432e7..2f3681b 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -165,6 +165,13 @@ class QcowHeaderExtension(Qcow2Struct): self.data = fd.read(padded) assert self.data is not None + data_str = self.data[:self.length] + if all(c in string.printable.encode('ascii') for c in data_str): + data_str = f"'{ data_str.decode('ascii') }'" + else: + data_str = '' + self.data_str = data_str + if self.magic == QCOW2_EXT_MAGIC_BITMAPS: self.obj = Qcow2BitmapExt(data=self.data) else: @@ -174,12 +181,7 @@ class QcowHeaderExtension(Qcow2Struct): super().dump() if self.obj is None: - data = self.data[:self.length] - if all(c in string.printable.encode('ascii') for c in data): - data = f"'{ data.decode('ascii') }'" - else: - data = '' - print(f'{"data":<25} {data}') + print(f'{"data":<25} {self.data_str}') else: self.obj.dump() From patchwork Fri Jul 17 08:14:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Shinkevich X-Patchwork-Id: 11669513 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4AA3D913 for ; Fri, 17 Jul 2020 08:20:56 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2A3A82071A for ; Fri, 17 Jul 2020 08:20:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2A3A82071A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=virtuozzo.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:35152 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jwLbz-0004DQ-Ej for patchwork-qemu-devel@patchwork.kernel.org; Fri, 17 Jul 2020 04:20:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60504) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWU-0001tm-C8; Fri, 17 Jul 2020 04:15:14 -0400 Received: from relay.sw.ru ([185.231.240.75]:50006 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWK-0007JU-Ch; Fri, 17 Jul 2020 04:15:14 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jwLWC-0003cP-J5; Fri, 17 Jul 2020 11:14:56 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v11 03/11] qcow2_format.py: change Qcow2BitmapExt initialization method Date: Fri, 17 Jul 2020 11:14:51 +0300 Message-Id: <1594973699-781898-4-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/17 04:15:01 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" There are two ways to initialize a class derived from Qcow2Struct: 1. Pass a block of binary data to the constructor. 2. Pass the file descriptor to allow reading the file from constructor. Let's change the Qcow2BitmapExt initialization method from 1 to 2 to support a scattered reading in the initialization chain. The implementation comes with the patch that follows. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 36 ++++++++++++++++++++++-------------- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index 2f3681b..d4a9974 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -113,6 +113,11 @@ class Qcow2BitmapExt(Qcow2Struct): ('u64', '{:#x}', 'bitmap_directory_offset') ) + def __init__(self, fd): + super().__init__(fd=fd) + tail = struct.calcsize(self.fmt) % 8 + if tail: + fd.seek(8 - tail, 1) QCOW2_EXT_MAGIC_BITMAPS = 0x23852875 @@ -161,21 +166,24 @@ class QcowHeaderExtension(Qcow2Struct): else: assert all(v is None for v in (magic, length, data)) super().__init__(fd=fd) - padded = (self.length + 7) & ~7 - self.data = fd.read(padded) - assert self.data is not None - - data_str = self.data[:self.length] - if all(c in string.printable.encode('ascii') for c in data_str): - data_str = f"'{ data_str.decode('ascii') }'" - else: - data_str = '' - self.data_str = data_str + if self.magic == QCOW2_EXT_MAGIC_BITMAPS: + self.obj = Qcow2BitmapExt(fd=fd) + self.data = None + else: + padded = (self.length + 7) & ~7 + self.data = fd.read(padded) + assert self.data is not None + self.obj = None + + if self.data is not None: + data_str = self.data[:self.length] + if all(c in string.printable.encode( + 'ascii') for c in data_str): + data_str = f"'{ data_str.decode('ascii') }'" + else: + data_str = '' + self.data_str = data_str - if self.magic == QCOW2_EXT_MAGIC_BITMAPS: - self.obj = Qcow2BitmapExt(data=self.data) - else: - self.obj = None def dump(self): super().dump() From patchwork Fri Jul 17 08:14:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Shinkevich X-Patchwork-Id: 11669509 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3B4CB13A4 for ; Fri, 17 Jul 2020 08:20:40 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 188372071A for ; Fri, 17 Jul 2020 08:20:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 188372071A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=virtuozzo.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:34114 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jwLbj-0003mz-CC for patchwork-qemu-devel@patchwork.kernel.org; Fri, 17 Jul 2020 04:20:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60546) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWg-0002Mh-1f; Fri, 17 Jul 2020 04:15:26 -0400 Received: from relay.sw.ru ([185.231.240.75]:49968 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWe-0007JD-Go; Fri, 17 Jul 2020 04:15:25 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jwLWC-0003cP-KC; Fri, 17 Jul 2020 11:14:56 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v11 04/11] qcow2_format.py: dump bitmap flags in human readable way. Date: Fri, 17 Jul 2020 11:14:52 +0300 Message-Id: <1594973699-781898-5-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/17 04:15:01 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Introduce the class BitmapFlags that parses a bitmap flags mask. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index d4a9974..b447344 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -40,6 +40,22 @@ class Flags64(Qcow2Field): return str(bits) +class BitmapFlags(Qcow2Field): + + flags = { + 0x1: 'in-use', + 0x2: 'auto' + } + + def __str__(self): + bits = [] + for bit in range(64): + flag = self.value & (1 << bit) + if flag: + bits.append(self.flags.get(flag, f'bit-{bit}')) + return f'{self.value:#x} ({bits})' + + class Enum(Qcow2Field): def __str__(self): From patchwork Fri Jul 17 08:14:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Shinkevich X-Patchwork-Id: 11669495 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2F80B13B1 for ; Fri, 17 Jul 2020 08:18:32 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0F95820737 for ; Fri, 17 Jul 2020 08:18:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0F95820737 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=virtuozzo.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:52372 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jwLZf-0007yF-9v for patchwork-qemu-devel@patchwork.kernel.org; Fri, 17 Jul 2020 04:18:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60548) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWg-0002OP-Nx; Fri, 17 Jul 2020 04:15:26 -0400 Received: from relay.sw.ru ([185.231.240.75]:49986 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWe-0007JM-FR; Fri, 17 Jul 2020 04:15:26 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jwLWC-0003cP-Me; Fri, 17 Jul 2020 11:14:56 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v11 06/11] qcow2_format.py: pass cluster size to substructures Date: Fri, 17 Jul 2020 11:14:54 +0300 Message-Id: <1594973699-781898-7-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/17 04:15:01 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" The cluster size of an image is the QcowHeader class member and may be obtained by dependent extension structures such as Qcow2BitmapExt for further bitmap table details print. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index 05a8aa9..ca0d350 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -129,19 +129,21 @@ class Qcow2BitmapExt(Qcow2Struct): ('u64', '{:#x}', 'bitmap_directory_offset') ) - def __init__(self, fd): + def __init__(self, fd, cluster_size): super().__init__(fd=fd) tail = struct.calcsize(self.fmt) % 8 if tail: fd.seek(8 - tail, 1) position = fd.tell() + self.cluster_size = cluster_size self.read_bitmap_directory(fd) fd.seek(position) def read_bitmap_directory(self, fd): fd.seek(self.bitmap_directory_offset) self.bitmap_directory = \ - [Qcow2BitmapDirEntry(fd) for _ in range(self.nb_bitmaps)] + [Qcow2BitmapDirEntry(fd, cluster_size=self.cluster_size) + for _ in range(self.nb_bitmaps)] def dump(self): super().dump() @@ -162,8 +164,9 @@ class Qcow2BitmapDirEntry(Qcow2Struct): ('u32', '{}', 'extra_data_size') ) - def __init__(self, fd): + def __init__(self, fd, cluster_size): super().__init__(fd=fd) + self.cluster_size = cluster_size # Seek relative to the current position in the file fd.seek(self.extra_data_size, 1) bitmap_name = fd.read(self.name_size) @@ -203,11 +206,13 @@ class QcowHeaderExtension(Qcow2Struct): # then padding to next multiply of 8 ) - def __init__(self, magic=None, length=None, data=None, fd=None): + def __init__(self, magic=None, length=None, data=None, fd=None, + cluster_size=None): """ Support both loading from fd and creation from user data. For fd-based creation current position in a file will be used to read the data. + The cluster_size value may be obtained by dependent structures. This should be somehow refactored and functionality should be moved to superclass (to allow creation of any qcow2 struct), but then, fields @@ -230,7 +235,7 @@ class QcowHeaderExtension(Qcow2Struct): assert all(v is None for v in (magic, length, data)) super().__init__(fd=fd) if self.magic == QCOW2_EXT_MAGIC_BITMAPS: - self.obj = Qcow2BitmapExt(fd=fd) + self.obj = Qcow2BitmapExt(fd=fd, cluster_size=cluster_size) self.data = None else: padded = (self.length + 7) & ~7 @@ -319,7 +324,7 @@ class QcowHeader(Qcow2Struct): end = self.cluster_size while fd.tell() < end: - ext = QcowHeaderExtension(fd=fd) + ext = QcowHeaderExtension(fd=fd, cluster_size=self.cluster_size) if ext.magic == 0: break else: From patchwork Fri Jul 17 08:14:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Shinkevich X-Patchwork-Id: 11669501 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9C7DB913 for ; Fri, 17 Jul 2020 08:19:31 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7C28C20737 for ; Fri, 17 Jul 2020 08:19:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7C28C20737 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=virtuozzo.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:58130 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jwLac-0001xs-Pr for patchwork-qemu-devel@patchwork.kernel.org; Fri, 17 Jul 2020 04:19:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60452) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWQ-0001rn-Mm; Fri, 17 Jul 2020 04:15:11 -0400 Received: from relay.sw.ru ([185.231.240.75]:49990 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWK-0007JS-G1; Fri, 17 Jul 2020 04:15:10 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jwLWC-0003cP-Nr; Fri, 17 Jul 2020 11:14:56 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v11 07/11] qcow2_format.py: Dump bitmap table serialized entries Date: Fri, 17 Jul 2020 11:14:55 +0300 Message-Id: <1594973699-781898-8-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/17 04:15:01 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Add bitmap table information to the QCOW2 metadata dump. Bitmap name bitmap-1 ... Bitmap table type offset size 0 serialized 4718592 65536 1 serialized 4294967296 65536 2 serialized 5348033147437056 65536 3 serialized 13792273858822144 65536 4 serialized 4718592 65536 5 serialized 4294967296 65536 6 serialized 4503608217305088 65536 7 serialized 14073748835532800 65536 Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 41 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index ca0d350..ad1918c 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -175,6 +175,10 @@ class Qcow2BitmapDirEntry(Qcow2Struct): entry_raw_size = self.bitmap_dir_entry_raw_size() padding = ((entry_raw_size + 7) & ~7) - entry_raw_size fd.seek(padding, 1) + self.bitmap_table = Qcow2BitmapTable(fd=fd, + offset=self.bitmap_table_offset, + size=self.bitmap_table_size, + cluster_size=self.cluster_size) def bitmap_dir_entry_raw_size(self): return struct.calcsize(self.fmt) + self.name_size + \ @@ -183,6 +187,43 @@ class Qcow2BitmapDirEntry(Qcow2Struct): def dump(self): print(f'{"Bitmap name":<25} {self.name}') super(Qcow2BitmapDirEntry, self).dump() + self.bitmap_table.dump() + + +class Qcow2BitmapTableEntry: + + BME_TABLE_ENTRY_OFFSET_MASK = 0x00fffffffffffe00 + BME_TABLE_ENTRY_FLAG_ALL_ONES = 1 + + def __init__(self, entry): + self.offset = entry & self.BME_TABLE_ENTRY_OFFSET_MASK + if self.offset: + self.type = 'serialized' + elif entry & self.BME_TABLE_ENTRY_FLAG_ALL_ONES: + self.type = 'all-ones' + else: + self.type = 'all-zeroes' + + +class Qcow2BitmapTable: + + def __init__(self, fd, offset, size, cluster_size): + self.entries = [] + self.cluster_size = cluster_size + table_size = size * 8 + position = fd.tell() + fd.seek(offset) + table = [e[0] for e in struct.iter_unpack('>Q', fd.read(table_size))] + fd.seek(position) + for entry in table: + self.entries.append(Qcow2BitmapTableEntry(entry)) + + def dump(self): + size = self.cluster_size + bitmap_table = enumerate(self.entries) + print(f'{"Bitmap table":<14} {"type":<15} {"offset":<24} {"size"}') + for i, entry in bitmap_table: + print(f'{i:<14} {entry.type:<15} {entry.offset:<24} {size}') QCOW2_EXT_MAGIC_BITMAPS = 0x23852875 From patchwork Fri Jul 17 08:14:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Shinkevich X-Patchwork-Id: 11669487 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2B992913 for ; Fri, 17 Jul 2020 08:16:49 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0B71220737 for ; Fri, 17 Jul 2020 08:16:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0B71220737 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=virtuozzo.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:45162 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jwLY0-00050v-7b for patchwork-qemu-devel@patchwork.kernel.org; Fri, 17 Jul 2020 04:16:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60474) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWT-0001sq-2B; Fri, 17 Jul 2020 04:15:13 -0400 Received: from relay.sw.ru ([185.231.240.75]:49970 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWK-0007JF-CE; Fri, 17 Jul 2020 04:15:12 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jwLWC-0003cP-P7; Fri, 17 Jul 2020 11:14:56 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v11 08/11] qcow2.py: Introduce '-j' key to dump in JSON format Date: Fri, 17 Jul 2020 11:14:56 +0300 Message-Id: <1594973699-781898-9-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/17 04:15:01 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Add the command key to the qcow2.py arguments list to dump QCOW2 metadata in JSON format. Here is the suggested way to do that. The implementation of the dump in JSON format is in the patch that follows. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2.py | 18 ++++++++++++++---- tests/qemu-iotests/qcow2_format.py | 5 +++-- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/qcow2.py b/tests/qemu-iotests/qcow2.py index 0910e6a..77ca59c 100755 --- a/tests/qemu-iotests/qcow2.py +++ b/tests/qemu-iotests/qcow2.py @@ -26,16 +26,19 @@ from qcow2_format import ( ) +is_json = False + + def cmd_dump_header(fd): h = QcowHeader(fd) - h.dump() + h.dump(is_json) print() - h.dump_extensions() + h.dump_extensions(is_json) def cmd_dump_header_exts(fd): h = QcowHeader(fd) - h.dump_extensions() + h.dump_extensions(is_json) def cmd_set_header(fd, name, value): @@ -151,11 +154,14 @@ def main(filename, cmd, args): def usage(): - print("Usage: %s [, ...]" % sys.argv[0]) + print("Usage: %s [, ...] [, ...]" % sys.argv[0]) print("") print("Supported commands:") for name, handler, num_args, desc in cmds: print(" %-20s - %s" % (name, desc)) + print("") + print("Supported keys:") + print(" %-20s - %s" % ('-j', 'Dump in JSON format')) if __name__ == '__main__': @@ -163,4 +169,8 @@ if __name__ == '__main__': usage() sys.exit(1) + is_json = '-j' in sys.argv + if is_json: + sys.argv.remove('-j') + main(sys.argv[1], sys.argv[2], sys.argv[3:]) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index ad1918c..2921a27 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -109,7 +109,7 @@ class Qcow2Struct(metaclass=Qcow2StructMeta): self.__dict__ = dict((field[2], values[i]) for i, field in enumerate(self.fields)) - def dump(self): + def dump(self, is_json=False): for f in self.fields: value = self.__dict__[f[2]] if isinstance(f[1], str): @@ -147,6 +147,7 @@ class Qcow2BitmapExt(Qcow2Struct): def dump(self): super().dump() + for entry in self.bitmap_directory: print() entry.dump() @@ -399,7 +400,7 @@ class QcowHeader(Qcow2Struct): buf = buf[0:header_bytes-1] fd.write(buf) - def dump_extensions(self): + def dump_extensions(self, is_json=False): for ex in self.extensions: print('Header extension:') ex.dump() From patchwork Fri Jul 17 08:14:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Shinkevich X-Patchwork-Id: 11669493 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7BD2513B1 for ; Fri, 17 Jul 2020 08:18:10 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5C4EB20737 for ; Fri, 17 Jul 2020 08:18:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5C4EB20737 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=virtuozzo.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:50972 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jwLZJ-0007PP-Hr for patchwork-qemu-devel@patchwork.kernel.org; Fri, 17 Jul 2020 04:18:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60426) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWO-0001qt-02; Fri, 17 Jul 2020 04:15:09 -0400 Received: from relay.sw.ru ([185.231.240.75]:49998 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWK-0007JR-DW; Fri, 17 Jul 2020 04:15:07 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jwLWC-0003cP-QU; Fri, 17 Jul 2020 11:14:56 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v11 09/11] qcow2_format.py: collect fields to dump in JSON format Date: Fri, 17 Jul 2020 11:14:57 +0300 Message-Id: <1594973699-781898-10-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/17 04:15:01 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" As __dict__ is being extended with class members we do not want to print, add the to_dict() method to classes that returns a dictionary with desired fields and their values. Extend it in subclass when necessary to print the final dictionary in the JSON output which follows. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index 2921a27..19d29b8 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -119,6 +119,9 @@ class Qcow2Struct(metaclass=Qcow2StructMeta): print('{:<25} {}'.format(f[2], value_str)) + def to_dict(self): + return dict((f[2], self.__dict__[f[2]]) for f in self.fields) + class Qcow2BitmapExt(Qcow2Struct): @@ -152,6 +155,11 @@ class Qcow2BitmapExt(Qcow2Struct): print() entry.dump() + def to_dict(self): + fields_dict = super().to_dict() + fields_dict.update(bitmap_directory=self.bitmap_directory) + return fields_dict + class Qcow2BitmapDirEntry(Qcow2Struct): @@ -190,6 +198,13 @@ class Qcow2BitmapDirEntry(Qcow2Struct): super(Qcow2BitmapDirEntry, self).dump() self.bitmap_table.dump() + def to_dict(self): + fields_dict = super().to_dict() + fields_dict.update(bitmap_table=self.bitmap_table) + bmp_name = dict(name=self.name) + bme_dict = {**bmp_name, **fields_dict} + return bme_dict + class Qcow2BitmapTableEntry: @@ -205,6 +220,9 @@ class Qcow2BitmapTableEntry: else: self.type = 'all-zeroes' + def to_dict(self): + return dict(type=self.type, offset=self.offset) + class Qcow2BitmapTable: @@ -226,6 +244,9 @@ class Qcow2BitmapTable: for i, entry in bitmap_table: print(f'{i:<14} {entry.type:<15} {entry.offset:<24} {size}') + def to_dict(self): + return dict(entries=self.entries) + QCOW2_EXT_MAGIC_BITMAPS = 0x23852875 @@ -241,6 +262,9 @@ class QcowHeaderExtension(Qcow2Struct): 0x44415441: 'Data file' } + def to_dict(self): + return self.mapping.get(self.value, "") + fields = ( ('u32', Magic, 'magic'), ('u32', '{}', 'length') @@ -303,6 +327,17 @@ class QcowHeaderExtension(Qcow2Struct): else: self.obj.dump() + def to_dict(self): + fields_dict = super().to_dict() + ext_name = dict(name=self.Magic(self.magic)) + he_dict = {**ext_name, **fields_dict} + if self.obj is not None: + he_dict.update(data=self.obj) + else: + he_dict.update(data_str=self.data_str) + + return he_dict + @classmethod def create(cls, magic, data): return QcowHeaderExtension(magic, len(data), data) @@ -405,3 +440,6 @@ class QcowHeader(Qcow2Struct): print('Header extension:') ex.dump() print() + + def to_dict(self): + return super().to_dict() From patchwork Fri Jul 17 08:14:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Shinkevich X-Patchwork-Id: 11669485 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 831E213B1 for ; Fri, 17 Jul 2020 08:16:29 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 62B1820737 for ; Fri, 17 Jul 2020 08:16:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 62B1820737 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=virtuozzo.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:43288 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jwLXg-0004F6-D7 for patchwork-qemu-devel@patchwork.kernel.org; Fri, 17 Jul 2020 04:16:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60458) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWQ-0001rs-Uv; Fri, 17 Jul 2020 04:15:11 -0400 Received: from relay.sw.ru ([185.231.240.75]:49980 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWK-0007JH-GM; Fri, 17 Jul 2020 04:15:10 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jwLWC-0003cP-SP; Fri, 17 Jul 2020 11:14:56 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v11 10/11] qcow2_format.py: introduce Qcow2HeaderExtensionsDoc class Date: Fri, 17 Jul 2020 11:14:58 +0300 Message-Id: <1594973699-781898-11-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/17 04:15:01 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Per original script design, QcowHeader class may dump the QCOW2 header info separately from the QCOW2 extensions info. To implement the to_dict() method for dumping extensions, let us introduce the class Qcow2HeaderExtensionsDoc. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index 19d29b8..d2a8659 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -248,6 +248,15 @@ class Qcow2BitmapTable: return dict(entries=self.entries) +class Qcow2HeaderExtensionsDoc: + + def __init__(self, extensions): + self.extensions = extensions + + def to_dict(self): + return dict(Header_extensions=self.extensions) + + QCOW2_EXT_MAGIC_BITMAPS = 0x23852875 From patchwork Fri Jul 17 08:14:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Shinkevich X-Patchwork-Id: 11669503 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 27E33913 for ; Fri, 17 Jul 2020 08:19:40 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 077F32074B for ; Fri, 17 Jul 2020 08:19:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 077F32074B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=virtuozzo.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:58450 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jwLal-00027A-9e for patchwork-qemu-devel@patchwork.kernel.org; Fri, 17 Jul 2020 04:19:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60472) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWR-0001sk-Lt; Fri, 17 Jul 2020 04:15:13 -0400 Received: from relay.sw.ru ([185.231.240.75]:49972 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jwLWK-0007JE-Ei; Fri, 17 Jul 2020 04:15:11 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jwLWC-0003cP-Tj; Fri, 17 Jul 2020 11:14:56 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v11 11/11] qcow2_format.py: support dumping metadata in JSON format Date: Fri, 17 Jul 2020 11:14:59 +0300 Message-Id: <1594973699-781898-12-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1594973699-781898-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/17 04:15:01 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Implementation of dumping QCOW2 image metadata. The sample output: { "Header_extensions": [ { "name": "Feature table", "magic": 1745090647, "length": 192, "data_str": "" }, { "name": "Bitmaps", "magic": 595929205, "length": 24, "data": { "nb_bitmaps": 2, "reserved32": 0, "bitmap_directory_size": 64, "bitmap_directory_offset": 1048576, "bitmap_directory": [ { "name": "bitmap-1", "bitmap_table_offset": 589824, "bitmap_table_size": 1, "flags": 2, "type": 1, "granularity_bits": 15, "name_size": 8, "extra_data_size": 0, "bitmap_table": { "entries": [ { "type": "serialized", "offset": 655360 }, ... Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py index d2a8659..d40eb49 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -19,6 +19,15 @@ import struct import string +import json + + +class ComplexEncoder(json.JSONEncoder): + def default(self, obj): + if hasattr(obj, 'to_dict'): + return obj.to_dict() + else: + return json.JSONEncoder.default(self, obj) class Qcow2Field: @@ -110,6 +119,11 @@ class Qcow2Struct(metaclass=Qcow2StructMeta): for i, field in enumerate(self.fields)) def dump(self, is_json=False): + if is_json: + print(json.dumps(self.to_dict(), indent=4, + cls=ComplexEncoder)) + return + for f in self.fields: value = self.__dict__[f[2]] if isinstance(f[1], str): @@ -445,6 +459,12 @@ class QcowHeader(Qcow2Struct): fd.write(buf) def dump_extensions(self, is_json=False): + if is_json: + ext_doc = Qcow2HeaderExtensionsDoc(self.extensions) + print(json.dumps(ext_doc.to_dict(), indent=4, + cls=ComplexEncoder)) + return + for ex in self.extensions: print('Header extension:') ex.dump()