From patchwork Wed Oct 16 19:24:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 11193981 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 65D9B139A for ; Wed, 16 Oct 2019 19:26:12 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 45F4A2053B for ; Wed, 16 Oct 2019 19:26:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 45F4A2053B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:47416 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKovz-00039V-5k for patchwork-qemu-devel@patchwork.kernel.org; Wed, 16 Oct 2019 15:26:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:32918) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKoug-0001Fo-J9 for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:24:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iKoue-0002Lx-Jl for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:24:50 -0400 Received: from mx1.redhat.com ([209.132.183.28]:41818) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iKouZ-0002GC-7r; Wed, 16 Oct 2019 15:24:44 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 1608481F0C; Wed, 16 Oct 2019 19:24:39 +0000 (UTC) Received: from localhost (ovpn-116-20.phx2.redhat.com [10.3.116.20]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4FE9060127; Wed, 16 Oct 2019 19:24:36 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Subject: [PATCH 01/10] image-fuzzer: Open image files in binary mode Date: Wed, 16 Oct 2019 16:24:21 -0300 Message-Id: <20191016192430.25098-2-ehabkost@redhat.com> In-Reply-To: <20191016192430.25098-1-ehabkost@redhat.com> References: <20191016192430.25098-1-ehabkost@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Wed, 16 Oct 2019 19:24:39 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Stefan Hajnoczi , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" This probably never caused problems because on Linux there's no actual newline conversion happening, but on Python 3 the binary/text distinction is stronger and we must explicitly open the image file in binary mode. Signed-off-by: Eduardo Habkost Reviewed-by: Philippe Mathieu-Daudé --- tests/image-fuzzer/qcow2/layout.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py index 675877da96..c57418fa15 100644 --- a/tests/image-fuzzer/qcow2/layout.py +++ b/tests/image-fuzzer/qcow2/layout.py @@ -503,7 +503,7 @@ class Image(object): def write(self, filename): """Write an entire image to the file.""" - image_file = open(filename, 'w') + image_file = open(filename, 'wb') for field in self: image_file.seek(field.offset) image_file.write(struct.pack(field.fmt, field.value)) From patchwork Wed Oct 16 19:24:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 11193983 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A44B7139A for ; Wed, 16 Oct 2019 19:26:14 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 84D3C2053B for ; Wed, 16 Oct 2019 19:26:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 84D3C2053B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:47418 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKow1-0003Dq-Lw for patchwork-qemu-devel@patchwork.kernel.org; Wed, 16 Oct 2019 15:26:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:32936) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKoui-0001GR-Jp for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:24:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iKoug-0002Mv-JN for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:24:52 -0400 Received: from mx1.redhat.com ([209.132.183.28]:54324) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iKoua-0002Ix-Nd; Wed, 16 Oct 2019 15:24:46 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6EAA918C890A; Wed, 16 Oct 2019 19:24:43 +0000 (UTC) Received: from localhost (ovpn-116-20.phx2.redhat.com [10.3.116.20]) by smtp.corp.redhat.com (Postfix) with ESMTP id B6FA319C6A; Wed, 16 Oct 2019 19:24:40 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Subject: [PATCH 02/10] image-fuzzer: Write bytes instead of string to image file Date: Wed, 16 Oct 2019 16:24:22 -0300 Message-Id: <20191016192430.25098-3-ehabkost@redhat.com> In-Reply-To: <20191016192430.25098-1-ehabkost@redhat.com> References: <20191016192430.25098-1-ehabkost@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.redhat.com [10.5.110.70]); Wed, 16 Oct 2019 19:24:43 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Stefan Hajnoczi , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" This is necessary for Python 3 compatibility. Signed-off-by: Eduardo Habkost Reviewed-by: Philippe Mathieu-Daudé --- tests/image-fuzzer/qcow2/layout.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py index c57418fa15..fe273d4143 100644 --- a/tests/image-fuzzer/qcow2/layout.py +++ b/tests/image-fuzzer/qcow2/layout.py @@ -518,7 +518,7 @@ class Image(object): rounded = (size + self.cluster_size - 1) & ~(self.cluster_size - 1) if rounded > size: image_file.seek(rounded - 1) - image_file.write("\0") + image_file.write(b'\x00') image_file.close() @staticmethod From patchwork Wed Oct 16 19:24:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 11193985 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AD5EB139A for ; Wed, 16 Oct 2019 19:28:06 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8C43E2053B for ; Wed, 16 Oct 2019 19:28:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8C43E2053B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:47436 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKoxp-0005ju-Ki for patchwork-qemu-devel@patchwork.kernel.org; Wed, 16 Oct 2019 15:28:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:32973) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKoum-0001Mn-3A for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:24:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iKouk-0002PS-Ak for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:24:55 -0400 Received: from mx1.redhat.com ([209.132.183.28]:56822) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iKoue-0002Lg-Js; Wed, 16 Oct 2019 15:24:50 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id DBE8889F302; Wed, 16 Oct 2019 19:24:47 +0000 (UTC) Received: from localhost (ovpn-116-20.phx2.redhat.com [10.3.116.20]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1567F60852; Wed, 16 Oct 2019 19:24:44 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Subject: [PATCH 03/10] image-fuzzer: Explicitly use integer division operator Date: Wed, 16 Oct 2019 16:24:23 -0300 Message-Id: <20191016192430.25098-4-ehabkost@redhat.com> In-Reply-To: <20191016192430.25098-1-ehabkost@redhat.com> References: <20191016192430.25098-1-ehabkost@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.redhat.com [10.5.110.68]); Wed, 16 Oct 2019 19:24:47 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Stefan Hajnoczi , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Most of the division expressions in image-fuzzer assume integer division. Use the // operator to keep the same behavior when we move to Python 3. Signed-off-by: Eduardo Habkost --- tests/image-fuzzer/qcow2/fuzz.py | 12 ++++----- tests/image-fuzzer/qcow2/layout.py | 40 +++++++++++++++--------------- 2 files changed, 26 insertions(+), 26 deletions(-) diff --git a/tests/image-fuzzer/qcow2/fuzz.py b/tests/image-fuzzer/qcow2/fuzz.py index abc4f0635d..154dc06cc0 100644 --- a/tests/image-fuzzer/qcow2/fuzz.py +++ b/tests/image-fuzzer/qcow2/fuzz.py @@ -27,14 +27,14 @@ UINT64 = 0xffffffffffffffff UINT32_M = 31 UINT64_M = 63 # Fuzz vectors -UINT8_V = [0, 0x10, UINT8/4, UINT8/2 - 1, UINT8/2, UINT8/2 + 1, UINT8 - 1, +UINT8_V = [0, 0x10, UINT8//4, UINT8//2 - 1, UINT8//2, UINT8//2 + 1, UINT8 - 1, UINT8] -UINT16_V = [0, 0x100, 0x1000, UINT16/4, UINT16/2 - 1, UINT16/2, UINT16/2 + 1, +UINT16_V = [0, 0x100, 0x1000, UINT16//4, UINT16//2 - 1, UINT16//2, UINT16//2 + 1, UINT16 - 1, UINT16] -UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32/4, UINT32/2 - 1, - UINT32/2, UINT32/2 + 1, UINT32 - 1, UINT32] -UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64/4, - UINT64/2 - 1, UINT64/2, UINT64/2 + 1, UINT64 - 1, +UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32//4, UINT32//2 - 1, + UINT32//2, UINT32//2 + 1, UINT32 - 1, UINT32] +UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64//4, + UINT64//2 - 1, UINT64//2, UINT64//2 + 1, UINT64 - 1, UINT64] STRING_V = ['%s%p%x%d', '.1024d', '%.2049d', '%p%p%p%p', '%x%x%x%x', '%d%d%d%d', '%s%s%s%s', '%99999999999s', '%08x', '%%20d', '%%20n', diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py index fe273d4143..6501c9fd4b 100644 --- a/tests/image-fuzzer/qcow2/layout.py +++ b/tests/image-fuzzer/qcow2/layout.py @@ -253,7 +253,7 @@ class Image(object): ['>I', self.ext_offset, 0x6803f857, 'ext_magic'], # One feature table contains 3 fields and takes 48 bytes ['>I', self.ext_offset + UINT32_S, - len(feature_tables) / 3 * 48, 'ext_length'] + len(feature_tables) // 3 * 48, 'ext_length'] ] + feature_tables) self.ext_offset = inner_offset @@ -271,7 +271,7 @@ class Image(object): def create_l2_entry(host, guest, l2_cluster): """Generate one L2 entry.""" offset = l2_cluster * self.cluster_size - l2_size = self.cluster_size / UINT64_S + l2_size = self.cluster_size // UINT64_S entry_offset = offset + UINT64_S * (guest % l2_size) cluster_descriptor = host * self.cluster_size if not self.header['version'][0].value == 2: @@ -283,8 +283,8 @@ class Image(object): def create_l1_entry(l2_cluster, l1_offset, guest): """Generate one L1 entry.""" - l2_size = self.cluster_size / UINT64_S - entry_offset = l1_offset + UINT64_S * (guest / l2_size) + l2_size = self.cluster_size // UINT64_S + entry_offset = l1_offset + UINT64_S * (guest // l2_size) # While snapshots are not supported bit #63 = 1 entry_val = (1 << 63) + l2_cluster * self.cluster_size return ['>Q', entry_offset, entry_val, 'l1_entry'] @@ -298,11 +298,11 @@ class Image(object): l2 = [] else: meta_data = self._get_metadata() - guest_clusters = random.sample(range(self.image_size / + guest_clusters = random.sample(range(self.image_size // self.cluster_size), len(self.data_clusters)) # Number of entries in a L1/L2 table - l_size = self.cluster_size / UINT64_S + l_size = self.cluster_size // UINT64_S # Number of clusters necessary for L1 table l1_size = int(ceil((max(guest_clusters) + 1) / float(l_size**2))) l1_start = self._get_adjacent_clusters(self.data_clusters | @@ -318,7 +318,7 @@ class Image(object): # L2 entries l2 = [] for host, guest in zip(self.data_clusters, guest_clusters): - l2_id = guest / l_size + l2_id = guest // l_size if l2_id not in l2_ids: l2_ids.append(l2_id) l2_clusters.append(self._get_adjacent_clusters( @@ -339,14 +339,14 @@ class Image(object): def allocate_rfc_blocks(data, size): """Return indices of clusters allocated for refcount blocks.""" cluster_ids = set() - diff = block_ids = set([x / size for x in data]) + diff = block_ids = set([x // size for x in data]) while len(diff) != 0: # Allocate all yet not allocated clusters new = self._get_available_clusters(data | cluster_ids, len(diff)) # Indices of new refcount blocks necessary to cover clusters # in 'new' - diff = set([x / size for x in new]) - block_ids + diff = set([x // size for x in new]) - block_ids cluster_ids |= new block_ids |= diff return cluster_ids, block_ids @@ -359,7 +359,7 @@ class Image(object): blocks = set(init_blocks) clusters = set() # Number of entries in one cluster of the refcount table - size = self.cluster_size / UINT64_S + size = self.cluster_size // UINT64_S # Number of clusters necessary for the refcount table based on # the current number of refcount blocks table_size = int(ceil((max(blocks) + 1) / float(size))) @@ -373,7 +373,7 @@ class Image(object): table_size + 1)) # New refcount blocks necessary for clusters occupied by the # refcount table - diff = set([c / block_size for c in table_clusters]) - blocks + diff = set([c // block_size for c in table_clusters]) - blocks blocks |= diff while len(diff) != 0: # Allocate clusters for new refcount blocks @@ -382,12 +382,12 @@ class Image(object): len(diff)) # Indices of new refcount blocks necessary to cover # clusters in 'new' - diff = set([x / block_size for x in new]) - blocks + diff = set([x // block_size for x in new]) - blocks clusters |= new blocks |= diff # Check if the refcount table needs one more cluster if int(ceil((max(blocks) + 1) / float(size))) > table_size: - new_block_id = (table_start + table_size) / block_size + new_block_id = (table_start + table_size) // block_size # Check if the additional table cluster needs # one more refcount block if new_block_id not in blocks: @@ -399,13 +399,13 @@ class Image(object): def create_table_entry(table_offset, block_cluster, block_size, cluster): """Generate a refcount table entry.""" - offset = table_offset + UINT64_S * (cluster / block_size) + offset = table_offset + UINT64_S * (cluster // block_size) return ['>Q', offset, block_cluster * self.cluster_size, 'refcount_table_entry'] def create_block_entry(block_cluster, block_size, cluster): """Generate a list of entries for the current block.""" - entry_size = self.cluster_size / block_size + entry_size = self.cluster_size // block_size offset = block_cluster * self.cluster_size entry_offset = offset + entry_size * (cluster % block_size) # While snapshots are not supported all refcounts are set to 1 @@ -415,7 +415,7 @@ class Image(object): # Number of refcount entries per refcount block # Convert self.cluster_size from bytes to bits to have the same # base for the numerator and denominator - block_size = self.cluster_size * 8 / refcount_bits + block_size = self.cluster_size * 8 // refcount_bits meta_data = self._get_metadata() if len(self.data_clusters) == 0: # All metadata for an empty guest image needs 4 clusters: @@ -452,8 +452,8 @@ class Image(object): rfc_blocks = [] for cluster in sorted(self.data_clusters | meta_data): - if cluster / block_size != block_id: - block_id = cluster / block_size + if cluster // block_size != block_id: + block_id = cluster // block_size block_cluster = block_clusters[block_ids.index(block_id)] rfc_table.append(create_table_entry(table_offset, block_cluster, @@ -587,7 +587,7 @@ class Image(object): def _alloc_data(img_size, cluster_size): """Return a set of random indices of clusters allocated for guest data. """ - num_of_cls = img_size/cluster_size + num_of_cls = img_size // cluster_size return set(random.sample(range(1, num_of_cls + 1), random.randint(0, num_of_cls))) @@ -595,7 +595,7 @@ class Image(object): """Return indices of clusters allocated for image metadata.""" ids = set() for x in self: - ids.add(x.offset/self.cluster_size) + ids.add(x.offset // self.cluster_size) return ids From patchwork Wed Oct 16 19:24:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 11194017 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 53C86139A for ; Wed, 16 Oct 2019 19:30:08 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 32B4E2053B for ; Wed, 16 Oct 2019 19:30:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 32B4E2053B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:47458 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKozn-0007qM-Cy for patchwork-qemu-devel@patchwork.kernel.org; Wed, 16 Oct 2019 15:30:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33007) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKoup-0001S3-5d for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:25:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iKoun-0002Rf-Ur for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:24:58 -0400 Received: from mx1.redhat.com ([209.132.183.28]:38570) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iKoul-0002Pa-4V; Wed, 16 Oct 2019 15:24:55 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5FF0B30860BE; Wed, 16 Oct 2019 19:24:54 +0000 (UTC) Received: from localhost (ovpn-116-20.phx2.redhat.com [10.3.116.20]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8021C5D9DC; Wed, 16 Oct 2019 19:24:49 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Subject: [PATCH 04/10] image-fuzzer: Use io.StringIO Date: Wed, 16 Oct 2019 16:24:24 -0300 Message-Id: <20191016192430.25098-5-ehabkost@redhat.com> In-Reply-To: <20191016192430.25098-1-ehabkost@redhat.com> References: <20191016192430.25098-1-ehabkost@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.44]); Wed, 16 Oct 2019 19:24:54 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Stefan Hajnoczi , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" StringIO.StringIO is not available on Python 3, but io.StringIO is available on both Python 2 and 3. io.StringIO is slightly different from the Python 2 StringIO module, though, so we need bytes coming from subprocess.Popen() to be explicitly decoded. Signed-off-by: Eduardo Habkost --- tests/image-fuzzer/runner.py | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py index 95d84f38f3..94cab5bd93 100755 --- a/tests/image-fuzzer/runner.py +++ b/tests/image-fuzzer/runner.py @@ -28,7 +28,7 @@ import shutil from itertools import count import time import getopt -import StringIO +import io import resource try: @@ -84,8 +84,12 @@ def run_app(fd, q_args): try: out, err = process.communicate() signal.alarm(0) - fd.write(out) - fd.write(err) + # fd is a text file, so we need to decode the process output before + # writing to it. + # We could be simply using the `errors` parameter of subprocess.Popen(), + # but this will be possible only after migrating to Python 3 + fd.write(out.decode(errors='replace')) + fd.write(err.decode(errors='replace')) fd.flush() return process.returncode @@ -183,7 +187,7 @@ class TestEnv(object): MAX_BACKING_FILE_SIZE) * (1 << 20) cmd = self.qemu_img + ['create', '-f', backing_file_fmt, backing_file_name, str(backing_file_size)] - temp_log = StringIO.StringIO() + temp_log = io.StringIO() retcode = run_app(temp_log, cmd) if retcode == 0: temp_log.close() @@ -240,7 +244,7 @@ class TestEnv(object): "Backing file: %s\n" \ % (self.seed, " ".join(current_cmd), self.current_dir, backing_file_name) - temp_log = StringIO.StringIO() + temp_log = io.StringIO() try: retcode = run_app(temp_log, current_cmd) except OSError as e: From patchwork Wed Oct 16 19:24:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 11194021 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 63B8114ED for ; Wed, 16 Oct 2019 19:32:35 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 40AD92064B for ; Wed, 16 Oct 2019 19:32:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 40AD92064B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:47484 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKp2A-0000qe-GO for patchwork-qemu-devel@patchwork.kernel.org; Wed, 16 Oct 2019 15:32:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33020) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKouq-0001UT-I3 for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:25:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iKoup-0002Sa-Fw for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:25:00 -0400 Received: from mx1.redhat.com ([209.132.183.28]:34688) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iKoun-0002Qv-7Y; Wed, 16 Oct 2019 15:24:57 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 74C0010DCC94; Wed, 16 Oct 2019 19:24:56 +0000 (UTC) Received: from localhost (ovpn-116-20.phx2.redhat.com [10.3.116.20]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1515C60852; Wed, 16 Oct 2019 19:24:55 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Subject: [PATCH 05/10] image-fuzzer: Use %r for all fiels at Field.__repr__() Date: Wed, 16 Oct 2019 16:24:25 -0300 Message-Id: <20191016192430.25098-6-ehabkost@redhat.com> In-Reply-To: <20191016192430.25098-1-ehabkost@redhat.com> References: <20191016192430.25098-1-ehabkost@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.redhat.com [10.5.110.64]); Wed, 16 Oct 2019 19:24:56 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Stefan Hajnoczi , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" This makes the formatting code simpler, and safer if we change the type of self.value from str to bytes. Signed-off-by: Eduardo Habkost --- tests/image-fuzzer/qcow2/layout.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py index 6501c9fd4b..0adcbd448d 100644 --- a/tests/image-fuzzer/qcow2/layout.py +++ b/tests/image-fuzzer/qcow2/layout.py @@ -53,8 +53,8 @@ class Field(object): return iter([self.fmt, self.offset, self.value, self.name]) def __repr__(self): - return "Field(fmt='%s', offset=%d, value=%s, name=%s)" % \ - (self.fmt, self.offset, str(self.value), self.name) + return "Field(fmt=%r, offset=%r, value=%r, name=%r)" % \ + (self.fmt, self.offset, self.value, self.name) class FieldsList(object): From patchwork Wed Oct 16 19:24:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 11194023 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4D958139A for ; Wed, 16 Oct 2019 19:33:39 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2D7462064B for ; Wed, 16 Oct 2019 19:33:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2D7462064B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:47494 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKp3C-0001ng-CD for patchwork-qemu-devel@patchwork.kernel.org; Wed, 16 Oct 2019 15:33:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33052) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKouy-0001jy-FN for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:25:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iKoux-0002W5-2i for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:25:08 -0400 Received: from mx1.redhat.com ([209.132.183.28]:20634) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iKouu-0002UI-8Z; Wed, 16 Oct 2019 15:25:04 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7D08F877A61; Wed, 16 Oct 2019 19:25:03 +0000 (UTC) Received: from localhost (ovpn-116-20.phx2.redhat.com [10.3.116.20]) by smtp.corp.redhat.com (Postfix) with ESMTP id 179145C1D4; Wed, 16 Oct 2019 19:24:57 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Subject: [PATCH 06/10] image-fuzzer: Return bytes objects on string fuzzing functions Date: Wed, 16 Oct 2019 16:24:26 -0300 Message-Id: <20191016192430.25098-7-ehabkost@redhat.com> In-Reply-To: <20191016192430.25098-1-ehabkost@redhat.com> References: <20191016192430.25098-1-ehabkost@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.redhat.com [10.5.110.69]); Wed, 16 Oct 2019 19:25:03 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Stefan Hajnoczi , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" No caller of fuzzer functions is interested in unicode string values, so replace them with bytes sequences. Signed-off-by: Eduardo Habkost --- tests/image-fuzzer/qcow2/fuzz.py | 42 ++++++++++++++++---------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/tests/image-fuzzer/qcow2/fuzz.py b/tests/image-fuzzer/qcow2/fuzz.py index 154dc06cc0..c58bf11005 100644 --- a/tests/image-fuzzer/qcow2/fuzz.py +++ b/tests/image-fuzzer/qcow2/fuzz.py @@ -36,11 +36,11 @@ UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32//4, UINT32//2 - 1, UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64//4, UINT64//2 - 1, UINT64//2, UINT64//2 + 1, UINT64 - 1, UINT64] -STRING_V = ['%s%p%x%d', '.1024d', '%.2049d', '%p%p%p%p', '%x%x%x%x', - '%d%d%d%d', '%s%s%s%s', '%99999999999s', '%08x', '%%20d', '%%20n', - '%%20x', '%%20s', '%s%s%s%s%s%s%s%s%s%s', '%p%p%p%p%p%p%p%p%p%p', - '%#0123456x%08x%x%s%p%d%n%o%u%c%h%l%q%j%z%Z%t%i%e%g%f%a%C%S%08x%%', - '%s x 129', '%x x 257'] +BYTES_V = [b'%s%p%x%d', b'.1024d', b'%.2049d', b'%p%p%p%p', b'%x%x%x%x', + b'%d%d%d%d', b'%s%s%s%s', b'%99999999999s', b'%08x', b'%%20d', b'%%20n', + b'%%20x', b'%%20s', b'%s%s%s%s%s%s%s%s%s%s', b'%p%p%p%p%p%p%p%p%p%p', + b'%#0123456x%08x%x%s%p%d%n%o%u%c%h%l%q%j%z%Z%t%i%e%g%f%a%C%S%08x%%', + b'%s x 129', b'%x x 257'] def random_from_intervals(intervals): @@ -76,12 +76,12 @@ def random_bits(bit_ranges): return val -def truncate_string(strings, length): - """Return strings truncated to specified length.""" - if type(strings) == list: - return [s[:length] for s in strings] +def truncate_bytes(sequences, length): + """Return sequences truncated to specified length.""" + if type(sequences) == list: + return [s[:length] for s in sequences] else: - return strings[:length] + return sequences[:length] def validator(current, pick, choices): @@ -110,12 +110,12 @@ def bit_validator(current, bit_ranges): return validator(current, random_bits, bit_ranges) -def string_validator(current, strings): - """Return a random string value from the list not equal to the current. +def bytes_validator(current, sequences): + """Return a random bytes value from the list not equal to the current. This function is useful for selection from valid values except current one. """ - return validator(current, random.choice, strings) + return validator(current, random.choice, sequences) def selector(current, constraints, validate=int_validator): @@ -283,9 +283,9 @@ def header_length(current): def bf_name(current): """Fuzz the backing file name.""" constraints = [ - truncate_string(STRING_V, len(current)) + truncate_bytes(BYTES_V, len(current)) ] - return selector(current, constraints, string_validator) + return selector(current, constraints, bytes_validator) def ext_magic(current): @@ -303,10 +303,10 @@ def ext_length(current): def bf_format(current): """Fuzz backing file format in the corresponding header extension.""" constraints = [ - truncate_string(STRING_V, len(current)), - truncate_string(STRING_V, (len(current) + 7) & ~7) # Fuzz padding + truncate_bytes(BYTES_V, len(current)), + truncate_bytes(BYTES_V, (len(current) + 7) & ~7) # Fuzz padding ] - return selector(current, constraints, string_validator) + return selector(current, constraints, bytes_validator) def feature_type(current): @@ -324,10 +324,10 @@ def feature_bit_number(current): def feature_name(current): """Fuzz feature name field of a feature name table header extension.""" constraints = [ - truncate_string(STRING_V, len(current)), - truncate_string(STRING_V, 46) # Fuzz padding (field length = 46) + truncate_bytes(BYTES_V, len(current)), + truncate_bytes(BYTES_V, 46) # Fuzz padding (field length = 46) ] - return selector(current, constraints, string_validator) + return selector(current, constraints, bytes_validator) def l1_entry(current): From patchwork Wed Oct 16 19:24:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 11194025 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EC6D3139A for ; Wed, 16 Oct 2019 19:34:54 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CBDC72064B for ; Wed, 16 Oct 2019 19:34:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CBDC72064B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:47500 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKp4Q-0002qc-2R for patchwork-qemu-devel@patchwork.kernel.org; Wed, 16 Oct 2019 15:34:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33088) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKov6-0001wN-4N for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:25:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iKov5-0002bH-0C for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:25:15 -0400 Received: from mx1.redhat.com ([209.132.183.28]:51722) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iKov2-0002YE-Kj; Wed, 16 Oct 2019 15:25:12 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D88343688E; Wed, 16 Oct 2019 19:25:11 +0000 (UTC) Received: from localhost (ovpn-116-20.phx2.redhat.com [10.3.116.20]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2763B60852; Wed, 16 Oct 2019 19:25:04 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Subject: [PATCH 07/10] image-fuzzer: Use bytes constant for field values Date: Wed, 16 Oct 2019 16:24:27 -0300 Message-Id: <20191016192430.25098-8-ehabkost@redhat.com> In-Reply-To: <20191016192430.25098-1-ehabkost@redhat.com> References: <20191016192430.25098-1-ehabkost@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Wed, 16 Oct 2019 19:25:11 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Stefan Hajnoczi , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Field values are supposed to be bytes objects, not unicode strings. Change two constants that were declared as strings. Signed-off-by: Eduardo Habkost Reviewed-by: Philippe Mathieu-Daudé --- tests/image-fuzzer/qcow2/layout.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py index 0adcbd448d..a0fd53c7ad 100644 --- a/tests/image-fuzzer/qcow2/layout.py +++ b/tests/image-fuzzer/qcow2/layout.py @@ -122,7 +122,7 @@ class Image(object): def create_header(self, cluster_bits, backing_file_name=None): """Generate a random valid header.""" meta_header = [ - ['>4s', 0, "QFI\xfb", 'magic'], + ['>4s', 0, b"QFI\xfb", 'magic'], ['>I', 4, random.randint(2, 3), 'version'], ['>Q', 8, 0, 'backing_file_offset'], ['>I', 16, 0, 'backing_file_size'], @@ -231,7 +231,7 @@ class Image(object): feature_tables = [] feature_ids = [] inner_offset = self.ext_offset + ext_header_len - feat_name = 'some cool feature' + feat_name = b'some cool feature' while len(feature_tables) < num_fnt_entries * 3: feat_type, feat_bit = gen_feat_ids() # Remove duplicates From patchwork Wed Oct 16 19:24:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 11194027 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B90BC1390 for ; Wed, 16 Oct 2019 19:35:54 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 990B4218DE for ; Wed, 16 Oct 2019 19:35:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 990B4218DE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:47510 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKp5N-00041m-Rh for patchwork-qemu-devel@patchwork.kernel.org; Wed, 16 Oct 2019 15:35:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33104) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKov8-00020C-6V for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:25:20 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iKov6-0002e2-Qb for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:25:18 -0400 Received: from mx1.redhat.com ([209.132.183.28]:43990) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iKov4-0002ZV-MN; Wed, 16 Oct 2019 15:25:14 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E894D87633; Wed, 16 Oct 2019 19:25:13 +0000 (UTC) Received: from localhost (ovpn-116-20.phx2.redhat.com [10.3.116.20]) by smtp.corp.redhat.com (Postfix) with ESMTP id 871ED60852; Wed, 16 Oct 2019 19:25:13 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Subject: [PATCH 08/10] image-fuzzer: Encode file name and file format to bytes Date: Wed, 16 Oct 2019 16:24:28 -0300 Message-Id: <20191016192430.25098-9-ehabkost@redhat.com> In-Reply-To: <20191016192430.25098-1-ehabkost@redhat.com> References: <20191016192430.25098-1-ehabkost@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Wed, 16 Oct 2019 19:25:13 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Stefan Hajnoczi , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Callers of create_image() will pass strings as arguments, but the Image class will expect bytes objects to be provided. Encode them inside create_image(). Signed-off-by: Eduardo Habkost --- tests/image-fuzzer/qcow2/layout.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py index a0fd53c7ad..01bff4d05e 100644 --- a/tests/image-fuzzer/qcow2/layout.py +++ b/tests/image-fuzzer/qcow2/layout.py @@ -602,8 +602,8 @@ class Image(object): def create_image(test_img_path, backing_file_name=None, backing_file_fmt=None, fields_to_fuzz=None): """Create a fuzzed image and write it to the specified file.""" - image = Image(backing_file_name) - image.set_backing_file_format(backing_file_fmt) + image = Image(backing_file_name.encode()) + image.set_backing_file_format(backing_file_fmt.encode()) image.create_feature_name_table() image.set_end_of_extension_area() image.create_l_structures() From patchwork Wed Oct 16 19:24:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 11193989 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D5854139A for ; Wed, 16 Oct 2019 19:28:29 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B58F721A49 for ; Wed, 16 Oct 2019 19:28:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B58F721A49 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:47440 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKoyC-0006Mv-Kl for patchwork-qemu-devel@patchwork.kernel.org; Wed, 16 Oct 2019 15:28:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33179) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKovF-0002Fh-L0 for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:25:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iKovE-0002kX-8j for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:25:25 -0400 Received: from mx1.redhat.com ([209.132.183.28]:33072) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iKovB-0002iH-LK; Wed, 16 Oct 2019 15:25:21 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id DCE3DC057F23; Wed, 16 Oct 2019 19:25:20 +0000 (UTC) Received: from localhost (ovpn-116-20.phx2.redhat.com [10.3.116.20]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8558160BF7; Wed, 16 Oct 2019 19:25:15 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Subject: [PATCH 09/10] image-fuzzer: Run using python3 Date: Wed, 16 Oct 2019 16:24:29 -0300 Message-Id: <20191016192430.25098-10-ehabkost@redhat.com> In-Reply-To: <20191016192430.25098-1-ehabkost@redhat.com> References: <20191016192430.25098-1-ehabkost@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Wed, 16 Oct 2019 19:25:20 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Stefan Hajnoczi , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" image-fuzzer is now supposed to be ready to run using Python 3. Remove the __future__ imports and change the interpreter line to "#!/usr/bin/env python3". Signed-off-by: Eduardo Habkost --- tests/image-fuzzer/qcow2/__init__.py | 1 - tests/image-fuzzer/qcow2/layout.py | 1 - tests/image-fuzzer/runner.py | 3 +-- 3 files changed, 1 insertion(+), 4 deletions(-) diff --git a/tests/image-fuzzer/qcow2/__init__.py b/tests/image-fuzzer/qcow2/__init__.py index 09ef59821b..ed3af5da86 100644 --- a/tests/image-fuzzer/qcow2/__init__.py +++ b/tests/image-fuzzer/qcow2/__init__.py @@ -1,2 +1 @@ -from __future__ import absolute_import from .layout import create_image diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py index 01bff4d05e..57ebe86e9a 100644 --- a/tests/image-fuzzer/qcow2/layout.py +++ b/tests/image-fuzzer/qcow2/layout.py @@ -16,7 +16,6 @@ # along with this program. If not, see . # -from __future__ import absolute_import import random import struct from . import fuzz diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py index 94cab5bd93..0793234815 100755 --- a/tests/image-fuzzer/runner.py +++ b/tests/image-fuzzer/runner.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Tool for running fuzz tests # @@ -18,7 +18,6 @@ # along with this program. If not, see . # -from __future__ import print_function import sys import os import signal From patchwork Wed Oct 16 19:24:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 11193987 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8B48B15AB for ; Wed, 16 Oct 2019 19:28:15 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6A7332053B for ; Wed, 16 Oct 2019 19:28:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6A7332053B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:47438 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKoxy-00060p-Ee for patchwork-qemu-devel@patchwork.kernel.org; Wed, 16 Oct 2019 15:28:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33203) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iKovH-0002IK-CI for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:25:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iKovF-0002ln-UL for qemu-devel@nongnu.org; Wed, 16 Oct 2019 15:25:27 -0400 Received: from mx1.redhat.com ([209.132.183.28]:60188) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iKovD-0002jd-M4; Wed, 16 Oct 2019 15:25:23 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E718B18CB904; Wed, 16 Oct 2019 19:25:22 +0000 (UTC) Received: from localhost (ovpn-116-20.phx2.redhat.com [10.3.116.20]) by smtp.corp.redhat.com (Postfix) with ESMTP id 854CE60BF7; Wed, 16 Oct 2019 19:25:22 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Subject: [PATCH 10/10] image-fuzzer: Use errors parameter of subprocess.Popen() Date: Wed, 16 Oct 2019 16:24:30 -0300 Message-Id: <20191016192430.25098-11-ehabkost@redhat.com> In-Reply-To: <20191016192430.25098-1-ehabkost@redhat.com> References: <20191016192430.25098-1-ehabkost@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.redhat.com [10.5.110.63]); Wed, 16 Oct 2019 19:25:22 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Stefan Hajnoczi , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Instead of manually encoding stderr and stdout output, use `errors` parameter of subprocess.Popen(). This will make process.communicate() return unicode strings instead of bytes objects. Signed-off-by: Eduardo Habkost --- tests/image-fuzzer/runner.py | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py index 0793234815..4ba5c79e13 100755 --- a/tests/image-fuzzer/runner.py +++ b/tests/image-fuzzer/runner.py @@ -79,16 +79,13 @@ def run_app(fd, q_args): devnull = open('/dev/null', 'r+') process = subprocess.Popen(q_args, stdin=devnull, stdout=subprocess.PIPE, - stderr=subprocess.PIPE) + stderr=subprocess.PIPE, + errors='replace') try: out, err = process.communicate() signal.alarm(0) - # fd is a text file, so we need to decode the process output before - # writing to it. - # We could be simply using the `errors` parameter of subprocess.Popen(), - # but this will be possible only after migrating to Python 3 - fd.write(out.decode(errors='replace')) - fd.write(err.decode(errors='replace')) + fd.write(out) + fd.write(err) fd.flush() return process.returncode