From patchwork Fri Jan 31 17:44:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360303 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DEDA392A for ; Fri, 31 Jan 2020 17:46:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B5CC3214D8 for ; Fri, 31 Jan 2020 17:46:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="RuFp133s" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B5CC3214D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57204 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaMk-0004U2-QU for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:46:02 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41207) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLW-0002c4-Dg for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:47 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLV-0004IY-H3 for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:46 -0500 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:28292 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLV-0004IC-E3 for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:45 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492685; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PYMN4/fJJtaeioiL4a/PkGI8sWoRPSVQYwn4SQ5i6MU=; b=RuFp133sicr6oJTjNf0syXSxWpryMOuspJych9mKSy6pDKZ+jy0sLt7eW58nzHcmwESnjX 5FrbsWKR02Jo4OQDDK/qdwrcwFgnxTO5fqIb0kmoQiZS5rsnTxEpfY6USqwEQjcxg5pKeE i0AtQbhFaAjwCyGFZieGkGEqmLZQmT4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-321-Vd7cCsdFMfOmNuSr9ew7AQ-1; Fri, 31 Jan 2020 12:44:41 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C8F87800D48; Fri, 31 Jan 2020 17:44:39 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id 51D28811E3; Fri, 31 Jan 2020 17:44:39 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 01/17] qcow2: Comment typo fixes Date: Fri, 31 Jan 2020 11:44:20 -0600 Message-Id: <20200131174436.2961874-2-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: Vd7cCsdFMfOmNuSr9ew7AQ-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.81 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Various trivial typos noticed while working on this file. Signed-off-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia --- block/qcow2.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index cef9d72b3a16..30fd3d13032a 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -174,7 +174,7 @@ static ssize_t qcow2_crypto_hdr_write_func(QCryptoBlock *block, size_t offset, } -/* +/* * read qcow2 extension and fill bs * start reading from start_offset * finish reading upon magic of value 0 or when end_offset reached @@ -3251,7 +3251,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) * inconsistency later. * * We do need a refcount table because growing the refcount table means - * allocating two new refcount blocks - the seconds of which would be at + * allocating two new refcount blocks - the second of which would be at * 2 GB for 64k clusters, and we don't want to have a 2 GB initial file * size for any qcow2 image. */ @@ -3495,7 +3495,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) goto out; } - /* Want a backing file? There you go.*/ + /* Want a backing file? There you go. */ if (qcow2_opts->has_backing_file) { const char *backing_format = NULL; From patchwork Fri Jan 31 17:44:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360311 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 889DA92A for ; Fri, 31 Jan 2020 17:47:46 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5FA26214D8 for ; Fri, 31 Jan 2020 17:47:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="aIgfrVii" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5FA26214D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57238 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaOP-0007rd-Ch for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:47:45 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41180) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLV-0002c1-SZ for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:47 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLT-0004Ga-Vy for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:45 -0500 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:59560 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLT-0004G4-Rv for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492683; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NECrzhLnAJqisY5S+eozbIOoiheU/vYYbDCkaJ8l/Y4=; b=aIgfrViieS9hm4aZVU27vBArdy6IDQE7T4OIm+l0BzXfwtX2FNNWijCRfwn26104O8387b JjBEhdQqkedpFF4Ot1CXV2MNo8NMIMJY5D+1/jjIwrjpAP9iYuT57AkYCpprEp+OI5X2A7 nETfBOvTBbrnrlfoT0mUcbAUQXWMfeE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-351-WJgYAmB_Ns-NgI9iyCbNKA-1; Fri, 31 Jan 2020 12:44:41 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 79C7D801E67; Fri, 31 Jan 2020 17:44:40 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id 079FC7FB60; Fri, 31 Jan 2020 17:44:39 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 02/17] qcow2: List autoclear bit names in header Date: Fri, 31 Jan 2020 11:44:21 -0600 Message-Id: <20200131174436.2961874-3-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: WJgYAmB_Ns-NgI9iyCbNKA-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.81 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" The feature table is supposed to advertise the name of all feature bits that we support; however, we forgot to update the table for autoclear bits. While at it, move the table to read-only memory in code, and tweak the qcow2 spec to name the second autoclear bit. Update iotests that are affected by the longer header length. Fixes: 88ddffae Fixes: 93c24936 Signed-off-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/qcow2.c | 12 +++++++++++- docs/interop/qcow2.txt | 3 ++- tests/qemu-iotests/031.out | 8 ++++---- tests/qemu-iotests/036.out | 4 ++-- tests/qemu-iotests/061.out | 14 +++++++------- 5 files changed, 26 insertions(+), 15 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 30fd3d13032a..d3e7709ac2b4 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -2821,7 +2821,7 @@ int qcow2_update_header(BlockDriverState *bs) /* Feature table */ if (s->qcow_version >= 3) { - Qcow2Feature features[] = { + static const Qcow2Feature features[] = { { .type = QCOW2_FEAT_TYPE_INCOMPATIBLE, .bit = QCOW2_INCOMPAT_DIRTY_BITNR, @@ -2842,6 +2842,16 @@ int qcow2_update_header(BlockDriverState *bs) .bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR, .name = "lazy refcounts", }, + { + .type = QCOW2_FEAT_TYPE_AUTOCLEAR, + .bit = QCOW2_AUTOCLEAR_BITMAPS_BITNR, + .name = "consistent bitmaps", + }, + { + .type = QCOW2_FEAT_TYPE_AUTOCLEAR, + .bit = QCOW2_AUTOCLEAR_DATA_FILE_RAW_BITNR, + .name = "raw external data", + }, }; ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE, diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index af5711e53371..8510d74c8079 100644 --- a/docs/interop/qcow2.txt +++ b/docs/interop/qcow2.txt @@ -138,7 +138,8 @@ in the description of a field. bit is unset, the bitmaps extension data must be considered inconsistent. - Bit 1: If this bit is set, the external data file can + Bit 1: Raw external data bit + If this bit is set, the external data file can be read as a consistent standalone raw image without looking at the qcow2 metadata. diff --git a/tests/qemu-iotests/031.out b/tests/qemu-iotests/031.out index d535e407bc30..46f97c5a4ea4 100644 --- a/tests/qemu-iotests/031.out +++ b/tests/qemu-iotests/031.out @@ -117,7 +117,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 192 +length 288 data Header extension: @@ -150,7 +150,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 192 +length 288 data Header extension: @@ -164,7 +164,7 @@ No errors were found on the image. magic 0x514649fb version 3 -backing_file_offset 0x178 +backing_file_offset 0x1d8 backing_file_size 0x17 cluster_bits 16 size 67108864 @@ -188,7 +188,7 @@ data 'host_device' Header extension: magic 0x6803f857 -length 192 +length 288 data Header extension: diff --git a/tests/qemu-iotests/036.out b/tests/qemu-iotests/036.out index 0b52b934e115..23b699ce0622 100644 --- a/tests/qemu-iotests/036.out +++ b/tests/qemu-iotests/036.out @@ -26,7 +26,7 @@ compatible_features [] autoclear_features [63] Header extension: magic 0x6803f857 -length 192 +length 288 data @@ -38,7 +38,7 @@ compatible_features [] autoclear_features [] Header extension: magic 0x6803f857 -length 192 +length 288 data *** done diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out index 8b3091a412bc..413cc4e0f4ab 100644 --- a/tests/qemu-iotests/061.out +++ b/tests/qemu-iotests/061.out @@ -26,7 +26,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 192 +length 288 data magic 0x514649fb @@ -84,7 +84,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 192 +length 288 data magic 0x514649fb @@ -140,7 +140,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 192 +length 288 data ERROR cluster 5 refcount=0 reference=1 @@ -195,7 +195,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 192 +length 288 data magic 0x514649fb @@ -264,7 +264,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 192 +length 288 data read 65536/65536 bytes at offset 44040192 @@ -298,7 +298,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 192 +length 288 data ERROR cluster 5 refcount=0 reference=1 @@ -327,7 +327,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 192 +length 288 data read 131072/131072 bytes at offset 0 From patchwork Fri Jan 31 17:44:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360349 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9449492A for ; Fri, 31 Jan 2020 17:52:07 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6AB17206D5 for ; Fri, 31 Jan 2020 17:52:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="IWikJnzq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6AB17206D5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57312 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaSc-0006O6-JX for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:52:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41361) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLb-0002jY-J7 for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLa-0004Qn-4j for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:51 -0500 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:30298 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLZ-0004Pu-Tk for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492689; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xOcmGmzDjNdL53A3beBj1FGQfubPjec9rtjHXJju37M=; b=IWikJnzqsG+PrVikzolD7EmPP7BaERKKlqNdTedYJmXv/EGhWh6rPdXVfdcuIibzmgd0oI XI2Wxw7+UgzQeVBqvOHGAI+S8MEbO1sWlcUghX/W33Y0Fjx16aeiOEBNZscD/rs6AfPX8v 4RFskVnF7hph0BD/2V687ZOg12UJYok= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-377-Jn7vHhgnNgidW9bvBLtCZQ-1; Fri, 31 Jan 2020 12:44:42 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 25436800D54; Fri, 31 Jan 2020 17:44:41 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id A95887FB60; Fri, 31 Jan 2020 17:44:40 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 03/17] qcow2: Avoid feature name extension on small cluster size Date: Fri, 31 Jan 2020 11:44:22 -0600 Message-Id: <20200131174436.2961874-4-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: Jn7vHhgnNgidW9bvBLtCZQ-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.81 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" As the feature name table can be quite large (over 9k if all 64 bits of all three feature fields have names; a mere 8 features leaves only 8 bytes for a backing file name in a 512-byte cluster), it is unwise to emit this optional header in images with small cluster sizes. Update iotest 036 to skip running on small cluster sizes; meanwhile, note that iotest 061 never passed on alternative cluster sizes (however, I limited this patch to tests with output affected by adding feature names, rather than auditing for other tests that are not robust to alternative cluster sizes). Signed-off-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia --- block/qcow2.c | 11 +++++++++-- tests/qemu-iotests/036 | 6 ++++-- tests/qemu-iotests/061 | 6 ++++-- 3 files changed, 17 insertions(+), 6 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index d3e7709ac2b4..6ea06dbdf48a 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -2819,8 +2819,15 @@ int qcow2_update_header(BlockDriverState *bs) buflen -= ret; } - /* Feature table */ - if (s->qcow_version >= 3) { + /* + * Feature table. A mere 8 feature names occupies 392 bytes, and + * when coupled with the v3 minimum header of 104 bytes plus the + * 8-byte end-of-extension marker, that would leave only 8 bytes + * for a backing file name in an image with 512-byte clusters. + * Thus, we choose to omit this header for cluster sizes 4k and + * smaller. + */ + if (s->qcow_version >= 3 && s->cluster_size > 4096) { static const Qcow2Feature features[] = { { .type = QCOW2_FEAT_TYPE_INCOMPATIBLE, diff --git a/tests/qemu-iotests/036 b/tests/qemu-iotests/036 index 512598421c20..cf522de7a1aa 100755 --- a/tests/qemu-iotests/036 +++ b/tests/qemu-iotests/036 @@ -44,8 +44,10 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file # Only qcow2v3 and later supports feature bits; -# qcow2.py does not support external data files -_unsupported_imgopts 'compat=0.10' data_file +# qcow2.py does not support external data files; +# this test requires a cluster size large enough for the feature table +_unsupported_imgopts 'compat=0.10' data_file \ + 'cluster_size=\(512\|1024\|2048\|4096\)' echo echo === Image with unknown incompatible feature bit === diff --git a/tests/qemu-iotests/061 b/tests/qemu-iotests/061 index 36b040491fef..ce285d308408 100755 --- a/tests/qemu-iotests/061 +++ b/tests/qemu-iotests/061 @@ -44,8 +44,10 @@ _supported_os Linux # Conversion between different compat versions can only really work # with refcount_bits=16; # we have explicit tests for data_file here, but the whole test does -# not work with it -_unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)' data_file +# not work with it; +# we have explicit tests for various cluster sizes, the remaining tests +# require the default 64k cluster +_unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)' data_file cluster_size echo echo "=== Testing version downgrade with zero expansion ===" From patchwork Fri Jan 31 17:44:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360301 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4D518924 for ; Fri, 31 Jan 2020 17:46:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 178EE214D8 for ; Fri, 31 Jan 2020 17:46:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="P4a+Z5Ce" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 178EE214D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57202 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaMj-0004Sj-6e for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:46:01 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41211) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLW-0002c5-H0 for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:47 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLV-0004Ix-Kb for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:46 -0500 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:27108 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLV-0004HX-HA for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:45 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492684; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=y3UezPAkLNH5PjZcf6ESIZM+p07nrMqbwQwde4FJnfw=; b=P4a+Z5CenmRzFF7kNHrABm+O4/k00axnwZV1mkUgNgOLtmHwm1pb0Q7zm1YHMoRBZlnrE9 /A4zn1EBMzh0HEcjFLwW8gVV4bThqEP3aAG6jk6xmgv2OwqL17hksUSmcY+U0+19mtVkfQ 6T9dOESt7dWDdWhGBPqjmoj/lqhnDI8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-296-WL7OkWzdOqKhTIAEP-OaqQ-1; Fri, 31 Jan 2020 12:44:42 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C2DA013F9; Fri, 31 Jan 2020 17:44:41 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id 530EC7FB60; Fri, 31 Jan 2020 17:44:41 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 04/17] block: Improve documentation of .bdrv_has_zero_init Date: Fri, 31 Jan 2020 11:44:23 -0600 Message-Id: <20200131174436.2961874-5-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: WL7OkWzdOqKhTIAEP-OaqQ-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.120 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Several drivers supply .bdrv_has_zero_init that returns 1, but lack the .bdrv_has_zero_init_truncate callback (parallels and qed outright, vdi in some scenarios). A literal reading of the existing documentation says such drivers are broken, because bdrv_has_zero_init_truncate() defaults to zero if the callback is missing; but in practice, the tie between the two functions is only relevant when truncate is supported. Clarify the documentation to make it obvious that this is okay. Fixes: 1dcaf527 Signed-off-by: Eric Blake --- include/block/block_int.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/block/block_int.h b/include/block/block_int.h index 640fb82c789e..77ab45dc87cf 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -444,7 +444,8 @@ struct BlockDriver { /* * Returns 1 if newly created images are guaranteed to contain only * zeros, 0 otherwise. - * Must return 0 if .bdrv_has_zero_init_truncate() returns 0. + * Must return 0 if .bdrv_co_truncate is set and + * .bdrv_has_zero_init_truncate() returns 0. */ int (*bdrv_has_zero_init)(BlockDriverState *bs); From patchwork Fri Jan 31 17:44:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360307 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DCCD792A for ; Fri, 31 Jan 2020 17:46:31 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B3EF7214D8 for ; Fri, 31 Jan 2020 17:46:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="gMrWUDp2" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B3EF7214D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57208 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaNC-0005UK-Qx for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:46:30 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41315) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLa-0002gO-AP for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLY-0004OM-VX for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:50 -0500 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:20904 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLY-0004Nf-Pi for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492688; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bKI/xOJyK4CEKumsYglI3UK/B8/0BuSux4WShyruRtU=; b=gMrWUDp21dmatYcclw3QRCEJ6vRQoGH+iaXED1ALKH8t/XmQgcw5cxvUCYZpiFII9E5L1i 3Xsvfzrn+bZpL/N7wm8GBPaOpka2h5kqgVSb1I291+mLQpeVqEg9+MtWUBUjyPR1xrXUCc UU2/b/qElbpfa6iSuf7tcrMpV9vyKqc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-415-5TsUAxCrOmeiVtF9B3E9pA-1; Fri, 31 Jan 2020 12:44:43 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6BEE01937FC5; Fri, 31 Jan 2020 17:44:42 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id EEF1584BCE; Fri, 31 Jan 2020 17:44:41 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 05/17] block: Don't advertise zero_init_truncate with encryption Date: Fri, 31 Jan 2020 11:44:24 -0600 Message-Id: <20200131174436.2961874-6-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: 5TsUAxCrOmeiVtF9B3E9pA-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.120 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Commit 38841dcd correctly argued that having qcow2 blindly return 1 for .bdrv_has_zero_init() is wrong for preallocated images built on block devices, while .bdrv_has_zero_init_truncate() can still return 1 because it is only relied on when changing size with PREALLOC_MODE_OFF (and this is true even for v2 images which lack the notion of an explicit zero cluster, since the block layer already filters out the case of a larger backing file leaking through). However, it missed the fact that encrypted images do not default to reading as zero in any case. However, instead of changing qcow2's .bdrv_has_zero_init_truncate() to point to a one-off function that special-cases bs->encryption, it is smarter to just move the logic about encryption directly to the block layer (that is, the driver callbacks will never be invoked for encrypted images, just like they are already not called when a backing file is present). This solution fixes the qcow2 issue, has no effect on the crypto driver (which already lacks .bdrv_has_zero_init* callbacks), and no other driver currently uses bs->encrypted. One other reason to fix this at the block layer: any information we expose about an encrypted image that in turn may alter timing of algorithms run on that image can be considered a (slight) information leak; refusing to optimize zero handling of encrypted images thus avoids the possibility of that being a security concern. Signed-off-by: Eric Blake Reviewed-by: Alberto Garcia --- block.c | 19 ++++++++++++++++--- block/qcow2.c | 2 -- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/block.c b/block.c index 6c2b2bd2e292..296845040e59 100644 --- a/block.c +++ b/block.c @@ -5077,9 +5077,12 @@ int bdrv_has_zero_init(BlockDriverState *bs) return 0; } - /* If BS is a copy on write image, it is initialized to - the contents of the base image, which may not be zeroes. */ - if (bs->backing) { + /* + * If BS is a copy on write image, it is initialized to the + * contents of the base image, which may not be zeroes. Likewise, + * encrypted images do not read as zero. + */ + if (bs->backing || bs->encrypted) { return 0; } if (bs->drv->bdrv_has_zero_init) { @@ -5099,6 +5102,16 @@ int bdrv_has_zero_init_truncate(BlockDriverState *bs) return 0; } + /* + * Encrypted images never default to reading all zero; and even if + * they did, advertising that fact might lead to an information + * leak based on timing comparisons of algorithms that change if + * our result were dynamic. + */ + if (bs->encrypted) { + return 0; + } + if (bs->backing) { /* Depends on the backing image length, but better safe than sorry */ return 0; diff --git a/block/qcow2.c b/block/qcow2.c index 6ea06dbdf48a..40aa751d1de7 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -4934,8 +4934,6 @@ static int qcow2_has_zero_init(BlockDriverState *bs) if (!preallocated) { return 1; - } else if (bs->encrypted) { - return 0; } else { return bdrv_has_zero_init(s->data_file->bs); } From patchwork Fri Jan 31 17:44:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360317 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6E58C924 for ; Fri, 31 Jan 2020 17:50:07 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 44A8222522 for ; Fri, 31 Jan 2020 17:50:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="IMkhNQJN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 44A8222522 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57266 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaQg-0002tc-FY for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:50:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41306) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLa-0002fr-3G for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLZ-0004Oa-1t for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:49 -0500 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:29443 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLY-0004Nk-S7 for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492688; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hityjKMfB0ls/+JKoTt1mzdgEMZFRSfbIhfAQvP0UgA=; b=IMkhNQJNBWIKEn4O7NM2S10isN1pNMhL732/i/P/Yrg1AMd7xS2KRf515RDBTj7PVHxlMX KIJ99zi78HOrnIYIuH6Ago9ANgZDsoyAMV5LeaZiJf7TqliFNgW2dw1MC6nO211Z0vSHCi IOmFYo53RaTdRko58DfOCVufseYWFVg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-319-DvDKizFLPamuqUujl0zeJg-1; Fri, 31 Jan 2020 12:44:44 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 13F9F1937FC6; Fri, 31 Jan 2020 17:44:43 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id 991B77FB60; Fri, 31 Jan 2020 17:44:42 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 06/17] block: Improve bdrv_has_zero_init_truncate with backing file Date: Fri, 31 Jan 2020 11:44:25 -0600 Message-Id: <20200131174436.2961874-7-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: DvDKizFLPamuqUujl0zeJg-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.120 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" When we added bdrv_has_zero_init_truncate(), we chose to blindly return 0 if a backing file was present, because we knew of the corner case where a backing layer larger than the current layer might leak the tail of the backing layer into the resized region. But as this setup is rare, it penalizes the more common case of a backing layer smaller than the current layer. Signed-off-by: Eric Blake Reviewed-by: Alberto Garcia --- block.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/block.c b/block.c index 296845040e59..d132662f3103 100644 --- a/block.c +++ b/block.c @@ -5112,9 +5112,19 @@ int bdrv_has_zero_init_truncate(BlockDriverState *bs) return 0; } + /* + * If the current layer is smaller than the backing layer, + * truncation may expose backing data; treat failure to query size + * in the same manner. Otherwise, we can trust the driver. + */ + if (bs->backing) { - /* Depends on the backing image length, but better safe than sorry */ - return 0; + int64_t back = bdrv_getlength(bs->backing->bs); + int64_t curr = bdrv_getlength(bs); + + if (back < 0 || curr < back) { + return 0; + } } if (bs->drv->bdrv_has_zero_init_truncate) { return bs->drv->bdrv_has_zero_init_truncate(bs); From patchwork Fri Jan 31 17:44:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360309 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E7B7792A for ; Fri, 31 Jan 2020 17:47:44 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BE2CD214D8 for ; Fri, 31 Jan 2020 17:47:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="KhtwGyRT" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BE2CD214D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57234 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaON-0007oO-TZ for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:47:43 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41274) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLZ-0002dH-2I for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLX-0004Me-TZ for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:48 -0500 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:37655 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLX-0004Lz-Q6 for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:47 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492687; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B51it5BXKL4ctGbDQxLb2aHXVzqtTtPhPQiH6qrqs8g=; b=KhtwGyRTLypocGK523GiaWvsKwbRIS6QVEYONWgyW3RpT+S64CBnzDsckOHKaxk/aDZoXh k2fxHRqvOVsR4VI83Fn7CxEk6Jnd4DTfcJO5UPtAgzCltCmp318sxfIVVgTfvIM4BA+IhE a/1ZT+AwYh1z3TmDr9rrQPUX3Rjcw/Q= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-340-ZmvyKsZBONawh1gVfeoFCg-1; Fri, 31 Jan 2020 12:44:45 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C03A88F4966; Fri, 31 Jan 2020 17:44:43 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3A166811E3; Fri, 31 Jan 2020 17:44:43 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 07/17] gluster: Drop useless has_zero_init callback Date: Fri, 31 Jan 2020 11:44:26 -0600 Message-Id: <20200131174436.2961874-8-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: ZmvyKsZBONawh1gVfeoFCg-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.61 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , "open list:GLUSTER" , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" block.c already defaults to 0 if we don't provide a callback; there's no need to write a callback that always fails. Signed-off-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Reviewed-by: Niels de Vos --- block/gluster.c | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/block/gluster.c b/block/gluster.c index 4fa4a77a4777..9d952c70981b 100644 --- a/block/gluster.c +++ b/block/gluster.c @@ -1357,12 +1357,6 @@ static int64_t qemu_gluster_allocated_file_size(BlockDriverState *bs) } } -static int qemu_gluster_has_zero_init(BlockDriverState *bs) -{ - /* GlusterFS volume could be backed by a block device */ - return 0; -} - /* * Find allocation range in @bs around offset @start. * May change underlying file descriptor's file offset. @@ -1567,8 +1561,6 @@ static BlockDriver bdrv_gluster = { .bdrv_co_readv = qemu_gluster_co_readv, .bdrv_co_writev = qemu_gluster_co_writev, .bdrv_co_flush_to_disk = qemu_gluster_co_flush_to_disk, - .bdrv_has_zero_init = qemu_gluster_has_zero_init, - .bdrv_has_zero_init_truncate = qemu_gluster_has_zero_init, #ifdef CONFIG_GLUSTERFS_DISCARD .bdrv_co_pdiscard = qemu_gluster_co_pdiscard, #endif @@ -1599,8 +1591,6 @@ static BlockDriver bdrv_gluster_tcp = { .bdrv_co_readv = qemu_gluster_co_readv, .bdrv_co_writev = qemu_gluster_co_writev, .bdrv_co_flush_to_disk = qemu_gluster_co_flush_to_disk, - .bdrv_has_zero_init = qemu_gluster_has_zero_init, - .bdrv_has_zero_init_truncate = qemu_gluster_has_zero_init, #ifdef CONFIG_GLUSTERFS_DISCARD .bdrv_co_pdiscard = qemu_gluster_co_pdiscard, #endif @@ -1631,8 +1621,6 @@ static BlockDriver bdrv_gluster_unix = { .bdrv_co_readv = qemu_gluster_co_readv, .bdrv_co_writev = qemu_gluster_co_writev, .bdrv_co_flush_to_disk = qemu_gluster_co_flush_to_disk, - .bdrv_has_zero_init = qemu_gluster_has_zero_init, - .bdrv_has_zero_init_truncate = qemu_gluster_has_zero_init, #ifdef CONFIG_GLUSTERFS_DISCARD .bdrv_co_pdiscard = qemu_gluster_co_pdiscard, #endif @@ -1669,8 +1657,6 @@ static BlockDriver bdrv_gluster_rdma = { .bdrv_co_readv = qemu_gluster_co_readv, .bdrv_co_writev = qemu_gluster_co_writev, .bdrv_co_flush_to_disk = qemu_gluster_co_flush_to_disk, - .bdrv_has_zero_init = qemu_gluster_has_zero_init, - .bdrv_has_zero_init_truncate = qemu_gluster_has_zero_init, #ifdef CONFIG_GLUSTERFS_DISCARD .bdrv_co_pdiscard = qemu_gluster_co_pdiscard, #endif From patchwork Fri Jan 31 17:44:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360305 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EFB0792A for ; Fri, 31 Jan 2020 17:46:17 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C5EE5214D8 for ; Fri, 31 Jan 2020 17:46:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LP2JcZRS" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C5EE5214D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57206 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaMy-0004vX-Sf for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:46:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41384) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLc-0002ld-8q for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLb-0004SP-7C for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:52 -0500 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:28460 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLb-0004Rd-1Y for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:51 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492690; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TAXHXGHUzOtsBvmo26uXaYT7Dh6K1j22oz09jejpdqU=; b=LP2JcZRSw7BFQWTaDOJd9zTVwP5ZfWYO2lsoLMhfmSn0Zy28DajLmIBog6p7hs3hC8m8Xe iEii1SoXjIf5vWlFYS5EtMDxqAIZixeqtVnOUA8ezH9fUpj5lrs3VdX3bx8cQhhC/++Z1Z OoSA6lvrYhHFjO4PWA2z0Pb092fFgRE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-226-v79gN-BqM_2enLWrON2g6g-1; Fri, 31 Jan 2020 12:44:46 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id DD0ED107ACC4; Fri, 31 Jan 2020 17:44:44 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id EA6DB811E3; Fri, 31 Jan 2020 17:44:43 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 08/17] sheepdog: Consistently set bdrv_has_zero_init_truncate Date: Fri, 31 Jan 2020 11:44:27 -0600 Message-Id: <20200131174436.2961874-9-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: v79gN-BqM_2enLWrON2g6g-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.120 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , "open list:Sheepdog" , qemu-block@nongnu.org, mreitz@redhat.com, david.edmondson@oracle.com, Liu Yuan Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" block_int.h claims that .bdrv_has_zero_init must return 0 if .bdrv_has_zero_init_truncate does likewise; but this is violated if only the former callback is provided if .bdrv_co_truncate also exists. When adding the latter callback, it was mistakenly added to only one of the three possible sheepdog instantiations. Fixes: 1dcaf527 Signed-off-by: Eric Blake --- block/sheepdog.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/block/sheepdog.c b/block/sheepdog.c index cfa84338a2d6..522c16a93676 100644 --- a/block/sheepdog.c +++ b/block/sheepdog.c @@ -3269,6 +3269,7 @@ static BlockDriver bdrv_sheepdog_tcp = { .bdrv_co_create = sd_co_create, .bdrv_co_create_opts = sd_co_create_opts, .bdrv_has_zero_init = bdrv_has_zero_init_1, + .bdrv_has_zero_init_truncate = bdrv_has_zero_init_1, .bdrv_getlength = sd_getlength, .bdrv_get_allocated_file_size = sd_get_allocated_file_size, .bdrv_co_truncate = sd_co_truncate, @@ -3307,6 +3308,7 @@ static BlockDriver bdrv_sheepdog_unix = { .bdrv_co_create = sd_co_create, .bdrv_co_create_opts = sd_co_create_opts, .bdrv_has_zero_init = bdrv_has_zero_init_1, + .bdrv_has_zero_init_truncate = bdrv_has_zero_init_1, .bdrv_getlength = sd_getlength, .bdrv_get_allocated_file_size = sd_get_allocated_file_size, .bdrv_co_truncate = sd_co_truncate, From patchwork Fri Jan 31 17:44:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360355 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 24A15924 for ; Fri, 31 Jan 2020 17:53:45 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E01D92082E for ; Fri, 31 Jan 2020 17:53:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="B0oyLrTt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E01D92082E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57354 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaUC-0001df-2J for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:53:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41527) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLj-00038N-OS for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLg-0004bi-OF for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:59 -0500 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:41280 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLg-0004aH-Gi for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:56 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492696; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DZotFooN7bPDhLT6K4q2mFBHquGbMfqpaB4DciS50rM=; b=B0oyLrTtR2OcBexazd6vi2ggJXrfYUTX83594fCH6IzGXk5MyvzawwywRY4bFB3YBBe49M vjPJNZiZCRUHIhtjjSYevcTTuMn+FXl7+K76mymXvCEgFQO++k2yiCsm5SmSIgbpN56Udk 6BTP8P6cXhJvxZLv+0BHSwuYMUPuZLo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-206-kBELA9_wMjS6MkzhPjSqJw-1; Fri, 31 Jan 2020 12:44:54 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A240F800D48; Fri, 31 Jan 2020 17:44:52 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id 453CE863AD; Fri, 31 Jan 2020 17:44:45 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 09/17] block: Refactor bdrv_has_zero_init{,_truncate} Date: Fri, 31 Jan 2020 11:44:28 -0600 Message-Id: <20200131174436.2961874-10-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: kBELA9_wMjS6MkzhPjSqJw-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.61 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Fam Zheng , "open list:Sheepdog" , qemu-block@nongnu.org, Jeff Cody , Stefan Weil , Peter Lieven , "Richard W.M. Jones" , mreitz@redhat.com, david.edmondson@oracle.com, Stefan Hajnoczi , Liu Yuan , "Denis V. Lunev" , Jason Dillaman , Markus Armbruster Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Having two slightly-different function names for related purposes is unwieldy, especially since I envision adding yet another notion of zero support in an upcoming patch. It doesn't help that bdrv_has_zero_init() is a misleading name (I originally thought that a driver could only return 1 when opening an already-existing image known to be all zeroes; but in reality many drivers always return 1 because it only applies to a just-created image). Refactor all uses to instead have a single function that returns multiple bits of information, with better naming and documentation. No semantic change, although some of the changes (such as to qcow2.c) require a careful reading to see how it remains the same. Signed-off-by: Eric Blake --- block.c | 49 ++++++++++++++------------------------ block/file-posix.c | 3 +-- block/file-win32.c | 3 +-- block/nfs.c | 7 +++--- block/parallels.c | 4 ++-- block/qcow.c | 2 +- block/qcow2.c | 10 ++++---- block/qed.c | 3 +-- block/raw-format.c | 12 +++------- block/rbd.c | 3 +-- block/sheepdog.c | 9 +++---- block/ssh.c | 7 +++--- block/vdi.c | 8 +++---- block/vhdx.c | 16 ++++++------- block/vmdk.c | 9 +++---- block/vpc.c | 8 +++---- blockdev.c | 2 +- include/block/block.h | 28 +++++++++++++++++++--- include/block/block_int.h | 15 ++---------- qemu-img.c | 3 ++- tests/qemu-iotests/122 | 2 +- tests/qemu-iotests/188 | 2 +- tests/qemu-iotests/188.out | 2 +- 23 files changed, 96 insertions(+), 111 deletions(-) diff --git a/block.c b/block.c index d132662f3103..fac0813140aa 100644 --- a/block.c +++ b/block.c @@ -5066,38 +5066,20 @@ int bdrv_get_flags(BlockDriverState *bs) return bs->open_flags; } -int bdrv_has_zero_init_1(BlockDriverState *bs) +int bdrv_known_zeroes_create(BlockDriverState *bs) { - return 1; + return BDRV_ZERO_CREATE; } -int bdrv_has_zero_init(BlockDriverState *bs) +int bdrv_known_zeroes_truncate(BlockDriverState *bs) { - if (!bs->drv) { - return 0; - } - - /* - * If BS is a copy on write image, it is initialized to the - * contents of the base image, which may not be zeroes. Likewise, - * encrypted images do not read as zero. - */ - if (bs->backing || bs->encrypted) { - return 0; - } - if (bs->drv->bdrv_has_zero_init) { - return bs->drv->bdrv_has_zero_init(bs); - } - if (bs->file && bs->drv->is_filter) { - return bdrv_has_zero_init(bs->file->bs); - } - - /* safe default */ - return 0; + return BDRV_ZERO_CREATE | BDRV_ZERO_TRUNCATE; } -int bdrv_has_zero_init_truncate(BlockDriverState *bs) +int bdrv_known_zeroes(BlockDriverState *bs) { + int mask = BDRV_ZERO_CREATE | BDRV_ZERO_TRUNCATE; + if (!bs->drv) { return 0; } @@ -5113,9 +5095,12 @@ int bdrv_has_zero_init_truncate(BlockDriverState *bs) } /* - * If the current layer is smaller than the backing layer, - * truncation may expose backing data; treat failure to query size - * in the same manner. Otherwise, we can trust the driver. + * If BS is a copy on write image, it is initialized to the + * contents of the base image, which may not be zeroes, so + * ZERO_CREATE is not viable. If the current layer is smaller + * than the backing layer, truncation may expose backing data, + * restricting ZERO_TRUNCATE; treat failure to query size in the + * same manner. Otherwise, we can trust the driver. */ if (bs->backing) { @@ -5125,12 +5110,14 @@ int bdrv_has_zero_init_truncate(BlockDriverState *bs) if (back < 0 || curr < back) { return 0; } + mask = BDRV_ZERO_TRUNCATE; } - if (bs->drv->bdrv_has_zero_init_truncate) { - return bs->drv->bdrv_has_zero_init_truncate(bs); + + if (bs->drv->bdrv_known_zeroes) { + return bs->drv->bdrv_known_zeroes(bs) & mask; } if (bs->file && bs->drv->is_filter) { - return bdrv_has_zero_init_truncate(bs->file->bs); + return bdrv_known_zeroes(bs->file->bs) & mask; } /* safe default */ diff --git a/block/file-posix.c b/block/file-posix.c index ab82ee1a6718..ff9e39ab882f 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -3071,8 +3071,7 @@ BlockDriver bdrv_file = { .bdrv_close = raw_close, .bdrv_co_create = raw_co_create, .bdrv_co_create_opts = raw_co_create_opts, - .bdrv_has_zero_init = bdrv_has_zero_init_1, - .bdrv_has_zero_init_truncate = bdrv_has_zero_init_1, + .bdrv_known_zeroes = bdrv_known_zeroes_truncate, .bdrv_co_block_status = raw_co_block_status, .bdrv_co_invalidate_cache = raw_co_invalidate_cache, .bdrv_co_pwrite_zeroes = raw_co_pwrite_zeroes, diff --git a/block/file-win32.c b/block/file-win32.c index 77e8ff7b68ae..e9b8f3b2370b 100644 --- a/block/file-win32.c +++ b/block/file-win32.c @@ -635,8 +635,7 @@ BlockDriver bdrv_file = { .bdrv_refresh_limits = raw_probe_alignment, .bdrv_close = raw_close, .bdrv_co_create_opts = raw_co_create_opts, - .bdrv_has_zero_init = bdrv_has_zero_init_1, - .bdrv_has_zero_init_truncate = bdrv_has_zero_init_1, + .bdrv_known_zeroes = bdrv_known_zeroes_truncate, .bdrv_aio_preadv = raw_aio_preadv, .bdrv_aio_pwritev = raw_aio_pwritev, diff --git a/block/nfs.c b/block/nfs.c index 9a6311e27066..34ebe91d5b39 100644 --- a/block/nfs.c +++ b/block/nfs.c @@ -702,10 +702,10 @@ out: return ret; } -static int nfs_has_zero_init(BlockDriverState *bs) +static int nfs_known_zeroes(BlockDriverState *bs) { NFSClient *client = bs->opaque; - return client->has_zero_init; + return client->has_zero_init ? BDRV_ZERO_CREATE | BDRV_ZERO_TRUNCATE : 0; } /* Called (via nfs_service) with QemuMutex held. */ @@ -869,8 +869,7 @@ static BlockDriver bdrv_nfs = { .bdrv_parse_filename = nfs_parse_filename, .create_opts = &nfs_create_opts, - .bdrv_has_zero_init = nfs_has_zero_init, - .bdrv_has_zero_init_truncate = nfs_has_zero_init, + .bdrv_known_zeroes = nfs_known_zeroes, .bdrv_get_allocated_file_size = nfs_get_allocated_file_size, .bdrv_co_truncate = nfs_file_co_truncate, diff --git a/block/parallels.c b/block/parallels.c index 7a01997659b0..dad6389c8481 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -835,7 +835,7 @@ static int parallels_open(BlockDriverState *bs, QDict *options, int flags, goto fail_options; } - if (!bdrv_has_zero_init_truncate(bs->file->bs)) { + if (!(bdrv_known_zeroes(bs->file->bs) & BDRV_ZERO_TRUNCATE)) { s->prealloc_mode = PRL_PREALLOC_MODE_FALLOCATE; } @@ -906,7 +906,7 @@ static BlockDriver bdrv_parallels = { .bdrv_close = parallels_close, .bdrv_child_perm = bdrv_format_default_perms, .bdrv_co_block_status = parallels_co_block_status, - .bdrv_has_zero_init = bdrv_has_zero_init_1, + .bdrv_known_zeroes = bdrv_known_zeroes_create, .bdrv_co_flush_to_os = parallels_co_flush_to_os, .bdrv_co_readv = parallels_co_readv, .bdrv_co_writev = parallels_co_writev, diff --git a/block/qcow.c b/block/qcow.c index fce89898681f..b0c9e212fdb1 100644 --- a/block/qcow.c +++ b/block/qcow.c @@ -1183,7 +1183,7 @@ static BlockDriver bdrv_qcow = { .bdrv_reopen_prepare = qcow_reopen_prepare, .bdrv_co_create = qcow_co_create, .bdrv_co_create_opts = qcow_co_create_opts, - .bdrv_has_zero_init = bdrv_has_zero_init_1, + .bdrv_known_zeroes = bdrv_known_zeroes_create, .supports_backing = true, .bdrv_refresh_limits = qcow_refresh_limits, diff --git a/block/qcow2.c b/block/qcow2.c index 40aa751d1de7..9f2371925737 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -4914,10 +4914,11 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs, return spec_info; } -static int qcow2_has_zero_init(BlockDriverState *bs) +static int qcow2_known_zeroes(BlockDriverState *bs) { BDRVQcow2State *s = bs->opaque; bool preallocated; + int r = BDRV_ZERO_TRUNCATE; if (qemu_in_coroutine()) { qemu_co_mutex_lock(&s->lock); @@ -4933,9 +4934,9 @@ static int qcow2_has_zero_init(BlockDriverState *bs) } if (!preallocated) { - return 1; + return r | BDRV_ZERO_CREATE; } else { - return bdrv_has_zero_init(s->data_file->bs); + return r | bdrv_known_zeroes(s->data_file->bs); } } @@ -5559,8 +5560,7 @@ BlockDriver bdrv_qcow2 = { .bdrv_child_perm = bdrv_format_default_perms, .bdrv_co_create_opts = qcow2_co_create_opts, .bdrv_co_create = qcow2_co_create, - .bdrv_has_zero_init = qcow2_has_zero_init, - .bdrv_has_zero_init_truncate = bdrv_has_zero_init_1, + .bdrv_known_zeroes = qcow2_known_zeroes, .bdrv_co_block_status = qcow2_co_block_status, .bdrv_co_preadv_part = qcow2_co_preadv_part, diff --git a/block/qed.c b/block/qed.c index d8c4e5fb1e85..b00cef2035b3 100644 --- a/block/qed.c +++ b/block/qed.c @@ -1672,8 +1672,7 @@ static BlockDriver bdrv_qed = { .bdrv_child_perm = bdrv_format_default_perms, .bdrv_co_create = bdrv_qed_co_create, .bdrv_co_create_opts = bdrv_qed_co_create_opts, - .bdrv_has_zero_init = bdrv_has_zero_init_1, - .bdrv_has_zero_init_truncate = bdrv_has_zero_init_1, + .bdrv_known_zeroes = bdrv_known_zeroes_truncate, .bdrv_co_block_status = bdrv_qed_co_block_status, .bdrv_co_readv = bdrv_qed_co_readv, .bdrv_co_writev = bdrv_qed_co_writev, diff --git a/block/raw-format.c b/block/raw-format.c index 3a76ec7dd21b..1334a7a2c224 100644 --- a/block/raw-format.c +++ b/block/raw-format.c @@ -409,14 +409,9 @@ static int raw_co_ioctl(BlockDriverState *bs, unsigned long int req, void *buf) return bdrv_co_ioctl(bs->file->bs, req, buf); } -static int raw_has_zero_init(BlockDriverState *bs) +static int raw_known_zeroes(BlockDriverState *bs) { - return bdrv_has_zero_init(bs->file->bs); -} - -static int raw_has_zero_init_truncate(BlockDriverState *bs) -{ - return bdrv_has_zero_init_truncate(bs->file->bs); + return bdrv_known_zeroes(bs->file->bs); } static int coroutine_fn raw_co_create_opts(const char *filename, QemuOpts *opts, @@ -577,8 +572,7 @@ BlockDriver bdrv_raw = { .bdrv_lock_medium = &raw_lock_medium, .bdrv_co_ioctl = &raw_co_ioctl, .create_opts = &raw_create_opts, - .bdrv_has_zero_init = &raw_has_zero_init, - .bdrv_has_zero_init_truncate = &raw_has_zero_init_truncate, + .bdrv_known_zeroes = &raw_known_zeroes, .strong_runtime_opts = raw_strong_runtime_opts, .mutable_opts = mutable_opts, }; diff --git a/block/rbd.c b/block/rbd.c index 027cbcc69520..6cd8e86bccec 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -1289,8 +1289,7 @@ static BlockDriver bdrv_rbd = { .bdrv_reopen_prepare = qemu_rbd_reopen_prepare, .bdrv_co_create = qemu_rbd_co_create, .bdrv_co_create_opts = qemu_rbd_co_create_opts, - .bdrv_has_zero_init = bdrv_has_zero_init_1, - .bdrv_has_zero_init_truncate = bdrv_has_zero_init_1, + .bdrv_known_zeroes = bdrv_known_zeroes_truncate, .bdrv_get_info = qemu_rbd_getinfo, .create_opts = &qemu_rbd_create_opts, .bdrv_getlength = qemu_rbd_getlength, diff --git a/block/sheepdog.c b/block/sheepdog.c index 522c16a93676..916e64abdd74 100644 --- a/block/sheepdog.c +++ b/block/sheepdog.c @@ -3229,8 +3229,7 @@ static BlockDriver bdrv_sheepdog = { .bdrv_close = sd_close, .bdrv_co_create = sd_co_create, .bdrv_co_create_opts = sd_co_create_opts, - .bdrv_has_zero_init = bdrv_has_zero_init_1, - .bdrv_has_zero_init_truncate = bdrv_has_zero_init_1, + .bdrv_known_zeroes = bdrv_known_zeroes_truncate, .bdrv_getlength = sd_getlength, .bdrv_get_allocated_file_size = sd_get_allocated_file_size, .bdrv_co_truncate = sd_co_truncate, @@ -3268,8 +3267,7 @@ static BlockDriver bdrv_sheepdog_tcp = { .bdrv_close = sd_close, .bdrv_co_create = sd_co_create, .bdrv_co_create_opts = sd_co_create_opts, - .bdrv_has_zero_init = bdrv_has_zero_init_1, - .bdrv_has_zero_init_truncate = bdrv_has_zero_init_1, + .bdrv_known_zeroes = bdrv_known_zeroes_truncate, .bdrv_getlength = sd_getlength, .bdrv_get_allocated_file_size = sd_get_allocated_file_size, .bdrv_co_truncate = sd_co_truncate, @@ -3307,8 +3305,7 @@ static BlockDriver bdrv_sheepdog_unix = { .bdrv_close = sd_close, .bdrv_co_create = sd_co_create, .bdrv_co_create_opts = sd_co_create_opts, - .bdrv_has_zero_init = bdrv_has_zero_init_1, - .bdrv_has_zero_init_truncate = bdrv_has_zero_init_1, + .bdrv_known_zeroes = bdrv_known_zeroes_truncate, .bdrv_getlength = sd_getlength, .bdrv_get_allocated_file_size = sd_get_allocated_file_size, .bdrv_co_truncate = sd_co_truncate, diff --git a/block/ssh.c b/block/ssh.c index b4375cf7d2e5..e89dae39800c 100644 --- a/block/ssh.c +++ b/block/ssh.c @@ -1007,14 +1007,14 @@ static void ssh_close(BlockDriverState *bs) ssh_state_free(s); } -static int ssh_has_zero_init(BlockDriverState *bs) +static int ssh_known_zeroes(BlockDriverState *bs) { BDRVSSHState *s = bs->opaque; /* Assume false, unless we can positively prove it's true. */ int has_zero_init = 0; if (s->attrs->type == SSH_FILEXFER_TYPE_REGULAR) { - has_zero_init = 1; + has_zero_init = BDRV_ZERO_CREATE | BDRV_ZERO_TRUNCATE; } return has_zero_init; @@ -1390,8 +1390,7 @@ static BlockDriver bdrv_ssh = { .bdrv_co_create = ssh_co_create, .bdrv_co_create_opts = ssh_co_create_opts, .bdrv_close = ssh_close, - .bdrv_has_zero_init = ssh_has_zero_init, - .bdrv_has_zero_init_truncate = ssh_has_zero_init, + .bdrv_known_zeroes = ssh_known_zeroes, .bdrv_co_readv = ssh_co_readv, .bdrv_co_writev = ssh_co_writev, .bdrv_getlength = ssh_getlength, diff --git a/block/vdi.c b/block/vdi.c index 0142da723315..df8f62624ccf 100644 --- a/block/vdi.c +++ b/block/vdi.c @@ -989,14 +989,14 @@ static void vdi_close(BlockDriverState *bs) error_free(s->migration_blocker); } -static int vdi_has_zero_init(BlockDriverState *bs) +static int vdi_known_zeroes(BlockDriverState *bs) { BDRVVdiState *s = bs->opaque; if (s->header.image_type == VDI_TYPE_STATIC) { - return bdrv_has_zero_init(bs->file->bs); + return bdrv_known_zeroes(bs->file->bs) & BDRV_ZERO_CREATE; } else { - return 1; + return BDRV_ZERO_CREATE; } } @@ -1040,7 +1040,7 @@ static BlockDriver bdrv_vdi = { .bdrv_child_perm = bdrv_format_default_perms, .bdrv_co_create = vdi_co_create, .bdrv_co_create_opts = vdi_co_create_opts, - .bdrv_has_zero_init = vdi_has_zero_init, + .bdrv_known_zeroes = vdi_known_zeroes, .bdrv_co_block_status = vdi_co_block_status, .bdrv_make_empty = vdi_make_empty, diff --git a/block/vhdx.c b/block/vhdx.c index f02d2611bef8..4e8320c1b855 100644 --- a/block/vhdx.c +++ b/block/vhdx.c @@ -1365,7 +1365,7 @@ static coroutine_fn int vhdx_co_writev(BlockDriverState *bs, int64_t sector_num, /* Queue another write of zero buffers if the underlying file * does not zero-fill on file extension */ - if (bdrv_has_zero_init_truncate(bs->file->bs) == 0) { + if (!(bdrv_known_zeroes(bs->file->bs) & BDRV_ZERO_TRUNCATE)) { use_zero_buffers = true; /* zero fill the front, if any */ @@ -1720,8 +1720,8 @@ static int vhdx_create_bat(BlockBackend *blk, BDRVVHDXState *s, } if (type == VHDX_TYPE_FIXED || - use_zero_blocks || - bdrv_has_zero_init(blk_bs(blk)) == 0) { + use_zero_blocks || + !(bdrv_known_zeroes(blk_bs(blk)) & BDRV_ZERO_CREATE)) { /* for a fixed file, the default BAT entry is not zero */ s->bat = g_try_malloc0(length); if (length && s->bat == NULL) { @@ -2162,7 +2162,7 @@ static int coroutine_fn vhdx_co_check(BlockDriverState *bs, return 0; } -static int vhdx_has_zero_init(BlockDriverState *bs) +static int vhdx_known_zeroes(BlockDriverState *bs) { BDRVVHDXState *s = bs->opaque; int state; @@ -2173,17 +2173,17 @@ static int vhdx_has_zero_init(BlockDriverState *bs) * therefore enough to check the first BAT entry. */ if (!s->bat_entries) { - return 1; + return BDRV_ZERO_CREATE; } state = s->bat[0] & VHDX_BAT_STATE_BIT_MASK; if (state == PAYLOAD_BLOCK_FULLY_PRESENT) { /* Fixed subformat */ - return bdrv_has_zero_init(bs->file->bs); + return bdrv_known_zeroes(bs->file->bs) & BDRV_ZERO_CREATE; } /* Dynamic subformat */ - return 1; + return BDRV_ZERO_CREATE; } static QemuOptsList vhdx_create_opts = { @@ -2239,7 +2239,7 @@ static BlockDriver bdrv_vhdx = { .bdrv_co_create_opts = vhdx_co_create_opts, .bdrv_get_info = vhdx_get_info, .bdrv_co_check = vhdx_co_check, - .bdrv_has_zero_init = vhdx_has_zero_init, + .bdrv_known_zeroes = vhdx_known_zeroes, .create_opts = &vhdx_create_opts, }; diff --git a/block/vmdk.c b/block/vmdk.c index 20e909d99794..ca59f50413d2 100644 --- a/block/vmdk.c +++ b/block/vmdk.c @@ -2815,7 +2815,7 @@ static int64_t vmdk_get_allocated_file_size(BlockDriverState *bs) return ret; } -static int vmdk_has_zero_init(BlockDriverState *bs) +static int vmdk_known_zeroes(BlockDriverState *bs) { int i; BDRVVmdkState *s = bs->opaque; @@ -2824,12 +2824,13 @@ static int vmdk_has_zero_init(BlockDriverState *bs) * return 0. */ for (i = 0; i < s->num_extents; i++) { if (s->extents[i].flat) { - if (!bdrv_has_zero_init(s->extents[i].file->bs)) { + if (!(bdrv_known_zeroes(s->extents[i].file->bs) & + BDRV_ZERO_CREATE)) { return 0; } } } - return 1; + return BDRV_ZERO_CREATE; } static ImageInfo *vmdk_get_extent_info(VmdkExtent *extent) @@ -3052,7 +3053,7 @@ static BlockDriver bdrv_vmdk = { .bdrv_co_flush_to_disk = vmdk_co_flush, .bdrv_co_block_status = vmdk_co_block_status, .bdrv_get_allocated_file_size = vmdk_get_allocated_file_size, - .bdrv_has_zero_init = vmdk_has_zero_init, + .bdrv_known_zeroes = vmdk_known_zeroes, .bdrv_get_specific_info = vmdk_get_specific_info, .bdrv_refresh_limits = vmdk_refresh_limits, .bdrv_get_info = vmdk_get_info, diff --git a/block/vpc.c b/block/vpc.c index a65550298e19..f4741e07bfb2 100644 --- a/block/vpc.c +++ b/block/vpc.c @@ -1173,15 +1173,15 @@ fail: } -static int vpc_has_zero_init(BlockDriverState *bs) +static int vpc_known_zeroes(BlockDriverState *bs) { BDRVVPCState *s = bs->opaque; VHDFooter *footer = (VHDFooter *) s->footer_buf; if (be32_to_cpu(footer->type) == VHD_FIXED) { - return bdrv_has_zero_init(bs->file->bs); + return bdrv_known_zeroes(bs->file->bs) & BDRV_ZERO_CREATE; } else { - return 1; + return BDRV_ZERO_CREATE; } } @@ -1249,7 +1249,7 @@ static BlockDriver bdrv_vpc = { .bdrv_get_info = vpc_get_info, .create_opts = &vpc_create_opts, - .bdrv_has_zero_init = vpc_has_zero_init, + .bdrv_known_zeroes = vpc_known_zeroes, .strong_runtime_opts = vpc_strong_runtime_opts, }; diff --git a/blockdev.c b/blockdev.c index c6a727cca99d..90a17e7f7bce 100644 --- a/blockdev.c +++ b/blockdev.c @@ -4001,7 +4001,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) zero_target = (arg->sync == MIRROR_SYNC_MODE_FULL && (arg->mode == NEW_IMAGE_MODE_EXISTING || - !bdrv_has_zero_init(target_bs))); + !(bdrv_known_zeroes(target_bs) & BDRV_ZERO_CREATE))); /* Honor bdrv_try_set_aio_context() context acquisition requirements. */ diff --git a/include/block/block.h b/include/block/block.h index 6cd566324d95..a6a227f50678 100644 --- a/include/block/block.h +++ b/include/block/block.h @@ -85,6 +85,28 @@ typedef enum { BDRV_REQ_MASK = 0x3ff, } BdrvRequestFlags; +typedef enum { + /* + * bdrv_known_zeroes() should include this bit if the contents of + * a freshly-created image with no backing file reads as all + * zeroes without any additional effort. If .bdrv_co_truncate is + * set, then this must be clear if BDRV_ZERO_TRUNCATE is clear. + * Since this bit is only reliable at image creation, a driver may + * return this bit even for existing images that do not currently + * read as zero. + */ + BDRV_ZERO_CREATE = 0x1, + + /* + * bdrv_known_zeroes() should include this bit if growing an image + * with PREALLOC_MODE_OFF (either with no backing file, or beyond + * the size of the backing file) will read the new data as all + * zeroes without any additional effort. This bit only matters + * for drivers that set .bdrv_co_truncate. + */ + BDRV_ZERO_TRUNCATE = 0x2, +} BdrvZeroFlags; + typedef struct BlockSizes { uint32_t phys; uint32_t log; @@ -430,9 +452,9 @@ void bdrv_drain_all(void); int bdrv_pdiscard(BdrvChild *child, int64_t offset, int64_t bytes); int bdrv_co_pdiscard(BdrvChild *child, int64_t offset, int64_t bytes); -int bdrv_has_zero_init_1(BlockDriverState *bs); -int bdrv_has_zero_init(BlockDriverState *bs); -int bdrv_has_zero_init_truncate(BlockDriverState *bs); +int bdrv_known_zeroes_create(BlockDriverState *bs); +int bdrv_known_zeroes_truncate(BlockDriverState *bs); +int bdrv_known_zeroes(BlockDriverState *bs); bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs); bool bdrv_can_write_zeroes_with_unmap(BlockDriverState *bs); int bdrv_block_status(BlockDriverState *bs, int64_t offset, diff --git a/include/block/block_int.h b/include/block/block_int.h index 77ab45dc87cf..47b34860bf95 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -441,19 +441,8 @@ struct BlockDriver { void (*bdrv_refresh_limits)(BlockDriverState *bs, Error **errp); - /* - * Returns 1 if newly created images are guaranteed to contain only - * zeros, 0 otherwise. - * Must return 0 if .bdrv_co_truncate is set and - * .bdrv_has_zero_init_truncate() returns 0. - */ - int (*bdrv_has_zero_init)(BlockDriverState *bs); - - /* - * Returns 1 if new areas added by growing the image with - * PREALLOC_MODE_OFF contain only zeros, 0 otherwise. - */ - int (*bdrv_has_zero_init_truncate)(BlockDriverState *bs); + /* Returns bitwise-OR of BdrvZeroFlags. */ + int (*bdrv_known_zeroes)(BlockDriverState *bs); /* Remove fd handlers, timers, and other event loop callbacks so the event * loop is no longer in use. Called with no in-flight requests and in diff --git a/qemu-img.c b/qemu-img.c index e0bfc33ef4f6..e60217e6c382 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -1987,7 +1987,8 @@ static int convert_do_copy(ImgConvertState *s) /* Check whether we have zero initialisation or can get it efficiently */ if (!s->has_zero_init && s->target_is_new && s->min_sparse && !s->target_has_backing) { - s->has_zero_init = bdrv_has_zero_init(blk_bs(s->target)); + s->has_zero_init = !!(bdrv_known_zeroes(blk_bs(s->target)) & + BDRV_ZERO_CREATE); } if (!s->has_zero_init && !s->target_has_backing && diff --git a/tests/qemu-iotests/122 b/tests/qemu-iotests/122 index dfa350936fe6..7cb09309948f 100755 --- a/tests/qemu-iotests/122 +++ b/tests/qemu-iotests/122 @@ -267,7 +267,7 @@ echo # Keep source zero _make_test_img 64M -# Output is not zero, but has bdrv_has_zero_init() == 1 +# Output is not zero, but has bdrv_known_zeroes() including BDRV_ZERO_CREATE TEST_IMG="$TEST_IMG".orig _make_test_img 64M $QEMU_IO -c "write -P 42 0 64k" "$TEST_IMG".orig | _filter_qemu_io diff --git a/tests/qemu-iotests/188 b/tests/qemu-iotests/188 index afca44df5427..9656969fef4a 100755 --- a/tests/qemu-iotests/188 +++ b/tests/qemu-iotests/188 @@ -71,7 +71,7 @@ $QEMU_IO --object $SECRETALT -c "read -P 0xa 0 $size" --image-opts $IMGSPEC | _f _cleanup_test_img echo -echo "== verify that has_zero_init returns false when preallocating ==" +echo "== verify that known_zeroes returns 0 when preallocating ==" # Empty source file if [ -n "$TEST_IMG_FILE" ]; then diff --git a/tests/qemu-iotests/188.out b/tests/qemu-iotests/188.out index c568ef370145..f7da30440c65 100644 --- a/tests/qemu-iotests/188.out +++ b/tests/qemu-iotests/188.out @@ -16,7 +16,7 @@ read 16777216/16777216 bytes at offset 0 == verify open failure with wrong password == qemu-io: can't open: Invalid password, cannot unlock any keyslot -== verify that has_zero_init returns false when preallocating == +== verify that known_zeroes returns 0 when preallocating == Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=16777216 Images are identical. *** done From patchwork Fri Jan 31 17:44:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360313 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ED60C92A for ; Fri, 31 Jan 2020 17:49:24 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C436E214D8 for ; Fri, 31 Jan 2020 17:49:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Dq6WucUY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C436E214D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57256 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaPz-0001OO-SS for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:49:23 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41535) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLj-00038w-VC for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:01 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLi-0004fF-L5 for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:59 -0500 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:46699 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLi-0004ec-Gx for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:58 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492698; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a4rFAJzXEu4+xQ1KGm8pisE5inMABop5rkbEiEgeSus=; b=Dq6WucUY7Oer/xMFxr0dg535DdCMYpooGvw4YAbFEPZW1F+NWclsIftYwyoShfjWqIuTQV 0q58SEWTXK92MHYI/enIRjl1EDg+3xU01i57alqE1wDBRXqVNvyXHAIuipfxAi6LI3bly1 O7TzoqUokToVGH5x9sP2/oJ61KZsWxg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-184-VmLEKIqBO4SMu_YsNn6m9A-1; Fri, 31 Jan 2020 12:44:54 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 51FC61937FC0; Fri, 31 Jan 2020 17:44:53 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id D4E1384D96; Fri, 31 Jan 2020 17:44:52 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 10/17] block: Add new BDRV_ZERO_OPEN flag Date: Fri, 31 Jan 2020 11:44:29 -0600 Message-Id: <20200131174436.2961874-11-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: VmLEKIqBO4SMu_YsNn6m9A-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.61 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Knowing that a file reads as all zeroes when created is useful, but limited in scope to drivers that can create images. However, there are also situations where pre-existing images can quickly be determined to read as all zeroes, even when the image was not just created by the same process. The optimization used in qemu-img convert to avoid a pre-zeroing pass on the destination is just as useful in such a scenario. As such, it is worth the block layer adding another bit to bdrv_known_zeroes(). Note that while BDRV_ZERO_CREATE cannot chase through backing layers (because it only applies at creation time, but the backing layer was not created at the same time as the active layer being created), it IS okay for BDRV_ZERO_OPEN to chase through layers (as long as all layers currently read as zero, the image reads as zero). Upcoming patches will update the qcow2, file-posix, and nbd drivers to advertise the new bit when appropriate. Signed-off-by: Eric Blake --- block.c | 12 ++++++------ include/block/block.h | 10 ++++++++++ qemu-img.c | 10 ++++++---- 3 files changed, 22 insertions(+), 10 deletions(-) diff --git a/block.c b/block.c index fac0813140aa..d68f527dc41f 100644 --- a/block.c +++ b/block.c @@ -5078,7 +5078,7 @@ int bdrv_known_zeroes_truncate(BlockDriverState *bs) int bdrv_known_zeroes(BlockDriverState *bs) { - int mask = BDRV_ZERO_CREATE | BDRV_ZERO_TRUNCATE; + int mask = BDRV_ZERO_CREATE | BDRV_ZERO_TRUNCATE | BDRV_ZERO_OPEN; if (!bs->drv) { return 0; @@ -5100,17 +5100,17 @@ int bdrv_known_zeroes(BlockDriverState *bs) * ZERO_CREATE is not viable. If the current layer is smaller * than the backing layer, truncation may expose backing data, * restricting ZERO_TRUNCATE; treat failure to query size in the - * same manner. Otherwise, we can trust the driver. + * same manner. For ZERO_OPEN, we insist that both backing and + * current layer report the bit. */ - if (bs->backing) { int64_t back = bdrv_getlength(bs->backing->bs); int64_t curr = bdrv_getlength(bs); - if (back < 0 || curr < back) { - return 0; + mask = bdrv_known_zeroes(bs->backing->bs) & BDRV_ZERO_OPEN; + if (back >= 0 && curr >= back) { + mask |= BDRV_ZERO_TRUNCATE; } - mask = BDRV_ZERO_TRUNCATE; } if (bs->drv->bdrv_known_zeroes) { diff --git a/include/block/block.h b/include/block/block.h index a6a227f50678..dafb8cc2bd80 100644 --- a/include/block/block.h +++ b/include/block/block.h @@ -105,6 +105,16 @@ typedef enum { * for drivers that set .bdrv_co_truncate. */ BDRV_ZERO_TRUNCATE = 0x2, + + /* + * bdrv_known_zeroes() should include this bit if an image is + * known to read as all zeroes when first opened; this bit should + * not be relied on after any writes to the image. This can be + * set even if BDRV_ZERO_INIT is clear, but should only be set if + * making the determination is more efficient than looping over + * block status for the image. + */ + BDRV_ZERO_OPEN = 0x4, } BdrvZeroFlags; typedef struct BlockSizes { diff --git a/qemu-img.c b/qemu-img.c index e60217e6c382..c8519a74f738 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -1985,10 +1985,12 @@ static int convert_do_copy(ImgConvertState *s) int64_t sector_num = 0; /* Check whether we have zero initialisation or can get it efficiently */ - if (!s->has_zero_init && s->target_is_new && s->min_sparse && - !s->target_has_backing) { - s->has_zero_init = !!(bdrv_known_zeroes(blk_bs(s->target)) & - BDRV_ZERO_CREATE); + if (!s->has_zero_init && s->min_sparse && !s->target_has_backing) { + ret = bdrv_known_zeroes(blk_bs(s->target)); + if (ret & BDRV_ZERO_OPEN || + (s->target_is_new && ret & BDRV_ZERO_CREATE)) { + s->has_zero_init = true; + } } if (!s->has_zero_init && !s->target_has_backing && From patchwork Fri Jan 31 17:44:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360345 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4D5FD159A for ; Fri, 31 Jan 2020 17:51:17 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 242992082E for ; Fri, 31 Jan 2020 17:51:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Rhha1wMP" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 242992082E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57296 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaRo-0004ob-8S for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:51:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41548) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLk-0003Au-KC for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:01 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLj-0004h3-Fr for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:00 -0500 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:34405 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLj-0004gT-BY for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:59 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492699; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Q5LRfQ5sKsOuscvW/woHxAejuTxhwKv2TY1mM/n+QBw=; b=Rhha1wMPG5U2jvSJGV7WlRNu3vtP/ysGFxBifZ9ordAb/dFLwgqZV9EB8HZdcmkRdX8DEw hEBPVLVIL1bRy2KTzHaV0ULTMMLVsFHYrnS4fRswJAj1qht/0EMp0QlIPzAgWy67gnzGVE EtJ2s7nJJLCKtYicaSIRxjc/MHVUW/U= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-31-7Hu66hmNPLCyKYsJr_Ghvg-1; Fri, 31 Jan 2020 12:44:55 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EEA6A107ACC4; Fri, 31 Jan 2020 17:44:53 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7D2E97FB60; Fri, 31 Jan 2020 17:44:53 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 11/17] file-posix: Support BDRV_ZERO_OPEN Date: Fri, 31 Jan 2020 11:44:30 -0600 Message-Id: <20200131174436.2961874-12-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: 7Hu66hmNPLCyKYsJr_Ghvg-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.81 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" A single lseek(SEEK_DATA) is sufficient to tell us if a raw file is completely sparse, in which case it reads as all zeroes. Not done here, but possible extension for the future: when working with block devices instead of files, there may be various ways with ioctl or similar to quickly probe if a given block device is known to be completely unmapped where unmapped regions read as zero. But for now, block devices remain without a .bdrv_known_zeroes, because most block devices have random content without an explicit pre-zeroing pass. Signed-off-by: Eric Blake --- block/file-posix.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/block/file-posix.c b/block/file-posix.c index ff9e39ab882f..b4d73dd0363b 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -2541,6 +2541,19 @@ static int find_allocation(BlockDriverState *bs, off_t start, #endif } +static int raw_known_zeroes(BlockDriverState *bs) +{ + /* This callback is only installed for files, not block devices. */ + int r = BDRV_ZERO_CREATE | BDRV_ZERO_TRUNCATE; + off_t data, hole; + + if (find_allocation(bs, 0, &data, &hole) == -ENXIO) { + r |= BDRV_ZERO_OPEN; + } + + return r; +} + /* * Returns the allocation status of the specified offset. * @@ -3071,7 +3084,7 @@ BlockDriver bdrv_file = { .bdrv_close = raw_close, .bdrv_co_create = raw_co_create, .bdrv_co_create_opts = raw_co_create_opts, - .bdrv_known_zeroes = bdrv_known_zeroes_truncate, + .bdrv_known_zeroes = raw_known_zeroes, .bdrv_co_block_status = raw_co_block_status, .bdrv_co_invalidate_cache = raw_co_invalidate_cache, .bdrv_co_pwrite_zeroes = raw_co_pwrite_zeroes, From patchwork Fri Jan 31 17:44:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360353 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CF551924 for ; Fri, 31 Jan 2020 17:52:58 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A5838206D5 for ; Fri, 31 Jan 2020 17:52:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Gk5b5oP4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A5838206D5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57334 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaTR-0008FB-QP for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:52:57 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41567) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLl-0003CB-78 for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:02 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLk-0004iP-0O for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:01 -0500 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:34232 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLj-0004ha-T9 for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:59 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492699; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Jf6rrmMMkG7rUq9+VULR4sD8iT5fg6b7OJ1gXa5pIec=; b=Gk5b5oP4N4bFG43EDlIrhGL6hAM4w6gK9nwiAMMqIS9u/YmpD8dalMVDOvT9EGTc/v6Dk6 uwOHre1290MBb1+pXLMgFNYWpSvnpwKz+B+HKiA1h/kY/WdgqLg/D6stdaOp+mwJ6MgZrL OAMoEbDLnSUv7rpl5zmyZLM+ECSCydo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-306-9WEUHAjVO7OdEaS2C4nYeA-1; Fri, 31 Jan 2020 12:44:55 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id AFB998F4964; Fri, 31 Jan 2020 17:44:54 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2835C7FB60; Fri, 31 Jan 2020 17:44:54 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 12/17] gluster: Support BDRV_ZERO_OPEN Date: Fri, 31 Jan 2020 11:44:31 -0600 Message-Id: <20200131174436.2961874-13-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: 9WEUHAjVO7OdEaS2C4nYeA-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.120 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , "open list:GLUSTER" , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Since gluster already copies file-posix for lseek usage in block status, it also makes sense to copy it for learning if the image currently reads as all zeroes. Signed-off-by: Eric Blake --- block/gluster.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/block/gluster.c b/block/gluster.c index 9d952c70981b..0417a86547c8 100644 --- a/block/gluster.c +++ b/block/gluster.c @@ -1464,6 +1464,22 @@ exit: return -ENOTSUP; } +static int qemu_gluster_known_zeroes(BlockDriverState *bs) +{ + /* + * GlusterFS volume could be backed by a block device, with no way + * to query if regions added by creation or truncation will read + * as zeroes. However, we can use lseek(SEEK_DATA) to check if + * contents currently read as zero. + */ + off_t data, hole; + + if (find_allocation(bs, 0, &data, &hole) == -ENXIO) { + return BDRV_ZERO_OPEN; + } + return 0; +} + /* * Returns the allocation status of the specified offset. * @@ -1561,6 +1577,7 @@ static BlockDriver bdrv_gluster = { .bdrv_co_readv = qemu_gluster_co_readv, .bdrv_co_writev = qemu_gluster_co_writev, .bdrv_co_flush_to_disk = qemu_gluster_co_flush_to_disk, + .bdrv_known_zeroes = qemu_gluster_known_zeroes, #ifdef CONFIG_GLUSTERFS_DISCARD .bdrv_co_pdiscard = qemu_gluster_co_pdiscard, #endif @@ -1591,6 +1608,7 @@ static BlockDriver bdrv_gluster_tcp = { .bdrv_co_readv = qemu_gluster_co_readv, .bdrv_co_writev = qemu_gluster_co_writev, .bdrv_co_flush_to_disk = qemu_gluster_co_flush_to_disk, + .bdrv_known_zeroes = qemu_gluster_known_zeroes, #ifdef CONFIG_GLUSTERFS_DISCARD .bdrv_co_pdiscard = qemu_gluster_co_pdiscard, #endif @@ -1621,6 +1639,7 @@ static BlockDriver bdrv_gluster_unix = { .bdrv_co_readv = qemu_gluster_co_readv, .bdrv_co_writev = qemu_gluster_co_writev, .bdrv_co_flush_to_disk = qemu_gluster_co_flush_to_disk, + .bdrv_known_zeroes = qemu_gluster_known_zeroes, #ifdef CONFIG_GLUSTERFS_DISCARD .bdrv_co_pdiscard = qemu_gluster_co_pdiscard, #endif @@ -1657,6 +1676,7 @@ static BlockDriver bdrv_gluster_rdma = { .bdrv_co_readv = qemu_gluster_co_readv, .bdrv_co_writev = qemu_gluster_co_writev, .bdrv_co_flush_to_disk = qemu_gluster_co_flush_to_disk, + .bdrv_known_zeroes = qemu_gluster_known_zeroes, #ifdef CONFIG_GLUSTERFS_DISCARD .bdrv_co_pdiscard = qemu_gluster_co_pdiscard, #endif From patchwork Fri Jan 31 17:44:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360357 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B70B792A for ; Fri, 31 Jan 2020 17:55:27 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8684F206D5 for ; Fri, 31 Jan 2020 17:55:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ebsTzS/D" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8684F206D5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57398 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaVq-0004D9-MU for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:55:26 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41579) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLl-0003Dc-NG for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLj-0004hx-Pk for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:01 -0500 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:42752 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLj-0004gr-Jn for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:44:59 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492699; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r15KjtoTfDHDeFA/sMgNA1KjglEt1D7DEzNP6tGdv84=; b=ebsTzS/D41xCCV1MH0PFd4PGJbPr0rkb2bQ2ojbHI7/x1g2NUbDvWtkROjmmbla4ZPL1CZ yFueSOsIZpQfAc961sfjSg15+NilGyvMY1x9Ac7Za4DUTqc2zddVnQZXlk7fHorRCt0mMk Hydqi1kVX5BZFW4l+GFkF910RaVFhyI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-43-lVqnbjmUMqqE3ODxGSAN9w-1; Fri, 31 Jan 2020 12:44:56 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 77BB1100550E; Fri, 31 Jan 2020 17:44:55 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id EBD3B7FB60; Fri, 31 Jan 2020 17:44:54 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 13/17] qcow2: Add new autoclear feature for all zero image Date: Fri, 31 Jan 2020 11:44:32 -0600 Message-Id: <20200131174436.2961874-14-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: lVqnbjmUMqqE3ODxGSAN9w-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.81 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , Markus Armbruster , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" With the recent introduction of BDRV_ZERO_OPEN, we can optimize various qemu-img operations if we know the destination starts life with all zero content. For an image with no cluster allocations and no backing file, this was already trivial with BDRV_ZERO_CREATE; but for a fully preallocated image, it does not scale to crawl through the entire L1/L2 tree to see if every cluster is currently marked as a zero cluster. But it is quite easy to add an autoclear bit to the qcow2 file itself: the bit will be set after newly creating an image or after qcow2_make_empty, and cleared on any other modification (including by an older qemu that doesn't recognize the bit). This patch documents the new bit, independently of implementing the places in code that should set it (which means that for bisection purposes, it is safer to still mask the bit out when opening an image with the bit set). A few iotests have updated output due to the larger number of named header features. Signed-off-by: Eric Blake --- RFC: As defined in this patch, I defined the bit to be clear if any cluster defers to a backing file. But the block layer would handle things just fine if we instead allowed the bit to be set if all clusters allocated in this image are zero, even if there are other clusters not allocated. Or maybe we want TWO bits: one if all clusters allocated here are known zero, and a second if we know that there are any clusters that defer to a backing image. --- block/qcow2.c | 9 +++++++++ block/qcow2.h | 3 +++ docs/interop/qcow2.txt | 12 +++++++++++- qapi/block-core.json | 4 ++++ tests/qemu-iotests/031.out | 8 ++++---- tests/qemu-iotests/036.out | 4 ++-- tests/qemu-iotests/061.out | 14 +++++++------- 7 files changed, 40 insertions(+), 14 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 9f2371925737..20cce9410c84 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -2859,6 +2859,11 @@ int qcow2_update_header(BlockDriverState *bs) .bit = QCOW2_AUTOCLEAR_DATA_FILE_RAW_BITNR, .name = "raw external data", }, + { + .type = QCOW2_FEAT_TYPE_AUTOCLEAR, + .bit = QCOW2_AUTOCLEAR_ALL_ZERO_BITNR, + .name = "all zero", + }, }; ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE, @@ -4874,6 +4879,10 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs, .corrupt = s->incompatible_features & QCOW2_INCOMPAT_CORRUPT, .has_corrupt = true, + .all_zero = s->autoclear_features & + QCOW2_AUTOCLEAR_ALL_ZERO, + .has_all_zero = s->autoclear_features & + QCOW2_AUTOCLEAR_ALL_ZERO, .refcount_bits = s->refcount_bits, .has_bitmaps = !!bitmaps, .bitmaps = bitmaps, diff --git a/block/qcow2.h b/block/qcow2.h index 094212623257..6fc2d323d753 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -237,11 +237,14 @@ enum { enum { QCOW2_AUTOCLEAR_BITMAPS_BITNR = 0, QCOW2_AUTOCLEAR_DATA_FILE_RAW_BITNR = 1, + QCOW2_AUTOCLEAR_ALL_ZERO_BITNR = 2, QCOW2_AUTOCLEAR_BITMAPS = 1 << QCOW2_AUTOCLEAR_BITMAPS_BITNR, QCOW2_AUTOCLEAR_DATA_FILE_RAW = 1 << QCOW2_AUTOCLEAR_DATA_FILE_RAW_BITNR, + QCOW2_AUTOCLEAR_ALL_ZERO = 1 << QCOW2_AUTOCLEAR_ALL_ZERO_BITNR, QCOW2_AUTOCLEAR_MASK = QCOW2_AUTOCLEAR_BITMAPS | QCOW2_AUTOCLEAR_DATA_FILE_RAW, + /* TODO: Add _ALL_ZERO to _MASK once it is handled correctly */ }; enum qcow2_discard_type { diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index 8510d74c8079..d435363a413c 100644 --- a/docs/interop/qcow2.txt +++ b/docs/interop/qcow2.txt @@ -153,7 +153,17 @@ in the description of a field. File bit (incompatible feature bit 1) is also set. - Bits 2-63: Reserved (set to 0) + Bit 2: All zero image bit + If this bit is set, the entire image reads + as all zeroes. This can be useful for + detecting just-created images even when + clusters are preallocated, which in turn + can be used to optimize image copying. + + This bit should not be set if any cluster + in the image defers to a backing file. + + Bits 3-63: Reserved (set to 0) 96 - 99: refcount_order Describes the width of a reference count block entry (width diff --git a/qapi/block-core.json b/qapi/block-core.json index ef94a296868f..af837ed5af33 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -71,6 +71,9 @@ # @corrupt: true if the image has been marked corrupt; only valid for # compat >= 1.1 (since 2.2) # +# @all-zero: present and true only if the image is known to read as all +# zeroes (since 5.0) +# # @refcount-bits: width of a refcount entry in bits (since 2.3) # # @encrypt: details about encryption parameters; only set if image @@ -87,6 +90,7 @@ '*data-file-raw': 'bool', '*lazy-refcounts': 'bool', '*corrupt': 'bool', + '*all-zero': 'bool', 'refcount-bits': 'int', '*encrypt': 'ImageInfoSpecificQCow2Encryption', '*bitmaps': ['Qcow2BitmapInfo'] diff --git a/tests/qemu-iotests/031.out b/tests/qemu-iotests/031.out index 46f97c5a4ea4..bb1afa7b87f6 100644 --- a/tests/qemu-iotests/031.out +++ b/tests/qemu-iotests/031.out @@ -117,7 +117,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 288 +length 336 data Header extension: @@ -150,7 +150,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 288 +length 336 data Header extension: @@ -164,7 +164,7 @@ No errors were found on the image. magic 0x514649fb version 3 -backing_file_offset 0x1d8 +backing_file_offset 0x208 backing_file_size 0x17 cluster_bits 16 size 67108864 @@ -188,7 +188,7 @@ data 'host_device' Header extension: magic 0x6803f857 -length 288 +length 336 data Header extension: diff --git a/tests/qemu-iotests/036.out b/tests/qemu-iotests/036.out index 23b699ce0622..e409acf60e2b 100644 --- a/tests/qemu-iotests/036.out +++ b/tests/qemu-iotests/036.out @@ -26,7 +26,7 @@ compatible_features [] autoclear_features [63] Header extension: magic 0x6803f857 -length 288 +length 336 data @@ -38,7 +38,7 @@ compatible_features [] autoclear_features [] Header extension: magic 0x6803f857 -length 288 +length 336 data *** done diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out index 413cc4e0f4ab..d873f79bb606 100644 --- a/tests/qemu-iotests/061.out +++ b/tests/qemu-iotests/061.out @@ -26,7 +26,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 288 +length 336 data magic 0x514649fb @@ -84,7 +84,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 288 +length 336 data magic 0x514649fb @@ -140,7 +140,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 288 +length 336 data ERROR cluster 5 refcount=0 reference=1 @@ -195,7 +195,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 288 +length 336 data magic 0x514649fb @@ -264,7 +264,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 288 +length 336 data read 65536/65536 bytes at offset 44040192 @@ -298,7 +298,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 288 +length 336 data ERROR cluster 5 refcount=0 reference=1 @@ -327,7 +327,7 @@ header_length 104 Header extension: magic 0x6803f857 -length 288 +length 336 data read 131072/131072 bytes at offset 0 From patchwork Fri Jan 31 17:44:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360347 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 30931924 for ; Fri, 31 Jan 2020 17:51:40 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 06AA5206D5 for ; Fri, 31 Jan 2020 17:51:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Zvypeutc" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 06AA5206D5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57304 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaSB-0005Ka-66 for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:51:39 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41740) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLu-0003Ws-3J for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:11 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLt-0004yi-5i for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:10 -0500 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:27447 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLt-0004yD-2C for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:09 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492708; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EOGOsxpR0p2OgwGamNOhNFGeIiUOgsZlr2K/1g0z2zM=; b=Zvypeutc971fTZln/w2lNWhtUJ63KXIetcIT+I38L60JX8UjGWAi+H+lad5MN4rIuxMF2t ztFJ7MAeD8N+TRrI3gsVoDWwJCFvBaNTTzJRek9ht1iAZpIQxOeN++pAxrrgVSfj65V+bf X7uY+hgxDReLY7exu391jd2Z+HGWVwE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-339-F9DMh8VkMUimTRjx07R7XA-1; Fri, 31 Jan 2020 12:44:57 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 23576800D48; Fri, 31 Jan 2020 17:44:56 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id A56E87FB60; Fri, 31 Jan 2020 17:44:55 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 14/17] qcow2: Expose all zero bit through .bdrv_known_zeroes Date: Fri, 31 Jan 2020 11:44:33 -0600 Message-Id: <20200131174436.2961874-15-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: F9DMh8VkMUimTRjx07R7XA-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.120 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Now that qcow2 images have a way to track when the contents are known to be all zero, it is worth exposing this to clients such as qemu-img convert. (Of course, until the next patch wires up qcow2 to actually set the bit, this patch has no immediate effect; however, keeping it as a separate patch allows for an easier revert when testing if the bit makes a difference in qemu-img behavior). Signed-off-by: Eric Blake --- block/qcow2.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/block/qcow2.c b/block/qcow2.c index 20cce9410c84..3f61d806a14b 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -4938,6 +4938,9 @@ static int qcow2_known_zeroes(BlockDriverState *bs) * therefore enough to check the first one. */ preallocated = s->l1_size > 0 && s->l1_table[0] != 0; + if (s->autoclear_features & QCOW2_AUTOCLEAR_ALL_ZERO) { + r |= BDRV_ZERO_OPEN; + } if (qemu_in_coroutine()) { qemu_co_mutex_unlock(&s->lock); } From patchwork Fri Jan 31 17:44:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360351 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 491FE92A for ; Fri, 31 Jan 2020 17:52:31 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 10036206D5 for ; Fri, 31 Jan 2020 17:52:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="KaxGas0y" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 10036206D5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57328 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaT0-0007Nk-9V for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:52:30 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41667) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLp-0003NX-Uf for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:09 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLm-0004nj-Mq for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:05 -0500 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:46514 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLm-0004m9-Fr for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:02 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492702; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vSXT9TeN+7rkm9Q6hKCJz30xc5WswAgdcGlCz6C8fUM=; b=KaxGas0yE6Wm4KGovhgFlpv7rcVikqWkLLJ/RWM9sHMIK8ZnS3BzqtgH23PUJcuVlwOhSt puZTo4BmxVgUqlzS1XwpvTYTbHHoHGtusbjYJ1kuZpTw5YXGPKyNaxmHTk94wOKa86nn6k xBKX280u05GThJ1+yv4L9d1ZWleQvlw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-44-u4T0iZxpMtCLKRls8FVvpg-1; Fri, 31 Jan 2020 12:44:58 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E358D107ACC4; Fri, 31 Jan 2020 17:44:56 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id 656D47FB60; Fri, 31 Jan 2020 17:44:56 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 15/17] qcow2: Implement all-zero autoclear bit Date: Fri, 31 Jan 2020 11:44:34 -0600 Message-Id: <20200131174436.2961874-16-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: u4T0iZxpMtCLKRls8FVvpg-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.120 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Wire up the autoclear bit just defined in the previous patch. When we create an image or clear it with .bdrv_make_empty, we know that it reads as all zeroes. Reading an image does not change the previous status, nor does writing zeroes, trimming (because we specifically set trimmed clusters to read as zero), or resize (because the new length reads as zero). This leaves normal writes, data copies, snapshot reverts, and altering the backing file that can change the status. Furthermore, it is not safe to claim that an encrypted image or an image with a backing file reads as all zeroes. Implementation-wise, we clear the bit from the file on the first modification, and then rewrite it when marking the image clean; some callers want to rewrite it (to either set or clear), while others want to preserve the current value; the modifications to qemu_mark_clean make it easier to consolidate the logic for when setting the bit is safe. A number of iotests have altered output, in situations where we have a provably zero image at that point in the test. Later, we may want to wire in further checks to qemu-img check that validates if the bit is set correctly, and/or to set the bit in images where it would be valid, but I did not do that here. Signed-off-by: Eric Blake --- block/qcow2-snapshot.c | 11 +++++ block/qcow2.c | 97 ++++++++++++++++++++++++++++++++++---- block/qcow2.h | 5 +- tests/qemu-iotests/031.out | 6 +-- tests/qemu-iotests/036.out | 6 +-- tests/qemu-iotests/061.out | 12 +++-- tests/qemu-iotests/065 | 12 ++--- tests/qemu-iotests/082.out | 7 +++ tests/qemu-iotests/206.out | 4 ++ tests/qemu-iotests/242.out | 1 + 10 files changed, 134 insertions(+), 27 deletions(-) diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c index 5ab64da1ec36..e19f1b3ef5fa 100644 --- a/block/qcow2-snapshot.c +++ b/block/qcow2-snapshot.c @@ -781,6 +781,16 @@ int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id) goto fail; } + /* + * With modification to the qcow2 spec, snapshots could store + * whether they are in an all zero state. But for now, we assume + * all snapshots are nonzero. + */ + ret = qcow2_mark_nonzero(bs); + if (ret < 0) { + goto fail; + } + /* * Make sure that the current L1 table is big enough to contain the whole * L1 table of the snapshot. If the snapshot L1 table is smaller, the @@ -1044,6 +1054,7 @@ int qcow2_snapshot_load_tmp(BlockDriverState *bs, s->l1_size = sn->l1_size; s->l1_table_offset = sn->l1_table_offset; s->l1_table = new_l1_table; + s->autoclear_features &= ~QCOW2_AUTOCLEAR_ALL_ZERO; for(i = 0;i < s->l1_size; i++) { be64_to_cpus(&s->l1_table[i]); diff --git a/block/qcow2.c b/block/qcow2.c index 3f61d806a14b..6b1969e4d90a 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -480,6 +480,40 @@ static void report_unsupported_feature(Error **errp, Qcow2Feature *table, g_free(features); } +/* + * Clear the all zero bit and flushes afterwards if necessary. + * + * If updating the header fails, it is not safe to proceed with + * modifying the image. + */ +int qcow2_mark_nonzero(BlockDriverState *bs) +{ + BDRVQcow2State *s = bs->opaque; + uint64_t val; + int ret; + + if (!(s->autoclear_features & QCOW2_AUTOCLEAR_ALL_ZERO)) { + return 0; /* already marked non-zero, including version 2 */ + } + + assert(s->qcow_version >= 3); + + val = cpu_to_be64(s->autoclear_features & ~QCOW2_AUTOCLEAR_ALL_ZERO); + ret = bdrv_pwrite(bs->file, offsetof(QCowHeader, autoclear_features), + &val, sizeof(val)); + if (ret < 0) { + return ret; + } + ret = bdrv_flush(bs->file->bs); + if (ret < 0) { + return ret; + } + + /* Only clear the in-memory flag if the header was updated successfully */ + s->autoclear_features &= ~QCOW2_AUTOCLEAR_ALL_ZERO; + return 0; +} + /* * Sets the dirty bit and flushes afterwards if necessary. * @@ -518,16 +552,27 @@ int qcow2_mark_dirty(BlockDriverState *bs) /* * Clears the dirty bit and flushes before if necessary. Only call this * function when there are no pending requests, it does not guard against - * concurrent requests dirtying the image. + * concurrent requests dirtying the image. If all_zero is 0 or 1, adjust + * the value of s->autoclear; if -1, preserve the cached value. */ -static int qcow2_mark_clean(BlockDriverState *bs) +static int qcow2_mark_clean(BlockDriverState *bs, int all_zero) { BDRVQcow2State *s = bs->opaque; - if (s->incompatible_features & QCOW2_INCOMPAT_DIRTY) { + if (all_zero == -1) { + all_zero = !!(s->autoclear_features & QCOW2_AUTOCLEAR_ALL_ZERO); + } + if (bs->backing || bs->encrypted || s->qcow_version < 3) { + all_zero = 0; + } + if (s->incompatible_features & QCOW2_INCOMPAT_DIRTY || + (all_zero && !(s->autoclear_features & QCOW2_AUTOCLEAR_ALL_ZERO))) { int ret; s->incompatible_features &= ~QCOW2_INCOMPAT_DIRTY; + if (all_zero) { + s->autoclear_features |= QCOW2_AUTOCLEAR_ALL_ZERO; + } ret = qcow2_flush_caches(bs); if (ret < 0) { @@ -616,7 +661,13 @@ static int coroutine_fn qcow2_co_check_locked(BlockDriverState *bs, } if (fix && result->check_errors == 0 && result->corruptions == 0) { - ret = qcow2_mark_clean(bs); + /* + * In the case of fixing an image, we've actually spent the + * time of traversing every cluster, and could thus turn the + * all_zero bit on if the check proves it is correct; but for + * now, it is easier to just always drop the all_zero bit. + */ + ret = qcow2_mark_clean(bs, 0); if (ret < 0) { return ret; } @@ -1069,7 +1120,7 @@ static int qcow2_update_options_prepare(BlockDriverState *bs, } if (s->use_lazy_refcounts && !r->use_lazy_refcounts) { - ret = qcow2_mark_clean(bs); + ret = qcow2_mark_clean(bs, -1); if (ret < 0) { error_setg_errno(errp, -ret, "Failed to disable lazy refcounts"); goto fail; @@ -1865,7 +1916,7 @@ static int qcow2_reopen_prepare(BDRVReopenState *state, goto fail; } - ret = qcow2_mark_clean(state->bs); + ret = qcow2_mark_clean(state->bs, -1); if (ret < 0) { goto fail; } @@ -2486,6 +2537,11 @@ static coroutine_fn int qcow2_co_pwritev_part( trace_qcow2_writev_start_req(qemu_coroutine_self(), offset, bytes); + ret = qcow2_mark_nonzero(bs); + if (ret < 0) { + goto fail_nometa; + } + while (bytes != 0 && aio_task_pool_status(aio) == 0) { l2meta = NULL; @@ -2586,7 +2642,7 @@ static int qcow2_inactivate(BlockDriverState *bs) } if (result == 0) { - qcow2_mark_clean(bs); + qcow2_mark_clean(bs, -1); } return result; @@ -3443,6 +3499,9 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) header->autoclear_features |= cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW); } + if (version >= 3 && !qcow2_opts->has_backing_file) { + header->autoclear_features |= cpu_to_be64(QCOW2_AUTOCLEAR_ALL_ZERO); + } ret = blk_pwrite(blk, 0, header, cluster_size, 0); g_free(header); @@ -3793,6 +3852,11 @@ static coroutine_fn int qcow2_co_pdiscard(BlockDriverState *bs, } qemu_co_mutex_lock(&s->lock); + /* + * No need to call qcow2_mark_nonzero: v2 images lack autoclear + * bits and so are already nonzero; v3 images pass full_discard=false + * so that discarded clusters still read as zero. + */ ret = qcow2_cluster_discard(bs, offset, bytes, QCOW2_DISCARD_REQUEST, false); qemu_co_mutex_unlock(&s->lock); @@ -3902,6 +3966,11 @@ qcow2_co_copy_range_to(BlockDriverState *bs, qemu_co_mutex_lock(&s->lock); + ret = qcow2_mark_nonzero(bs); + if (ret < 0) { + goto fail; + } + while (bytes != 0) { l2meta = NULL; @@ -4334,6 +4403,11 @@ qcow2_co_pwritev_compressed_part(BlockDriverState *bs, return -ENOTSUP; } + ret = qcow2_mark_nonzero(bs); + if (ret < 0) { + return ret; + } + if (bytes == 0) { /* * align end of file to a sector boundary to ease reading with @@ -4547,7 +4621,7 @@ static int make_completely_empty(BlockDriverState *bs) /* Now finally the in-memory information corresponds to the on-disk * structures and is correct */ - ret = qcow2_mark_clean(bs); + ret = qcow2_mark_clean(bs, 1); if (ret < 0) { goto fail; } @@ -4615,6 +4689,9 @@ static int qcow2_make_empty(BlockDriverState *bs) break; } } + if (!bs->backing && !bs->encrypted && s->qcow_version >= 3) { + s->autoclear_features |= QCOW2_AUTOCLEAR_ALL_ZERO; + } return ret; } @@ -5002,7 +5079,7 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version, /* clear incompatible features */ if (s->incompatible_features & QCOW2_INCOMPAT_DIRTY) { - ret = qcow2_mark_clean(bs); + ret = qcow2_mark_clean(bs, 0); if (ret < 0) { error_setg_errno(errp, -ret, "Failed to make the image clean"); return ret; @@ -5372,7 +5449,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, s->use_lazy_refcounts = true; } else { /* make image clean first */ - ret = qcow2_mark_clean(bs); + ret = qcow2_mark_clean(bs, -1); if (ret < 0) { error_setg_errno(errp, -ret, "Failed to make the image clean"); return ret; diff --git a/block/qcow2.h b/block/qcow2.h index 6fc2d323d753..7b971ed825ed 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -243,8 +243,8 @@ enum { QCOW2_AUTOCLEAR_ALL_ZERO = 1 << QCOW2_AUTOCLEAR_ALL_ZERO_BITNR, QCOW2_AUTOCLEAR_MASK = QCOW2_AUTOCLEAR_BITMAPS - | QCOW2_AUTOCLEAR_DATA_FILE_RAW, - /* TODO: Add _ALL_ZERO to _MASK once it is handled correctly */ + | QCOW2_AUTOCLEAR_DATA_FILE_RAW + | QCOW2_AUTOCLEAR_ALL_ZERO, }; enum qcow2_discard_type { @@ -610,6 +610,7 @@ int64_t qcow2_refcount_metadata_size(int64_t clusters, size_t cluster_size, int qcow2_mark_dirty(BlockDriverState *bs); int qcow2_mark_corrupt(BlockDriverState *bs); +int qcow2_mark_nonzero(BlockDriverState *bs); int qcow2_mark_consistent(BlockDriverState *bs); int qcow2_update_header(BlockDriverState *bs); diff --git a/tests/qemu-iotests/031.out b/tests/qemu-iotests/031.out index bb1afa7b87f6..293f67e96bb6 100644 --- a/tests/qemu-iotests/031.out +++ b/tests/qemu-iotests/031.out @@ -111,7 +111,7 @@ nb_snapshots 0 snapshot_offset 0x0 incompatible_features [] compatible_features [] -autoclear_features [] +autoclear_features [2] refcount_order 4 header_length 104 @@ -144,7 +144,7 @@ nb_snapshots 0 snapshot_offset 0x0 incompatible_features [] compatible_features [] -autoclear_features [] +autoclear_features [2] refcount_order 4 header_length 104 @@ -177,7 +177,7 @@ nb_snapshots 0 snapshot_offset 0x0 incompatible_features [] compatible_features [] -autoclear_features [] +autoclear_features [2] refcount_order 4 header_length 104 diff --git a/tests/qemu-iotests/036.out b/tests/qemu-iotests/036.out index e409acf60e2b..5eea8b2bb547 100644 --- a/tests/qemu-iotests/036.out +++ b/tests/qemu-iotests/036.out @@ -5,7 +5,7 @@ QA output created by 036 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 incompatible_features [63] compatible_features [] -autoclear_features [] +autoclear_features [2] qemu-img: Could not open 'TEST_DIR/t.IMGFMT': Unsupported IMGFMT feature(s): Unknown incompatible feature: 8000000000000000 qemu-img: Could not open 'TEST_DIR/t.IMGFMT': Unsupported IMGFMT feature(s): Test feature @@ -23,7 +23,7 @@ qemu-img: Could not open 'TEST_DIR/t.IMGFMT': Unsupported IMGFMT feature(s): tes Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 incompatible_features [] compatible_features [] -autoclear_features [63] +autoclear_features [2, 63] Header extension: magic 0x6803f857 length 336 @@ -35,7 +35,7 @@ data No errors were found on the image. incompatible_features [] compatible_features [] -autoclear_features [] +autoclear_features [2] Header extension: magic 0x6803f857 length 336 diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out index d873f79bb606..3d471c2bde14 100644 --- a/tests/qemu-iotests/061.out +++ b/tests/qemu-iotests/061.out @@ -20,7 +20,7 @@ nb_snapshots 0 snapshot_offset 0x0 incompatible_features [] compatible_features [0] -autoclear_features [] +autoclear_features [2] refcount_order 4 header_length 104 @@ -78,7 +78,7 @@ nb_snapshots 0 snapshot_offset 0x0 incompatible_features [] compatible_features [0] -autoclear_features [] +autoclear_features [2] refcount_order 4 header_length 104 @@ -189,7 +189,7 @@ nb_snapshots 0 snapshot_offset 0x0 incompatible_features [] compatible_features [42] -autoclear_features [42] +autoclear_features [2, 42] refcount_order 4 header_length 104 @@ -491,6 +491,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + all zero: true lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data @@ -511,6 +512,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + all zero: true lazy refcounts: false refcount bits: 16 data file: foo @@ -524,6 +526,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + all zero: true lazy refcounts: false refcount bits: 16 data file raw: false @@ -538,6 +541,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + all zero: true lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data @@ -550,6 +554,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + all zero: true lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data @@ -563,6 +568,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + all zero: true lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065 index 5b21eb96bd09..d47b3d30d0de 100755 --- a/tests/qemu-iotests/065 +++ b/tests/qemu-iotests/065 @@ -94,17 +94,17 @@ class TestQCow2(TestQemuImgInfo): class TestQCow3NotLazy(TestQemuImgInfo): '''Testing a qcow2 version 3 image with lazy refcounts disabled''' img_options = 'compat=1.1,lazy_refcounts=off' - json_compare = { 'compat': '1.1', 'lazy-refcounts': False, + json_compare = { 'compat': '1.1', 'all-zero': True, 'lazy-refcounts': False, 'refcount-bits': 16, 'corrupt': False } - human_compare = [ 'compat: 1.1', 'lazy refcounts: false', + human_compare = [ 'compat: 1.1', 'all zero: true', 'lazy refcounts: false', 'refcount bits: 16', 'corrupt: false' ] class TestQCow3Lazy(TestQemuImgInfo): '''Testing a qcow2 version 3 image with lazy refcounts enabled''' img_options = 'compat=1.1,lazy_refcounts=on' - json_compare = { 'compat': '1.1', 'lazy-refcounts': True, + json_compare = { 'compat': '1.1', 'all-zero': True, 'lazy-refcounts': True, 'refcount-bits': 16, 'corrupt': False } - human_compare = [ 'compat: 1.1', 'lazy refcounts: true', + human_compare = [ 'compat: 1.1', 'all zero: true', 'lazy refcounts: true', 'refcount bits: 16', 'corrupt: false' ] class TestQCow3NotLazyQMP(TestQMP): @@ -112,7 +112,7 @@ class TestQCow3NotLazyQMP(TestQMP): with lazy refcounts enabled''' img_options = 'compat=1.1,lazy_refcounts=off' qemu_options = 'lazy-refcounts=on' - compare = { 'compat': '1.1', 'lazy-refcounts': False, + compare = { 'compat': '1.1', 'all-zero': True, 'lazy-refcounts': False, 'refcount-bits': 16, 'corrupt': False } @@ -121,7 +121,7 @@ class TestQCow3LazyQMP(TestQMP): with lazy refcounts disabled''' img_options = 'compat=1.1,lazy_refcounts=on' qemu_options = 'lazy-refcounts=off' - compare = { 'compat': '1.1', 'lazy-refcounts': True, + compare = { 'compat': '1.1', 'all-zero': True, 'lazy-refcounts': True, 'refcount-bits': 16, 'corrupt': False } TestImageInfoSpecific = None diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out index 9d4ed4dc9d61..6729a43712f2 100644 --- a/tests/qemu-iotests/082.out +++ b/tests/qemu-iotests/082.out @@ -17,6 +17,7 @@ virtual size: 128 MiB (134217728 bytes) cluster_size: 4096 Format specific information: compat: 1.1 + all zero: true lazy refcounts: true refcount bits: 16 corrupt: false @@ -29,6 +30,7 @@ virtual size: 128 MiB (134217728 bytes) cluster_size: 8192 Format specific information: compat: 1.1 + all zero: true lazy refcounts: true refcount bits: 16 corrupt: false @@ -299,6 +301,7 @@ virtual size: 128 MiB (134217728 bytes) cluster_size: 4096 Format specific information: compat: 1.1 + all zero: true lazy refcounts: true refcount bits: 16 corrupt: false @@ -310,6 +313,7 @@ virtual size: 128 MiB (134217728 bytes) cluster_size: 8192 Format specific information: compat: 1.1 + all zero: true lazy refcounts: true refcount bits: 16 corrupt: false @@ -579,6 +583,7 @@ virtual size: 128 MiB (134217728 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + all zero: true lazy refcounts: true refcount bits: 16 corrupt: false @@ -590,6 +595,7 @@ virtual size: 130 MiB (136314880 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + all zero: true lazy refcounts: false refcount bits: 16 corrupt: false @@ -601,6 +607,7 @@ virtual size: 132 MiB (138412032 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + all zero: true lazy refcounts: true refcount bits: 16 corrupt: false diff --git a/tests/qemu-iotests/206.out b/tests/qemu-iotests/206.out index 61e7241e0bf3..aa27d75d12b1 100644 --- a/tests/qemu-iotests/206.out +++ b/tests/qemu-iotests/206.out @@ -18,6 +18,7 @@ virtual size: 128 MiB (134217728 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + all zero: true lazy refcounts: false refcount bits: 16 corrupt: false @@ -40,6 +41,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + all zero: true lazy refcounts: false refcount bits: 16 corrupt: false @@ -62,6 +64,7 @@ virtual size: 32 MiB (33554432 bytes) cluster_size: 2097152 Format specific information: compat: 1.1 + all zero: true lazy refcounts: true refcount bits: 1 corrupt: false @@ -102,6 +105,7 @@ encrypted: yes cluster_size: 65536 Format specific information: compat: 1.1 + all zero: true lazy refcounts: false refcount bits: 16 encrypt: diff --git a/tests/qemu-iotests/242.out b/tests/qemu-iotests/242.out index 7ac8404d11c8..807f24549e89 100644 --- a/tests/qemu-iotests/242.out +++ b/tests/qemu-iotests/242.out @@ -153,6 +153,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + all zero: true lazy refcounts: false bitmaps: [0]: From patchwork Fri Jan 31 17:44:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360315 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D5C7A92A for ; Fri, 31 Jan 2020 17:49:34 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 99483214D8 for ; Fri, 31 Jan 2020 17:49:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="edmyrVdG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 99483214D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57260 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaQ9-0001ni-Ou for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:49:33 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41719) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLs-0003TK-Hv for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLp-0004u2-Ro for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:08 -0500 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:48990 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLp-0004tY-LL for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492705; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cSq7DecclquLgo3YGvTsrqUS/ylJr0pUGFtgCoasKd0=; b=edmyrVdGXxGZTmq/REkoNzE8kdeYR0uap4VSUuj036JEf5nQXPgKPOem6MorUFghOCpZ+r yZdxVBpYoBnUEIGgLhC+5hukJhUVjbzg6n60lJBHWGIx1K3MVyGNZjrqlscBEvdQwLGRLM 6R9sY8WAnyngJr4x5t/7SC17gvtZyvI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-40-Lc1a6eBuNwuGlmU6qd7oHg-1; Fri, 31 Jan 2020 12:45:00 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D63981937FC0; Fri, 31 Jan 2020 17:44:59 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1FCC87FB60; Fri, 31 Jan 2020 17:44:57 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 16/17] iotests: Add new test for qcow2 all-zero bit Date: Fri, 31 Jan 2020 11:44:35 -0600 Message-Id: <20200131174436.2961874-17-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: Lc1a6eBuNwuGlmU6qd7oHg-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.61 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Cover various scenarios to show that the bit gets set even for fully-allocated images, as well as scenarios where it is properly cleared. Signed-off-by: Eric Blake --- tests/qemu-iotests/285 | 107 +++++++++++++++ tests/qemu-iotests/285.out | 257 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 365 insertions(+) create mode 100755 tests/qemu-iotests/285 create mode 100644 tests/qemu-iotests/285.out diff --git a/tests/qemu-iotests/285 b/tests/qemu-iotests/285 new file mode 100755 index 000000000000..66037af237a1 --- /dev/null +++ b/tests/qemu-iotests/285 @@ -0,0 +1,107 @@ +#!/usr/bin/env bash +# +# Test qcow2 all-zero autoclear bit +# +# Copyright (C) 2020 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +seq=$(basename $0) +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux +# Autoclear bit is not available in compat=0.10; +# encrypted images never advertise all-zero bit +_unsupported_imgopts 'compat=0.10' encrypt + +for mode in off metadata falloc full; do + + echo + echo "=== preallocation=$mode ===" + echo + + _make_test_img -o "preallocation=$mode" 32M + + # Actions that do not lose the all-zero nature of the image: + $QEMU_IO -c 'w -z 0 16M' -c 'discard 8M 16M' "$TEST_IMG" | _filter_qemu_io + $QEMU_IMG resize --preallocation=$mode "$TEST_IMG" +8M + $QEMU_IO -c 'r -P 0 0 40M' "$TEST_IMG" | _filter_qemu_io + $QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific + + # Writing data must clear the all-zero bit: + $QEMU_IO -c 'w -P 1 32M 1M' "$TEST_IMG" | _filter_qemu_io + $QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific + + # Alas, rewriting the image back to zero does not restore the bit + # (checking if each write gets us back to zero does not scale) + $QEMU_IO -c 'w -z 32M 1M' "$TEST_IMG" | _filter_qemu_io + $QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific + +done + +echo +echo "=== backing files ===" +echo + +# Even when a backing file is all zero, we do not set all-zero bit; +# this is true whether we create with a backing file or rebase later +TEST_IMG_SAVE=$TEST_IMG +TEST_IMG=$TEST_IMG.base +_make_test_img 32M +TEST_IMG=$TEST_IMG_SAVE +_make_test_img -b "$TEST_IMG.base" -F qcow2 32M +$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific +_make_test_img 32M +$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific +$QEMU_IMG rebase -u -F qcow2 -b "$TEST_IMG.base" "$TEST_IMG" +$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific + +# qemu-img commit clears an image, but because it still has a backing file, +# setting the all-zero bit is not correct +$QEMU_IO -c 'w -P 1 0 1M' "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG commit "$TEST_IMG" +$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific + +echo +echo "=== internal snapshots ===" +echo + +# For now, internal snapshots do not remember the all-zero bit +_make_test_img 32M +$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific +$QEMU_IMG snapshot -c snap "$TEST_IMG" +$QEMU_IO -c 'w -P 1 0 1M' "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG snapshot -l snap "$TEST_IMG" +$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific \ + | _filter_date | _filter_vmstate_size + +# success, all done +echo "*** done" +rm -f $seq.full +status=0 diff --git a/tests/qemu-iotests/285.out b/tests/qemu-iotests/285.out new file mode 100644 index 000000000000..e43ff9906b5f --- /dev/null +++ b/tests/qemu-iotests/285.out @@ -0,0 +1,257 @@ +QA output created by 285 + +=== preallocation=off === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 preallocation=off +wrote 16777216/16777216 bytes at offset 0 +16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +discard 16777216/16777216 bytes at offset 8388608 +16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Image resized. +read 41943040/41943040 bytes at offset 0 +40 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 40 MiB (41943040 bytes) +disk size: 260 KiB +Format specific information: + compat: 1.1 + all zero: true + lazy refcounts: false + refcount bits: 16 + corrupt: false +wrote 1048576/1048576 bytes at offset 33554432 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 40 MiB (41943040 bytes) +disk size: 1.25 MiB +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false +wrote 1048576/1048576 bytes at offset 33554432 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 40 MiB (41943040 bytes) +disk size: 1.25 MiB +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false + +=== preallocation=metadata === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 preallocation=metadata +wrote 16777216/16777216 bytes at offset 0 +16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +discard 16777216/16777216 bytes at offset 8388608 +16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Image resized. +read 41943040/41943040 bytes at offset 0 +40 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 40 MiB (41943040 bytes) +disk size: 260 KiB +Format specific information: + compat: 1.1 + all zero: true + lazy refcounts: false + refcount bits: 16 + corrupt: false +wrote 1048576/1048576 bytes at offset 33554432 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 40 MiB (41943040 bytes) +disk size: 1.25 MiB +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false +wrote 1048576/1048576 bytes at offset 33554432 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 40 MiB (41943040 bytes) +disk size: 1.25 MiB +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false + +=== preallocation=falloc === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 preallocation=falloc +wrote 16777216/16777216 bytes at offset 0 +16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +discard 16777216/16777216 bytes at offset 8388608 +16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Image resized. +read 41943040/41943040 bytes at offset 0 +40 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 40 MiB (41943040 bytes) +disk size: 24.3 MiB +Format specific information: + compat: 1.1 + all zero: true + lazy refcounts: false + refcount bits: 16 + corrupt: false +wrote 1048576/1048576 bytes at offset 33554432 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 40 MiB (41943040 bytes) +disk size: 24.3 MiB +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false +wrote 1048576/1048576 bytes at offset 33554432 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 40 MiB (41943040 bytes) +disk size: 24.3 MiB +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false + +=== preallocation=full === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 preallocation=full +wrote 16777216/16777216 bytes at offset 0 +16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +discard 16777216/16777216 bytes at offset 8388608 +16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Image resized. +read 41943040/41943040 bytes at offset 0 +40 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 40 MiB (41943040 bytes) +disk size: 24.3 MiB +Format specific information: + compat: 1.1 + all zero: true + lazy refcounts: false + refcount bits: 16 + corrupt: false +wrote 1048576/1048576 bytes at offset 33554432 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 40 MiB (41943040 bytes) +disk size: 24.3 MiB +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false +wrote 1048576/1048576 bytes at offset 33554432 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 40 MiB (41943040 bytes) +disk size: 24.3 MiB +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false + +=== backing files === + +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=33554432 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 32 MiB (33554432 bytes) +disk size: 196 KiB +backing file: TEST_DIR/t.IMGFMT.base +backing file format: IMGFMT +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 32 MiB (33554432 bytes) +disk size: 196 KiB +Format specific information: + compat: 1.1 + all zero: true + lazy refcounts: false + refcount bits: 16 + corrupt: false +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 32 MiB (33554432 bytes) +disk size: 196 KiB +backing file: TEST_DIR/t.IMGFMT.base +backing file format: IMGFMT +Format specific information: + compat: 1.1 + all zero: true + lazy refcounts: false + refcount bits: 16 + corrupt: false +wrote 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Image committed. +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 32 MiB (33554432 bytes) +disk size: 260 KiB +backing file: TEST_DIR/t.IMGFMT.base +backing file format: IMGFMT +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false + +=== internal snapshots === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 32 MiB (33554432 bytes) +disk size: 196 KiB +Format specific information: + compat: 1.1 + all zero: true + lazy refcounts: false + refcount bits: 16 + corrupt: false +wrote 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +qemu-img: Expecting one image file name +Try 'qemu-img --help' for more information +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 32 MiB (33554432 bytes) +disk size: SIZE +Snapshot list: +ID TAG VM SIZE DATE VM CLOCK +1 snap SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000 +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false +*** done diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index e041cc1ee360..e9b20818fad5 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -289,3 +289,4 @@ 279 rw backing quick 280 rw migration quick 281 rw quick +285 rw quick From patchwork Fri Jan 31 17:44:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 11360361 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6F535924 for ; Fri, 31 Jan 2020 17:56:46 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 045AF214D8 for ; Fri, 31 Jan 2020 17:56:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Zoe9M4ZK" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 045AF214D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:57424 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaX6-0005U6-TN for patchwork-qemu-devel@patchwork.kernel.org; Fri, 31 Jan 2020 12:56:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41725) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ixaLs-0003Tk-Oj for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ixaLq-0004ve-TG for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:08 -0500 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:51879 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ixaLq-0004uw-LG for qemu-devel@nongnu.org; Fri, 31 Jan 2020 12:45:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580492706; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bVVLLfNB4yRmFqSpWcv4l5srCFwjCup1lKXgjD1c2ws=; b=Zoe9M4ZKOenDy1ki995rWyUF1EyETzQZvTG021+q2G1JbfYUEaBh7H+nzbNlzwNd1V8nOU CqMQKn0ir4UezF6IoNddbthaH3+pzPrOEmRqomACEY1dSlkIIRBTKaO0++GwsdYGO0+fFz +saLJqc6wS6GY4NY+rAXJm3k7Gm49Hs= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-49-1Rngf_udOfuu1wt8eE04bA-1; Fri, 31 Jan 2020 12:45:01 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8D3B61005510; Fri, 31 Jan 2020 17:45:00 +0000 (UTC) Received: from blue.redhat.com (ovpn-116-181.phx2.redhat.com [10.3.116.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1BB34863AD; Fri, 31 Jan 2020 17:45:00 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH 17/17] qcow2: Let qemu-img check cover all-zero bit Date: Fri, 31 Jan 2020 11:44:36 -0600 Message-Id: <20200131174436.2961874-18-eblake@redhat.com> In-Reply-To: <20200131174436.2961874-1-eblake@redhat.com> References: <20200131174436.2961874-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: 1Rngf_udOfuu1wt8eE04bA-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.120 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, Kevin Wolf , qemu-block@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Since checking an images refcounts already visits every cluster, it's basically free to also check that the all-zero bit is correctly set. Only check for the active L1 table, and only output an error on the first non-zero cluster found. Signed-off-by: Eric Blake --- block/qcow2-refcount.c | 60 +++++++++++++++++++++++++++++++++++--- tests/qemu-iotests/060.out | 6 ++-- tests/qemu-iotests/285 | 17 +++++++++++ tests/qemu-iotests/285.out | 20 +++++++++++++ 4 files changed, 97 insertions(+), 6 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index f67ac6b2d893..95c8101df365 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -1583,6 +1583,7 @@ int qcow2_inc_refcounts_imrt(BlockDriverState *bs, BdrvCheckResult *res, /* Flags for check_refcounts_l1() and check_refcounts_l2() */ enum { CHECK_FRAG_INFO = 0x2, /* update BlockFragInfo counters */ + CHECK_ALL_ZERO = 0x4, /* check autoclear all_zero bit */ }; /* @@ -1596,12 +1597,14 @@ enum { static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, void **refcount_table, int64_t *refcount_table_size, int64_t l2_offset, - int flags, BdrvCheckMode fix, bool active) + int flags, BdrvCheckMode fix, bool active, + bool *all_zero) { BDRVQcow2State *s = bs->opaque; uint64_t *l2_table, l2_entry; uint64_t next_contiguous_offset = 0; int i, l2_size, nb_csectors, ret; + bool check_all_zero; /* Read L2 table from disk */ l2_size = s->l2_size * sizeof(uint64_t); @@ -1615,8 +1618,9 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, } /* Do the actual checks */ - for(i = 0; i < s->l2_size; i++) { + for (i = 0; i < s->l2_size; i++) { l2_entry = be64_to_cpu(l2_table[i]); + check_all_zero = *all_zero; switch (qcow2_get_cluster_type(bs, l2_entry)) { case QCOW2_CLUSTER_COMPRESSED: @@ -1662,6 +1666,8 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, break; case QCOW2_CLUSTER_ZERO_ALLOC: + check_all_zero = false; + /* fall through */ case QCOW2_CLUSTER_NORMAL: { uint64_t offset = l2_entry & L2E_OFFSET_MASK; @@ -1740,12 +1746,51 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, } case QCOW2_CLUSTER_ZERO_PLAIN: + check_all_zero = false; + break; + case QCOW2_CLUSTER_UNALLOCATED: + if (!bs->backing) { + check_all_zero = false; + } break; default: abort(); } + + if (check_all_zero) { + fprintf(stderr, "%s: all zero bit set but L2 table at offset " + "0x%" PRIx64" contains non-zero cluster at entry %d\n", + fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR", + l2_offset, i); + *all_zero = false; + if (fix & BDRV_FIX_ERRORS) { + uint64_t feat; + + ret = bdrv_pread(bs->file, + offsetof(QCowHeader, autoclear_features), + &feat, sizeof(feat)); + if (ret >= 0) { + feat &= ~cpu_to_be64(QCOW2_AUTOCLEAR_ALL_ZERO); + ret = bdrv_pwrite(bs->file, + offsetof(QCowHeader, autoclear_features), + &feat, sizeof(feat)); + } + if (ret < 0) { + fprintf(stderr, + "ERROR: Failed to update all zero bit: %s\n", + strerror(-ret)); + res->check_errors++; + /* Continue checking the rest of this L2 table */ + } else { + res->corruptions_fixed++; + } + s->autoclear_features &= ~QCOW2_AUTOCLEAR_ALL_ZERO; + } else { + res->corruptions++; + } + } } g_free(l2_table); @@ -1774,6 +1819,12 @@ static int check_refcounts_l1(BlockDriverState *bs, BDRVQcow2State *s = bs->opaque; uint64_t *l1_table = NULL, l2_offset, l1_size2; int i, ret; + bool all_zero = false; + + if (flags & CHECK_ALL_ZERO && + s->autoclear_features & QCOW2_AUTOCLEAR_ALL_ZERO) { + all_zero = true; + } l1_size2 = l1_size * sizeof(uint64_t); @@ -1825,7 +1876,7 @@ static int check_refcounts_l1(BlockDriverState *bs, /* Process and check L2 entries */ ret = check_refcounts_l2(bs, res, refcount_table, refcount_table_size, l2_offset, flags, - fix, active); + fix, active, &all_zero); if (ret < 0) { goto fail; } @@ -2114,7 +2165,8 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res, /* current L1 table */ ret = check_refcounts_l1(bs, res, refcount_table, nb_clusters, - s->l1_table_offset, s->l1_size, CHECK_FRAG_INFO, + s->l1_table_offset, s->l1_size, + CHECK_FRAG_INFO | CHECK_ALL_ZERO, fix, true); if (ret < 0) { return ret; diff --git a/tests/qemu-iotests/060.out b/tests/qemu-iotests/060.out index d27692a33c0d..d82aca458544 100644 --- a/tests/qemu-iotests/060.out +++ b/tests/qemu-iotests/060.out @@ -3,9 +3,10 @@ QA output created by 060 === Testing L2 reference into L1 === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +ERROR: all zero bit set but L2 table at offset 0x30000 contains non-zero cluster at entry 0 ERROR cluster 3 refcount=1 reference=3 -1 errors were found on the image. +2 errors were found on the image. Data may be corrupted, or further writes to the image may corrupt it. incompatible_features [] qcow2: Marking image as corrupt: Preventing invalid write on metadata (overlaps with active L1 table); further corruption events will be suppressed @@ -28,10 +29,11 @@ read 512/512 bytes at offset 0 === Testing cluster data reference into refcount block === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +ERROR: all zero bit set but L2 table at offset 0x40000 contains non-zero cluster at entry 0 ERROR refcount block 0 refcount=2 ERROR cluster 2 refcount=1 reference=2 -2 errors were found on the image. +3 errors were found on the image. Data may be corrupted, or further writes to the image may corrupt it. incompatible_features [] qcow2: Marking image as corrupt: Preventing invalid write on metadata (overlaps with refcount block); further corruption events will be suppressed diff --git a/tests/qemu-iotests/285 b/tests/qemu-iotests/285 index 66037af237a1..c435bb57d749 100755 --- a/tests/qemu-iotests/285 +++ b/tests/qemu-iotests/285 @@ -101,6 +101,23 @@ $QEMU_IMG snapshot -l snap "$TEST_IMG" $QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific \ | _filter_date | _filter_vmstate_size +echo +echo "=== qemu-img check ===" +echo + +_make_test_img 32M +$QEMU_IO -c 'w -P 1 0 1M' "$TEST_IMG" | _filter_qemu_io +# Image should be clean +_check_test_img +# Manually corrupt the image by setting the bit +$PYTHON qcow2.py "$TEST_IMG" set-feature-bit autoclear 2 +# check should detect the problem +_check_test_img +# repair should fix it +_check_test_img -r all +# the image should be clean again +_check_test_img + # success, all done echo "*** done" rm -f $seq.full diff --git a/tests/qemu-iotests/285.out b/tests/qemu-iotests/285.out index e43ff9906b5f..b28c9e266bf6 100644 --- a/tests/qemu-iotests/285.out +++ b/tests/qemu-iotests/285.out @@ -254,4 +254,24 @@ Format specific information: lazy refcounts: false refcount bits: 16 corrupt: false + +=== qemu-img check === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 +wrote 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +No errors were found on the image. +ERROR: all zero bit set but L2 table at offset 0x40000 contains non-zero cluster at entry 0 + +1 errors were found on the image. +Data may be corrupted, or further writes to the image may corrupt it. +Repairing: all zero bit set but L2 table at offset 0x40000 contains non-zero cluster at entry 0 +The following inconsistencies were found and repaired: + + 0 leaked clusters + 1 corruptions + +Double checking the fixed image now... +No errors were found on the image. +No errors were found on the image. *** done