From patchwork Wed Feb 27 17:22:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Wolf X-Patchwork-Id: 10832095 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3564015AC for ; Wed, 27 Feb 2019 17:24:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 180FD2E8B4 for ; Wed, 27 Feb 2019 17:24:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0D7F32E8C9; Wed, 27 Feb 2019 17:24:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6E31D2E907 for ; Wed, 27 Feb 2019 17:24:54 +0000 (UTC) Received: from localhost ([127.0.0.1]:47896 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gz2wv-00048Y-2K for patchwork-qemu-devel@patchwork.kernel.org; Wed, 27 Feb 2019 12:24:53 -0500 Received: from eggs.gnu.org ([209.51.188.92]:51190) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gz2vY-0002wW-0W for qemu-devel@nongnu.org; Wed, 27 Feb 2019 12:23:29 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gz2vM-0002wx-1u for qemu-devel@nongnu.org; Wed, 27 Feb 2019 12:23:19 -0500 Received: from mx1.redhat.com ([209.132.183.28]:51850) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gz2vF-0002om-Lb; Wed, 27 Feb 2019 12:23:09 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 389CB44; Wed, 27 Feb 2019 17:23:05 +0000 (UTC) Received: from linux.fritz.box.com (ovpn-117-19.ams2.redhat.com [10.36.117.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id C9F91171A0; Wed, 27 Feb 2019 17:23:03 +0000 (UTC) From: Kevin Wolf To: qemu-block@nongnu.org Date: Wed, 27 Feb 2019 18:22:36 +0100 Message-Id: <20190227172256.30368-1-kwolf@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Wed, 27 Feb 2019 17:23:05 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 00/20] qcow2: External data files X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, qemu-devel@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP There are use cases where raw images are given (e.g. existing physical disks), but advanced features like dirty bitmaps or backing files are wanted that require use of a proper image format like qcow2. This series adds an incompatible feature bit to qcow2 which allows to use an external data file: Metadata is kept in the qcow2 file like usual, but guest data is written to an external file. Clusters in the data file are not reference counted, instead we use a flat layout where host cluster offset == guest cluster offset. The external data file is therefore readable as a raw image (though writing to it invalidates the associated qcow2 metadata). Features that require refcounting such as internal snapshots or compression are not supposed in such setups. Major changes since the RFC (many more minor changes): - Added a 'data-file-raw' flag to the spec to keep the data file a valid self-consistent raw image (write_zeroes must always be propagated) - Added QAPI documentation - Added some test cases - Implemented data file support or error paths for discard, check, compressed writes, snapshots - Fixed qcow2_co_block_status() - Rearranged qcow2_do_open() code for data files for better error handling (e.g. no more fallback to default data file if an explicit, but invalid data-file option is given) git-backport-diff compared to the RFC: Key: [----] : patches are identical [####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively 001/20:[down] 'qemu-iotests: Test qcow2 preallocation modes' 002/20:[down] 'qcow2: Simplify preallocation code' 003/20:[0027] [FC] 'qcow2: Extend spec for external data files' 004/20:[0014] [FC] 'qcow2: Basic definitions for external data files' 005/20:[----] [--] 'qcow2: Pass bs to qcow2_get_cluster_type()' 006/20:[----] [--] 'qcow2: Prepare qcow2_get_cluster_type() for external data file' 007/20:[----] [--] 'qcow2: Prepare count_contiguous_clusters() for external data file' 008/20:[----] [--] 'qcow2: Don't assume 0 is an invalid cluster offset' 009/20:[down] 'qcow2: Return 0/-errno in qcow2_alloc_compressed_cluster_offset()' 010/20:[down] 'qcow2: Prepare qcow2_co_block_status() for data file' 011/20:[0024] [FC] 'qcow2: External file I/O' 012/20:[down] 'qcow2: Return error for snapshot operation with data file' 013/20:[down] 'qcow2: Support external data file in qemu-img check' 014/20:[0035] [FC] 'qcow2: Add basic data-file infrastructure' 015/20:[0003] [FC] 'qcow2: Creating images with external data file' 016/20:[0057] [FC] 'qcow2: Store data file name in the image' 017/20:[down] 'qcow2: Implement data-file-raw create option' 018/20:[down] 'qemu-iotests: Preallocation with external data file' 019/20:[down] 'qemu-iotests: General tests for qcow2 with external data file' 020/20:[down] 'qemu-iotests: amend with external data file' Kevin Wolf (20): qemu-iotests: Test qcow2 preallocation modes qcow2: Simplify preallocation code qcow2: Extend spec for external data files qcow2: Basic definitions for external data files qcow2: Pass bs to qcow2_get_cluster_type() qcow2: Prepare qcow2_get_cluster_type() for external data file qcow2: Prepare count_contiguous_clusters() for external data file qcow2: Don't assume 0 is an invalid cluster offset qcow2: Return 0/-errno in qcow2_alloc_compressed_cluster_offset() qcow2: Prepare qcow2_co_block_status() for data file qcow2: External file I/O qcow2: Return error for snapshot operation with data file qcow2: Support external data file in qemu-img check qcow2: Add basic data-file infrastructure qcow2: Creating images with external data file qcow2: Store data file name in the image qcow2: Implement data-file-raw create option qemu-iotests: Preallocation with external data file qemu-iotests: General tests for qcow2 with external data file qemu-iotests: amend with external data file qapi/block-core.json | 26 ++- docs/interop/qcow2.txt | 38 ++++- block/qcow2.h | 66 ++++++-- include/block/block_int.h | 2 + block/qcow2-bitmap.c | 7 +- block/qcow2-cache.c | 6 +- block/qcow2-cluster.c | 182 ++++++++++++--------- block/qcow2-refcount.c | 88 ++++++++--- block/qcow2-snapshot.c | 22 ++- block/qcow2.c | 314 +++++++++++++++++++++++++++++++------ tests/qemu-iotests/031.out | 8 +- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/061 | 45 +++++- tests/qemu-iotests/061.out | 103 +++++++++++- tests/qemu-iotests/082.out | 54 +++++++ tests/qemu-iotests/220.out | 2 +- tests/qemu-iotests/243 | 85 ++++++++++ tests/qemu-iotests/243.out | 58 +++++++ tests/qemu-iotests/244 | 200 +++++++++++++++++++++++ tests/qemu-iotests/244.out | 125 +++++++++++++++ tests/qemu-iotests/group | 2 + 21 files changed, 1243 insertions(+), 194 deletions(-) create mode 100755 tests/qemu-iotests/243 create mode 100644 tests/qemu-iotests/243.out create mode 100755 tests/qemu-iotests/244 create mode 100644 tests/qemu-iotests/244.out