From patchwork Mon Jun 5 15:45:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267709 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ED586C7EE2C for ; Mon, 5 Jun 2023 15:47:31 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CP3-0007jP-3B; Mon, 05 Jun 2023 11:45:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CP0-0007iV-Lz for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:45:50 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6COy-0001HZ-JB for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:45:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979948; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FyIrEWo26B59V0T8JUOyas8lfhiEpkvThYsnLnUy6bk=; b=MFQj7/nkMtQ2YsfERhWWiPuf/6FRwQpvspciq9YLO+DhQIeueIHk3gl1ZUfIVcm7vhelw/ qlvsLJIYmHUFvlhjOJP2JKiyqQyEZPAFhSzgSLs8rJdLU+MD6fZprzoxGnnEc9JGZUmWIt 2hklvf2YXPZ3TVWDnQEv/FBlwzZOjLA= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-668-8_nfyQdLPtWTrw4OQz8nOg-1; Mon, 05 Jun 2023 11:45:45 -0400 X-MC-Unique: 8_nfyQdLPtWTrw4OQz8nOg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6602D1C05153; Mon, 5 Jun 2023 15:45:45 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 277321121314; Mon, 5 Jun 2023 15:45:45 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 01/17] util/iov: Make qiov_slice() public Date: Mon, 5 Jun 2023 17:45:25 +0200 Message-Id: <20230605154541.1043261-2-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org We want to inline qemu_iovec_init_extended() in block/io.c for padding requests, and having access to qiov_slice() is useful for this. As a public function, it is renamed to qemu_iovec_slice(). (We will need to count the number of I/O vector elements of a slice there, and then later process this slice. Without qiov_slice(), we would need to call qemu_iovec_subvec_niov(), and all further IOV-processing functions may need to skip prefixing elements to accomodate for a qiov_offset. Because qemu_iovec_subvec_niov() internally calls qiov_slice(), we can just have the block/io.c code call qiov_slice() itself, thus get the number of elements, and also create an iovec array with the superfluous prefixing elements stripped, so the following processing functions no longer need to skip them.) Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Hanna Czenczek Message-Id: <20230411173418.19549-2-hreitz@redhat.com> --- include/qemu/iov.h | 3 +++ util/iov.c | 14 +++++++------- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/include/qemu/iov.h b/include/qemu/iov.h index 9330746680..46fadfb27a 100644 --- a/include/qemu/iov.h +++ b/include/qemu/iov.h @@ -229,6 +229,9 @@ int qemu_iovec_init_extended( void *tail_buf, size_t tail_len); void qemu_iovec_init_slice(QEMUIOVector *qiov, QEMUIOVector *source, size_t offset, size_t len); +struct iovec *qemu_iovec_slice(QEMUIOVector *qiov, + size_t offset, size_t len, + size_t *head, size_t *tail, int *niov); int qemu_iovec_subvec_niov(QEMUIOVector *qiov, size_t offset, size_t len); void qemu_iovec_add(QEMUIOVector *qiov, void *base, size_t len); void qemu_iovec_concat(QEMUIOVector *dst, diff --git a/util/iov.c b/util/iov.c index b4be580022..65a70449da 100644 --- a/util/iov.c +++ b/util/iov.c @@ -378,15 +378,15 @@ static struct iovec *iov_skip_offset(struct iovec *iov, size_t offset, } /* - * qiov_slice + * qemu_iovec_slice * * Find subarray of iovec's, containing requested range. @head would * be offset in first iov (returned by the function), @tail would be * count of extra bytes in last iovec (returned iov + @niov - 1). */ -static struct iovec *qiov_slice(QEMUIOVector *qiov, - size_t offset, size_t len, - size_t *head, size_t *tail, int *niov) +struct iovec *qemu_iovec_slice(QEMUIOVector *qiov, + size_t offset, size_t len, + size_t *head, size_t *tail, int *niov) { struct iovec *iov, *end_iov; @@ -411,7 +411,7 @@ int qemu_iovec_subvec_niov(QEMUIOVector *qiov, size_t offset, size_t len) size_t head, tail; int niov; - qiov_slice(qiov, offset, len, &head, &tail, &niov); + qemu_iovec_slice(qiov, offset, len, &head, &tail, &niov); return niov; } @@ -439,8 +439,8 @@ int qemu_iovec_init_extended( } if (mid_len) { - mid_iov = qiov_slice(mid_qiov, mid_offset, mid_len, - &mid_head, &mid_tail, &mid_niov); + mid_iov = qemu_iovec_slice(mid_qiov, mid_offset, mid_len, + &mid_head, &mid_tail, &mid_niov); } total_niov = !!head_len + mid_niov + !!tail_len; From patchwork Mon Jun 5 15:45:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5C347C7EE23 for ; Mon, 5 Jun 2023 15:48:02 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CP6-0007lP-0n; Mon, 05 Jun 2023 11:45:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CP3-0007jR-37 for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:45:53 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CP0-0001I0-Oe for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:45:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979949; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W9IdGHYywvBoHrKHd/prbEpTXxdeXSE8arL8UjZXsMM=; b=efFpAWUq93MckUKC0Z759MfDwIVc543xQrnEFJ9Rg8CoMqXi2OyEwI49ndnogDZTtWmcni If69hY/7eLr3RUM2NSTxOjDsQNIhTnSJ6HCQlzh8RiC7W3tCywGRa/YDsz128WOO0mSwyQ GqEWElsPM0r/TCTZeaBckw1rvBMubsI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-46-i3X0gkECMPe-5YC_EWk2ow-1; Mon, 05 Jun 2023 11:45:47 -0400 X-MC-Unique: i3X0gkECMPe-5YC_EWk2ow-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id ED2932A5956A; Mon, 5 Jun 2023 15:45:46 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 91D9C140E958; Mon, 5 Jun 2023 15:45:46 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 02/17] block: Collapse padded I/O vecs exceeding IOV_MAX Date: Mon, 5 Jun 2023 17:45:26 +0200 Message-Id: <20230605154541.1043261-3-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org When processing vectored guest requests that are not aligned to the storage request alignment, we pad them by adding head and/or tail buffers for a read-modify-write cycle. The guest can submit I/O vectors up to IOV_MAX (1024) in length, but with this padding, the vector can exceed that limit. As of 4c002cef0e9abe7135d7916c51abce47f7fc1ee2 ("util/iov: make qemu_iovec_init_extended() honest"), we refuse to pad vectors beyond the limit, instead returning an error to the guest. To the guest, this appears as a random I/O error. We should not return an I/O error to the guest when it issued a perfectly valid request. Before 4c002cef0e9abe7135d7916c51abce47f7fc1ee2, we just made the vector longer than IOV_MAX, which generally seems to work (because the guest assumes a smaller alignment than we really have, file-posix's raw_co_prw() will generally see bdrv_qiov_is_aligned() return false, and so emulate the request, so that the IOV_MAX does not matter). However, that does not seem exactly great. I see two ways to fix this problem: 1. We split such long requests into two requests. 2. We join some elements of the vector into new buffers to make it shorter. I am wary of (1), because it seems like it may have unintended side effects. (2) on the other hand seems relatively simple to implement, with hopefully few side effects, so this patch does that. To do this, the use of qemu_iovec_init_extended() in bdrv_pad_request() is effectively replaced by the new function bdrv_create_padded_qiov(), which not only wraps the request IOV with padding head/tail, but also ensures that the resulting vector will not have more than IOV_MAX elements. Putting that functionality into qemu_iovec_init_extended() is infeasible because it requires allocating a bounce buffer; doing so would require many more parameters (buffer alignment, how to initialize the buffer, and out parameters like the buffer, its length, and the original elements), which is not reasonable. Conversely, it is not difficult to move qemu_iovec_init_extended()'s functionality into bdrv_create_padded_qiov() by using public qemu_iovec_* functions, so that is what this patch does. Because bdrv_pad_request() was the only "serious" user of qemu_iovec_init_extended(), the next patch will remove the latter function, so the functionality is not implemented twice. Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2141964 Signed-off-by: Hanna Czenczek Message-Id: <20230411173418.19549-3-hreitz@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/io.c | 166 ++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 151 insertions(+), 15 deletions(-) diff --git a/block/io.c b/block/io.c index f2dfc7c405..30748f0b59 100644 --- a/block/io.c +++ b/block/io.c @@ -1441,6 +1441,14 @@ out: * @merge_reads is true for small requests, * if @buf_len == @head + bytes + @tail. In this case it is possible that both * head and tail exist but @buf_len == align and @tail_buf == @buf. + * + * @write is true for write requests, false for read requests. + * + * If padding makes the vector too long (exceeding IOV_MAX), then we need to + * merge existing vector elements into a single one. @collapse_bounce_buf acts + * as the bounce buffer in such cases. @pre_collapse_qiov has the pre-collapse + * I/O vector elements so for read requests, the data can be copied back after + * the read is done. */ typedef struct BdrvRequestPadding { uint8_t *buf; @@ -1449,11 +1457,17 @@ typedef struct BdrvRequestPadding { size_t head; size_t tail; bool merge_reads; + bool write; QEMUIOVector local_qiov; + + uint8_t *collapse_bounce_buf; + size_t collapse_len; + QEMUIOVector pre_collapse_qiov; } BdrvRequestPadding; static bool bdrv_init_padding(BlockDriverState *bs, int64_t offset, int64_t bytes, + bool write, BdrvRequestPadding *pad) { int64_t align = bs->bl.request_alignment; @@ -1485,6 +1499,8 @@ static bool bdrv_init_padding(BlockDriverState *bs, pad->tail_buf = pad->buf + pad->buf_len - align; } + pad->write = write; + return true; } @@ -1549,8 +1565,23 @@ zero_mem: return 0; } -static void bdrv_padding_destroy(BdrvRequestPadding *pad) +/** + * Free *pad's associated buffers, and perform any necessary finalization steps. + */ +static void bdrv_padding_finalize(BdrvRequestPadding *pad) { + if (pad->collapse_bounce_buf) { + if (!pad->write) { + /* + * If padding required elements in the vector to be collapsed into a + * bounce buffer, copy the bounce buffer content back + */ + qemu_iovec_from_buf(&pad->pre_collapse_qiov, 0, + pad->collapse_bounce_buf, pad->collapse_len); + } + qemu_vfree(pad->collapse_bounce_buf); + qemu_iovec_destroy(&pad->pre_collapse_qiov); + } if (pad->buf) { qemu_vfree(pad->buf); qemu_iovec_destroy(&pad->local_qiov); @@ -1558,6 +1589,101 @@ static void bdrv_padding_destroy(BdrvRequestPadding *pad) memset(pad, 0, sizeof(*pad)); } +/* + * Create pad->local_qiov by wrapping @iov in the padding head and tail, while + * ensuring that the resulting vector will not exceed IOV_MAX elements. + * + * To ensure this, when necessary, the first two or three elements of @iov are + * merged into pad->collapse_bounce_buf and replaced by a reference to that + * bounce buffer in pad->local_qiov. + * + * After performing a read request, the data from the bounce buffer must be + * copied back into pad->pre_collapse_qiov (e.g. by bdrv_padding_finalize()). + */ +static int bdrv_create_padded_qiov(BlockDriverState *bs, + BdrvRequestPadding *pad, + struct iovec *iov, int niov, + size_t iov_offset, size_t bytes) +{ + int padded_niov, surplus_count, collapse_count; + + /* Assert this invariant */ + assert(niov <= IOV_MAX); + + /* + * Cannot pad if resulting length would exceed SIZE_MAX. Returning an error + * to the guest is not ideal, but there is little else we can do. At least + * this will practically never happen on 64-bit systems. + */ + if (SIZE_MAX - pad->head < bytes || + SIZE_MAX - pad->head - bytes < pad->tail) + { + return -EINVAL; + } + + /* Length of the resulting IOV if we just concatenated everything */ + padded_niov = !!pad->head + niov + !!pad->tail; + + qemu_iovec_init(&pad->local_qiov, MIN(padded_niov, IOV_MAX)); + + if (pad->head) { + qemu_iovec_add(&pad->local_qiov, pad->buf, pad->head); + } + + /* + * If padded_niov > IOV_MAX, we cannot just concatenate everything. + * Instead, merge the first two or three elements of @iov to reduce the + * number of vector elements as necessary. + */ + if (padded_niov > IOV_MAX) { + /* + * Only head and tail can have lead to the number of entries exceeding + * IOV_MAX, so we can exceed it by the head and tail at most. We need + * to reduce the number of elements by `surplus_count`, so we merge that + * many elements plus one into one element. + */ + surplus_count = padded_niov - IOV_MAX; + assert(surplus_count <= !!pad->head + !!pad->tail); + collapse_count = surplus_count + 1; + + /* + * Move the elements to collapse into `pad->pre_collapse_qiov`, then + * advance `iov` (and associated variables) by those elements. + */ + qemu_iovec_init(&pad->pre_collapse_qiov, collapse_count); + qemu_iovec_concat_iov(&pad->pre_collapse_qiov, iov, + collapse_count, iov_offset, SIZE_MAX); + iov += collapse_count; + iov_offset = 0; + niov -= collapse_count; + bytes -= pad->pre_collapse_qiov.size; + + /* + * Construct the bounce buffer to match the length of the to-collapse + * vector elements, and for write requests, initialize it with the data + * from those elements. Then add it to `pad->local_qiov`. + */ + pad->collapse_len = pad->pre_collapse_qiov.size; + pad->collapse_bounce_buf = qemu_blockalign(bs, pad->collapse_len); + if (pad->write) { + qemu_iovec_to_buf(&pad->pre_collapse_qiov, 0, + pad->collapse_bounce_buf, pad->collapse_len); + } + qemu_iovec_add(&pad->local_qiov, + pad->collapse_bounce_buf, pad->collapse_len); + } + + qemu_iovec_concat_iov(&pad->local_qiov, iov, niov, iov_offset, bytes); + + if (pad->tail) { + qemu_iovec_add(&pad->local_qiov, + pad->buf + pad->buf_len - pad->tail, pad->tail); + } + + assert(pad->local_qiov.niov == MIN(padded_niov, IOV_MAX)); + return 0; +} + /* * bdrv_pad_request * @@ -1565,6 +1691,8 @@ static void bdrv_padding_destroy(BdrvRequestPadding *pad) * read of padding, bdrv_padding_rmw_read() should be called separately if * needed. * + * @write is true for write requests, false for read requests. + * * Request parameters (@qiov, &qiov_offset, &offset, &bytes) are in-out: * - on function start they represent original request * - on failure or when padding is not needed they are unchanged @@ -1573,26 +1701,34 @@ static void bdrv_padding_destroy(BdrvRequestPadding *pad) static int bdrv_pad_request(BlockDriverState *bs, QEMUIOVector **qiov, size_t *qiov_offset, int64_t *offset, int64_t *bytes, + bool write, BdrvRequestPadding *pad, bool *padded, BdrvRequestFlags *flags) { int ret; + struct iovec *sliced_iov; + int sliced_niov; + size_t sliced_head, sliced_tail; bdrv_check_qiov_request(*offset, *bytes, *qiov, *qiov_offset, &error_abort); - if (!bdrv_init_padding(bs, *offset, *bytes, pad)) { + if (!bdrv_init_padding(bs, *offset, *bytes, write, pad)) { if (padded) { *padded = false; } return 0; } - ret = qemu_iovec_init_extended(&pad->local_qiov, pad->buf, pad->head, - *qiov, *qiov_offset, *bytes, - pad->buf + pad->buf_len - pad->tail, - pad->tail); + sliced_iov = qemu_iovec_slice(*qiov, *qiov_offset, *bytes, + &sliced_head, &sliced_tail, + &sliced_niov); + + /* Guaranteed by bdrv_check_qiov_request() */ + assert(*bytes <= SIZE_MAX); + ret = bdrv_create_padded_qiov(bs, pad, sliced_iov, sliced_niov, + sliced_head, *bytes); if (ret < 0) { - bdrv_padding_destroy(pad); + bdrv_padding_finalize(pad); return ret; } *bytes += pad->head + pad->tail; @@ -1659,8 +1795,8 @@ int coroutine_fn bdrv_co_preadv_part(BdrvChild *child, flags |= BDRV_REQ_COPY_ON_READ; } - ret = bdrv_pad_request(bs, &qiov, &qiov_offset, &offset, &bytes, &pad, - NULL, &flags); + ret = bdrv_pad_request(bs, &qiov, &qiov_offset, &offset, &bytes, false, + &pad, NULL, &flags); if (ret < 0) { goto fail; } @@ -1670,7 +1806,7 @@ int coroutine_fn bdrv_co_preadv_part(BdrvChild *child, bs->bl.request_alignment, qiov, qiov_offset, flags); tracked_request_end(&req); - bdrv_padding_destroy(&pad); + bdrv_padding_finalize(&pad); fail: bdrv_dec_in_flight(bs); @@ -2002,7 +2138,7 @@ bdrv_co_do_zero_pwritev(BdrvChild *child, int64_t offset, int64_t bytes, /* This flag doesn't make sense for padding or zero writes */ flags &= ~BDRV_REQ_REGISTERED_BUF; - padding = bdrv_init_padding(bs, offset, bytes, &pad); + padding = bdrv_init_padding(bs, offset, bytes, true, &pad); if (padding) { assert(!(flags & BDRV_REQ_NO_WAIT)); bdrv_make_request_serialising(req, align); @@ -2050,7 +2186,7 @@ bdrv_co_do_zero_pwritev(BdrvChild *child, int64_t offset, int64_t bytes, } out: - bdrv_padding_destroy(&pad); + bdrv_padding_finalize(&pad); return ret; } @@ -2118,8 +2254,8 @@ int coroutine_fn bdrv_co_pwritev_part(BdrvChild *child, * bdrv_co_do_zero_pwritev() does aligning by itself, so, we do * alignment only if there is no ZERO flag. */ - ret = bdrv_pad_request(bs, &qiov, &qiov_offset, &offset, &bytes, &pad, - &padded, &flags); + ret = bdrv_pad_request(bs, &qiov, &qiov_offset, &offset, &bytes, true, + &pad, &padded, &flags); if (ret < 0) { return ret; } @@ -2149,7 +2285,7 @@ int coroutine_fn bdrv_co_pwritev_part(BdrvChild *child, ret = bdrv_aligned_pwritev(child, &req, offset, bytes, align, qiov, qiov_offset, flags); - bdrv_padding_destroy(&pad); + bdrv_padding_finalize(&pad); out: tracked_request_end(&req); From patchwork Mon Jun 5 15:45:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267710 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C7A89C7EE2D for ; Mon, 5 Jun 2023 15:47:32 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CPA-0007nF-EV; Mon, 05 Jun 2023 11:46:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CP3-0007jc-MT for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:45:55 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CP0-0001I3-Rj for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:45:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979950; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BfJ02aGvCqfV0zX1lhxaJrquq3qvSsjbP7hNNd0fv+0=; b=BbtXVwVErc1KIsGJw90o5APUDIGEwMHfRqgWXd0MiBJixhFuKyVtRuHLNgnghSqeIY0i3k tlKTSp15WyyKve3QhAYcHHYLxncFACfUr42AOZt2F4gEtNkaApqAIUvM24hLZ6ZDQGnM0w PzGaf80K9KxHSNVxT0ZJUtzh67qjulo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-509-m6-dilYeNxSLrzB3du-WFQ-1; Mon, 05 Jun 2023 11:45:48 -0400 X-MC-Unique: m6-dilYeNxSLrzB3du-WFQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 86AA4801182; Mon, 5 Jun 2023 15:45:48 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 465302166B25; Mon, 5 Jun 2023 15:45:48 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 03/17] util/iov: Remove qemu_iovec_init_extended() Date: Mon, 5 Jun 2023 17:45:27 +0200 Message-Id: <20230605154541.1043261-4-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org bdrv_pad_request() was the main user of qemu_iovec_init_extended(). HEAD^ has removed that use, so we can remove qemu_iovec_init_extended() now. The only remaining user is qemu_iovec_init_slice(), which can easily inline the small part it really needs. Note that qemu_iovec_init_extended() offered a memcpy() optimization to initialize the new I/O vector. qemu_iovec_concat_iov(), which is used to replace its functionality, does not, but calls qemu_iovec_add() for every single element. If we decide this optimization was important, we will need to re-implement it in qemu_iovec_concat_iov(), which might also benefit its pre-existing users. Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Hanna Czenczek Message-Id: <20230411173418.19549-4-hreitz@redhat.com> --- include/qemu/iov.h | 5 --- util/iov.c | 79 +++++++--------------------------------------- 2 files changed, 11 insertions(+), 73 deletions(-) diff --git a/include/qemu/iov.h b/include/qemu/iov.h index 46fadfb27a..63a1c01965 100644 --- a/include/qemu/iov.h +++ b/include/qemu/iov.h @@ -222,11 +222,6 @@ static inline void *qemu_iovec_buf(QEMUIOVector *qiov) void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint); void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov); -int qemu_iovec_init_extended( - QEMUIOVector *qiov, - void *head_buf, size_t head_len, - QEMUIOVector *mid_qiov, size_t mid_offset, size_t mid_len, - void *tail_buf, size_t tail_len); void qemu_iovec_init_slice(QEMUIOVector *qiov, QEMUIOVector *source, size_t offset, size_t len); struct iovec *qemu_iovec_slice(QEMUIOVector *qiov, diff --git a/util/iov.c b/util/iov.c index 65a70449da..866fb577f3 100644 --- a/util/iov.c +++ b/util/iov.c @@ -416,70 +416,6 @@ int qemu_iovec_subvec_niov(QEMUIOVector *qiov, size_t offset, size_t len) return niov; } -/* - * Compile new iovec, combining @head_buf buffer, sub-qiov of @mid_qiov, - * and @tail_buf buffer into new qiov. - */ -int qemu_iovec_init_extended( - QEMUIOVector *qiov, - void *head_buf, size_t head_len, - QEMUIOVector *mid_qiov, size_t mid_offset, size_t mid_len, - void *tail_buf, size_t tail_len) -{ - size_t mid_head, mid_tail; - int total_niov, mid_niov = 0; - struct iovec *p, *mid_iov = NULL; - - assert(mid_qiov->niov <= IOV_MAX); - - if (SIZE_MAX - head_len < mid_len || - SIZE_MAX - head_len - mid_len < tail_len) - { - return -EINVAL; - } - - if (mid_len) { - mid_iov = qemu_iovec_slice(mid_qiov, mid_offset, mid_len, - &mid_head, &mid_tail, &mid_niov); - } - - total_niov = !!head_len + mid_niov + !!tail_len; - if (total_niov > IOV_MAX) { - return -EINVAL; - } - - if (total_niov == 1) { - qemu_iovec_init_buf(qiov, NULL, 0); - p = &qiov->local_iov; - } else { - qiov->niov = qiov->nalloc = total_niov; - qiov->size = head_len + mid_len + tail_len; - p = qiov->iov = g_new(struct iovec, qiov->niov); - } - - if (head_len) { - p->iov_base = head_buf; - p->iov_len = head_len; - p++; - } - - assert(!mid_niov == !mid_len); - if (mid_niov) { - memcpy(p, mid_iov, mid_niov * sizeof(*p)); - p[0].iov_base = (uint8_t *)p[0].iov_base + mid_head; - p[0].iov_len -= mid_head; - p[mid_niov - 1].iov_len -= mid_tail; - p += mid_niov; - } - - if (tail_len) { - p->iov_base = tail_buf; - p->iov_len = tail_len; - } - - return 0; -} - /* * Check if the contents of subrange of qiov data is all zeroes. */ @@ -511,14 +447,21 @@ bool qemu_iovec_is_zero(QEMUIOVector *qiov, size_t offset, size_t bytes) void qemu_iovec_init_slice(QEMUIOVector *qiov, QEMUIOVector *source, size_t offset, size_t len) { - int ret; + struct iovec *slice_iov; + int slice_niov; + size_t slice_head, slice_tail; assert(source->size >= len); assert(source->size - len >= offset); - /* We shrink the request, so we can't overflow neither size_t nor MAX_IOV */ - ret = qemu_iovec_init_extended(qiov, NULL, 0, source, offset, len, NULL, 0); - assert(ret == 0); + slice_iov = qemu_iovec_slice(source, offset, len, + &slice_head, &slice_tail, &slice_niov); + if (slice_niov == 1) { + qemu_iovec_init_buf(qiov, slice_iov[0].iov_base + slice_head, len); + } else { + qemu_iovec_init(qiov, slice_niov); + qemu_iovec_concat_iov(qiov, slice_iov, slice_niov, slice_head, len); + } } void qemu_iovec_destroy(QEMUIOVector *qiov) From patchwork Mon Jun 5 15:45:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267702 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2DB88C7EE23 for ; Mon, 5 Jun 2023 15:46:45 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CP7-0007m1-CS; Mon, 05 Jun 2023 11:45:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CP4-0007jh-Nr for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:45:55 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CP2-0001IT-JY for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:45:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979951; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OJ5wn2P538auDnJD+SuCeINdH4RRyR0W835mOsRnRm4=; b=EFP6ORSB1cjNSsBSaZtftaB0Bve8JwJrJihmSAXAYrrYNlyqwDQA9Iq3HvgYzIfGvanT6Z qJAcMNREeBgkfk1iwncDm6q4Sd+JzhiSmxKiconwwsVDOv+mqcVzeL8MIrZLifjoXuuqP+ T4rAKShcRq5rLjBImF0gcaeiAG2qvzM= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-449-hPtlzKnAOWqfY90joyBilA-1; Mon, 05 Jun 2023 11:45:50 -0400 X-MC-Unique: hPtlzKnAOWqfY90joyBilA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3BA461C03369; Mon, 5 Jun 2023 15:45:50 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B3DA12166B25; Mon, 5 Jun 2023 15:45:49 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 04/17] iotests/iov-padding: New test Date: Mon, 5 Jun 2023 17:45:28 +0200 Message-Id: <20230605154541.1043261-5-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Test that even vectored IO requests with 1024 vector elements that are not aligned to the device's request alignment will succeed. Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Hanna Czenczek Message-Id: <20230411173418.19549-5-hreitz@redhat.com> --- tests/qemu-iotests/tests/iov-padding | 85 ++++++++++++++++++++++++ tests/qemu-iotests/tests/iov-padding.out | 59 ++++++++++++++++ 2 files changed, 144 insertions(+) create mode 100755 tests/qemu-iotests/tests/iov-padding create mode 100644 tests/qemu-iotests/tests/iov-padding.out diff --git a/tests/qemu-iotests/tests/iov-padding b/tests/qemu-iotests/tests/iov-padding new file mode 100755 index 0000000000..b9604900c7 --- /dev/null +++ b/tests/qemu-iotests/tests/iov-padding @@ -0,0 +1,85 @@ +#!/usr/bin/env bash +# group: rw quick +# +# Check the interaction of request padding (to fit alignment restrictions) with +# vectored I/O from the guest +# +# Copyright Red Hat +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +seq=$(basename $0) +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +cd .. +. ./common.rc +. ./common.filter + +_supported_fmt raw +_supported_proto file + +_make_test_img 1M + +IMGSPEC="driver=blkdebug,align=4096,image.driver=file,image.filename=$TEST_IMG" + +# Four combinations: +# - Offset 4096, length 1023 * 512 + 512: Fully aligned to 4k +# - Offset 4096, length 1023 * 512 + 4096: Head is aligned, tail is not +# - Offset 512, length 1023 * 512 + 512: Neither head nor tail are aligned +# - Offset 512, length 1023 * 512 + 4096: Tail is aligned, head is not +for start_offset in 4096 512; do + for last_element_length in 512 4096; do + length=$((1023 * 512 + $last_element_length)) + + echo + echo "== performing 1024-element vectored requests to image (offset: $start_offset; length: $length) ==" + + # Fill with data for testing + $QEMU_IO -c 'write -P 1 0 1M' "$TEST_IMG" | _filter_qemu_io + + # 1023 512-byte buffers, and then one with length $last_element_length + cmd_params="-P 2 $start_offset $(yes 512 | head -n 1023 | tr '\n' ' ') $last_element_length" + QEMU_IO_OPTIONS="$QEMU_IO_OPTIONS_NO_FMT" $QEMU_IO \ + -c "writev $cmd_params" \ + --image-opts \ + "$IMGSPEC" \ + | _filter_qemu_io + + # Read all patterns -- read the part we just wrote with writev twice, + # once "normally", and once with a readv, so we see that that works, too + QEMU_IO_OPTIONS="$QEMU_IO_OPTIONS_NO_FMT" $QEMU_IO \ + -c "read -P 1 0 $start_offset" \ + -c "read -P 2 $start_offset $length" \ + -c "readv $cmd_params" \ + -c "read -P 1 $((start_offset + length)) $((1024 * 1024 - length - start_offset))" \ + --image-opts \ + "$IMGSPEC" \ + | _filter_qemu_io + done +done + +# success, all done +echo "*** done" +rm -f $seq.full +status=0 diff --git a/tests/qemu-iotests/tests/iov-padding.out b/tests/qemu-iotests/tests/iov-padding.out new file mode 100644 index 0000000000..e07a91fac7 --- /dev/null +++ b/tests/qemu-iotests/tests/iov-padding.out @@ -0,0 +1,59 @@ +QA output created by iov-padding +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 + +== performing 1024-element vectored requests to image (offset: 4096; length: 524288) == +wrote 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 524288/524288 bytes at offset 4096 +512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 4096/4096 bytes at offset 0 +4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 524288/524288 bytes at offset 4096 +512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 524288/524288 bytes at offset 4096 +512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 520192/520192 bytes at offset 528384 +508 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +== performing 1024-element vectored requests to image (offset: 4096; length: 527872) == +wrote 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 527872/527872 bytes at offset 4096 +515.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 4096/4096 bytes at offset 0 +4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 527872/527872 bytes at offset 4096 +515.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 527872/527872 bytes at offset 4096 +515.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 516608/516608 bytes at offset 531968 +504.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +== performing 1024-element vectored requests to image (offset: 512; length: 524288) == +wrote 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 524288/524288 bytes at offset 512 +512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 512/512 bytes at offset 0 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 524288/524288 bytes at offset 512 +512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 524288/524288 bytes at offset 512 +512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 523776/523776 bytes at offset 524800 +511.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +== performing 1024-element vectored requests to image (offset: 512; length: 527872) == +wrote 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 527872/527872 bytes at offset 512 +515.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 512/512 bytes at offset 0 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 527872/527872 bytes at offset 512 +515.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 527872/527872 bytes at offset 512 +515.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 520192/520192 bytes at offset 528384 +508 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +*** done From patchwork Mon Jun 5 15:45:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267704 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 41BFFC7EE2F for ; Mon, 5 Jun 2023 15:46:45 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CPD-0007oR-WB; Mon, 05 Jun 2023 11:46:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CP5-0007km-Ka for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:45:55 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CP3-0001Iu-Sg for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:45:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979953; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z8HstDAZVktTJkrB//J+fQO/MgjmPuN3y1/ZrbZM3A8=; b=gUc1UVyDjdVpmk8pPMguoPjOGvPNjLoWmGlTgndjTs4pyCPH4a00lbgXYlgWzeOlT4DI0k uzG2w6PZf+7DngCuNb4c8vBUJu/2aH5dbrokb0DY+NS8gbV3RQzgOwAVOPqkzTB5QHX7aQ 1acn4PCbXUzC0ybQfDliwuMEgq1sU4k= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-515-SPCaotr_MbCcIy92dSEmNQ-1; Mon, 05 Jun 2023 11:45:52 -0400 X-MC-Unique: SPCaotr_MbCcIy92dSEmNQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A7303801585; Mon, 5 Jun 2023 15:45:51 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 689881121314; Mon, 5 Jun 2023 15:45:51 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 05/17] parallels: Out of image offset in BAT leads to image inflation Date: Mon, 5 Jun 2023 17:45:29 +0200 Message-Id: <20230605154541.1043261-6-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Alexander Ivanov data_end field in BDRVParallelsState is set to the biggest offset present in BAT. If this offset is outside of the image, any further write will create the cluster at this offset and/or the image will be truncated to this offset on close. This is definitely not correct. Raise an error in parallels_open() if data_end points outside the image and it is not a check (let the check to repaire the image). Set data_end to the end of the cluster with the last correct offset. Signed-off-by: Alexander Ivanov Message-Id: <20230424093147.197643-2-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek Signed-off-by: Hanna Czenczek --- block/parallels.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/block/parallels.c b/block/parallels.c index d8a3f13e24..7b6d770f8e 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -733,6 +733,7 @@ static int parallels_open(BlockDriverState *bs, QDict *options, int flags, BDRVParallelsState *s = bs->opaque; ParallelsHeader ph; int ret, size, i; + int64_t file_nb_sectors; QemuOpts *opts = NULL; Error *local_err = NULL; char *buf; @@ -742,6 +743,11 @@ static int parallels_open(BlockDriverState *bs, QDict *options, int flags, return ret; } + file_nb_sectors = bdrv_nb_sectors(bs->file->bs); + if (file_nb_sectors < 0) { + return -EINVAL; + } + ret = bdrv_pread(bs->file, 0, sizeof(ph), &ph, 0); if (ret < 0) { goto fail; @@ -806,6 +812,17 @@ static int parallels_open(BlockDriverState *bs, QDict *options, int flags, for (i = 0; i < s->bat_size; i++) { int64_t off = bat2sect(s, i); + if (off >= file_nb_sectors) { + if (flags & BDRV_O_CHECK) { + continue; + } + error_setg(errp, "parallels: Offset %" PRIi64 " in BAT[%d] entry " + "is larger than file size (%" PRIi64 ")", + off << BDRV_SECTOR_BITS, i, + file_nb_sectors << BDRV_SECTOR_BITS); + ret = -EINVAL; + goto fail; + } if (off >= s->data_end) { s->data_end = off + s->tracks; } From patchwork Mon Jun 5 15:45:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 40049C7EE24 for ; Mon, 5 Jun 2023 15:46:45 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CPN-0007ry-NE; Mon, 05 Jun 2023 11:46:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPC-0007o2-BR for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:02 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CP8-0001LE-C0 for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979957; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dmkqy+yiuUBc+PXnJnUOXg+0u9s1I12UE+YYrguSpOw=; b=M9rfhqwNusNf0SrAiYtBbwhV85XZKDPBTee8jtmBi+9G59NkSEOMFSqPCP34CVv7tyBAMg sP0zUv1BAN0f7lqrysOZ6Jo9adpunYFyrJ7XPUqcF7wRT17z5ObYMiAbSZwF+Jzh5hs4oX X2ePz8ChEVSHaFmSsm+rbzlSH45CCCs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-537-soljBuEXNSKwgLDdREwAFw-1; Mon, 05 Jun 2023 11:45:53 -0400 X-MC-Unique: soljBuEXNSKwgLDdREwAFw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1AEB6101A531; Mon, 5 Jun 2023 15:45:53 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D12CF40218C; Mon, 5 Jun 2023 15:45:52 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 06/17] parallels: Fix high_off calculation in parallels_co_check() Date: Mon, 5 Jun 2023 17:45:30 +0200 Message-Id: <20230605154541.1043261-7-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Alexander Ivanov Don't let high_off be more than the file size even if we don't fix the image. Signed-off-by: Alexander Ivanov Reviewed-by: Denis V. Lunev Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <20230424093147.197643-3-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek Signed-off-by: Hanna Czenczek --- block/parallels.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/parallels.c b/block/parallels.c index 7b6d770f8e..204d20685b 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -462,12 +462,12 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR", i); res->corruptions++; if (fix & BDRV_FIX_ERRORS) { - prev_off = 0; s->bat_bitmap[i] = 0; res->corruptions_fixed++; flush_bat = true; - continue; } + prev_off = 0; + continue; } res->bfi.allocated_clusters++; From patchwork Mon Jun 5 15:45:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267706 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CCC00C7EE23 for ; Mon, 5 Jun 2023 15:46:48 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CPd-00080u-Tp; Mon, 05 Jun 2023 11:46:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPG-0007pr-21 for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:07 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPD-0001LX-0n for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979960; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=04pv5oWfRMY/HzoR/mhhjMw1YcRoMEudYrFfPVYCe2g=; b=iEpVGNp9IvT6dBAZdDN0n4eU2phF2e3CeBAmlaJd4NZJKdHYyzYFCX7+1crM7DjbKA4ogr EWpyGAV0dnSRXliRkYdImRGfFcUKd8G5UssDjVrMzni/7Tsb1GzEzCY5DBy07SUzGM/6Rr jG+7k5rbRPxuef6yaWpyGlzFVKBHyRM= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-317-2m2rymClOIOgFr2mh87Gpw-1; Mon, 05 Jun 2023 11:45:56 -0400 X-MC-Unique: 2m2rymClOIOgFr2mh87Gpw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0F1663801ED8; Mon, 5 Jun 2023 15:45:56 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C47391121314; Mon, 5 Jun 2023 15:45:55 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 07/17] parallels: Fix image_end_offset and data_end after out-of-image check Date: Mon, 5 Jun 2023 17:45:31 +0200 Message-Id: <20230605154541.1043261-8-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Alexander Ivanov Set data_end to the end of the last cluster inside the image. In such a way we can be sure that corrupted offsets in the BAT can't affect on the image size. If there are no allocated clusters set image_end_offset by data_end. Signed-off-by: Alexander Ivanov Reviewed-by: Denis V. Lunev Message-Id: <20230424093147.197643-4-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek Signed-off-by: Hanna Czenczek --- block/parallels.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/block/parallels.c b/block/parallels.c index 204d20685b..ea382e8382 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -490,7 +490,13 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, } } - res->image_end_offset = high_off + s->cluster_size; + if (high_off == 0) { + res->image_end_offset = s->data_end << BDRV_SECTOR_BITS; + } else { + res->image_end_offset = high_off + s->cluster_size; + s->data_end = res->image_end_offset >> BDRV_SECTOR_BITS; + } + if (size > res->image_end_offset) { int64_t count; count = DIV_ROUND_UP(size - res->image_end_offset, s->cluster_size); From patchwork Mon Jun 5 15:45:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267707 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8F53DC7EE24 for ; Mon, 5 Jun 2023 15:47:28 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CPN-0007s1-Uy; Mon, 05 Jun 2023 11:46:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPE-0007oy-3F for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:04 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPA-0001LQ-5s for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979959; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RAYYsp1WmlOGq274EViIoxRctL5gJmI09RH9IeK3iU8=; b=FF1UHbbWNMpWXq6BrDVz7Z5HjJ5t4i2/OmPUJVIIcbvl8M7nIM5FTXQrl9AT+PrEEecZyF b3oWN8niOol561Wncm2qIj2FtsAxqqtgrWlQ/41eLvPBl8dbbHDIn+FCgO4h00xWAcHxWe LGoUnFM9YIVvXl0UsmVy5j3oizFLG04= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-297-HvYHfSjkOx6NsuWy3ws-aA-1; Mon, 05 Jun 2023 11:45:58 -0400 X-MC-Unique: HvYHfSjkOx6NsuWy3ws-aA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 850A785A5A8; Mon, 5 Jun 2023 15:45:57 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 47307492B00; Mon, 5 Jun 2023 15:45:57 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 08/17] parallels: create parallels_set_bat_entry_helper() to assign BAT value Date: Mon, 5 Jun 2023 17:45:32 +0200 Message-Id: <20230605154541.1043261-9-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Alexander Ivanov This helper will be reused in next patches during parallels_co_check rework to simplify its code. Signed-off-by: Alexander Ivanov Reviewed-by: Denis V. Lunev Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <20230424093147.197643-5-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek Signed-off-by: Hanna Czenczek --- block/parallels.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/block/parallels.c b/block/parallels.c index ea382e8382..db6426e5f9 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -165,6 +165,13 @@ static int64_t block_status(BDRVParallelsState *s, int64_t sector_num, return start_off; } +static void parallels_set_bat_entry(BDRVParallelsState *s, + uint32_t index, uint32_t offset) +{ + s->bat_bitmap[index] = cpu_to_le32(offset); + bitmap_set(s->bat_dirty_bmap, bat_entry_off(index) / s->bat_dirty_block, 1); +} + static int64_t coroutine_fn GRAPH_RDLOCK allocate_clusters(BlockDriverState *bs, int64_t sector_num, int nb_sectors, int *pnum) @@ -251,10 +258,8 @@ allocate_clusters(BlockDriverState *bs, int64_t sector_num, } for (i = 0; i < to_allocate; i++) { - s->bat_bitmap[idx + i] = cpu_to_le32(s->data_end / s->off_multiplier); + parallels_set_bat_entry(s, idx + i, s->data_end / s->off_multiplier); s->data_end += s->tracks; - bitmap_set(s->bat_dirty_bmap, - bat_entry_off(idx + i) / s->bat_dirty_block, 1); } return bat2sect(s, idx) + sector_num % s->tracks; From patchwork Mon Jun 5 15:45:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267708 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A5E61C7EE24 for ; Mon, 5 Jun 2023 15:47:31 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CPZ-0007wz-0F; Mon, 05 Jun 2023 11:46:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPG-0007pq-1s for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:07 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPC-0001Lg-1v for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979961; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xkomnlOoVcp56VSOijJDIFig922LR7JqMs93U2Hg0RA=; b=Vfh44HiqJJXWjYnDc8RFazEg69gIjEL9TUB0koY9M41BEHJSeroFap9un3amlzBIL9YHo7 /jB3TSYBSs7vm/1Y7fdxqZpAt8h1z8s+rHRwerWI3kFG1z3QggyUjnEKsqMPkmTf5L9Tgc 6nJkoM85/qk52NPK7UVSJZ+hEsT1zcs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-483-kNRQEoQ5NTyHoeQmbU6lAA-1; Mon, 05 Jun 2023 11:45:59 -0400 X-MC-Unique: kNRQEoQ5NTyHoeQmbU6lAA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E90AE101A53A; Mon, 5 Jun 2023 15:45:58 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id ABA5A40CFD47; Mon, 5 Jun 2023 15:45:58 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 09/17] parallels: Use generic infrastructure for BAT writing in parallels_co_check() Date: Mon, 5 Jun 2023 17:45:33 +0200 Message-Id: <20230605154541.1043261-10-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Alexander Ivanov BAT is written in the context of conventional operations over the image inside bdrv_co_flush() when it calls parallels_co_flush_to_os() callback. Thus we should not modify BAT array directly, but call parallels_set_bat_entry() helper and bdrv_co_flush() further on. After that there is no need to manually write BAT and track its modification. This makes code more generic and allows to split parallels_set_bat_entry() for independent pieces. Signed-off-by: Alexander Ivanov Reviewed-by: Denis V. Lunev Message-Id: <20230424093147.197643-6-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek Signed-off-by: Hanna Czenczek --- block/parallels.c | 23 ++++++++++------------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/block/parallels.c b/block/parallels.c index db6426e5f9..fa8a56a948 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -427,9 +427,8 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, { BDRVParallelsState *s = bs->opaque; int64_t size, prev_off, high_off; - int ret; + int ret = 0; uint32_t i; - bool flush_bat = false; size = bdrv_getlength(bs->file->bs); if (size < 0) { @@ -467,9 +466,8 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR", i); res->corruptions++; if (fix & BDRV_FIX_ERRORS) { - s->bat_bitmap[i] = 0; + parallels_set_bat_entry(s, i, 0); res->corruptions_fixed++; - flush_bat = true; } prev_off = 0; continue; @@ -486,15 +484,6 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, prev_off = off; } - ret = 0; - if (flush_bat) { - ret = bdrv_co_pwrite_sync(bs->file, 0, s->header_size, s->header, 0); - if (ret < 0) { - res->check_errors++; - goto out; - } - } - if (high_off == 0) { res->image_end_offset = s->data_end << BDRV_SECTOR_BITS; } else { @@ -529,6 +518,14 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, out: qemu_co_mutex_unlock(&s->lock); + + if (ret == 0) { + ret = bdrv_co_flush(bs); + if (ret < 0) { + res->check_errors++; + } + } + return ret; } From patchwork Mon Jun 5 15:45:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267712 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9EE00C7EE2D for ; Mon, 5 Jun 2023 15:47:51 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CPw-0008Mx-Ei; Mon, 05 Jun 2023 11:46:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPU-0007tM-G5 for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:23 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPD-0001Lw-TY for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979962; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XxmLxQpEJrjP10A1yWPKilviO+OTFPazcS5ehkNvh0s=; b=hL4Ziz7pjKAI2jFXbgkAMNelKYrwtuabAHZgU2FMyhDu1DYNXMIiNqjw2bFJ1xRUnpFmCs /mnMbEx/C6g5UXx9VyMOLlQ0PQudmYJ4KmnwBHy1WTixAombtdBVSvE1cxF0VrZ2C0QCEO W5nTNRYqZ3PV8pBxceXGVn5FcKh/L7k= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-416-Z_qAoFtHOseH7nfdEGaG7A-1; Mon, 05 Jun 2023 11:46:00 -0400 X-MC-Unique: Z_qAoFtHOseH7nfdEGaG7A-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 616E5802E58; Mon, 5 Jun 2023 15:46:00 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2362A40CFD47; Mon, 5 Jun 2023 15:46:00 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 10/17] parallels: Move check of unclean image to a separate function Date: Mon, 5 Jun 2023 17:45:34 +0200 Message-Id: <20230605154541.1043261-11-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Alexander Ivanov We will add more and more checks so we need a better code structure in parallels_co_check. Let each check performs in a separate loop in a separate helper. Signed-off-by: Alexander Ivanov Reviewed-by: Denis V. Lunev Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <20230424093147.197643-7-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek Signed-off-by: Hanna Czenczek --- block/parallels.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/block/parallels.c b/block/parallels.c index fa8a56a948..94b7408b6b 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -420,6 +420,25 @@ parallels_co_readv(BlockDriverState *bs, int64_t sector_num, int nb_sectors, return ret; } +static void parallels_check_unclean(BlockDriverState *bs, + BdrvCheckResult *res, + BdrvCheckMode fix) +{ + BDRVParallelsState *s = bs->opaque; + + if (!s->header_unclean) { + return; + } + + fprintf(stderr, "%s image was not closed correctly\n", + fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR"); + res->corruptions++; + if (fix & BDRV_FIX_ERRORS) { + /* parallels_close will do the job right */ + res->corruptions_fixed++; + s->header_unclean = false; + } +} static int coroutine_fn GRAPH_RDLOCK parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, @@ -437,16 +456,8 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, } qemu_co_mutex_lock(&s->lock); - if (s->header_unclean) { - fprintf(stderr, "%s image was not closed correctly\n", - fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR"); - res->corruptions++; - if (fix & BDRV_FIX_ERRORS) { - /* parallels_close will do the job right */ - res->corruptions_fixed++; - s->header_unclean = false; - } - } + + parallels_check_unclean(bs, res, fix); res->bfi.total_clusters = s->bat_size; res->bfi.compressed_clusters = 0; /* compression is not supported */ From patchwork Mon Jun 5 15:45:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267714 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8147CC7EE23 for ; Mon, 5 Jun 2023 15:48:31 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CPh-00081X-Sc; Mon, 05 Jun 2023 11:46:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPN-0007rv-Ke for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:13 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPG-0001Mf-Rg for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979965; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZUQr/xmQi6Lc41sX7GRbuzs/MT4gD1nKwSC2k47KZnE=; b=MKgkTHDMvlGpuvGU3L/KhM1+8h4wSiPJBv7PXfvCA0csQXcKwDeplVVTZyhBL6JEuYzyed VfsIgZg5Eu2zTCWWERidoUwmcyB4wiZdrrDhyAV7WTZ2ZoDLaW3gCiVoENagrLZybHYYJ7 d0Uiee8FV7MXAM4VnOI/H4K5L7NSHBM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-561-c-V3yZIUNUec6P5aU3q6KQ-1; Mon, 05 Jun 2023 11:46:02 -0400 X-MC-Unique: c-V3yZIUNUec6P5aU3q6KQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EA31D811E8D; Mon, 5 Jun 2023 15:46:01 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AB5EC2166B27; Mon, 5 Jun 2023 15:46:01 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 11/17] parallels: Move check of cluster outside image to a separate function Date: Mon, 5 Jun 2023 17:45:35 +0200 Message-Id: <20230605154541.1043261-12-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Alexander Ivanov We will add more and more checks so we need a better code structure in parallels_co_check. Let each check performs in a separate loop in a separate helper. Signed-off-by: Alexander Ivanov Reviewed-by: Denis V. Lunev Message-Id: <20230424093147.197643-8-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek Signed-off-by: Hanna Czenczek --- block/parallels.c | 75 +++++++++++++++++++++++++++++++---------------- 1 file changed, 49 insertions(+), 26 deletions(-) diff --git a/block/parallels.c b/block/parallels.c index 94b7408b6b..7f0f72e879 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -440,13 +440,55 @@ static void parallels_check_unclean(BlockDriverState *bs, } } +static int coroutine_fn GRAPH_RDLOCK +parallels_check_outside_image(BlockDriverState *bs, BdrvCheckResult *res, + BdrvCheckMode fix) +{ + BDRVParallelsState *s = bs->opaque; + uint32_t i; + int64_t off, high_off, size; + + size = bdrv_getlength(bs->file->bs); + if (size < 0) { + res->check_errors++; + return size; + } + + high_off = 0; + for (i = 0; i < s->bat_size; i++) { + off = bat2sect(s, i) << BDRV_SECTOR_BITS; + if (off > size) { + fprintf(stderr, "%s cluster %u is outside image\n", + fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR", i); + res->corruptions++; + if (fix & BDRV_FIX_ERRORS) { + parallels_set_bat_entry(s, i, 0); + res->corruptions_fixed++; + } + continue; + } + if (high_off < off) { + high_off = off; + } + } + + if (high_off == 0) { + res->image_end_offset = s->data_end << BDRV_SECTOR_BITS; + } else { + res->image_end_offset = high_off + s->cluster_size; + s->data_end = res->image_end_offset >> BDRV_SECTOR_BITS; + } + + return 0; +} + static int coroutine_fn GRAPH_RDLOCK parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, BdrvCheckMode fix) { BDRVParallelsState *s = bs->opaque; - int64_t size, prev_off, high_off; - int ret = 0; + int64_t size, prev_off; + int ret; uint32_t i; size = bdrv_getlength(bs->file->bs); @@ -459,10 +501,14 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, parallels_check_unclean(bs, res, fix); + ret = parallels_check_outside_image(bs, res, fix); + if (ret < 0) { + goto out; + } + res->bfi.total_clusters = s->bat_size; res->bfi.compressed_clusters = 0; /* compression is not supported */ - high_off = 0; prev_off = 0; for (i = 0; i < s->bat_size; i++) { int64_t off = bat2sect(s, i) << BDRV_SECTOR_BITS; @@ -471,23 +517,7 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, continue; } - /* cluster outside the image */ - if (off > size) { - fprintf(stderr, "%s cluster %u is outside image\n", - fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR", i); - res->corruptions++; - if (fix & BDRV_FIX_ERRORS) { - parallels_set_bat_entry(s, i, 0); - res->corruptions_fixed++; - } - prev_off = 0; - continue; - } - res->bfi.allocated_clusters++; - if (off > high_off) { - high_off = off; - } if (prev_off != 0 && (prev_off + s->cluster_size) != off) { res->bfi.fragmented_clusters++; @@ -495,13 +525,6 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, prev_off = off; } - if (high_off == 0) { - res->image_end_offset = s->data_end << BDRV_SECTOR_BITS; - } else { - res->image_end_offset = high_off + s->cluster_size; - s->data_end = res->image_end_offset >> BDRV_SECTOR_BITS; - } - if (size > res->image_end_offset) { int64_t count; count = DIV_ROUND_UP(size - res->image_end_offset, s->cluster_size); From patchwork Mon Jun 5 15:45:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 06CB0C7EE32 for ; Mon, 5 Jun 2023 15:46:45 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CPj-000834-1a; Mon, 05 Jun 2023 11:46:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPN-0007rx-Ka for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:13 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPI-0001Mv-En for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979967; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=57nffnxmzVa15uFCO61B1MOGlJpdbfkEl3eLQ3KLneg=; b=HJys1q5ln7NdR15MPhz7Q31MenElyWZ5/CQfFSJELbEqn1aScdZTgvExQxE+lBJ+N5h8TR 8CNVt6aIy/IcS7El4IgcvxQTsCUQgdprm5ka4XPIkm6s09Mw95uEEY6GMI8STtuNIXRikd 3DH/au548TthT1F1eUhNjM2rpADGsoQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-498-9LPd79g6MZCGSQQladdcbQ-1; Mon, 05 Jun 2023 11:46:03 -0400 X-MC-Unique: 9LPd79g6MZCGSQQladdcbQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5C03F811E85; Mon, 5 Jun 2023 15:46:03 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1ED5B403174; Mon, 5 Jun 2023 15:46:02 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 12/17] parallels: Fix statistics calculation Date: Mon, 5 Jun 2023 17:45:36 +0200 Message-Id: <20230605154541.1043261-13-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Alexander Ivanov Exclude out-of-image clusters from allocated and fragmented clusters calculation. Signed-off-by: Alexander Ivanov Message-Id: <20230424093147.197643-9-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek Signed-off-by: Hanna Czenczek --- block/parallels.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/block/parallels.c b/block/parallels.c index 7f0f72e879..fd1e2860d0 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -512,7 +512,11 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, prev_off = 0; for (i = 0; i < s->bat_size; i++) { int64_t off = bat2sect(s, i) << BDRV_SECTOR_BITS; - if (off == 0) { + /* + * If BDRV_FIX_ERRORS is not set, out-of-image BAT entries were not + * fixed. Skip not allocated and out-of-image BAT entries. + */ + if (off == 0 || off + s->cluster_size > res->image_end_offset) { prev_off = 0; continue; } From patchwork Mon Jun 5 15:45:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7FAD1C7EE24 for ; Mon, 5 Jun 2023 15:48:54 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CPs-00088a-Hi; Mon, 05 Jun 2023 11:46:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPT-0007tJ-TI for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:23 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPQ-0001P2-AQ for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979975; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XyEA4iClLdS3Fl+6nsOtSoLAGTr2JLz4Ogv1Kf/SCuE=; b=VGD6bwiX18eTOoRrvTZqTofkmQFzvAEBmDBBKSzwr6cfM5bespAMrmrKxLpXrtPuPLtf4Y UW+qdkqBggDP7LhkHoV7ILaqYoGfZWlJ/h1xGkRVb96dn0SUjg8lVP7288jg/z+tnOnMrk MKEx2q8ZR7M7IarPvFhb8JPbBXCBVw4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-148-jBjhxTMnO8CrpZwCacjJLQ-1; Mon, 05 Jun 2023 11:46:05 -0400 X-MC-Unique: jBjhxTMnO8CrpZwCacjJLQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C22E885A5BD; Mon, 5 Jun 2023 15:46:04 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 842F440CFD47; Mon, 5 Jun 2023 15:46:04 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 13/17] parallels: Move check of leaks to a separate function Date: Mon, 5 Jun 2023 17:45:37 +0200 Message-Id: <20230605154541.1043261-14-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Alexander Ivanov We will add more and more checks so we need a better code structure in parallels_co_check. Let each check performs in a separate loop in a separate helper. Signed-off-by: Alexander Ivanov Message-Id: <20230424093147.197643-10-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek Signed-off-by: Hanna Czenczek --- block/parallels.c | 74 ++++++++++++++++++++++++++++------------------- 1 file changed, 45 insertions(+), 29 deletions(-) diff --git a/block/parallels.c b/block/parallels.c index fd1e2860d0..ba766e5e86 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -483,13 +483,12 @@ parallels_check_outside_image(BlockDriverState *bs, BdrvCheckResult *res, } static int coroutine_fn GRAPH_RDLOCK -parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, - BdrvCheckMode fix) +parallels_check_leak(BlockDriverState *bs, BdrvCheckResult *res, + BdrvCheckMode fix) { BDRVParallelsState *s = bs->opaque; - int64_t size, prev_off; + int64_t size; int ret; - uint32_t i; size = bdrv_getlength(bs->file->bs); if (size < 0) { @@ -497,6 +496,43 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, return size; } + if (size > res->image_end_offset) { + int64_t count; + count = DIV_ROUND_UP(size - res->image_end_offset, s->cluster_size); + fprintf(stderr, "%s space leaked at the end of the image %" PRId64 "\n", + fix & BDRV_FIX_LEAKS ? "Repairing" : "ERROR", + size - res->image_end_offset); + res->leaks += count; + if (fix & BDRV_FIX_LEAKS) { + Error *local_err = NULL; + + /* + * In order to really repair the image, we must shrink it. + * That means we have to pass exact=true. + */ + ret = bdrv_co_truncate(bs->file, res->image_end_offset, true, + PREALLOC_MODE_OFF, 0, &local_err); + if (ret < 0) { + error_report_err(local_err); + res->check_errors++; + return ret; + } + res->leaks_fixed += count; + } + } + + return 0; +} + +static int coroutine_fn GRAPH_RDLOCK +parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, + BdrvCheckMode fix) +{ + BDRVParallelsState *s = bs->opaque; + int64_t prev_off; + int ret; + uint32_t i; + qemu_co_mutex_lock(&s->lock); parallels_check_unclean(bs, res, fix); @@ -506,6 +542,11 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, goto out; } + ret = parallels_check_leak(bs, res, fix); + if (ret < 0) { + goto out; + } + res->bfi.total_clusters = s->bat_size; res->bfi.compressed_clusters = 0; /* compression is not supported */ @@ -529,31 +570,6 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, prev_off = off; } - if (size > res->image_end_offset) { - int64_t count; - count = DIV_ROUND_UP(size - res->image_end_offset, s->cluster_size); - fprintf(stderr, "%s space leaked at the end of the image %" PRId64 "\n", - fix & BDRV_FIX_LEAKS ? "Repairing" : "ERROR", - size - res->image_end_offset); - res->leaks += count; - if (fix & BDRV_FIX_LEAKS) { - Error *local_err = NULL; - - /* - * In order to really repair the image, we must shrink it. - * That means we have to pass exact=true. - */ - ret = bdrv_co_truncate(bs->file, res->image_end_offset, true, - PREALLOC_MODE_OFF, 0, &local_err); - if (ret < 0) { - error_report_err(local_err); - res->check_errors++; - goto out; - } - res->leaks_fixed += count; - } - } - out: qemu_co_mutex_unlock(&s->lock); From patchwork Mon Jun 5 15:45:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EE06DC7EE24 for ; Mon, 5 Jun 2023 15:48:44 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CPs-00087W-6p; Mon, 05 Jun 2023 11:46:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPQ-0007ss-Tg for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:19 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPN-0001NT-CS for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979971; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hVbVYqScEn9lACkqQPjDiDNKCo88rOi2CcyO4vYlv2g=; b=dkewiOsrqXioOEviPZigSYkEbGv7EFUjsEeOt4AWAOIJF/saNXa3Dtyg5f/Lz6jaJxVSdN 1LRrDC1VOhhF5et6j4Cag76gLo2oSkTm/tXrWaMsRMWDONR3zWCw9zaJ886vvPM7Hy2RCa fXkaYvPMj6xubKWf+Ir1dg34Himc3Ok= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-283-KF5Yo4VCMYCT3MPl7QyoDw-1; Mon, 05 Jun 2023 11:46:07 -0400 X-MC-Unique: KF5Yo4VCMYCT3MPl7QyoDw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 46AAA811E8D; Mon, 5 Jun 2023 15:46:06 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 074912166B27; Mon, 5 Jun 2023 15:46:05 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 14/17] parallels: Move statistic collection to a separate function Date: Mon, 5 Jun 2023 17:45:38 +0200 Message-Id: <20230605154541.1043261-15-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Alexander Ivanov We will add more and more checks so we need a better code structure in parallels_co_check. Let each check performs in a separate loop in a separate helper. Signed-off-by: Alexander Ivanov Reviewed-by: Denis V. Lunev Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <20230424093147.197643-11-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek Signed-off-by: Hanna Czenczek --- block/parallels.c | 52 +++++++++++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 22 deletions(-) diff --git a/block/parallels.c b/block/parallels.c index ba766e5e86..bb279ddc70 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -524,35 +524,20 @@ parallels_check_leak(BlockDriverState *bs, BdrvCheckResult *res, return 0; } -static int coroutine_fn GRAPH_RDLOCK -parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, - BdrvCheckMode fix) +static void parallels_collect_statistics(BlockDriverState *bs, + BdrvCheckResult *res, + BdrvCheckMode fix) { BDRVParallelsState *s = bs->opaque; - int64_t prev_off; - int ret; + int64_t off, prev_off; uint32_t i; - qemu_co_mutex_lock(&s->lock); - - parallels_check_unclean(bs, res, fix); - - ret = parallels_check_outside_image(bs, res, fix); - if (ret < 0) { - goto out; - } - - ret = parallels_check_leak(bs, res, fix); - if (ret < 0) { - goto out; - } - res->bfi.total_clusters = s->bat_size; res->bfi.compressed_clusters = 0; /* compression is not supported */ prev_off = 0; for (i = 0; i < s->bat_size; i++) { - int64_t off = bat2sect(s, i) << BDRV_SECTOR_BITS; + off = bat2sect(s, i) << BDRV_SECTOR_BITS; /* * If BDRV_FIX_ERRORS is not set, out-of-image BAT entries were not * fixed. Skip not allocated and out-of-image BAT entries. @@ -562,13 +547,36 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, continue; } - res->bfi.allocated_clusters++; - if (prev_off != 0 && (prev_off + s->cluster_size) != off) { res->bfi.fragmented_clusters++; } prev_off = off; + res->bfi.allocated_clusters++; } +} + +static int coroutine_fn GRAPH_RDLOCK +parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, + BdrvCheckMode fix) +{ + BDRVParallelsState *s = bs->opaque; + int ret; + + qemu_co_mutex_lock(&s->lock); + + parallels_check_unclean(bs, res, fix); + + ret = parallels_check_outside_image(bs, res, fix); + if (ret < 0) { + goto out; + } + + ret = parallels_check_leak(bs, res, fix); + if (ret < 0) { + goto out; + } + + parallels_collect_statistics(bs, res, fix); out: qemu_co_mutex_unlock(&s->lock); From patchwork Mon Jun 5 15:45:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2DBF8C7EE2C for ; Mon, 5 Jun 2023 15:46:45 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CPZ-0007zw-Or; Mon, 05 Jun 2023 11:46:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPP-0007si-Ht for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:20 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPL-0001NH-AK for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979970; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RL1aMDPCB3s7lKj0NNf108vM/HPWOasIX6/48GNDtXY=; b=GZR2gZOXSzOWe4XKd9OjLC/sI/BY9tUTjR3WWySzwxjIfdIpRZUs1v9xyY8z2vgKbHlUTO YwrzG50zxohTtDGO6gmRYW3Ute27fs+x7rIKwJc0iKOnB4A3PXEkpB54Rraoa8Z7rNTzMd sYqvLTb56sAUZ0aTu9I26kOLRC5E638= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-610-kM1vT01fNUSIUiXi3gdLLg-1; Mon, 05 Jun 2023 11:46:08 -0400 X-MC-Unique: kM1vT01fNUSIUiXi3gdLLg-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 89643811E9E; Mon, 5 Jun 2023 15:46:08 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8F19940218C; Mon, 5 Jun 2023 15:46:07 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 15/17] parallels: Replace qemu_co_mutex_lock by WITH_QEMU_LOCK_GUARD Date: Mon, 5 Jun 2023 17:45:39 +0200 Message-Id: <20230605154541.1043261-16-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Received-SPF: pass client-ip=170.10.133.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Alexander Ivanov Replace the way we use mutex in parallels_co_check() for simplier and less error prone code. Signed-off-by: Alexander Ivanov Reviewed-by: Denis V. Lunev Message-Id: <20230424093147.197643-12-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek Signed-off-by: Hanna Czenczek --- block/parallels.c | 33 ++++++++++++++------------------- 1 file changed, 14 insertions(+), 19 deletions(-) diff --git a/block/parallels.c b/block/parallels.c index bb279ddc70..84d1d045ad 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -562,30 +562,25 @@ parallels_co_check(BlockDriverState *bs, BdrvCheckResult *res, BDRVParallelsState *s = bs->opaque; int ret; - qemu_co_mutex_lock(&s->lock); + WITH_QEMU_LOCK_GUARD(&s->lock) { + parallels_check_unclean(bs, res, fix); - parallels_check_unclean(bs, res, fix); + ret = parallels_check_outside_image(bs, res, fix); + if (ret < 0) { + return ret; + } - ret = parallels_check_outside_image(bs, res, fix); - if (ret < 0) { - goto out; - } + ret = parallels_check_leak(bs, res, fix); + if (ret < 0) { + return ret; + } - ret = parallels_check_leak(bs, res, fix); - if (ret < 0) { - goto out; + parallels_collect_statistics(bs, res, fix); } - parallels_collect_statistics(bs, res, fix); - -out: - qemu_co_mutex_unlock(&s->lock); - - if (ret == 0) { - ret = bdrv_co_flush(bs); - if (ret < 0) { - res->check_errors++; - } + ret = bdrv_co_flush(bs); + if (ret < 0) { + res->check_errors++; } return ret; From patchwork Mon Jun 5 15:45:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5EF57C7EE24 for ; Mon, 5 Jun 2023 15:47:50 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CPf-000812-BL; Mon, 05 Jun 2023 11:46:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPS-0007t1-UV for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:23 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPP-0001OK-9l for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979973; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bZX+hAUdVnkgKpp0SMujUojyhmybQyH26HBjH1Izbek=; b=TEcJEjoLs3xSgfouG6vTP7OEqJnUAXdRL8B1uqqgjU+kVMFlVUIzPrVX2IsTgOr90jfWXJ ymCKg2HFNfpYhYLqDJSZKpZ5SpyH7GGPelKKoO++cSIYx406GzegmszArs7ejpPu9BPX0J cfYLBNwfB4TWcyiQI9ji82Z6RziLt1o= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-5-rnydjblMMEaBPmpYAlIsrg-1; Mon, 05 Jun 2023 11:46:10 -0400 X-MC-Unique: rnydjblMMEaBPmpYAlIsrg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 02539185A791; Mon, 5 Jun 2023 15:46:10 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B810140D1B60; Mon, 5 Jun 2023 15:46:09 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 16/17] parallels: Incorrect condition in out-of-image check Date: Mon, 5 Jun 2023 17:45:40 +0200 Message-Id: <20230605154541.1043261-17-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Alexander Ivanov All the offsets in the BAT must be lower than the file size. Fix the check condition for correct check. Signed-off-by: Alexander Ivanov Reviewed-by: Denis V. Lunev Message-Id: <20230424093147.197643-13-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek Signed-off-by: Hanna Czenczek --- block/parallels.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/parallels.c b/block/parallels.c index 84d1d045ad..7c263d5085 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -457,7 +457,7 @@ parallels_check_outside_image(BlockDriverState *bs, BdrvCheckResult *res, high_off = 0; for (i = 0; i < s->bat_size; i++) { off = bat2sect(s, i) << BDRV_SECTOR_BITS; - if (off > size) { + if (off + s->cluster_size > size) { fprintf(stderr, "%s cluster %u is outside image\n", fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR", i); res->corruptions++; From patchwork Mon Jun 5 15:45:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanna Czenczek X-Patchwork-Id: 13267718 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 67E1BC7EE2C for ; Mon, 5 Jun 2023 15:49:32 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q6CPr-00086X-Fg; Mon, 05 Jun 2023 11:46:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPU-0007tK-4k for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:23 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q6CPO-0001OA-2l for qemu-devel@nongnu.org; Mon, 05 Jun 2023 11:46:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685979973; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ID3ZTafvHgJJoU3XOFcYLiDtyI20na1z/ZbUr4+K/rA=; b=cbAzv0LY50vErLZ8eahXWVYppfxaRoImieTHSwPI+4AgmW/2JXTkI2qJdWtiLNMOhH3jjw OPwHdeRBQu6BjkErJ/Q8GbYWieSzGxqJz587Sa4B7b85T/D3F61iTm7jHt4fVKslMN+ef1 sEopasK2pMRFKJ4wXo5YYJZWnWE3TOI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-313-Q9kguHI-PVa5SKas4w2OAQ-1; Mon, 05 Jun 2023 11:46:11 -0400 X-MC-Unique: Q9kguHI-PVa5SKas4w2OAQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 924003801EDF; Mon, 5 Jun 2023 15:46:11 +0000 (UTC) Received: from localhost (unknown [10.39.194.13]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 34EE02166B27; Mon, 5 Jun 2023 15:46:11 +0000 (UTC) From: Hanna Czenczek To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, Hanna Czenczek , Kevin Wolf , Richard Henderson Subject: [PULL 17/17] qcow2: add discard-no-unref option Date: Mon, 5 Jun 2023 17:45:41 +0200 Message-Id: <20230605154541.1043261-18-hreitz@redhat.com> In-Reply-To: <20230605154541.1043261-1-hreitz@redhat.com> References: <20230605154541.1043261-1-hreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Received-SPF: pass client-ip=170.10.129.124; envelope-from=hreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jean-Louis Dupond When we for example have a sparse qcow2 image and discard: unmap is enabled, there can be a lot of fragmentation in the image after some time. Especially on VM's that do a lot of writes/deletes. This causes the qcow2 image to grow even over 110% of its virtual size, because the free gaps in the image get too small to allocate new continuous clusters. So it allocates new space at the end of the image. Disabling discard is not an option, as discard is needed to keep the incremental backup size as low as possible. Without discard, the incremental backups would become large, as qemu thinks it's just dirty blocks but it doesn't know the blocks are unneeded. So we need to avoid fragmentation but also 'empty' the unneeded blocks in the image to have a small incremental backup. In addition, we also want to send the discards further down the stack, so the underlying blocks are still discarded. Therefor we introduce a new qcow2 option "discard-no-unref". When setting this option to true, discards will no longer have the qcow2 driver relinquish cluster allocations. Other than that, the request is handled as normal: All clusters in range are marked as zero, and, if pass-discard-request is true, it is passed further down the stack. The only difference is that the now-zero clusters are preallocated instead of being unallocated. This will avoid fragmentation on the qcow2 image. Fixes: https://gitlab.com/qemu-project/qemu/-/issues/1621 Signed-off-by: Jean-Louis Dupond Message-Id: <20230605084523.34134-2-jean-louis@dupond.be> Reviewed-by: Hanna Czenczek Signed-off-by: Hanna Czenczek --- qapi/block-core.json | 12 ++++++++++++ block/qcow2.h | 3 +++ block/qcow2-cluster.c | 32 ++++++++++++++++++++++++++++---- block/qcow2.c | 18 ++++++++++++++++++ qemu-options.hx | 12 ++++++++++++ 5 files changed, 73 insertions(+), 4 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index 4bf89171c6..5dd5f7e4b0 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -3478,6 +3478,17 @@ # @pass-discard-other: whether discard requests for the data source # should be issued on other occasions where a cluster gets freed # +# @discard-no-unref: when enabled, discards from the guest will not cause +# cluster allocations to be relinquished. This prevents qcow2 fragmentation +# that would be caused by such discards. Besides potential +# performance degradation, such fragmentation can lead to increased +# allocation of clusters past the end of the image file, +# resulting in image files whose file length can grow much larger +# than their guest disk size would suggest. +# If image file length is of concern (e.g. when storing qcow2 +# images directly on block devices), you should consider enabling +# this option. (since 8.1) +# # @overlap-check: which overlap checks to perform for writes to the # image, defaults to 'cached' (since 2.2) # @@ -3516,6 +3527,7 @@ '*pass-discard-request': 'bool', '*pass-discard-snapshot': 'bool', '*pass-discard-other': 'bool', + '*discard-no-unref': 'bool', '*overlap-check': 'Qcow2OverlapChecks', '*cache-size': 'int', '*l2-cache-size': 'int', diff --git a/block/qcow2.h b/block/qcow2.h index 4f67eb912a..ea9adb5706 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -133,6 +133,7 @@ #define QCOW2_OPT_DISCARD_REQUEST "pass-discard-request" #define QCOW2_OPT_DISCARD_SNAPSHOT "pass-discard-snapshot" #define QCOW2_OPT_DISCARD_OTHER "pass-discard-other" +#define QCOW2_OPT_DISCARD_NO_UNREF "discard-no-unref" #define QCOW2_OPT_OVERLAP "overlap-check" #define QCOW2_OPT_OVERLAP_TEMPLATE "overlap-check.template" #define QCOW2_OPT_OVERLAP_MAIN_HEADER "overlap-check.main-header" @@ -385,6 +386,8 @@ typedef struct BDRVQcow2State { bool discard_passthrough[QCOW2_DISCARD_MAX]; + bool discard_no_unref; + int overlap_check; /* bitmask of Qcow2MetadataOverlap values */ bool signaled_corruption; diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 39cda7f907..2e76de027c 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -1925,6 +1925,10 @@ static int discard_in_l2_slice(BlockDriverState *bs, uint64_t offset, uint64_t new_l2_bitmap = old_l2_bitmap; QCow2ClusterType cluster_type = qcow2_get_cluster_type(bs, old_l2_entry); + bool keep_reference = (cluster_type != QCOW2_CLUSTER_COMPRESSED) && + !full_discard && + (s->discard_no_unref && + type == QCOW2_DISCARD_REQUEST); /* * If full_discard is true, the cluster should not read back as zeroes, @@ -1943,10 +1947,22 @@ static int discard_in_l2_slice(BlockDriverState *bs, uint64_t offset, new_l2_entry = new_l2_bitmap = 0; } else if (bs->backing || qcow2_cluster_is_allocated(cluster_type)) { if (has_subclusters(s)) { - new_l2_entry = 0; + if (keep_reference) { + new_l2_entry = old_l2_entry; + } else { + new_l2_entry = 0; + } new_l2_bitmap = QCOW_L2_BITMAP_ALL_ZEROES; } else { - new_l2_entry = s->qcow_version >= 3 ? QCOW_OFLAG_ZERO : 0; + if (s->qcow_version >= 3) { + if (keep_reference) { + new_l2_entry |= QCOW_OFLAG_ZERO; + } else { + new_l2_entry = QCOW_OFLAG_ZERO; + } + } else { + new_l2_entry = 0; + } } } @@ -1960,8 +1976,16 @@ static int discard_in_l2_slice(BlockDriverState *bs, uint64_t offset, if (has_subclusters(s)) { set_l2_bitmap(s, l2_slice, l2_index + i, new_l2_bitmap); } - /* Then decrease the refcount */ - qcow2_free_any_cluster(bs, old_l2_entry, type); + if (!keep_reference) { + /* Then decrease the refcount */ + qcow2_free_any_cluster(bs, old_l2_entry, type); + } else if (s->discard_passthrough[type] && + (cluster_type == QCOW2_CLUSTER_NORMAL || + cluster_type == QCOW2_CLUSTER_ZERO_ALLOC)) { + /* If we keep the reference, pass on the discard still */ + bdrv_pdiscard(s->data_file, old_l2_entry & L2E_OFFSET_MASK, + s->cluster_size); + } } qcow2_cache_put(s->l2_table_cache, (void **) &l2_slice); diff --git a/block/qcow2.c b/block/qcow2.c index 7f3948360d..e23edd48c2 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -682,6 +682,7 @@ static const char *const mutable_opts[] = { QCOW2_OPT_DISCARD_REQUEST, QCOW2_OPT_DISCARD_SNAPSHOT, QCOW2_OPT_DISCARD_OTHER, + QCOW2_OPT_DISCARD_NO_UNREF, QCOW2_OPT_OVERLAP, QCOW2_OPT_OVERLAP_TEMPLATE, QCOW2_OPT_OVERLAP_MAIN_HEADER, @@ -726,6 +727,11 @@ static QemuOptsList qcow2_runtime_opts = { .type = QEMU_OPT_BOOL, .help = "Generate discard requests when other clusters are freed", }, + { + .name = QCOW2_OPT_DISCARD_NO_UNREF, + .type = QEMU_OPT_BOOL, + .help = "Do not unreference discarded clusters", + }, { .name = QCOW2_OPT_OVERLAP, .type = QEMU_OPT_STRING, @@ -969,6 +975,7 @@ typedef struct Qcow2ReopenState { bool use_lazy_refcounts; int overlap_check; bool discard_passthrough[QCOW2_DISCARD_MAX]; + bool discard_no_unref; uint64_t cache_clean_interval; QCryptoBlockOpenOptions *crypto_opts; /* Disk encryption runtime options */ } Qcow2ReopenState; @@ -1140,6 +1147,15 @@ static int qcow2_update_options_prepare(BlockDriverState *bs, r->discard_passthrough[QCOW2_DISCARD_OTHER] = qemu_opt_get_bool(opts, QCOW2_OPT_DISCARD_OTHER, false); + r->discard_no_unref = qemu_opt_get_bool(opts, QCOW2_OPT_DISCARD_NO_UNREF, + false); + if (r->discard_no_unref && s->qcow_version < 3) { + error_setg(errp, + "discard-no-unref is only supported since qcow2 version 3"); + ret = -EINVAL; + goto fail; + } + switch (s->crypt_method_header) { case QCOW_CRYPT_NONE: if (encryptfmt) { @@ -1220,6 +1236,8 @@ static void qcow2_update_options_commit(BlockDriverState *bs, s->discard_passthrough[i] = r->discard_passthrough[i]; } + s->discard_no_unref = r->discard_no_unref; + if (s->cache_clean_interval != r->cache_clean_interval) { cache_clean_timer_del(bs); s->cache_clean_interval = r->cache_clean_interval; diff --git a/qemu-options.hx b/qemu-options.hx index b37eb9662b..b57489d7ca 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -1431,6 +1431,18 @@ SRST issued on other occasions where a cluster gets freed (on/off; default: off) + ``discard-no-unref`` + When enabled, discards from the guest will not cause cluster + allocations to be relinquished. This prevents qcow2 fragmentation + that would be caused by such discards. Besides potential + performance degradation, such fragmentation can lead to increased + allocation of clusters past the end of the image file, + resulting in image files whose file length can grow much larger + than their guest disk size would suggest. + If image file length is of concern (e.g. when storing qcow2 + images directly on block devices), you should consider enabling + this option. + ``overlap-check`` Which overlap checks to perform for writes to the image (none/constant/cached/all; default: cached). For details or