From patchwork Tue Jun 18 11:43:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 11001415 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0942214BB for ; Tue, 18 Jun 2019 11:44:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E9357289F2 for ; Tue, 18 Jun 2019 11:44:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D8BCE28A1B; Tue, 18 Jun 2019 11:44:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 10DAA289F2 for ; Tue, 18 Jun 2019 11:44:52 +0000 (UTC) Received: from localhost ([::1]:55844 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCXj-0005sW-Rr for patchwork-qemu-devel@patchwork.kernel.org; Tue, 18 Jun 2019 07:44:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48334) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCWX-00041s-AF for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdCWV-0000N3-6G for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:37 -0400 Received: from relay.sw.ru ([185.231.240.75]:58034) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdCWU-0000Jn-Rc; Tue, 18 Jun 2019 07:43:35 -0400 Received: from [10.94.3.0] (helo=kvm.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.92) (envelope-from ) id 1hdCWO-00016b-Tv; Tue, 18 Jun 2019 14:43:29 +0300 From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Tue, 18 Jun 2019 14:43:20 +0300 Message-Id: <20190618114328.55249-2-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20190618114328.55249-1-vsementsov@virtuozzo.com> References: <20190618114328.55249-1-vsementsov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v7 1/9] block/nbd: split connection_co start out of nbd_client_connect X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, stefanha@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP nbd_client_connect is going to be used from connection_co, so, let's refactor nbd_client_connect in advance, leaving io channel configuration all in nbd_client_connect. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- block/nbd.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/block/nbd.c b/block/nbd.c index 81edabbf35..46a1de7220 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -1232,14 +1232,8 @@ static int nbd_client_connect(BlockDriverState *bs, object_ref(OBJECT(s->ioc)); } - /* - * Now that we're connected, set the socket to be non-blocking and - * kick the reply mechanism. - */ qio_channel_set_blocking(QIO_CHANNEL(sioc), false, NULL); - s->connection_co = qemu_coroutine_create(nbd_connection_entry, s); - bdrv_inc_in_flight(bs); - nbd_client_attach_aio_context(bs, bdrv_get_aio_context(bs)); + qio_channel_attach_aio_context(s->ioc, bdrv_get_aio_context(bs)); trace_nbd_client_connect_success(export); @@ -1270,14 +1264,24 @@ static int nbd_client_init(BlockDriverState *bs, const char *x_dirty_bitmap, Error **errp) { + int ret; BDRVNBDState *s = (BDRVNBDState *)bs->opaque; s->bs = bs; qemu_co_mutex_init(&s->send_mutex); qemu_co_queue_init(&s->free_sema); - return nbd_client_connect(bs, saddr, export, tlscreds, hostname, - x_dirty_bitmap, errp); + ret = nbd_client_connect(bs, saddr, export, tlscreds, hostname, + x_dirty_bitmap, errp); + if (ret < 0) { + return ret; + } + + s->connection_co = qemu_coroutine_create(nbd_connection_entry, s); + bdrv_inc_in_flight(bs); + aio_co_schedule(bdrv_get_aio_context(bs), s->connection_co); + + return 0; } static int nbd_parse_uri(const char *filename, QDict *options) From patchwork Tue Jun 18 11:43:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 11001423 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E20291398 for ; Tue, 18 Jun 2019 11:49:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D24D528A05 for ; Tue, 18 Jun 2019 11:49:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C623828A53; Tue, 18 Jun 2019 11:49:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4B74B28A5C for ; Tue, 18 Jun 2019 11:49:39 +0000 (UTC) Received: from localhost ([::1]:55882 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCcM-00016r-Lk for patchwork-qemu-devel@patchwork.kernel.org; Tue, 18 Jun 2019 07:49:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48338) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCWX-00041v-AX for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdCWV-0000Me-4v for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:37 -0400 Received: from relay.sw.ru ([185.231.240.75]:57996) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdCWU-0000Jf-RU; Tue, 18 Jun 2019 07:43:35 -0400 Received: from [10.94.3.0] (helo=kvm.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.92) (envelope-from ) id 1hdCWP-00016b-2t; Tue, 18 Jun 2019 14:43:29 +0300 From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Tue, 18 Jun 2019 14:43:21 +0300 Message-Id: <20190618114328.55249-3-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20190618114328.55249-1-vsementsov@virtuozzo.com> References: <20190618114328.55249-1-vsementsov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v7 2/9] block/nbd: use non-blocking io channel for nbd negotiation X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, stefanha@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP No reason to use blocking channel for negotiation and we'll benefit in further reconnect feature, as qio_channel reads and writes will do qemu_coroutine_yield while waiting for io completion. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- include/block/nbd.h | 3 ++- block/nbd.c | 16 +++++++--------- nbd/client.c | 16 +++++++++++----- qemu-nbd.c | 2 +- 4 files changed, 21 insertions(+), 16 deletions(-) diff --git a/include/block/nbd.h b/include/block/nbd.h index bb9f5bc021..7b36d672f0 100644 --- a/include/block/nbd.h +++ b/include/block/nbd.h @@ -304,7 +304,8 @@ struct NBDExportInfo { }; typedef struct NBDExportInfo NBDExportInfo; -int nbd_receive_negotiate(QIOChannel *ioc, QCryptoTLSCreds *tlscreds, +int nbd_receive_negotiate(AioContext *aio_context, QIOChannel *ioc, + QCryptoTLSCreds *tlscreds, const char *hostname, QIOChannel **outioc, NBDExportInfo *info, Error **errp); void nbd_free_export_list(NBDExportInfo *info, int count); diff --git a/block/nbd.c b/block/nbd.c index 46a1de7220..16a38373fd 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -1176,6 +1176,7 @@ static int nbd_client_connect(BlockDriverState *bs, Error **errp) { BDRVNBDState *s = (BDRVNBDState *)bs->opaque; + AioContext *aio_context = bdrv_get_aio_context(bs); int ret; /* @@ -1190,15 +1191,16 @@ static int nbd_client_connect(BlockDriverState *bs, /* NBD handshake */ trace_nbd_client_connect(export); - qio_channel_set_blocking(QIO_CHANNEL(sioc), true, NULL); + qio_channel_set_blocking(QIO_CHANNEL(sioc), false, NULL); + qio_channel_attach_aio_context(QIO_CHANNEL(sioc), aio_context); s->info.request_sizes = true; s->info.structured_reply = true; s->info.base_allocation = true; s->info.x_dirty_bitmap = g_strdup(x_dirty_bitmap); s->info.name = g_strdup(export ?: ""); - ret = nbd_receive_negotiate(QIO_CHANNEL(sioc), tlscreds, hostname, - &s->ioc, &s->info, errp); + ret = nbd_receive_negotiate(aio_context, QIO_CHANNEL(sioc), tlscreds, + hostname, &s->ioc, &s->info, errp); g_free(s->info.x_dirty_bitmap); g_free(s->info.name); if (ret < 0) { @@ -1232,18 +1234,14 @@ static int nbd_client_connect(BlockDriverState *bs, object_ref(OBJECT(s->ioc)); } - qio_channel_set_blocking(QIO_CHANNEL(sioc), false, NULL); - qio_channel_attach_aio_context(s->ioc, bdrv_get_aio_context(bs)); - trace_nbd_client_connect_success(export); return 0; fail: /* - * We have connected, but must fail for other reasons. The - * connection is still blocking; send NBD_CMD_DISC as a courtesy - * to the server. + * We have connected, but must fail for other reasons. + * Send NBD_CMD_DISC as a courtesy to the server. */ { NBDRequest request = { .type = NBD_CMD_DISC }; diff --git a/nbd/client.c b/nbd/client.c index 4de30630c7..8f524c3e35 100644 --- a/nbd/client.c +++ b/nbd/client.c @@ -867,7 +867,8 @@ static int nbd_list_meta_contexts(QIOChannel *ioc, * 2: server is newstyle, but lacks structured replies * 3: server is newstyle and set up for structured replies */ -static int nbd_start_negotiate(QIOChannel *ioc, QCryptoTLSCreds *tlscreds, +static int nbd_start_negotiate(AioContext *aio_context, QIOChannel *ioc, + QCryptoTLSCreds *tlscreds, const char *hostname, QIOChannel **outioc, bool structured_reply, bool *zeroes, Error **errp) @@ -934,6 +935,10 @@ static int nbd_start_negotiate(QIOChannel *ioc, QCryptoTLSCreds *tlscreds, return -EINVAL; } ioc = *outioc; + if (aio_context) { + qio_channel_set_blocking(ioc, false, NULL); + qio_channel_attach_aio_context(ioc, aio_context); + } } else { error_setg(errp, "Server does not support STARTTLS"); return -EINVAL; @@ -998,7 +1003,8 @@ static int nbd_negotiate_finish_oldstyle(QIOChannel *ioc, NBDExportInfo *info, * Returns: negative errno: failure talking to server * 0: server is connected */ -int nbd_receive_negotiate(QIOChannel *ioc, QCryptoTLSCreds *tlscreds, +int nbd_receive_negotiate(AioContext *aio_context, QIOChannel *ioc, + QCryptoTLSCreds *tlscreds, const char *hostname, QIOChannel **outioc, NBDExportInfo *info, Error **errp) { @@ -1009,7 +1015,7 @@ int nbd_receive_negotiate(QIOChannel *ioc, QCryptoTLSCreds *tlscreds, assert(info->name); trace_nbd_receive_negotiate_name(info->name); - result = nbd_start_negotiate(ioc, tlscreds, hostname, outioc, + result = nbd_start_negotiate(aio_context, ioc, tlscreds, hostname, outioc, info->structured_reply, &zeroes, errp); info->structured_reply = false; @@ -1129,8 +1135,8 @@ int nbd_receive_export_list(QIOChannel *ioc, QCryptoTLSCreds *tlscreds, QIOChannel *sioc = NULL; *info = NULL; - result = nbd_start_negotiate(ioc, tlscreds, hostname, &sioc, true, NULL, - errp); + result = nbd_start_negotiate(NULL, ioc, tlscreds, hostname, &sioc, true, + NULL, errp); if (tlscreds && sioc) { ioc = sioc; } diff --git a/qemu-nbd.c b/qemu-nbd.c index a8cb39e510..049645491d 100644 --- a/qemu-nbd.c +++ b/qemu-nbd.c @@ -362,7 +362,7 @@ static void *nbd_client_thread(void *arg) goto out; } - ret = nbd_receive_negotiate(QIO_CHANNEL(sioc), + ret = nbd_receive_negotiate(NULL, QIO_CHANNEL(sioc), NULL, NULL, NULL, &info, &local_error); if (ret < 0) { if (local_error) { From patchwork Tue Jun 18 11:43:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 11001427 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F3D021398 for ; Tue, 18 Jun 2019 11:50:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C085328702 for ; Tue, 18 Jun 2019 11:50:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B44E528918; Tue, 18 Jun 2019 11:50:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 1EF7028702 for ; Tue, 18 Jun 2019 11:50:43 +0000 (UTC) Received: from localhost ([::1]:55886 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCdO-0001dC-Ei for patchwork-qemu-devel@patchwork.kernel.org; Tue, 18 Jun 2019 07:50:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48388) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCWZ-00041z-7W for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdCWV-0000Mx-5u for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:39 -0400 Received: from relay.sw.ru ([185.231.240.75]:58018) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdCWU-0000Ji-Rl; Tue, 18 Jun 2019 07:43:35 -0400 Received: from [10.94.3.0] (helo=kvm.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.92) (envelope-from ) id 1hdCWP-00016b-68; Tue, 18 Jun 2019 14:43:29 +0300 From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Tue, 18 Jun 2019 14:43:22 +0300 Message-Id: <20190618114328.55249-4-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20190618114328.55249-1-vsementsov@virtuozzo.com> References: <20190618114328.55249-1-vsementsov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v7 3/9] block/nbd: move from quit to state X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, stefanha@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP To implement reconnect we need several states for the client: CONNECTED, QUIT and two different CONNECTING states. CONNECTING states will be added in the following patches. This patch implements CONNECTED and QUIT. QUIT means, that we should close the connection and fail all current and further requests (like old quit = true). CONNECTED means that connection is ok, we can send requests (like old quit = false). For receiving loop we use a comparison of the current state with QUIT, because reconnect will be in the same loop, so it should be looping until the end. Opposite, for requests we use a comparison of the current state with CONNECTED, as we don't want to send requests in future CONNECTING states. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- block/nbd.c | 58 ++++++++++++++++++++++++++++++++++------------------- 1 file changed, 37 insertions(+), 21 deletions(-) diff --git a/block/nbd.c b/block/nbd.c index 16a38373fd..9e505c895a 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -53,6 +53,11 @@ typedef struct { bool receiving; /* waiting for connection_co? */ } NBDClientRequest; +typedef enum NBDClientState { + NBD_CLIENT_CONNECTED, + NBD_CLIENT_QUIT +} NBDClientState; + typedef struct BDRVNBDState { QIOChannelSocket *sioc; /* The master data channel */ QIOChannel *ioc; /* The current I/O channel which may differ (eg TLS) */ @@ -62,17 +67,23 @@ typedef struct BDRVNBDState { CoQueue free_sema; Coroutine *connection_co; int in_flight; + NBDClientState state; NBDClientRequest requests[MAX_NBD_REQUESTS]; NBDReply reply; BlockDriverState *bs; - bool quit; /* For nbd_refresh_filename() */ SocketAddress *saddr; char *export, *tlscredsid; } BDRVNBDState; +/* @ret will be used for reconnect in future */ +static void nbd_channel_error(BDRVNBDState *s, int ret) +{ + s->state = NBD_CLIENT_QUIT; +} + static void nbd_recv_coroutines_wake_all(BDRVNBDState *s) { int i; @@ -151,7 +162,7 @@ static coroutine_fn void nbd_connection_entry(void *opaque) int ret = 0; Error *local_err = NULL; - while (!s->quit) { + while (s->state != NBD_CLIENT_QUIT) { /* * The NBD client can only really be considered idle when it has * yielded from qio_channel_readv_all_eof(), waiting for data. This is @@ -169,6 +180,7 @@ static coroutine_fn void nbd_connection_entry(void *opaque) error_free(local_err); } if (ret <= 0) { + nbd_channel_error(s, ret ? ret : -EIO); break; } @@ -183,6 +195,7 @@ static coroutine_fn void nbd_connection_entry(void *opaque) !s->requests[i].receiving || (nbd_reply_is_structured(&s->reply) && !s->info.structured_reply)) { + nbd_channel_error(s, -EINVAL); break; } @@ -202,7 +215,6 @@ static coroutine_fn void nbd_connection_entry(void *opaque) qemu_coroutine_yield(); } - s->quit = true; nbd_recv_coroutines_wake_all(s); bdrv_dec_in_flight(s->bs); @@ -215,12 +227,18 @@ static int nbd_co_send_request(BlockDriverState *bs, QEMUIOVector *qiov) { BDRVNBDState *s = (BDRVNBDState *)bs->opaque; - int rc, i; + int rc, i = -1; qemu_co_mutex_lock(&s->send_mutex); while (s->in_flight == MAX_NBD_REQUESTS) { qemu_co_queue_wait(&s->free_sema, &s->send_mutex); } + + if (s->state != NBD_CLIENT_CONNECTED) { + rc = -EIO; + goto err; + } + s->in_flight++; for (i = 0; i < MAX_NBD_REQUESTS; i++) { @@ -238,16 +256,12 @@ static int nbd_co_send_request(BlockDriverState *bs, request->handle = INDEX_TO_HANDLE(s, i); - if (s->quit) { - rc = -EIO; - goto err; - } assert(s->ioc); if (qiov) { qio_channel_set_cork(s->ioc, true); rc = nbd_send_request(s->ioc, request); - if (rc >= 0 && !s->quit) { + if (rc >= 0 && s->state == NBD_CLIENT_CONNECTED) { if (qio_channel_writev_all(s->ioc, qiov->iov, qiov->niov, NULL) < 0) { rc = -EIO; @@ -262,9 +276,11 @@ static int nbd_co_send_request(BlockDriverState *bs, err: if (rc < 0) { - s->quit = true; - s->requests[i].coroutine = NULL; - s->in_flight--; + nbd_channel_error(s, rc); + if (i != -1) { + s->requests[i].coroutine = NULL; + s->in_flight--; + } qemu_co_queue_next(&s->free_sema); } qemu_co_mutex_unlock(&s->send_mutex); @@ -556,7 +572,7 @@ static coroutine_fn int nbd_co_do_receive_one_chunk( s->requests[i].receiving = true; qemu_coroutine_yield(); s->requests[i].receiving = false; - if (s->quit) { + if (s->state != NBD_CLIENT_CONNECTED) { error_setg(errp, "Connection closed"); return -EIO; } @@ -640,7 +656,7 @@ static coroutine_fn int nbd_co_receive_one_chunk( request_ret, qiov, payload, errp); if (ret < 0) { - s->quit = true; + nbd_channel_error(s, ret); } else { /* For assert at loop start in nbd_connection_entry */ if (reply) { @@ -710,7 +726,7 @@ static bool nbd_reply_chunk_iter_receive(BDRVNBDState *s, NBDReply local_reply; NBDStructuredReplyChunk *chunk; Error *local_err = NULL; - if (s->quit) { + if (s->state != NBD_CLIENT_CONNECTED) { error_setg(&local_err, "Connection closed"); nbd_iter_channel_error(iter, -EIO, &local_err); goto break_loop; @@ -735,7 +751,7 @@ static bool nbd_reply_chunk_iter_receive(BDRVNBDState *s, } /* Do not execute the body of NBD_FOREACH_REPLY_CHUNK for simple reply. */ - if (nbd_reply_is_simple(reply) || s->quit) { + if (nbd_reply_is_simple(reply) || s->state != NBD_CLIENT_CONNECTED) { goto break_loop; } @@ -809,14 +825,14 @@ static int nbd_co_receive_cmdread_reply(BDRVNBDState *s, uint64_t handle, ret = nbd_parse_offset_hole_payload(s, &reply.structured, payload, offset, qiov, &local_err); if (ret < 0) { - s->quit = true; + nbd_channel_error(s, ret); nbd_iter_channel_error(&iter, ret, &local_err); } break; default: if (!nbd_reply_type_is_error(chunk->type)) { /* not allowed reply type */ - s->quit = true; + nbd_channel_error(s, -EINVAL); error_setg(&local_err, "Unexpected reply type: %d (%s) for CMD_READ", chunk->type, nbd_reply_type_lookup(chunk->type)); @@ -854,7 +870,7 @@ static int nbd_co_receive_blockstatus_reply(BDRVNBDState *s, switch (chunk->type) { case NBD_REPLY_TYPE_BLOCK_STATUS: if (received) { - s->quit = true; + nbd_channel_error(s, -EINVAL); error_setg(&local_err, "Several BLOCK_STATUS chunks in reply"); nbd_iter_channel_error(&iter, -EINVAL, &local_err); } @@ -864,13 +880,13 @@ static int nbd_co_receive_blockstatus_reply(BDRVNBDState *s, payload, length, extent, &local_err); if (ret < 0) { - s->quit = true; + nbd_channel_error(s, ret); nbd_iter_channel_error(&iter, ret, &local_err); } break; default: if (!nbd_reply_type_is_error(chunk->type)) { - s->quit = true; + nbd_channel_error(s, -EINVAL); error_setg(&local_err, "Unexpected reply type: %d (%s) " "for CMD_BLOCK_STATUS", From patchwork Tue Jun 18 11:43:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 11001431 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D64C614B6 for ; Tue, 18 Jun 2019 11:53:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C6C9326247 for ; Tue, 18 Jun 2019 11:53:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BAD4028898; Tue, 18 Jun 2019 11:53:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 5C0DA26247 for ; Tue, 18 Jun 2019 11:53:34 +0000 (UTC) Received: from localhost ([::1]:55914 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCg9-0004bP-OB for patchwork-qemu-devel@patchwork.kernel.org; Tue, 18 Jun 2019 07:53:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48336) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCWX-00041t-AH for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdCWV-0000NB-7b for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:37 -0400 Received: from relay.sw.ru ([185.231.240.75]:58004) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdCWU-0000Jj-Rn; Tue, 18 Jun 2019 07:43:35 -0400 Received: from [10.94.3.0] (helo=kvm.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.92) (envelope-from ) id 1hdCWP-00016b-9e; Tue, 18 Jun 2019 14:43:29 +0300 From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Tue, 18 Jun 2019 14:43:23 +0300 Message-Id: <20190618114328.55249-5-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20190618114328.55249-1-vsementsov@virtuozzo.com> References: <20190618114328.55249-1-vsementsov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v7 4/9] block/nbd: add cmdline and qapi parameter reconnect-delay X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, stefanha@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Reconnect will be implemented in the following commit, so for now, in semantics below, disconnect itself is a "serious error". Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- qapi/block-core.json | 11 ++++++++++- block/nbd.c | 16 +++++++++++++++- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index 61124431d8..17faf723e0 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -3858,13 +3858,22 @@ # traditional "base:allocation" block status (see # NBD_OPT_LIST_META_CONTEXT in the NBD protocol) (since 3.0) # +# @reconnect-delay: On an unexpected disconnect, the nbd client tries to +# connect again until succeeding or encountering a serious +# error. During the first @reconnect-delay seconds, all +# requests are paused and will be rerun on a successful +# reconnect. After that time, any delayed requests and all +# future requests before a successful reconnect will +# immediately fail. Default 0 (Since 4.1) +# # Since: 2.9 ## { 'struct': 'BlockdevOptionsNbd', 'data': { 'server': 'SocketAddress', '*export': 'str', '*tls-creds': 'str', - '*x-dirty-bitmap': 'str' } } + '*x-dirty-bitmap': 'str', + '*reconnect-delay': 'uint32' } } ## # @BlockdevOptionsRaw: diff --git a/block/nbd.c b/block/nbd.c index 9e505c895a..df9b0a2c8a 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -1276,6 +1276,7 @@ static int nbd_client_init(BlockDriverState *bs, QCryptoTLSCreds *tlscreds, const char *hostname, const char *x_dirty_bitmap, + uint32_t reconnect_delay, Error **errp) { int ret; @@ -1601,6 +1602,17 @@ static QemuOptsList nbd_runtime_opts = { .help = "experimental: expose named dirty bitmap in place of " "block status", }, + { + .name = "reconnect-delay", + .type = QEMU_OPT_NUMBER, + .help = "On an unexpected disconnect, the nbd client tries to " + "connect again until succeeding or encountering a serious " + "error. During the first @reconnect-delay seconds, all " + "requests are paused and will be rerun on a successful " + "reconnect. After that time, any delayed requests and all " + "future requests before a successful reconnect will " + "immediately fail. Default 0", + }, { /* end of list */ } }, }; @@ -1652,7 +1664,9 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, /* NBD handshake */ ret = nbd_client_init(bs, s->saddr, s->export, tlscreds, hostname, - qemu_opt_get(opts, "x-dirty-bitmap"), errp); + qemu_opt_get(opts, "x-dirty-bitmap"), + qemu_opt_get_number(opts, "reconnect-delay", 0), + errp); error: if (tlscreds) { From patchwork Tue Jun 18 11:43:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 11001425 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18CFC1398 for ; Tue, 18 Jun 2019 11:50:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0540028A60 for ; Tue, 18 Jun 2019 11:50:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E94B328A53; Tue, 18 Jun 2019 11:50:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 5A85E28A60 for ; Tue, 18 Jun 2019 11:50:06 +0000 (UTC) Received: from localhost ([::1]:55884 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCcn-00018N-D0 for patchwork-qemu-devel@patchwork.kernel.org; Tue, 18 Jun 2019 07:50:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48337) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCWX-00041u-AJ for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdCWV-0000N8-7t for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:37 -0400 Received: from relay.sw.ru ([185.231.240.75]:58024) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdCWU-0000Jo-Re; Tue, 18 Jun 2019 07:43:35 -0400 Received: from [10.94.3.0] (helo=kvm.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.92) (envelope-from ) id 1hdCWP-00016b-Fo; Tue, 18 Jun 2019 14:43:29 +0300 From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Tue, 18 Jun 2019 14:43:24 +0300 Message-Id: <20190618114328.55249-6-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20190618114328.55249-1-vsementsov@virtuozzo.com> References: <20190618114328.55249-1-vsementsov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v7 5/9] block/nbd: refactor nbd connection parameters X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, stefanha@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP We'll need some connection parameters to be available all the time to implement nbd reconnect. So, let's refactor them: define additional parameters in BDRVNBDState, drop them from function parameters, drop nbd_client_init and separate options parsing instead from nbd_open. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- block/nbd.c | 125 +++++++++++++++++++++++++++------------------------- 1 file changed, 64 insertions(+), 61 deletions(-) diff --git a/block/nbd.c b/block/nbd.c index df9b0a2c8a..68e0e168e8 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -73,9 +73,13 @@ typedef struct BDRVNBDState { NBDReply reply; BlockDriverState *bs; - /* For nbd_refresh_filename() */ + /* Connection parameters */ + uint32_t reconnect_delay; SocketAddress *saddr; char *export, *tlscredsid; + QCryptoTLSCreds *tlscreds; + const char *hostname; + char *x_dirty_bitmap; } BDRVNBDState; /* @ret will be used for reconnect in future */ @@ -1183,13 +1187,7 @@ static QIOChannelSocket *nbd_establish_connection(SocketAddress *saddr, return sioc; } -static int nbd_client_connect(BlockDriverState *bs, - SocketAddress *saddr, - const char *export, - QCryptoTLSCreds *tlscreds, - const char *hostname, - const char *x_dirty_bitmap, - Error **errp) +static int nbd_client_connect(BlockDriverState *bs, Error **errp) { BDRVNBDState *s = (BDRVNBDState *)bs->opaque; AioContext *aio_context = bdrv_get_aio_context(bs); @@ -1199,33 +1197,33 @@ static int nbd_client_connect(BlockDriverState *bs, * establish TCP connection, return error if it fails * TODO: Configurable retry-until-timeout behaviour. */ - QIOChannelSocket *sioc = nbd_establish_connection(saddr, errp); + QIOChannelSocket *sioc = nbd_establish_connection(s->saddr, errp); if (!sioc) { return -ECONNREFUSED; } /* NBD handshake */ - trace_nbd_client_connect(export); + trace_nbd_client_connect(s->export); qio_channel_set_blocking(QIO_CHANNEL(sioc), false, NULL); qio_channel_attach_aio_context(QIO_CHANNEL(sioc), aio_context); s->info.request_sizes = true; s->info.structured_reply = true; s->info.base_allocation = true; - s->info.x_dirty_bitmap = g_strdup(x_dirty_bitmap); - s->info.name = g_strdup(export ?: ""); - ret = nbd_receive_negotiate(aio_context, QIO_CHANNEL(sioc), tlscreds, - hostname, &s->ioc, &s->info, errp); + s->info.x_dirty_bitmap = g_strdup(s->x_dirty_bitmap); + s->info.name = g_strdup(s->export ?: ""); + ret = nbd_receive_negotiate(aio_context, QIO_CHANNEL(sioc), s->tlscreds, + s->hostname, &s->ioc, &s->info, errp); g_free(s->info.x_dirty_bitmap); g_free(s->info.name); if (ret < 0) { object_unref(OBJECT(sioc)); return ret; } - if (x_dirty_bitmap && !s->info.base_allocation) { + if (s->x_dirty_bitmap && !s->info.base_allocation) { error_setg(errp, "requested x-dirty-bitmap %s not found", - x_dirty_bitmap); + s->x_dirty_bitmap); ret = -EINVAL; goto fail; } @@ -1250,7 +1248,7 @@ static int nbd_client_connect(BlockDriverState *bs, object_ref(OBJECT(s->ioc)); } - trace_nbd_client_connect_success(export); + trace_nbd_client_connect_success(s->export); return 0; @@ -1270,34 +1268,9 @@ static int nbd_client_connect(BlockDriverState *bs, } } -static int nbd_client_init(BlockDriverState *bs, - SocketAddress *saddr, - const char *export, - QCryptoTLSCreds *tlscreds, - const char *hostname, - const char *x_dirty_bitmap, - uint32_t reconnect_delay, - Error **errp) -{ - int ret; - BDRVNBDState *s = (BDRVNBDState *)bs->opaque; - - s->bs = bs; - qemu_co_mutex_init(&s->send_mutex); - qemu_co_queue_init(&s->free_sema); - - ret = nbd_client_connect(bs, saddr, export, tlscreds, hostname, - x_dirty_bitmap, errp); - if (ret < 0) { - return ret; - } - - s->connection_co = qemu_coroutine_create(nbd_connection_entry, s); - bdrv_inc_in_flight(bs); - aio_co_schedule(bdrv_get_aio_context(bs), s->connection_co); - - return 0; -} +/* + * Parse nbd_open options + */ static int nbd_parse_uri(const char *filename, QDict *options) { @@ -1617,14 +1590,12 @@ static QemuOptsList nbd_runtime_opts = { }, }; -static int nbd_open(BlockDriverState *bs, QDict *options, int flags, - Error **errp) +static int nbd_process_options(BlockDriverState *bs, QDict *options, + Error **errp) { BDRVNBDState *s = bs->opaque; - QemuOpts *opts = NULL; + QemuOpts *opts; Error *local_err = NULL; - QCryptoTLSCreds *tlscreds = NULL; - const char *hostname = NULL; int ret = -EINVAL; opts = qemu_opts_create(&nbd_runtime_opts, NULL, 0, &error_abort); @@ -1649,8 +1620,8 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, s->tlscredsid = g_strdup(qemu_opt_get(opts, "tls-creds")); if (s->tlscredsid) { - tlscreds = nbd_get_tls_creds(s->tlscredsid, errp); - if (!tlscreds) { + s->tlscreds = nbd_get_tls_creds(s->tlscredsid, errp); + if (!s->tlscreds) { goto error; } @@ -1659,20 +1630,19 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, error_setg(errp, "TLS only supported over IP sockets"); goto error; } - hostname = s->saddr->u.inet.host; + s->hostname = s->saddr->u.inet.host; } - /* NBD handshake */ - ret = nbd_client_init(bs, s->saddr, s->export, tlscreds, hostname, - qemu_opt_get(opts, "x-dirty-bitmap"), - qemu_opt_get_number(opts, "reconnect-delay", 0), - errp); + s->x_dirty_bitmap = g_strdup(qemu_opt_get(opts, "x-dirty-bitmap")); + s->reconnect_delay = qemu_opt_get_number(opts, "reconnect-delay", 0); + + ret = 0; error: - if (tlscreds) { - object_unref(OBJECT(tlscreds)); - } if (ret < 0) { + if (s->tlscreds) { + object_unref(OBJECT(s->tlscreds)); + } qapi_free_SocketAddress(s->saddr); g_free(s->export); g_free(s->tlscredsid); @@ -1681,6 +1651,35 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, return ret; } +static int nbd_open(BlockDriverState *bs, QDict *options, int flags, + Error **errp) +{ + int ret; + BDRVNBDState *s = (BDRVNBDState *)bs->opaque; + + ret = nbd_process_options(bs, options, errp); + if (ret < 0) { + return ret; + } + + s->bs = bs; + qemu_co_mutex_init(&s->send_mutex); + qemu_co_queue_init(&s->free_sema); + + ret = nbd_client_connect(bs, errp); + if (ret < 0) { + return ret; + } + /* successfully connected */ + s->state = NBD_CLIENT_CONNECTED; + + s->connection_co = qemu_coroutine_create(nbd_connection_entry, s); + bdrv_inc_in_flight(bs); + aio_co_schedule(bdrv_get_aio_context(bs), s->connection_co); + + return 0; +} + static int nbd_co_flush(BlockDriverState *bs) { return nbd_client_co_flush(bs); @@ -1726,9 +1725,13 @@ static void nbd_close(BlockDriverState *bs) nbd_client_close(bs); + if (s->tlscreds) { + object_unref(OBJECT(s->tlscreds)); + } qapi_free_SocketAddress(s->saddr); g_free(s->export); g_free(s->tlscredsid); + g_free(s->x_dirty_bitmap); } static int64_t nbd_getlength(BlockDriverState *bs) From patchwork Tue Jun 18 11:43:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 11001419 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 22ABE1398 for ; Tue, 18 Jun 2019 11:45:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0F2D0289F1 for ; Tue, 18 Jun 2019 11:45:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F10A9289F5; Tue, 18 Jun 2019 11:45:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6273F289F1 for ; Tue, 18 Jun 2019 11:45:02 +0000 (UTC) Received: from localhost ([::1]:55848 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCXt-0006H1-IC for patchwork-qemu-devel@patchwork.kernel.org; Tue, 18 Jun 2019 07:45:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48333) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCWX-00041r-A3 for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdCWV-0000MS-4q for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:37 -0400 Received: from relay.sw.ru ([185.231.240.75]:58028) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdCWU-0000Jk-QV; Tue, 18 Jun 2019 07:43:35 -0400 Received: from [10.94.3.0] (helo=kvm.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.92) (envelope-from ) id 1hdCWP-00016b-Ip; Tue, 18 Jun 2019 14:43:29 +0300 From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Tue, 18 Jun 2019 14:43:25 +0300 Message-Id: <20190618114328.55249-7-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20190618114328.55249-1-vsementsov@virtuozzo.com> References: <20190618114328.55249-1-vsementsov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v7 6/9] qemu-coroutine-sleep: introduce qemu_co_sleep_wake X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, stefanha@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Introduce a function to gracefully wake a coroutine sleeping in qemu_co_sleep_ns(). Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/qemu/coroutine.h | 17 ++++++++++++-- block/null.c | 2 +- block/sheepdog.c | 2 +- tests/test-bdrv-drain.c | 6 ++--- tests/test-block-iothread.c | 2 +- util/qemu-coroutine-sleep.c | 47 +++++++++++++++++++++++++++---------- 6 files changed, 55 insertions(+), 21 deletions(-) diff --git a/include/qemu/coroutine.h b/include/qemu/coroutine.h index 9801e7f5a4..96780a4902 100644 --- a/include/qemu/coroutine.h +++ b/include/qemu/coroutine.h @@ -274,9 +274,22 @@ void qemu_co_rwlock_wrlock(CoRwlock *lock); void qemu_co_rwlock_unlock(CoRwlock *lock); /** - * Yield the coroutine for a given duration + * Yield the coroutine for a given duration. During this yield @sleep_state (if + * not NULL) is set to opaque pointer, which may be used for + * qemu_co_sleep_wake(). Be careful, the pointer is set back to zero when timer + * shoots. Don't save obtained value to other variables and don't call + * qemu_co_sleep_wake from another aio context. */ -void coroutine_fn qemu_co_sleep_ns(QEMUClockType type, int64_t ns); +void coroutine_fn qemu_co_sleep_ns(QEMUClockType type, int64_t ns, + void **sleep_state); + +/** + * Wake a coroutine if it is sleeping by qemu_co_sleep_ns. Timer will be + * deleted. @sleep_state must be the variable which address was given to + * qemu_co_sleep_ns() and should be checked to be non-NULL before calling + * qemu_co_sleep_wake(). + */ +void qemu_co_sleep_wake(void *sleep_state); /** * Yield until a file descriptor becomes readable diff --git a/block/null.c b/block/null.c index 699aa295cb..1e3f26b07e 100644 --- a/block/null.c +++ b/block/null.c @@ -109,7 +109,7 @@ static coroutine_fn int null_co_common(BlockDriverState *bs) BDRVNullState *s = bs->opaque; if (s->latency_ns) { - qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, s->latency_ns); + qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, s->latency_ns, NULL); } return 0; } diff --git a/block/sheepdog.c b/block/sheepdog.c index 6f402e5d4d..e7cab41c95 100644 --- a/block/sheepdog.c +++ b/block/sheepdog.c @@ -742,7 +742,7 @@ static coroutine_fn void reconnect_to_sdog(void *opaque) if (s->fd < 0) { trace_sheepdog_reconnect_to_sdog(); error_report_err(local_err); - qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 1000000000ULL); + qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 1000000000ULL, NULL); } }; diff --git a/tests/test-bdrv-drain.c b/tests/test-bdrv-drain.c index 12e2ecf517..aabe9e4264 100644 --- a/tests/test-bdrv-drain.c +++ b/tests/test-bdrv-drain.c @@ -42,7 +42,7 @@ static void coroutine_fn bdrv_test_co_drain_begin(BlockDriverState *bs) BDRVTestState *s = bs->opaque; s->drain_count++; if (s->sleep_in_drain_begin) { - qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 100000); + qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 100000, NULL); } } @@ -73,7 +73,7 @@ static int coroutine_fn bdrv_test_co_preadv(BlockDriverState *bs, * it to complete. We need to sleep a while as bdrv_drain_invoke() comes * first and polls its result, too, but it shouldn't accidentally complete * this request yet. */ - qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 100000); + qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 100000, NULL); if (s->bh_indirection_ctx) { aio_bh_schedule_oneshot(s->bh_indirection_ctx, co_reenter_bh, @@ -818,7 +818,7 @@ static int coroutine_fn test_job_run(Job *job, Error **errp) /* Avoid job_sleep_ns() because it marks the job as !busy. We want to * emulate some actual activity (probably some I/O) here so that drain * has to wait for this activity to stop. */ - qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 1000000); + qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 1000000, NULL); job_pause_point(&s->common.job); } diff --git a/tests/test-block-iothread.c b/tests/test-block-iothread.c index 79d9cf8a57..e0e871d53a 100644 --- a/tests/test-block-iothread.c +++ b/tests/test-block-iothread.c @@ -380,7 +380,7 @@ static int coroutine_fn test_job_run(Job *job, Error **errp) * emulate some actual activity (probably some I/O) here so that the * drain involved in AioContext switches has to wait for this activity * to stop. */ - qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 1000000); + qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 1000000, NULL); job_pause_point(&s->common.job); } diff --git a/util/qemu-coroutine-sleep.c b/util/qemu-coroutine-sleep.c index 4bfdd30cbf..48a64bb8d8 100644 --- a/util/qemu-coroutine-sleep.c +++ b/util/qemu-coroutine-sleep.c @@ -17,31 +17,52 @@ #include "qemu/timer.h" #include "block/aio.h" -static void co_sleep_cb(void *opaque) -{ - Coroutine *co = opaque; +const char *qemu_co_sleep_ns__scheduled = "qemu_co_sleep_ns"; + +typedef struct QemuCoSleepState { + Coroutine *co; + QEMUTimer *ts; + void **user_state_pointer; +} QemuCoSleepState; +void qemu_co_sleep_wake(void *sleep_state) +{ + QemuCoSleepState *s = (QemuCoSleepState *)sleep_state; /* Write of schedule protected by barrier write in aio_co_schedule */ - atomic_set(&co->scheduled, NULL); - aio_co_wake(co); + const char *scheduled = atomic_cmpxchg(&s->co->scheduled, + qemu_co_sleep_ns__scheduled, NULL); + + assert(scheduled == qemu_co_sleep_ns__scheduled); + if (s->user_state_pointer) { + *s->user_state_pointer = NULL; + } + timer_del(s->ts); + aio_co_wake(s->co); } -void coroutine_fn qemu_co_sleep_ns(QEMUClockType type, int64_t ns) +void coroutine_fn qemu_co_sleep_ns(QEMUClockType type, int64_t ns, + void **sleep_state) { AioContext *ctx = qemu_get_current_aio_context(); - QEMUTimer *ts; - Coroutine *co = qemu_coroutine_self(); + QemuCoSleepState state = { + .co = qemu_coroutine_self(), + .ts = aio_timer_new(ctx, type, SCALE_NS, qemu_co_sleep_wake, &state), + .user_state_pointer = sleep_state, + }; - const char *scheduled = atomic_cmpxchg(&co->scheduled, NULL, __func__); + const char *scheduled = atomic_cmpxchg(&state.co->scheduled, NULL, + qemu_co_sleep_ns__scheduled); if (scheduled) { fprintf(stderr, "%s: Co-routine was already scheduled in '%s'\n", __func__, scheduled); abort(); } - ts = aio_timer_new(ctx, type, SCALE_NS, co_sleep_cb, co); - timer_mod(ts, qemu_clock_get_ns(type) + ns); + + if (sleep_state) { + *sleep_state = &state; + } + timer_mod(state.ts, qemu_clock_get_ns(type) + ns); qemu_coroutine_yield(); - timer_del(ts); - timer_free(ts); + timer_free(state.ts); } From patchwork Tue Jun 18 11:43:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 11001417 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6C77B924 for ; Tue, 18 Jun 2019 11:44:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5C8C9289F1 for ; Tue, 18 Jun 2019 11:44:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4F701289F4; Tue, 18 Jun 2019 11:44:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F2B5D289F1 for ; Tue, 18 Jun 2019 11:44:52 +0000 (UTC) Received: from localhost ([::1]:55846 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCXj-0005tc-PT for patchwork-qemu-devel@patchwork.kernel.org; Tue, 18 Jun 2019 07:44:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48332) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCWX-00041q-9p for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdCWV-0000Mh-5g for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:37 -0400 Received: from relay.sw.ru ([185.231.240.75]:58032) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdCWU-0000Jm-Rj; Tue, 18 Jun 2019 07:43:35 -0400 Received: from [10.94.3.0] (helo=kvm.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.92) (envelope-from ) id 1hdCWP-00016b-OR; Tue, 18 Jun 2019 14:43:29 +0300 From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Tue, 18 Jun 2019 14:43:26 +0300 Message-Id: <20190618114328.55249-8-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20190618114328.55249-1-vsementsov@virtuozzo.com> References: <20190618114328.55249-1-vsementsov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v7 7/9] qemu/units: add SI decimal units X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, stefanha@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/qemu/units.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/include/qemu/units.h b/include/qemu/units.h index 692db3fbb2..52ccc7445c 100644 --- a/include/qemu/units.h +++ b/include/qemu/units.h @@ -17,4 +17,11 @@ #define PiB (INT64_C(1) << 50) #define EiB (INT64_C(1) << 60) +#define SI_k 1000LL +#define SI_M 1000000LL +#define SI_G 1000000000LL +#define SI_T 1000000000000LL +#define SI_P 1000000000000000LL +#define SI_E 1000000000000000000LL + #endif From patchwork Tue Jun 18 11:43:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 11001421 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2B1EF1398 for ; Tue, 18 Jun 2019 11:45:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1A23B289F1 for ; Tue, 18 Jun 2019 11:45:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0DD7D289F4; Tue, 18 Jun 2019 11:45:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id D418D289F2 for ; Tue, 18 Jun 2019 11:45:02 +0000 (UTC) Received: from localhost ([::1]:55850 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCXu-0006HF-5i for patchwork-qemu-devel@patchwork.kernel.org; Tue, 18 Jun 2019 07:45:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48387) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCWZ-00041y-7J for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdCWV-0000NL-AS for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:39 -0400 Received: from relay.sw.ru ([185.231.240.75]:58022) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdCWU-0000Jl-RV; Tue, 18 Jun 2019 07:43:35 -0400 Received: from [10.94.3.0] (helo=kvm.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.92) (envelope-from ) id 1hdCWQ-00016b-0B; Tue, 18 Jun 2019 14:43:30 +0300 From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Tue, 18 Jun 2019 14:43:27 +0300 Message-Id: <20190618114328.55249-9-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20190618114328.55249-1-vsementsov@virtuozzo.com> References: <20190618114328.55249-1-vsementsov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v7 8/9] block/nbd: nbd reconnect X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, stefanha@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Implement reconnect. To achieve this: 1. add new modes: connecting-wait: means, that reconnecting is in progress, and there were small number of reconnect attempts, so all requests are waiting for the connection. connecting-nowait: reconnecting is in progress, there were a lot of attempts of reconnect, all requests will return errors. two old modes are used too: connected: normal state quit: exiting after fatal error or on close Possible transitions are: * -> quit connecting-* -> connected connecting-wait -> connecting-nowait (transition is done after reconnect-delay seconds in connecting-wait mode) connected -> connecting-wait 2. Implement reconnect in connection_co. So, in connecting-* mode, connection_co, tries to reconnect unlimited times. 3. Retry nbd queries on channel error, if we are in connecting-wait state. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/nbd.c | 336 ++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 272 insertions(+), 64 deletions(-) diff --git a/block/nbd.c b/block/nbd.c index 68e0e168e8..42a9d14ea0 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -1,6 +1,7 @@ /* * QEMU Block driver for NBD * + * Copyright (c) 2019 Virtuozzo International GmbH. * Copyright (C) 2016 Red Hat, Inc. * Copyright (C) 2008 Bull S.A.S. * Author: Laurent Vivier @@ -33,6 +34,7 @@ #include "qemu/uri.h" #include "qemu/option.h" #include "qemu/cutils.h" +#include "qemu/units.h" #include "qapi/qapi-visit-sockets.h" #include "qapi/qmp/qstring.h" @@ -54,6 +56,8 @@ typedef struct { } NBDClientRequest; typedef enum NBDClientState { + NBD_CLIENT_CONNECTING_WAIT, + NBD_CLIENT_CONNECTING_NOWAIT, NBD_CLIENT_CONNECTED, NBD_CLIENT_QUIT } NBDClientState; @@ -66,8 +70,14 @@ typedef struct BDRVNBDState { CoMutex send_mutex; CoQueue free_sema; Coroutine *connection_co; + void *connection_co_sleep_ns_state; + bool drained; + bool wait_drained_end; int in_flight; NBDClientState state; + int connect_status; + Error *connect_err; + bool wait_in_flight; NBDClientRequest requests[MAX_NBD_REQUESTS]; NBDReply reply; @@ -82,10 +92,21 @@ typedef struct BDRVNBDState { char *x_dirty_bitmap; } BDRVNBDState; -/* @ret will be used for reconnect in future */ +static int nbd_client_connect(BlockDriverState *bs, Error **errp); + static void nbd_channel_error(BDRVNBDState *s, int ret) { - s->state = NBD_CLIENT_QUIT; + if (ret == -EIO) { + if (s->state == NBD_CLIENT_CONNECTED) { + s->state = s->reconnect_delay ? NBD_CLIENT_CONNECTING_WAIT : + NBD_CLIENT_CONNECTING_NOWAIT; + } + } else { + if (s->state == NBD_CLIENT_CONNECTED) { + qio_channel_shutdown(s->ioc, QIO_CHANNEL_SHUTDOWN_BOTH, NULL); + } + s->state = NBD_CLIENT_QUIT; + } } static void nbd_recv_coroutines_wake_all(BDRVNBDState *s) @@ -128,7 +149,13 @@ static void nbd_client_attach_aio_context(BlockDriverState *bs, { BDRVNBDState *s = (BDRVNBDState *)bs->opaque; - qio_channel_attach_aio_context(QIO_CHANNEL(s->ioc), new_context); + /* + * s->connection_co is either yielded from nbd_receive_reply or from + * nbd_reconnect_loop() + */ + if (s->state == NBD_CLIENT_CONNECTED) { + qio_channel_attach_aio_context(QIO_CHANNEL(s->ioc), new_context); + } bdrv_inc_in_flight(bs); @@ -139,29 +166,157 @@ static void nbd_client_attach_aio_context(BlockDriverState *bs, aio_wait_bh_oneshot(new_context, nbd_client_attach_aio_context_bh, bs); } +static void coroutine_fn nbd_client_co_drain_begin(BlockDriverState *bs) +{ + BDRVNBDState *s = (BDRVNBDState *)bs->opaque; -static void nbd_teardown_connection(BlockDriverState *bs) + s->drained = true; + if (s->connection_co_sleep_ns_state) { + qemu_co_sleep_wake(s->connection_co_sleep_ns_state); + } +} + +static void coroutine_fn nbd_client_co_drain_end(BlockDriverState *bs) { BDRVNBDState *s = (BDRVNBDState *)bs->opaque; - assert(s->ioc); + s->drained = false; + if (s->wait_drained_end) { + s->wait_drained_end = false; + aio_co_wake(s->connection_co); + } +} + - /* finish any pending coroutines */ - qio_channel_shutdown(s->ioc, - QIO_CHANNEL_SHUTDOWN_BOTH, - NULL); +static void nbd_teardown_connection(BlockDriverState *bs) +{ + BDRVNBDState *s = (BDRVNBDState *)bs->opaque; + + if (s->state == NBD_CLIENT_CONNECTED) { + /* finish any pending coroutines */ + assert(s->ioc); + qio_channel_shutdown(s->ioc, QIO_CHANNEL_SHUTDOWN_BOTH, NULL); + } + s->state = NBD_CLIENT_QUIT; + if (s->connection_co) { + if (s->connection_co_sleep_ns_state) { + qemu_co_sleep_wake(s->connection_co_sleep_ns_state); + } + } BDRV_POLL_WHILE(bs, s->connection_co); +} - nbd_client_detach_aio_context(bs); - object_unref(OBJECT(s->sioc)); - s->sioc = NULL; - object_unref(OBJECT(s->ioc)); - s->ioc = NULL; +static bool nbd_client_connecting(BDRVNBDState *s) +{ + return s->state == NBD_CLIENT_CONNECTING_WAIT || + s->state == NBD_CLIENT_CONNECTING_NOWAIT; +} + +static bool nbd_client_connecting_wait(BDRVNBDState *s) +{ + return s->state == NBD_CLIENT_CONNECTING_WAIT; +} + +static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) +{ + Error *local_err = NULL; + + if (!nbd_client_connecting(s)) { + return; + } + assert(nbd_client_connecting(s)); + + /* Wait for completion of all in-flight requests */ + + qemu_co_mutex_lock(&s->send_mutex); + + while (s->in_flight > 0) { + qemu_co_mutex_unlock(&s->send_mutex); + nbd_recv_coroutines_wake_all(s); + s->wait_in_flight = true; + qemu_coroutine_yield(); + s->wait_in_flight = false; + qemu_co_mutex_lock(&s->send_mutex); + } + + qemu_co_mutex_unlock(&s->send_mutex); + + if (!nbd_client_connecting(s)) { + return; + } + + /* + * Now we are sure that nobody is accessing the channel, and no one will + * try until we set the state to CONNECTED. + */ + + /* Finalize previous connection if any */ + if (s->ioc) { + nbd_client_detach_aio_context(s->bs); + object_unref(OBJECT(s->sioc)); + s->sioc = NULL; + object_unref(OBJECT(s->ioc)); + s->ioc = NULL; + } + + s->connect_status = nbd_client_connect(s->bs, &local_err); + error_free(s->connect_err); + s->connect_err = NULL; + error_propagate(&s->connect_err, local_err); + local_err = NULL; + + if (s->connect_status < 0) { + /* failed attempt */ + return; + } + + /* successfully connected */ + s->state = NBD_CLIENT_CONNECTED; + qemu_co_queue_restart_all(&s->free_sema); +} + +static coroutine_fn void nbd_reconnect_loop(BDRVNBDState *s) +{ + uint64_t start_time_ns = qemu_clock_get_ns(QEMU_CLOCK_REALTIME); + uint64_t delay_ns = s->reconnect_delay * SI_G; + uint64_t timeout = SI_G; /* 1 sec */ + uint64_t max_timeout = 16 * SI_G; + + nbd_reconnect_attempt(s); + + while (nbd_client_connecting(s)) { + if (s->state == NBD_CLIENT_CONNECTING_WAIT && + qemu_clock_get_ns(QEMU_CLOCK_REALTIME) - start_time_ns > delay_ns) + { + s->state = NBD_CLIENT_CONNECTING_NOWAIT; + qemu_co_queue_restart_all(&s->free_sema); + } + + qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, timeout, + &s->connection_co_sleep_ns_state); + if (s->drained) { + bdrv_dec_in_flight(s->bs); + s->wait_drained_end = true; + while (s->drained) { + /* + * We may be entered once from nbd_client_attach_aio_context_bh + * and then from nbd_client_co_drain_end. So here is a loop. + */ + qemu_coroutine_yield(); + } + bdrv_inc_in_flight(s->bs); + } + if (timeout < max_timeout) { + timeout *= 2; + } + + nbd_reconnect_attempt(s); + } } static coroutine_fn void nbd_connection_entry(void *opaque) { - BDRVNBDState *s = opaque; + BDRVNBDState *s = (BDRVNBDState *)opaque; uint64_t i; int ret = 0; Error *local_err = NULL; @@ -176,16 +331,26 @@ static coroutine_fn void nbd_connection_entry(void *opaque) * Therefore we keep an additional in_flight reference all the time and * only drop it temporarily here. */ + + if (nbd_client_connecting(s)) { + nbd_reconnect_loop(s); + } + + if (s->state != NBD_CLIENT_CONNECTED) { + continue; + } + assert(s->reply.handle == 0); ret = nbd_receive_reply(s->bs, s->ioc, &s->reply, &local_err); if (local_err) { trace_nbd_read_reply_entry_fail(ret, error_get_pretty(local_err)); error_free(local_err); + local_err = NULL; } if (ret <= 0) { nbd_channel_error(s, ret ? ret : -EIO); - break; + continue; } /* @@ -200,7 +365,7 @@ static coroutine_fn void nbd_connection_entry(void *opaque) (nbd_reply_is_structured(&s->reply) && !s->info.structured_reply)) { nbd_channel_error(s, -EINVAL); - break; + continue; } /* @@ -219,10 +384,19 @@ static coroutine_fn void nbd_connection_entry(void *opaque) qemu_coroutine_yield(); } + qemu_co_queue_restart_all(&s->free_sema); nbd_recv_coroutines_wake_all(s); bdrv_dec_in_flight(s->bs); s->connection_co = NULL; + if (s->ioc) { + nbd_client_detach_aio_context(s->bs); + object_unref(OBJECT(s->sioc)); + s->sioc = NULL; + object_unref(OBJECT(s->ioc)); + s->ioc = NULL; + } + aio_wait_kick(); } @@ -234,7 +408,7 @@ static int nbd_co_send_request(BlockDriverState *bs, int rc, i = -1; qemu_co_mutex_lock(&s->send_mutex); - while (s->in_flight == MAX_NBD_REQUESTS) { + while (s->in_flight == MAX_NBD_REQUESTS || nbd_client_connecting_wait(s)) { qemu_co_queue_wait(&s->free_sema, &s->send_mutex); } @@ -285,7 +459,11 @@ err: s->requests[i].coroutine = NULL; s->in_flight--; } - qemu_co_queue_next(&s->free_sema); + if (s->in_flight == 0 && s->wait_in_flight) { + aio_co_wake(s->connection_co); + } else { + qemu_co_queue_next(&s->free_sema); + } } qemu_co_mutex_unlock(&s->send_mutex); return rc; @@ -666,10 +844,15 @@ static coroutine_fn int nbd_co_receive_one_chunk( if (reply) { *reply = s->reply; } - s->reply.handle = 0; } + s->reply.handle = 0; - if (s->connection_co) { + if (s->connection_co && !s->wait_in_flight) { + /* + * We must check s->wait_in_flight, because we may entered by + * nbd_recv_coroutines_wake_all(), in this case we should not + * wake connection_co here, it will woken by last request. + */ aio_co_wake(s->connection_co); } @@ -781,7 +964,11 @@ break_loop: qemu_co_mutex_lock(&s->send_mutex); s->in_flight--; - qemu_co_queue_next(&s->free_sema); + if (s->in_flight == 0 && s->wait_in_flight) { + aio_co_wake(s->connection_co); + } else { + qemu_co_queue_next(&s->free_sema); + } qemu_co_mutex_unlock(&s->send_mutex); return false; @@ -927,20 +1114,26 @@ static int nbd_co_request(BlockDriverState *bs, NBDRequest *request, } else { assert(request->type != NBD_CMD_WRITE); } - ret = nbd_co_send_request(bs, request, write_qiov); - if (ret < 0) { - return ret; - } - ret = nbd_co_receive_return_code(s, request->handle, - &request_ret, &local_err); - if (local_err) { - trace_nbd_co_request_fail(request->from, request->len, request->handle, - request->flags, request->type, - nbd_cmd_lookup(request->type), - ret, error_get_pretty(local_err)); - error_free(local_err); - } + do { + ret = nbd_co_send_request(bs, request, write_qiov); + if (ret < 0) { + continue; + } + + ret = nbd_co_receive_return_code(s, request->handle, + &request_ret, &local_err); + if (local_err) { + trace_nbd_co_request_fail(request->from, request->len, + request->handle, request->flags, + request->type, + nbd_cmd_lookup(request->type), + ret, error_get_pretty(local_err)); + error_free(local_err); + local_err = NULL; + } + } while (ret < 0 && nbd_client_connecting_wait(s)); + return ret ? ret : request_ret; } @@ -981,20 +1174,24 @@ static int nbd_client_co_preadv(BlockDriverState *bs, uint64_t offset, request.len -= slop; } - ret = nbd_co_send_request(bs, &request, NULL); - if (ret < 0) { - return ret; - } + do { + ret = nbd_co_send_request(bs, &request, NULL); + if (ret < 0) { + continue; + } + + ret = nbd_co_receive_cmdread_reply(s, request.handle, offset, qiov, + &request_ret, &local_err); + if (local_err) { + trace_nbd_co_request_fail(request.from, request.len, request.handle, + request.flags, request.type, + nbd_cmd_lookup(request.type), + ret, error_get_pretty(local_err)); + error_free(local_err); + local_err = NULL; + } + } while (ret < 0 && nbd_client_connecting_wait(s)); - ret = nbd_co_receive_cmdread_reply(s, request.handle, offset, qiov, - &request_ret, &local_err); - if (local_err) { - trace_nbd_co_request_fail(request.from, request.len, request.handle, - request.flags, request.type, - nbd_cmd_lookup(request.type), - ret, error_get_pretty(local_err)); - error_free(local_err); - } return ret ? ret : request_ret; } @@ -1127,20 +1324,25 @@ static int coroutine_fn nbd_client_co_block_status( if (s->info.min_block) { assert(QEMU_IS_ALIGNED(request.len, s->info.min_block)); } - ret = nbd_co_send_request(bs, &request, NULL); - if (ret < 0) { - return ret; - } + do { + ret = nbd_co_send_request(bs, &request, NULL); + if (ret < 0) { + continue; + } + + ret = nbd_co_receive_blockstatus_reply(s, request.handle, bytes, + &extent, &request_ret, + &local_err); + if (local_err) { + trace_nbd_co_request_fail(request.from, request.len, request.handle, + request.flags, request.type, + nbd_cmd_lookup(request.type), + ret, error_get_pretty(local_err)); + error_free(local_err); + local_err = NULL; + } + } while (ret < 0 && nbd_client_connecting_wait(s)); - ret = nbd_co_receive_blockstatus_reply(s, request.handle, bytes, - &extent, &request_ret, &local_err); - if (local_err) { - trace_nbd_co_request_fail(request.from, request.len, request.handle, - request.flags, request.type, - nbd_cmd_lookup(request.type), - ret, error_get_pretty(local_err)); - error_free(local_err); - } if (ret < 0 || request_ret < 0) { return ret ? ret : request_ret; } @@ -1159,9 +1361,9 @@ static void nbd_client_close(BlockDriverState *bs) BDRVNBDState *s = (BDRVNBDState *)bs->opaque; NBDRequest request = { .type = NBD_CMD_DISC }; - assert(s->ioc); - - nbd_send_request(s->ioc, &request); + if (s->ioc) { + nbd_send_request(s->ioc, &request); + } nbd_teardown_connection(bs); } @@ -1808,6 +2010,8 @@ static BlockDriver bdrv_nbd = { .bdrv_getlength = nbd_getlength, .bdrv_detach_aio_context = nbd_client_detach_aio_context, .bdrv_attach_aio_context = nbd_client_attach_aio_context, + .bdrv_co_drain_begin = nbd_client_co_drain_begin, + .bdrv_co_drain_end = nbd_client_co_drain_end, .bdrv_refresh_filename = nbd_refresh_filename, .bdrv_co_block_status = nbd_client_co_block_status, .bdrv_dirname = nbd_dirname, @@ -1830,6 +2034,8 @@ static BlockDriver bdrv_nbd_tcp = { .bdrv_getlength = nbd_getlength, .bdrv_detach_aio_context = nbd_client_detach_aio_context, .bdrv_attach_aio_context = nbd_client_attach_aio_context, + .bdrv_co_drain_begin = nbd_client_co_drain_begin, + .bdrv_co_drain_end = nbd_client_co_drain_end, .bdrv_refresh_filename = nbd_refresh_filename, .bdrv_co_block_status = nbd_client_co_block_status, .bdrv_dirname = nbd_dirname, @@ -1852,6 +2058,8 @@ static BlockDriver bdrv_nbd_unix = { .bdrv_getlength = nbd_getlength, .bdrv_detach_aio_context = nbd_client_detach_aio_context, .bdrv_attach_aio_context = nbd_client_attach_aio_context, + .bdrv_co_drain_begin = nbd_client_co_drain_begin, + .bdrv_co_drain_end = nbd_client_co_drain_end, .bdrv_refresh_filename = nbd_refresh_filename, .bdrv_co_block_status = nbd_client_co_block_status, .bdrv_dirname = nbd_dirname, From patchwork Tue Jun 18 11:43:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 11001461 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A137514B6 for ; Tue, 18 Jun 2019 11:55:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 914B928971 for ; Tue, 18 Jun 2019 11:55:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 85A8E28A16; Tue, 18 Jun 2019 11:55:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 070C128A14 for ; Tue, 18 Jun 2019 11:55:23 +0000 (UTC) Received: from localhost ([::1]:55938 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdChu-0007Ef-Cn for patchwork-qemu-devel@patchwork.kernel.org; Tue, 18 Jun 2019 07:55:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48330) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdCWX-00041p-9j for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdCWV-0000Mr-66 for qemu-devel@nongnu.org; Tue, 18 Jun 2019 07:43:37 -0400 Received: from relay.sw.ru ([185.231.240.75]:58014) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdCWU-0000Jg-Rj; Tue, 18 Jun 2019 07:43:35 -0400 Received: from [10.94.3.0] (helo=kvm.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.92) (envelope-from ) id 1hdCWQ-00016b-3V; Tue, 18 Jun 2019 14:43:30 +0300 From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Tue, 18 Jun 2019 14:43:28 +0300 Message-Id: <20190618114328.55249-10-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20190618114328.55249-1-vsementsov@virtuozzo.com> References: <20190618114328.55249-1-vsementsov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v7 9/9] iotests: test nbd reconnect X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, stefanha@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Add test, which starts backup to nbd target and restarts nbd server during backup. Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/257 | 63 +++++++++++++++++++++++++++++++++++ tests/qemu-iotests/257.out | 10 ++++++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 4 +++ 4 files changed, 78 insertions(+) create mode 100755 tests/qemu-iotests/257 create mode 100644 tests/qemu-iotests/257.out diff --git a/tests/qemu-iotests/257 b/tests/qemu-iotests/257 new file mode 100755 index 0000000000..a29a276207 --- /dev/null +++ b/tests/qemu-iotests/257 @@ -0,0 +1,63 @@ +#!/usr/bin/env python +# +# Test nbd reconnect +# +# Copyright (c) 2019 Virtuozzo International GmbH. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +import time + +import iotests +from iotests import qemu_img_create, file_path, qemu_nbd_popen, log + +disk_a, disk_b, nbd_sock = file_path('disk_a', 'disk_b', 'nbd-sock') + +qemu_img_create('-f', iotests.imgfmt, disk_a, '5M') +qemu_img_create('-f', iotests.imgfmt, disk_b, '5M') +srv = qemu_nbd_popen('-k', nbd_sock, '-f', iotests.imgfmt, disk_b) +time.sleep(1) + +vm = iotests.VM().add_drive(disk_a) +vm.launch() +vm.hmp_qemu_io('drive0', 'write 0 5M') + +vm.qmp_log('blockdev-add', filters=[iotests.filter_qmp_testfiles], + **{'node_name': 'backup0', + 'driver': 'raw', + 'file': {'driver': 'nbd', + 'server': {'type': 'unix', 'path': nbd_sock}, + 'reconnect-delay': 10}}) +vm.qmp_log('blockdev-backup', device='drive0', sync='full', target='backup0') + +time.sleep(1) +log('Kill NBD server') +srv.kill() + +jobs = vm.qmp('query-block-jobs')['return'] +if jobs and jobs[0]['offset'] < jobs[0]['len']: + log('Backup job is still in progress') + +time.sleep(1) + +log('Start NBD server') +srv = qemu_nbd_popen('-k', nbd_sock, '-f', iotests.imgfmt, disk_b) + +e = vm.event_wait('BLOCK_JOB_COMPLETED') +log('Backup completed: {}'.format(e['data']['offset'])) + +vm.qmp_log('blockdev-del', node_name='backup0') +srv.kill() +vm.shutdown() diff --git a/tests/qemu-iotests/257.out b/tests/qemu-iotests/257.out new file mode 100644 index 0000000000..15a82ddefb --- /dev/null +++ b/tests/qemu-iotests/257.out @@ -0,0 +1,10 @@ +{"execute": "blockdev-add", "arguments": {"driver": "raw", "file": {"driver": "nbd", "reconnect-delay": 10, "server": {"path": "TEST_DIR/PID-nbd-sock", "type": "unix"}}, "node-name": "backup0"}} +{"return": {}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "sync": "full", "target": "backup0"}} +{"return": {}} +Kill NBD server +Backup job is still in progress +Start NBD server +Backup completed: 5242880 +{"execute": "blockdev-del", "arguments": {"node-name": "backup0"}} +{"return": {}} diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index b34c8e3c0c..00fbfefc8e 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -269,3 +269,4 @@ 254 rw auto backing quick 255 rw auto quick 256 rw auto quick +257 rw auto quick diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 3ecef5bc90..3c77950877 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -226,6 +226,10 @@ def qemu_nbd_early_pipe(*args): else: return exitcode, subp.communicate()[0] +def qemu_nbd_popen(*args): + '''Run qemu-nbd in daemon mode and return the parent's exit code''' + return subprocess.Popen(qemu_nbd_args + ['--persistent'] + list(args)) + def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt): '''Return True if two image files are identical''' return qemu_img('compare', '-f', fmt1,