From patchwork Wed Jan 9 11:27:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yongji Xie X-Patchwork-Id: 10753879 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0BFF013B5 for ; Wed, 9 Jan 2019 11:29:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EA39428ECF for ; Wed, 9 Jan 2019 11:29:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D876A28EE2; Wed, 9 Jan 2019 11:29:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4AC4428ECF for ; Wed, 9 Jan 2019 11:29:34 +0000 (UTC) Received: from localhost ([127.0.0.1]:34588 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghC3B-0001AP-GC for patchwork-qemu-devel@patchwork.kernel.org; Wed, 09 Jan 2019 06:29:33 -0500 Received: from eggs.gnu.org ([209.51.188.92]:51598) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghC1X-0000k9-54 for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:27:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ghC1W-0004Dx-1h for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:27:51 -0500 Received: from mail-pf1-x433.google.com ([2607:f8b0:4864:20::433]:40784) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ghC1V-0004DT-Po for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:27:49 -0500 Received: by mail-pf1-x433.google.com with SMTP id i12so3528419pfo.7 for ; Wed, 09 Jan 2019 03:27:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=1P7kOT3G9jsWY4SFY8k+LRZvUpdqXQzKe3OdHcgbz24=; b=gFf9JKKGC3etC0JfaRxb6zP6gwLwWV9TdIW0XQMEZ+vcldncYFUN3HLLs79jZfdT5o Mm+koio6WZSBLthHdAD7VRwEQ43Qmvst/svS9VWq4g8jalNj2mo1uL5/sjUjAZbf637m +kaTA1m4fqLfL/EAas4z+/fJxNX3GzTV58ch/rz96qDeLACPI6sBUTSJt+X0jv64lfd5 HbwYT8AemJj4r19GB7mDgaCQ6PGpQ6mmRCHvI1vdJHOEgmYUwDjyoQqHcf/2W5zKk7/X ihvSStbsx6K73YF7PypQYTRjMh5cIVRs1rVrurUo43ar+oGbs9n1HysR5BUYqjWwYi4U B+wA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=1P7kOT3G9jsWY4SFY8k+LRZvUpdqXQzKe3OdHcgbz24=; b=LcZVuudYpQ6jl51aKLxznMxjOg5QryKsvEMEKhpC1kVj9Zn86lxWXMu19jGzn6dT48 18n8bKlXPeO4q1pqg1wypiZ/k8Y+9waFuTFIRs/n5ViPOiaLXLzupthJXTXYOBxVUKb/ yIpJu0d8Z1UKtPMSrttP580V3EoZP8nG5JJhEYUYLgsK02E0KWMOLxogCcN9NAw7cdHQ sbXSmzX0/navaVC2WNnoX/XI8x18s62OsgmXK/E4ptL5Lm0gLRPcadfjE8pw8jysPs6o tccmiKBeN77OMYRttFZy9NXFPBpdWtc9u8zvLN+Aba0CY+ZNTOPn6++yOPXRhuiYj3hs bM+Q== X-Gm-Message-State: AJcUukeRdJNsiJz12sYuCaGkHHFV94e5R13f/XF7kNwH3FoeHKwy9Sej EaOJlNqwotquf3E7fXFNt08= X-Google-Smtp-Source: ALg8bN4oKH6FoQozWvOcgD/RlS0/SA/dhCYYBHnDGKMF58JQWS7mh83864wIMXKQ6Lmoaaov5D/2kQ== X-Received: by 2002:a63:c303:: with SMTP id c3mr5032848pgd.268.1547033268859; Wed, 09 Jan 2019 03:27:48 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id a17sm90790258pgm.26.2019.01.09.03.27.48 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 09 Jan 2019 03:27:48 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, marcandre.lureau@redhat.com, berrange@redhat.com, jasowang@redhat.com, maxime.coquelin@redhat.com, yury-kotov@yandex-team.ru, wrfsh@yandex-team.ru Date: Wed, 9 Jan 2019 19:27:22 +0800 Message-Id: <20190109112728.9214-2-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190109112728.9214-1-xieyongji@baidu.com> References: <20190109112728.9214-1-xieyongji@baidu.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::433 Subject: [Qemu-devel] [PATCH v4 for-4.0 1/7] char-socket: Enable "nowait" option on client sockets X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Xie Yongji Enable "nowait" option to make QEMU not do a connect on client sockets during initialization of the chardev. Then we can use qemu_chr_fe_wait_connected() to connect when necessary. Now it would be used for unix domain socket of vhost-user-blk device to support reconnect. Signed-off-by: Xie Yongji Signed-off-by: Zhang Yu --- chardev/char-socket.c | 56 +++++++++++++++++++++---------------------- qapi/char.json | 3 +-- qemu-options.hx | 9 ++++--- 3 files changed, 35 insertions(+), 33 deletions(-) diff --git a/chardev/char-socket.c b/chardev/char-socket.c index eaa8e8b68f..f803f4f7d3 100644 --- a/chardev/char-socket.c +++ b/chardev/char-socket.c @@ -1072,37 +1072,37 @@ static void qmp_chardev_open_socket(Chardev *chr, s->reconnect_time = reconnect; } - if (s->reconnect_time) { - tcp_chr_connect_async(chr); - } else { - if (s->is_listen) { - char *name; - s->listener = qio_net_listener_new(); + if (s->is_listen) { + char *name; + s->listener = qio_net_listener_new(); - name = g_strdup_printf("chardev-tcp-listener-%s", chr->label); - qio_net_listener_set_name(s->listener, name); - g_free(name); + name = g_strdup_printf("chardev-tcp-listener-%s", chr->label); + qio_net_listener_set_name(s->listener, name); + g_free(name); - if (qio_net_listener_open_sync(s->listener, s->addr, errp) < 0) { - object_unref(OBJECT(s->listener)); - s->listener = NULL; - goto error; - } + if (qio_net_listener_open_sync(s->listener, s->addr, errp) < 0) { + object_unref(OBJECT(s->listener)); + s->listener = NULL; + goto error; + } - qapi_free_SocketAddress(s->addr); - s->addr = socket_local_address(s->listener->sioc[0]->fd, errp); - update_disconnected_filename(s); + qapi_free_SocketAddress(s->addr); + s->addr = socket_local_address(s->listener->sioc[0]->fd, errp); + update_disconnected_filename(s); - if (is_waitconnect && - qemu_chr_wait_connected(chr, errp) < 0) { - return; - } - if (!s->ioc) { - qio_net_listener_set_client_func_full(s->listener, - tcp_chr_accept, - chr, NULL, - chr->gcontext); - } + if (is_waitconnect && + qemu_chr_wait_connected(chr, errp) < 0) { + return; + } + if (!s->ioc) { + qio_net_listener_set_client_func_full(s->listener, + tcp_chr_accept, + chr, NULL, + chr->gcontext); + } + } else if (is_waitconnect) { + if (s->reconnect_time) { + tcp_chr_connect_async(chr); } else if (qemu_chr_wait_connected(chr, errp) < 0) { goto error; } @@ -1120,7 +1120,7 @@ static void qemu_chr_parse_socket(QemuOpts *opts, ChardevBackend *backend, Error **errp) { bool is_listen = qemu_opt_get_bool(opts, "server", false); - bool is_waitconnect = is_listen && qemu_opt_get_bool(opts, "wait", true); + bool is_waitconnect = qemu_opt_get_bool(opts, "wait", true); bool is_telnet = qemu_opt_get_bool(opts, "telnet", false); bool is_tn3270 = qemu_opt_get_bool(opts, "tn3270", false); bool is_websock = qemu_opt_get_bool(opts, "websocket", false); diff --git a/qapi/char.json b/qapi/char.json index 77ed847972..6a3b5bcd71 100644 --- a/qapi/char.json +++ b/qapi/char.json @@ -249,8 +249,7 @@ # or connect to (server=false) # @tls-creds: the ID of the TLS credentials object (since 2.6) # @server: create server socket (default: true) -# @wait: wait for incoming connection on server -# sockets (default: false). +# @wait: wait for being connected or connecting to (default: false) # @nodelay: set TCP_NODELAY socket option (default: false) # @telnet: enable telnet protocol on server # sockets (default: false) diff --git a/qemu-options.hx b/qemu-options.hx index d4f3564b78..ebd11220c4 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -2556,8 +2556,9 @@ undefined if TCP options are specified for a unix socket. @option{server} specifies that the socket shall be a listening socket. -@option{nowait} specifies that QEMU should not block waiting for a client to -connect to a listening socket. +@option{nowait} specifies that QEMU should not wait for being connected on +server sockets or try to do a sync/async connect on client sockets during +initialization of the chardev. @option{telnet} specifies that traffic on the socket should interpret telnet escape sequences. @@ -3093,7 +3094,9 @@ I/O to a location or wait for a connection from a location. By default the TCP Net Console is sent to @var{host} at the @var{port}. If you use the @var{server} option QEMU will wait for a client socket application to connect to the port before continuing, unless the @code{nowait} -option was specified. The @code{nodelay} option disables the Nagle buffering +option was specified. And the @code{nowait} option could also be +used when @var{noserver} is set to disallow QEMU to connect during +initialization. The @code{nodelay} option disables the Nagle buffering algorithm. The @code{reconnect} option only applies if @var{noserver} is set, if the connection goes down it will attempt to reconnect at the given interval. If @var{host} is omitted, 0.0.0.0 is assumed. Only From patchwork Wed Jan 9 11:27:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yongji Xie X-Patchwork-Id: 10753893 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 566156C5 for ; Wed, 9 Jan 2019 11:32:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4067628F4B for ; Wed, 9 Jan 2019 11:32:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 31E3528F5B; Wed, 9 Jan 2019 11:32:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 29CB828F4B for ; Wed, 9 Jan 2019 11:32:16 +0000 (UTC) Received: from localhost ([127.0.0.1]:35594 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghC5n-0004JQ-DQ for patchwork-qemu-devel@patchwork.kernel.org; Wed, 09 Jan 2019 06:32:15 -0500 Received: from eggs.gnu.org ([209.51.188.92]:51631) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghC1b-0000nS-FT for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:27:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ghC1Z-0004FY-LP for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:27:55 -0500 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:45061) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ghC1Z-0004Ez-CZ for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:27:53 -0500 Received: by mail-pg1-x541.google.com with SMTP id y4so3170493pgc.12 for ; Wed, 09 Jan 2019 03:27:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=7+E3XkSQX5+npljE1Xi0VMmkkXpL/jQ2wFwf6goThvw=; b=R+YaotBZ7VZcT9lE2WZi6sdLV4SLKi0eiJdljWy4YMDKlm+4D6o4QS0fNDgkgTbBrM 7ou88wZNPaA6df4hHGfdvNb1KCg4K1lC/DjYaQYHMcg/uusD+IzMkjiFtAynw5ep7ZQB UZ5m9JBqPLw9dvBs2RHQP354iQPGZA7cK7pkYBpa4UHzS+MgB9BTvRTkcs1SMM+zr5Q+ eVmp2+8uhLnHxGXa0zr0mcS2YN853H5WgdDEZsZtgqwxuMrz6QjzvTl5G8Jc5m8NBOt8 3vFw+qO+SVvTZW3QQW1UOU0xcB9FTwrwZqSQmcs04/E1GVRTBlmBR8/5vbJJcpHIlwzm kEQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=7+E3XkSQX5+npljE1Xi0VMmkkXpL/jQ2wFwf6goThvw=; b=mPzWLDMFnF0TgVkscyunChe9FNh8J6g4MiOWgpLXWDskqsrqn/uKTj9pfLf7OFhQGa ILlt3h2XrSSSkmvjYSIOM9SQBF2rmITsrwzLipe93E02HPw9W31RDH8Aq1GbUXVw7g4Z I8uBXJ7T8wDxH5KxpgNBQbjeaQtYsi7tW7AWLthfQZ6rv8jrdB79+l2nFh1q3zAguVBV MhCn7mOC4tFW5YhGuGRJOFmkjOwSEHYw8hmVwLlEzEMvNGaZGJ0KaW5SBNR0U0gJ8Dvz lD9ZmG8ZolvP1g5oWDKrMlC3k2tqCjcrWQ8kn2uvm9pwSZzG2yl50Yr6aZ6aMg/xjO0R Q5lg== X-Gm-Message-State: AJcUukeHAaECuKz5h5VUzs2svqkOJpRfhxHyalY9xccjGOKP1FES6xFM S8V5ca11uc/XtDvXQ803mHpssnTBRds= X-Google-Smtp-Source: ALg8bN7EDymJ2srZhWmxIWAnexjFVuMol9xl7M1rhXQRDb0f+/nw3nLbWwNJnFR30jN+xDeLCvpHbg== X-Received: by 2002:a63:8ac4:: with SMTP id y187mr5022473pgd.446.1547033272267; Wed, 09 Jan 2019 03:27:52 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id n186sm99032843pfn.137.2019.01.09.03.27.51 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 09 Jan 2019 03:27:51 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, marcandre.lureau@redhat.com, berrange@redhat.com, jasowang@redhat.com, maxime.coquelin@redhat.com, yury-kotov@yandex-team.ru, wrfsh@yandex-team.ru Date: Wed, 9 Jan 2019 19:27:23 +0800 Message-Id: <20190109112728.9214-3-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190109112728.9214-1-xieyongji@baidu.com> References: <20190109112728.9214-1-xieyongji@baidu.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::541 Subject: [Qemu-devel] [PATCH v4 for-4.0 2/7] vhost-user: Support transferring inflight buffer between qemu and backend X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Xie Yongji This patch introduces two new messages VHOST_USER_GET_INFLIGHT_FD and VHOST_USER_SET_INFLIGHT_FD to support transferring a shared buffer between qemu and backend. Firstly, qemu uses VHOST_USER_GET_INFLIGHT_FD to get the shared buffer from backend. Then qemu should send it back through VHOST_USER_SET_INFLIGHT_FD each time we start vhost-user. This shared buffer is used to track inflight I/O by backend. Qemu should clear it when vm reset. Signed-off-by: Xie Yongji Signed-off-by: Chai Wen Signed-off-by: Zhang Yu --- docs/interop/vhost-user.txt | 60 +++++++++++++++++ hw/virtio/vhost-user.c | 108 ++++++++++++++++++++++++++++++ hw/virtio/vhost.c | 108 ++++++++++++++++++++++++++++++ include/hw/virtio/vhost-backend.h | 9 +++ include/hw/virtio/vhost.h | 19 ++++++ 5 files changed, 304 insertions(+) diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt index c2194711d9..67da41fdd2 100644 --- a/docs/interop/vhost-user.txt +++ b/docs/interop/vhost-user.txt @@ -142,6 +142,18 @@ Depending on the request type, payload can be: Offset: a 64-bit offset of this area from the start of the supplied file descriptor + * Inflight description + ---------------------------------------------------------- + | mmap size | mmap offset | align | num queues | version | + ---------------------------------------------------------- + + mmap size: a 64-bit size of area to track inflight I/O + mmap offset: a 64-bit offset of this area from the start + of the supplied file descriptor + align: a 32-bit align of each region in this area + num queues: a 16-bit number of virtqueues + version: a 16-bit version of this area + In QEMU the vhost-user message is implemented with the following struct: typedef struct VhostUserMsg { @@ -157,6 +169,7 @@ typedef struct VhostUserMsg { struct vhost_iotlb_msg iotlb; VhostUserConfig config; VhostUserVringArea area; + VhostUserInflight inflight; }; } QEMU_PACKED VhostUserMsg; @@ -175,6 +188,7 @@ the ones that do: * VHOST_USER_GET_PROTOCOL_FEATURES * VHOST_USER_GET_VRING_BASE * VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD) + * VHOST_USER_GET_INFLIGHT_FD (if VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD) [ Also see the section on REPLY_ACK protocol extension. ] @@ -188,6 +202,7 @@ in the ancillary data: * VHOST_USER_SET_VRING_CALL * VHOST_USER_SET_VRING_ERR * VHOST_USER_SET_SLAVE_REQ_FD + * VHOST_USER_SET_INFLIGHT_FD (if VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD) If Master is unable to send the full message or receives a wrong reply it will close the connection. An optional reconnection mechanism can be implemented. @@ -382,6 +397,30 @@ If VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD protocol feature is negotiated, slave can send file descriptors (at most 8 descriptors in each message) to master via ancillary data using this fd communication channel. +Inflight I/O tracking +--------------------- + +To support slave reconnecting, slave need to track inflight I/O in a +shared memory. VHOST_USER_GET_INFLIGHT_FD and VHOST_USER_SET_INFLIGHT_FD +are used to transfer the memory between master and slave. And to encourage +consistency, we provide a recommended format for this memory: + +offset width description +0x0 0x400 region for queue0 +0x400 0x400 region for queue1 +0x800 0x400 region for queue2 +... ... ... + +For each virtqueue, we have a 1024 bytes region. The region's format is like: + +offset width description +0x0 0x1 descriptor 0 is in use or not +0x1 0x1 descriptor 1 is in use or not +0x2 0x1 descriptor 2 is in use or not +... ... ... + +For each descriptor, we use one byte to specify whether it's in use or not. + Protocol features ----------------- @@ -397,6 +436,7 @@ Protocol features #define VHOST_USER_PROTOCOL_F_CONFIG 9 #define VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD 10 #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11 +#define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12 Master message types -------------------- @@ -761,6 +801,26 @@ Master message types was previously sent. The value returned is an error indication; 0 is success. + * VHOST_USER_GET_INFLIGHT_FD + Id: 31 + Equivalent ioctl: N/A + Master payload: inflight description + + When VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD protocol feature has been + successfully negotiated, this message is submitted by master to get + a shared memory from slave. The shared memory will be used to track + inflight I/O by slave. Master should clear it when vm reset. + + * VHOST_USER_SET_INFLIGHT_FD + Id: 32 + Equivalent ioctl: N/A + Master payload: inflight description + + When VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD protocol feature has been + successfully negotiated, this message is submitted by master to send + the shared inflight buffer back to slave so that slave could get + inflight I/O after a crash or restart. + Slave message types ------------------- diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index e09bed0e4a..4d118c6e14 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -52,6 +52,7 @@ enum VhostUserProtocolFeature { VHOST_USER_PROTOCOL_F_CONFIG = 9, VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD = 10, VHOST_USER_PROTOCOL_F_HOST_NOTIFIER = 11, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD = 12, VHOST_USER_PROTOCOL_F_MAX }; @@ -89,6 +90,8 @@ typedef enum VhostUserRequest { VHOST_USER_POSTCOPY_ADVISE = 28, VHOST_USER_POSTCOPY_LISTEN = 29, VHOST_USER_POSTCOPY_END = 30, + VHOST_USER_GET_INFLIGHT_FD = 31, + VHOST_USER_SET_INFLIGHT_FD = 32, VHOST_USER_MAX } VhostUserRequest; @@ -147,6 +150,14 @@ typedef struct VhostUserVringArea { uint64_t offset; } VhostUserVringArea; +typedef struct VhostUserInflight { + uint64_t mmap_size; + uint64_t mmap_offset; + uint32_t align; + uint16_t num_queues; + uint16_t version; +} VhostUserInflight; + typedef struct { VhostUserRequest request; @@ -169,6 +180,7 @@ typedef union { VhostUserConfig config; VhostUserCryptoSession session; VhostUserVringArea area; + VhostUserInflight inflight; } VhostUserPayload; typedef struct VhostUserMsg { @@ -1739,6 +1751,100 @@ static bool vhost_user_mem_section_filter(struct vhost_dev *dev, return result; } +static int vhost_user_get_inflight_fd(struct vhost_dev *dev, + struct vhost_inflight *inflight) +{ + void *addr; + int fd; + struct vhost_user *u = dev->opaque; + CharBackend *chr = u->user->chr; + VhostUserMsg msg = { + .hdr.request = VHOST_USER_GET_INFLIGHT_FD, + .hdr.flags = VHOST_USER_VERSION, + .payload.inflight.num_queues = dev->nvqs, + .hdr.size = sizeof(msg.payload.inflight), + }; + + if (!virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) { + return 0; + } + + if (vhost_user_write(dev, &msg, NULL, 0) < 0) { + return -1; + } + + if (vhost_user_read(dev, &msg) < 0) { + return -1; + } + + if (msg.hdr.request != VHOST_USER_GET_INFLIGHT_FD) { + error_report("Received unexpected msg type. " + "Expected %d received %d", + VHOST_USER_GET_INFLIGHT_FD, msg.hdr.request); + return -1; + } + + if (msg.hdr.size != sizeof(msg.payload.inflight)) { + error_report("Received bad msg size."); + return -1; + } + + if (!msg.payload.inflight.mmap_size) { + return 0; + } + + fd = qemu_chr_fe_get_msgfd(chr); + if (fd < 0) { + error_report("Failed to get mem fd"); + return -1; + } + + addr = mmap(0, msg.payload.inflight.mmap_size, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, msg.payload.inflight.mmap_offset); + + if (addr == MAP_FAILED) { + error_report("Failed to mmap mem fd"); + close(fd); + return -1; + } + + inflight->addr = addr; + inflight->fd = fd; + inflight->size = msg.payload.inflight.mmap_size; + inflight->offset = msg.payload.inflight.mmap_offset; + inflight->align = msg.payload.inflight.align; + inflight->version = msg.payload.inflight.version; + + return 0; +} + +static int vhost_user_set_inflight_fd(struct vhost_dev *dev, + struct vhost_inflight *inflight) +{ + VhostUserMsg msg = { + .hdr.request = VHOST_USER_SET_INFLIGHT_FD, + .hdr.flags = VHOST_USER_VERSION, + .payload.inflight.mmap_size = inflight->size, + .payload.inflight.mmap_offset = inflight->offset, + .payload.inflight.align = inflight->align, + .payload.inflight.num_queues = dev->nvqs, + .payload.inflight.version = inflight->version, + .hdr.size = sizeof(msg.payload.inflight), + }; + + if (!virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) { + return 0; + } + + if (vhost_user_write(dev, &msg, &inflight->fd, 1) < 0) { + return -1; + } + + return 0; +} + VhostUserState *vhost_user_init(void) { VhostUserState *user = g_new0(struct VhostUserState, 1); @@ -1790,4 +1896,6 @@ const VhostOps user_ops = { .vhost_crypto_create_session = vhost_user_crypto_create_session, .vhost_crypto_close_session = vhost_user_crypto_close_session, .vhost_backend_mem_section_filter = vhost_user_mem_section_filter, + .vhost_get_inflight_fd = vhost_user_get_inflight_fd, + .vhost_set_inflight_fd = vhost_user_set_inflight_fd, }; diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c index 569c4053ea..730f436692 100644 --- a/hw/virtio/vhost.c +++ b/hw/virtio/vhost.c @@ -1481,6 +1481,114 @@ void vhost_dev_set_config_notifier(struct vhost_dev *hdev, hdev->config_ops = ops; } +void vhost_dev_reset_inflight(struct vhost_inflight *inflight) +{ + if (inflight->addr) { + memset(inflight->addr, 0, inflight->size); + } +} + +void vhost_dev_free_inflight(struct vhost_inflight *inflight) +{ + if (inflight->addr) { + qemu_memfd_free(inflight->addr, inflight->size, inflight->fd); + inflight->addr = NULL; + inflight->fd = -1; + } +} + +static int vhost_dev_resize_inflight(struct vhost_inflight *inflight, + uint64_t new_size) +{ + Error *err = NULL; + int fd = -1; + void *addr = qemu_memfd_alloc("vhost-inflight", new_size, + F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL, + &fd, &err); + + if (err) { + error_report_err(err); + return -1; + } + + vhost_dev_free_inflight(inflight); + inflight->offset = 0; + inflight->addr = addr; + inflight->fd = fd; + inflight->size = new_size; + + return 0; +} + +void vhost_dev_save_inflight(struct vhost_inflight *inflight, QEMUFile *f) +{ + if (inflight->addr) { + qemu_put_be64(f, inflight->size); + qemu_put_be64(f, inflight->offset); + qemu_put_be32(f, inflight->align); + qemu_put_be16(f, inflight->version); + qemu_put_buffer(f, inflight->addr, inflight->size); + } else { + qemu_put_be64(f, 0); + } +} + +int vhost_dev_load_inflight(struct vhost_inflight *inflight, QEMUFile *f) +{ + uint64_t size; + + size = qemu_get_be64(f); + if (!size) { + return 0; + } + + if (inflight->size != size) { + if (vhost_dev_resize_inflight(inflight, size)) { + return -1; + } + } + inflight->size = size; + inflight->offset = qemu_get_be64(f); + inflight->align = qemu_get_be32(f); + inflight->version = qemu_get_be16(f); + + qemu_get_buffer(f, inflight->addr, size); + + return 0; +} + +int vhost_dev_set_inflight(struct vhost_dev *dev, + struct vhost_inflight *inflight) +{ + int r; + + if (dev->vhost_ops->vhost_set_inflight_fd && inflight->addr) { + r = dev->vhost_ops->vhost_set_inflight_fd(dev, inflight); + if (r) { + VHOST_OPS_DEBUG("vhost_set_inflight_fd failed"); + return -errno; + } + } + + return 0; +} + +int vhost_dev_get_inflight(struct vhost_dev *dev, + struct vhost_inflight *inflight) +{ + int r; + + if (dev->vhost_ops->vhost_get_inflight_fd) { + r = dev->vhost_ops->vhost_get_inflight_fd(dev, inflight); + if (r) { + VHOST_OPS_DEBUG("vhost_get_inflight_fd failed"); + return -errno; + } + } + + return 0; +} + /* Host notifiers must be enabled at this point. */ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev) { diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h index 81283ec50f..97676bd237 100644 --- a/include/hw/virtio/vhost-backend.h +++ b/include/hw/virtio/vhost-backend.h @@ -25,6 +25,7 @@ typedef enum VhostSetConfigType { VHOST_SET_CONFIG_TYPE_MIGRATION = 1, } VhostSetConfigType; +struct vhost_inflight; struct vhost_dev; struct vhost_log; struct vhost_memory; @@ -104,6 +105,12 @@ typedef int (*vhost_crypto_close_session_op)(struct vhost_dev *dev, typedef bool (*vhost_backend_mem_section_filter_op)(struct vhost_dev *dev, MemoryRegionSection *section); +typedef int (*vhost_get_inflight_fd_op)(struct vhost_dev *dev, + struct vhost_inflight *inflight); + +typedef int (*vhost_set_inflight_fd_op)(struct vhost_dev *dev, + struct vhost_inflight *inflight); + typedef struct VhostOps { VhostBackendType backend_type; vhost_backend_init vhost_backend_init; @@ -142,6 +149,8 @@ typedef struct VhostOps { vhost_crypto_create_session_op vhost_crypto_create_session; vhost_crypto_close_session_op vhost_crypto_close_session; vhost_backend_mem_section_filter_op vhost_backend_mem_section_filter; + vhost_get_inflight_fd_op vhost_get_inflight_fd; + vhost_set_inflight_fd_op vhost_set_inflight_fd; } VhostOps; extern const VhostOps user_ops; diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h index a7f449fa87..0a71596d8b 100644 --- a/include/hw/virtio/vhost.h +++ b/include/hw/virtio/vhost.h @@ -7,6 +7,16 @@ #include "exec/memory.h" /* Generic structures common for any vhost based device. */ + +struct vhost_inflight { + int fd; + void *addr; + uint64_t size; + uint64_t offset; + uint32_t align; + uint16_t version; +}; + struct vhost_virtqueue { int kick; int call; @@ -120,4 +130,13 @@ int vhost_dev_set_config(struct vhost_dev *dev, const uint8_t *data, */ void vhost_dev_set_config_notifier(struct vhost_dev *dev, const VhostDevConfigOps *ops); + +void vhost_dev_reset_inflight(struct vhost_inflight *inflight); +void vhost_dev_free_inflight(struct vhost_inflight *inflight); +void vhost_dev_save_inflight(struct vhost_inflight *inflight, QEMUFile *f); +int vhost_dev_load_inflight(struct vhost_inflight *inflight, QEMUFile *f); +int vhost_dev_set_inflight(struct vhost_dev *dev, + struct vhost_inflight *inflight); +int vhost_dev_get_inflight(struct vhost_dev *dev, + struct vhost_inflight *inflight); #endif From patchwork Wed Jan 9 11:27:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yongji Xie X-Patchwork-Id: 10753897 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 73EAA13BF for ; Wed, 9 Jan 2019 11:35:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E80428C29 for ; Wed, 9 Jan 2019 11:35:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4B57128E77; Wed, 9 Jan 2019 11:35:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id C100A28C29 for ; Wed, 9 Jan 2019 11:35:05 +0000 (UTC) Received: from localhost ([127.0.0.1]:36242 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghC8W-0006Oi-Ti for patchwork-qemu-devel@patchwork.kernel.org; Wed, 09 Jan 2019 06:35:04 -0500 Received: from eggs.gnu.org ([209.51.188.92]:51642) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghC1d-0000oq-F4 for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:27:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ghC1c-0004GH-Ht for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:27:57 -0500 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:38283) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ghC1c-0004G3-Bp for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:27:56 -0500 Received: by mail-pl1-x644.google.com with SMTP id e5so3456077plb.5 for ; Wed, 09 Jan 2019 03:27:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GKc3qU8HujCH0kBZ1CDODJ6Og5HhIYG0TFOCATmd5UE=; b=KZKdbeLzb6l4WNT/JpucEENAwGddj8vOpq4kmUKpYv6UDheOy/K9tpAm8ZLC6AsgkJ SxESxU4eZ7qwtvVBTrXrP2AygtEt6XOYOQnOKCsdoKnN2za0ndteaRqEQqVvQZ1+vR63 mbkPbZqz7nJV2ej1+SRiQ0ZYWRJfSxGLro8VSg6jsxsUTSbB6BM1C5jzPtMwVEkizR2y bj8ILeHfpj9OvwLXAk2AYzs8ApvyO99yv8UEEqGZNdS1ni2MvubXHq9XuFdBEarjqJ9U 9hF+k9yuGq0nMrgOI1BhnFN5YC0AMexorbGZk2LWtw8IkZY1zjGGo5V3PS+jG7+bcZzb mAZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GKc3qU8HujCH0kBZ1CDODJ6Og5HhIYG0TFOCATmd5UE=; b=Hr4OFW/hk6AJa+5rjdsNTc7NJkcuOdOnSnfe3p23KONKy9xm+nT6HYtsg6UDtR672z AcbkXdGHwuSHm4nWezifB+xTqIVpDkV81xsCySddLhpqy73VPhIDNVCSLJER+DlMk5lK rvJ88aSglotCiAE54z7+bg0bYhgwRcau6xFMAYLToeImNwoOs1QKLvoBBkz1rDh5os/W NgSnj4cxBiKV4BjNQz5xDOD3yUzoHAaf1CZ2Ut7kdMQREWfEf2i98C5yRZHYwAoOSW75 OkV+8uv5psCIJg+sJGQrJ9wAVqqMDrYxMdYs2aNtHeFZ7mRq9K9vXStq7ORB5Ri+wuhr ltkA== X-Gm-Message-State: AJcUukeUrfLN1pmRvgPufWG/k4jpMrONeoRxI/e/DmpjFKpJsFDWU1bi spuz3EAYoKzqN12oCKObmRA= X-Google-Smtp-Source: ALg8bN6cc3BX+rGFujaT/eSMFcuto7xPFzii822RYWL3y1z/zu/H/xElzweAHUvSx7WSggiedD3LDA== X-Received: by 2002:a17:902:66e6:: with SMTP id e93mr5526310plk.92.1547033275459; Wed, 09 Jan 2019 03:27:55 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id f67sm107909155pfc.141.2019.01.09.03.27.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 09 Jan 2019 03:27:55 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, marcandre.lureau@redhat.com, berrange@redhat.com, jasowang@redhat.com, maxime.coquelin@redhat.com, yury-kotov@yandex-team.ru, wrfsh@yandex-team.ru Date: Wed, 9 Jan 2019 19:27:24 +0800 Message-Id: <20190109112728.9214-4-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190109112728.9214-1-xieyongji@baidu.com> References: <20190109112728.9214-1-xieyongji@baidu.com> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::644 Subject: [Qemu-devel] [PATCH v4 for-4.0 3/7] libvhost-user: Introduce vu_queue_map_desc() X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Xie Yongji Introduce vu_queue_map_desc() which should be independent with vu_queue_pop(); Signed-off-by: Xie Yongji Signed-off-by: Zhang Yu Reviewed-by: Marc-André Lureau --- contrib/libvhost-user/libvhost-user.c | 88 ++++++++++++++++----------- 1 file changed, 51 insertions(+), 37 deletions(-) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/libvhost-user.c index a6b46cdc03..23bd52264c 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -1853,49 +1853,20 @@ virtqueue_alloc_element(size_t sz, return elem; } -void * -vu_queue_pop(VuDev *dev, VuVirtq *vq, size_t sz) +static void * +vu_queue_map_desc(VuDev *dev, VuVirtq *vq, unsigned int idx, size_t sz) { - unsigned int i, head, max, desc_len; + struct vring_desc *desc = vq->vring.desc; uint64_t desc_addr, read_len; + unsigned int desc_len; + unsigned int max = vq->vring.num; + unsigned int i = idx; VuVirtqElement *elem; - unsigned out_num, in_num; + unsigned int out_num = 0, in_num = 0; struct iovec iov[VIRTQUEUE_MAX_SIZE]; struct vring_desc desc_buf[VIRTQUEUE_MAX_SIZE]; - struct vring_desc *desc; int rc; - if (unlikely(dev->broken) || - unlikely(!vq->vring.avail)) { - return NULL; - } - - if (vu_queue_empty(dev, vq)) { - return NULL; - } - /* Needed after virtio_queue_empty(), see comment in - * virtqueue_num_heads(). */ - smp_rmb(); - - /* When we start there are none of either input nor output. */ - out_num = in_num = 0; - - max = vq->vring.num; - if (vq->inuse >= vq->vring.num) { - vu_panic(dev, "Virtqueue size exceeded"); - return NULL; - } - - if (!virtqueue_get_head(dev, vq, vq->last_avail_idx++, &head)) { - return NULL; - } - - if (vu_has_feature(dev, VIRTIO_RING_F_EVENT_IDX)) { - vring_set_avail_event(vq, vq->last_avail_idx); - } - - i = head; - desc = vq->vring.desc; if (desc[i].flags & VRING_DESC_F_INDIRECT) { if (desc[i].len % sizeof(struct vring_desc)) { vu_panic(dev, "Invalid size for indirect buffer table"); @@ -1947,12 +1918,13 @@ vu_queue_pop(VuDev *dev, VuVirtq *vq, size_t sz) } while (rc == VIRTQUEUE_READ_DESC_MORE); if (rc == VIRTQUEUE_READ_DESC_ERROR) { + vu_panic(dev, "read descriptor error"); return NULL; } /* Now copy what we have collected and mapped */ elem = virtqueue_alloc_element(sz, out_num, in_num); - elem->index = head; + elem->index = idx; for (i = 0; i < out_num; i++) { elem->out_sg[i] = iov[i]; } @@ -1960,6 +1932,48 @@ vu_queue_pop(VuDev *dev, VuVirtq *vq, size_t sz) elem->in_sg[i] = iov[out_num + i]; } + return elem; +} + +void * +vu_queue_pop(VuDev *dev, VuVirtq *vq, size_t sz) +{ + unsigned int head; + VuVirtqElement *elem; + + if (unlikely(dev->broken) || + unlikely(!vq->vring.avail)) { + return NULL; + } + + if (vu_queue_empty(dev, vq)) { + return NULL; + } + /* + * Needed after virtio_queue_empty(), see comment in + * virtqueue_num_heads(). + */ + smp_rmb(); + + if (vq->inuse >= vq->vring.num) { + vu_panic(dev, "Virtqueue size exceeded"); + return NULL; + } + + if (!virtqueue_get_head(dev, vq, vq->last_avail_idx++, &head)) { + return NULL; + } + + if (vu_has_feature(dev, VIRTIO_RING_F_EVENT_IDX)) { + vring_set_avail_event(vq, vq->last_avail_idx); + } + + elem = vu_queue_map_desc(dev, vq, head, sz); + + if (!elem) { + return NULL; + } + vq->inuse++; return elem; From patchwork Wed Jan 9 11:27:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yongji Xie X-Patchwork-Id: 10753899 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B00666C5 for ; Wed, 9 Jan 2019 11:37:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 98D8128F67 for ; Wed, 9 Jan 2019 11:37:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 89AC728F77; Wed, 9 Jan 2019 11:37:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9511628F67 for ; Wed, 9 Jan 2019 11:37:16 +0000 (UTC) Received: from localhost ([127.0.0.1]:36815 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghCAd-0007t6-Qu for patchwork-qemu-devel@patchwork.kernel.org; Wed, 09 Jan 2019 06:37:15 -0500 Received: from eggs.gnu.org ([209.51.188.92]:51653) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghC1h-0000rw-HU for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:28:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ghC1g-0004HB-2m for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:28:01 -0500 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:43776) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ghC1f-0004Gz-QA for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:28:00 -0500 Received: by mail-pl1-x644.google.com with SMTP id gn14so3449838plb.10 for ; Wed, 09 Jan 2019 03:27:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/sT7adtr6Zk4fs0cXVzE5IeuPhzEShk4fL7MtlRsPkE=; b=NLfJT9/7If50g8mybkQfaVkVEx/agecHWWyEap69ORM7rOmaT+zSH2V6zw4qaMntKi ya7ybGb1cS0bNga0cBZGepFKpOzaxYYzJF61ne+Nld4FsM8eOaiAVbkQ64pRS18iuqmP AjJDD6/8g/91Y44W6qr447xudQtPIwJ4XCGcNBfcwPXKtiFM1227riEEcZ1EI1ZP1WhH 3jeVFmjMbMGoXYLfzkJT4eoR/6LaOcLEXpZJY8pXhZ/lygXfqybPKjTFVfBoeWg2qGs9 tjHKP+1VyS4xq5aav1nOdghmMKtdpdUf+Vwumhse4Tp0bH3DHbm7FA9INTAisWPFF98H YGbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/sT7adtr6Zk4fs0cXVzE5IeuPhzEShk4fL7MtlRsPkE=; b=F6G2Mk0t6qfjEcAvl0Tsp5Z6fFx2vA2b/iHuSp/B36H6ejU+51kMMEfPOLg0imD+CK 0V3Z3YHi9HWkSv/Ddf4dXOgNJElD6JYtNRN0NrEjs/CNrEUVHqtf47Q4JUYsK3I/Od48 GC3yy1/f2kQamyJDiqYrMEGncwTYE4OtuQnPJn3E1PvxNTsLyz4paMlnHbEj5QLtY3ov c1lQ/PfDh0yILa3zYXPZzlaz2UccNNJ2/csGzJdbqkZRgo4c0liuu2ZEFkzGH/CqG/7h w8dcuSZdxOlcXLjCeNZUjpPDQaohAslhhFpoQsneGxbLdP+vCSoQnRPRYLWlkg37g/VB T2CQ== X-Gm-Message-State: AJcUukdq+17kf+69lYiDYUIIlA6KpF1RF66f3gNjQWvRVFMs47HsI1QF gqYqTCrIsw/6xgimBn2MOKc= X-Google-Smtp-Source: ALg8bN4UxRDP5aZ2zAucR2Qmip321prckveJIT/GXnk1yjUbBYAVnyRFt9AWv1pzeAhCfHpTOxrePQ== X-Received: by 2002:a17:902:8a95:: with SMTP id p21mr5772981plo.183.1547033278357; Wed, 09 Jan 2019 03:27:58 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id e23sm106149383pfh.68.2019.01.09.03.27.57 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 09 Jan 2019 03:27:57 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, marcandre.lureau@redhat.com, berrange@redhat.com, jasowang@redhat.com, maxime.coquelin@redhat.com, yury-kotov@yandex-team.ru, wrfsh@yandex-team.ru Date: Wed, 9 Jan 2019 19:27:25 +0800 Message-Id: <20190109112728.9214-5-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190109112728.9214-1-xieyongji@baidu.com> References: <20190109112728.9214-1-xieyongji@baidu.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::644 Subject: [Qemu-devel] [PATCH v4 for-4.0 4/7] libvhost-user: Support tracking inflight I/O in shared memory X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Xie Yongji This patch adds support for VHOST_USER_GET_INFLIGHT_FD and VHOST_USER_SET_INFLIGHT_FD message to set/get shared memory to/from qemu. Then we maintain a "bitmap" of all descriptors in the shared memory for each queue to track inflight I/O. Signed-off-by: Xie Yongji Signed-off-by: Zhang Yu --- Makefile | 2 +- contrib/libvhost-user/libvhost-user.c | 258 ++++++++++++++++++++++++-- contrib/libvhost-user/libvhost-user.h | 29 +++ 3 files changed, 268 insertions(+), 21 deletions(-) diff --git a/Makefile b/Makefile index dd53965f77..b5c9092605 100644 --- a/Makefile +++ b/Makefile @@ -473,7 +473,7 @@ Makefile: $(version-obj-y) # Build libraries libqemuutil.a: $(util-obj-y) $(trace-obj-y) $(stub-obj-y) -libvhost-user.a: $(libvhost-user-obj-y) +libvhost-user.a: $(libvhost-user-obj-y) $(util-obj-y) $(stub-obj-y) ###################################################################### diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/libvhost-user.c index 23bd52264c..e73ce04619 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -41,6 +41,8 @@ #endif #include "qemu/atomic.h" +#include "qemu/osdep.h" +#include "qemu/memfd.h" #include "libvhost-user.h" @@ -53,6 +55,18 @@ _min1 < _min2 ? _min1 : _min2; }) #endif +/* Round number down to multiple */ +#define ALIGN_DOWN(n, m) ((n) / (m) * (m)) + +/* Round number up to multiple */ +#define ALIGN_UP(n, m) ALIGN_DOWN((n) + (m) - 1, (m)) + +/* Align each region to cache line size in inflight buffer */ +#define INFLIGHT_ALIGNMENT 64 + +/* The version of inflight buffer */ +#define INFLIGHT_VERSION 1 + #define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64) /* The version of the protocol we support */ @@ -66,6 +80,20 @@ } \ } while (0) +static inline +bool has_feature(uint64_t features, unsigned int fbit) +{ + assert(fbit < 64); + return !!(features & (1ULL << fbit)); +} + +static inline +bool vu_has_feature(VuDev *dev, + unsigned int fbit) +{ + return has_feature(dev->features, fbit); +} + static const char * vu_request_to_string(unsigned int req) { @@ -100,6 +128,8 @@ vu_request_to_string(unsigned int req) REQ(VHOST_USER_POSTCOPY_ADVISE), REQ(VHOST_USER_POSTCOPY_LISTEN), REQ(VHOST_USER_POSTCOPY_END), + REQ(VHOST_USER_GET_INFLIGHT_FD), + REQ(VHOST_USER_SET_INFLIGHT_FD), REQ(VHOST_USER_MAX), }; #undef REQ @@ -890,6 +920,41 @@ vu_check_queue_msg_file(VuDev *dev, VhostUserMsg *vmsg) return true; } +static int +vu_check_queue_inflights(VuDev *dev, VuVirtq *vq) +{ + int i = 0; + + if (!has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) { + return 0; + } + + if (unlikely(!vq->inflight)) { + return -1; + } + + vq->used_idx = vq->vring.used->idx; + vq->inflight_num = 0; + for (i = 0; i < vq->vring.num; i++) { + if (vq->inflight->desc[i] == 0) { + continue; + } + + vq->inflight_desc[vq->inflight_num++] = i; + vq->inuse++; + } + vq->shadow_avail_idx = vq->last_avail_idx = vq->inuse + vq->used_idx; + + /* in case of I/O hang after reconnecting */ + if (eventfd_write(vq->kick_fd, 1) || + eventfd_write(vq->call_fd, 1)) { + return -1; + } + + return 0; +} + static bool vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) { @@ -925,6 +990,10 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) dev->vq[index].kick_fd, index); } + if (vu_check_queue_inflights(dev, &dev->vq[index])) { + vu_panic(dev, "Failed to check inflights for vq: %d\n", index); + } + return false; } @@ -1215,6 +1284,117 @@ vu_set_postcopy_end(VuDev *dev, VhostUserMsg *vmsg) return true; } +static bool +vu_get_inflight_fd(VuDev *dev, VhostUserMsg *vmsg) +{ + int fd; + void *addr; + uint64_t mmap_size; + + if (vmsg->size != sizeof(vmsg->payload.inflight)) { + vu_panic(dev, "Invalid get_inflight_fd message:%d", vmsg->size); + vmsg->payload.inflight.mmap_size = 0; + return true; + } + + DPRINT("set_inflight_fd num_queues: %"PRId16"\n", + vmsg->payload.inflight.num_queues); + + mmap_size = vmsg->payload.inflight.num_queues * + ALIGN_UP(sizeof(VuVirtqInflight), INFLIGHT_ALIGNMENT); + + addr = qemu_memfd_alloc("vhost-inflight", mmap_size, + F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL, + &fd, NULL); + + if (!addr) { + vu_panic(dev, "Failed to alloc vhost inflight area"); + vmsg->payload.inflight.mmap_size = 0; + return true; + } + + dev->inflight_info.addr = addr; + dev->inflight_info.size = vmsg->payload.inflight.mmap_size = mmap_size; + vmsg->payload.inflight.mmap_offset = 0; + vmsg->payload.inflight.align = INFLIGHT_ALIGNMENT; + vmsg->payload.inflight.version = INFLIGHT_VERSION; + vmsg->fd_num = 1; + dev->inflight_info.fd = vmsg->fds[0] = fd; + + DPRINT("send inflight mmap_size: %"PRId64"\n", + vmsg->payload.inflight.mmap_size); + DPRINT("send inflight mmap offset: %"PRId64"\n", + vmsg->payload.inflight.mmap_offset); + DPRINT("send inflight align: %"PRId32"\n", + vmsg->payload.inflight.align); + DPRINT("send inflight version: %"PRId16"\n", + vmsg->payload.inflight.version); + + return true; +} + +static bool +vu_set_inflight_fd(VuDev *dev, VhostUserMsg *vmsg) +{ + int fd, i; + uint64_t mmap_size, mmap_offset; + uint32_t align; + uint16_t num_queues, version; + void *rc; + + if (vmsg->fd_num != 1 || + vmsg->size != sizeof(vmsg->payload.inflight)) { + vu_panic(dev, "Invalid set_inflight_fd message size:%d fds:%d", + vmsg->size, vmsg->fd_num); + return false; + } + + fd = vmsg->fds[0]; + mmap_size = vmsg->payload.inflight.mmap_size; + mmap_offset = vmsg->payload.inflight.mmap_offset; + align = vmsg->payload.inflight.align; + num_queues = vmsg->payload.inflight.num_queues; + version = vmsg->payload.inflight.version; + + DPRINT("set_inflight_fd mmap_size: %"PRId64"\n", mmap_size); + DPRINT("set_inflight_fd mmap_offset: %"PRId64"\n", mmap_offset); + DPRINT("set_inflight_fd align: %"PRId32"\n", align); + DPRINT("set_inflight_fd num_queues: %"PRId16"\n", num_queues); + DPRINT("set_inflight_fd version: %"PRId16"\n", version); + + rc = mmap(0, mmap_size, PROT_READ | PROT_WRITE, MAP_SHARED, + fd, mmap_offset); + + if (rc == MAP_FAILED) { + vu_panic(dev, "set_inflight_fd mmap error: %s", strerror(errno)); + return false; + } + + if (version != INFLIGHT_VERSION) { + vu_panic(dev, "Invalid set_inflight_fd version: %d", version); + return false; + } + + if (dev->inflight_info.fd) { + close(dev->inflight_info.fd); + } + + if (dev->inflight_info.addr) { + munmap(dev->inflight_info.addr, dev->inflight_info.size); + } + + dev->inflight_info.fd = fd; + dev->inflight_info.addr = rc; + dev->inflight_info.size = mmap_size; + + for (i = 0; i < num_queues; i++) { + dev->vq[i].inflight = (VuVirtqInflight *)rc; + rc = (void *)((char *)rc + ALIGN_UP(sizeof(VuVirtqInflight), align)); + } + + return false; +} + static bool vu_process_message(VuDev *dev, VhostUserMsg *vmsg) { @@ -1292,6 +1472,10 @@ vu_process_message(VuDev *dev, VhostUserMsg *vmsg) return vu_set_postcopy_listen(dev, vmsg); case VHOST_USER_POSTCOPY_END: return vu_set_postcopy_end(dev, vmsg); + case VHOST_USER_GET_INFLIGHT_FD: + return vu_get_inflight_fd(dev, vmsg); + case VHOST_USER_SET_INFLIGHT_FD: + return vu_set_inflight_fd(dev, vmsg); default: vmsg_close_fds(vmsg); vu_panic(dev, "Unhandled request: %d", vmsg->request); @@ -1359,8 +1543,18 @@ vu_deinit(VuDev *dev) close(vq->err_fd); vq->err_fd = -1; } + vq->inflight = NULL; } + if (dev->inflight_info.addr) { + munmap(dev->inflight_info.addr, dev->inflight_info.size); + dev->inflight_info.addr = NULL; + } + + if (dev->inflight_info.fd > 0) { + close(dev->inflight_info.fd); + dev->inflight_info.fd = -1; + } vu_close_log(dev); if (dev->slave_fd != -1) { @@ -1687,20 +1881,6 @@ vu_queue_empty(VuDev *dev, VuVirtq *vq) return vring_avail_idx(vq) == vq->last_avail_idx; } -static inline -bool has_feature(uint64_t features, unsigned int fbit) -{ - assert(fbit < 64); - return !!(features & (1ULL << fbit)); -} - -static inline -bool vu_has_feature(VuDev *dev, - unsigned int fbit) -{ - return has_feature(dev->features, fbit); -} - static bool vring_notify(VuDev *dev, VuVirtq *vq) { @@ -1829,12 +2009,6 @@ virtqueue_map_desc(VuDev *dev, *p_num_sg = num_sg; } -/* Round number down to multiple */ -#define ALIGN_DOWN(n, m) ((n) / (m) * (m)) - -/* Round number up to multiple */ -#define ALIGN_UP(n, m) ALIGN_DOWN((n) + (m) - 1, (m)) - static void * virtqueue_alloc_element(size_t sz, unsigned out_num, unsigned in_num) @@ -1935,9 +2109,44 @@ vu_queue_map_desc(VuDev *dev, VuVirtq *vq, unsigned int idx, size_t sz) return elem; } +static int +vu_queue_inflight_get(VuDev *dev, VuVirtq *vq, int desc_idx) +{ + if (!has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) { + return 0; + } + + if (unlikely(!vq->inflight)) { + return -1; + } + + vq->inflight->desc[desc_idx] = 1; + + return 0; +} + +static int +vu_queue_inflight_put(VuDev *dev, VuVirtq *vq, int desc_idx) +{ + if (!has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) { + return 0; + } + + if (unlikely(!vq->inflight)) { + return -1; + } + + vq->inflight->desc[desc_idx] = 0; + + return 0; +} + void * vu_queue_pop(VuDev *dev, VuVirtq *vq, size_t sz) { + int i; unsigned int head; VuVirtqElement *elem; @@ -1946,6 +2155,12 @@ vu_queue_pop(VuDev *dev, VuVirtq *vq, size_t sz) return NULL; } + if (unlikely(vq->inflight_num > 0)) { + i = (--vq->inflight_num); + elem = vu_queue_map_desc(dev, vq, vq->inflight_desc[i], sz); + return elem; + } + if (vu_queue_empty(dev, vq)) { return NULL; } @@ -1976,6 +2191,8 @@ vu_queue_pop(VuDev *dev, VuVirtq *vq, size_t sz) vq->inuse++; + vu_queue_inflight_get(dev, vq, head); + return elem; } @@ -2121,4 +2338,5 @@ vu_queue_push(VuDev *dev, VuVirtq *vq, { vu_queue_fill(dev, vq, elem, len, 0); vu_queue_flush(dev, vq, 1); + vu_queue_inflight_put(dev, vq, elem->index); } diff --git a/contrib/libvhost-user/libvhost-user.h b/contrib/libvhost-user/libvhost-user.h index 4aa55b4d2d..5afb80ea5c 100644 --- a/contrib/libvhost-user/libvhost-user.h +++ b/contrib/libvhost-user/libvhost-user.h @@ -53,6 +53,7 @@ enum VhostUserProtocolFeature { VHOST_USER_PROTOCOL_F_CONFIG = 9, VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD = 10, VHOST_USER_PROTOCOL_F_HOST_NOTIFIER = 11, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD = 12, VHOST_USER_PROTOCOL_F_MAX }; @@ -91,6 +92,8 @@ typedef enum VhostUserRequest { VHOST_USER_POSTCOPY_ADVISE = 28, VHOST_USER_POSTCOPY_LISTEN = 29, VHOST_USER_POSTCOPY_END = 30, + VHOST_USER_GET_INFLIGHT_FD = 31, + VHOST_USER_SET_INFLIGHT_FD = 32, VHOST_USER_MAX } VhostUserRequest; @@ -138,6 +141,14 @@ typedef struct VhostUserVringArea { uint64_t offset; } VhostUserVringArea; +typedef struct VhostUserInflight { + uint64_t mmap_size; + uint64_t mmap_offset; + uint32_t align; + uint16_t num_queues; + uint16_t version; +} VhostUserInflight; + #if defined(_WIN32) # define VU_PACKED __attribute__((gcc_struct, packed)) #else @@ -163,6 +174,7 @@ typedef struct VhostUserMsg { VhostUserLog log; VhostUserConfig config; VhostUserVringArea area; + VhostUserInflight inflight; } payload; int fds[VHOST_MEMORY_MAX_NREGIONS]; @@ -234,9 +246,19 @@ typedef struct VuRing { uint32_t flags; } VuRing; +typedef struct VuVirtqInflight { + char desc[VIRTQUEUE_MAX_SIZE]; +} VuVirtqInflight; + typedef struct VuVirtq { VuRing vring; + VuVirtqInflight *inflight; + + uint16_t inflight_desc[VIRTQUEUE_MAX_SIZE]; + + uint16_t inflight_num; + /* Next head to pop */ uint16_t last_avail_idx; @@ -279,11 +301,18 @@ typedef void (*vu_set_watch_cb) (VuDev *dev, int fd, int condition, vu_watch_cb cb, void *data); typedef void (*vu_remove_watch_cb) (VuDev *dev, int fd); +typedef struct VuDevInflightInfo { + int fd; + void *addr; + uint64_t size; +} VuDevInflightInfo; + struct VuDev { int sock; uint32_t nregions; VuDevRegion regions[VHOST_MEMORY_MAX_NREGIONS]; VuVirtq vq[VHOST_MAX_NR_VIRTQUEUE]; + VuDevInflightInfo inflight_info; int log_call_fd; int slave_fd; uint64_t log_size; From patchwork Wed Jan 9 11:27:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yongji Xie X-Patchwork-Id: 10753887 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ED23A17FB for ; Wed, 9 Jan 2019 11:31:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DCD8B28F4B for ; Wed, 9 Jan 2019 11:31:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D17BF28F5C; Wed, 9 Jan 2019 11:31:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 11EBD28F5B for ; Wed, 9 Jan 2019 11:31:09 +0000 (UTC) Received: from localhost ([127.0.0.1]:35354 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghC4i-0003cY-AM for patchwork-qemu-devel@patchwork.kernel.org; Wed, 09 Jan 2019 06:31:08 -0500 Received: from eggs.gnu.org ([209.51.188.92]:51665) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghC1j-0000tD-9T for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:28:04 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ghC1i-0004J3-ET for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:28:03 -0500 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:44669) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ghC1i-0004Hf-7g for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:28:02 -0500 Received: by mail-pl1-x644.google.com with SMTP id e11so3453309plt.11 for ; Wed, 09 Jan 2019 03:28:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=R1acYzi45ES3MQAv7iC3m1nLc5Pfp8ailgufHkJhY+4=; b=aQ61dFPxHduDjrbqoAZ1znrhXtOW8a92LSaXIHcFA2VmrIO8ftWic/GAwFOf0f6xzg tMuW+ufj3Cl82dE44kzBm9qm2ZWgk66512CYaPOu1wYeEHcmPoJ+bv8lhmmtmo3x/Cem RQBvqMcIicHrF1pum1FykJ+mJZ0mCZ7617yO+2DzssoNBIyAm6wReXbholms/pJlXlO5 Vqq5+7tyq71/4BSN7JkWseiO3knQ7o9SyaTrNee1SOm4J1HiBsDlFLDl38mLPUPziPEV cPkqF5SqigmungX4LOPdC1JE1qBqnTEhfKZT/fF5oOTRlPcvmtfUxZ7bJOxVjSGOhtTP dnpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=R1acYzi45ES3MQAv7iC3m1nLc5Pfp8ailgufHkJhY+4=; b=E8RDmhxlDlDR+trf+ZgZacqBFdEFSKlWKibzm7wNNmyATFAF2AiNZhvL744gEDkwbm R0b/kxk8ASY0d602IajZo7HPiED0AGUObCcc8LivA/giOPtCSVVWdJRUVVOirB6tqPvc AN6uRlgamPG/yL6NubC5/c5Jfq6ZaCJxAvC5HLHPh5ORBwH82Djw6Nu59TFKhq8XzsAP wxHJDqWIzJR2fkLGKJeb0FYBjryzutD6wsge4rTEd7lpiwl9rl6UZaE4OZmVORulhnvC FvrzSN+4kwJPSxSQxeShu2YmGDqRJH/uGnDIJiNWeFvyJ1+GZsug8BTLZYJ+LuAqpjNJ 2AYw== X-Gm-Message-State: AJcUukeqkaopDkBGaG/GwR3/LjQ0PwQ0fVQVPEC93PZg4mdfT9DJgG4f 1XiD+xZUKaNqutboW5VIZ0A= X-Google-Smtp-Source: ALg8bN4akOKfFO6TgK+PHeB3uXwO6XCEYYN5UCluIlXzHapXpWzDchoTYtuwFWS10oCDxYjXo0+O/A== X-Received: by 2002:a17:902:3f81:: with SMTP id a1mr5596986pld.258.1547033281458; Wed, 09 Jan 2019 03:28:01 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id d18sm109326422pfj.47.2019.01.09.03.28.00 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 09 Jan 2019 03:28:01 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, marcandre.lureau@redhat.com, berrange@redhat.com, jasowang@redhat.com, maxime.coquelin@redhat.com, yury-kotov@yandex-team.ru, wrfsh@yandex-team.ru Date: Wed, 9 Jan 2019 19:27:26 +0800 Message-Id: <20190109112728.9214-6-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190109112728.9214-1-xieyongji@baidu.com> References: <20190109112728.9214-1-xieyongji@baidu.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::644 Subject: [Qemu-devel] [PATCH v4 for-4.0 5/7] vhost-user-blk: Add support to get/set inflight buffer X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Xie Yongji This patch adds support for vhost-user-blk device to get/set inflight buffer from/to backend. Signed-off-by: Xie Yongji Signed-off-by: Zhang Yu --- hw/block/vhost-user-blk.c | 26 ++++++++++++++++++++++++++ include/hw/virtio/vhost-user-blk.h | 1 + 2 files changed, 27 insertions(+) diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c index 1451940845..e1c48b938c 100644 --- a/hw/block/vhost-user-blk.c +++ b/hw/block/vhost-user-blk.c @@ -126,6 +126,13 @@ static void vhost_user_blk_start(VirtIODevice *vdev) } s->dev.acked_features = vdev->guest_features; + + ret = vhost_dev_set_inflight(&s->dev, s->inflight); + if (ret < 0) { + error_report("Error set inflight: %d", -ret); + goto err_guest_notifiers; + } + ret = vhost_dev_start(&s->dev, vdev); if (ret < 0) { error_report("Error starting vhost: %d", -ret); @@ -245,6 +252,13 @@ static void vhost_user_blk_handle_output(VirtIODevice *vdev, VirtQueue *vq) } } +static void vhost_user_blk_reset(VirtIODevice *vdev) +{ + VHostUserBlk *s = VHOST_USER_BLK(vdev); + + vhost_dev_reset_inflight(s->inflight); +} + static void vhost_user_blk_device_realize(DeviceState *dev, Error **errp) { VirtIODevice *vdev = VIRTIO_DEVICE(dev); @@ -284,6 +298,8 @@ static void vhost_user_blk_device_realize(DeviceState *dev, Error **errp) vhost_user_blk_handle_output); } + s->inflight = g_new0(struct vhost_inflight, 1); + s->dev.nvqs = s->num_queues; s->dev.vqs = g_new(struct vhost_virtqueue, s->dev.nvqs); s->dev.vq_index = 0; @@ -309,12 +325,19 @@ static void vhost_user_blk_device_realize(DeviceState *dev, Error **errp) s->blkcfg.num_queues = s->num_queues; } + ret = vhost_dev_get_inflight(&s->dev, s->inflight); + if (ret < 0) { + error_setg(errp, "vhost-user-blk: get inflight failed"); + goto vhost_err; + } + return; vhost_err: vhost_dev_cleanup(&s->dev); virtio_err: g_free(s->dev.vqs); + g_free(s->inflight); virtio_cleanup(vdev); vhost_user_cleanup(user); @@ -329,7 +352,9 @@ static void vhost_user_blk_device_unrealize(DeviceState *dev, Error **errp) vhost_user_blk_set_status(vdev, 0); vhost_dev_cleanup(&s->dev); + vhost_dev_free_inflight(s->inflight); g_free(s->dev.vqs); + g_free(s->inflight); virtio_cleanup(vdev); if (s->vhost_user) { @@ -379,6 +404,7 @@ static void vhost_user_blk_class_init(ObjectClass *klass, void *data) vdc->set_config = vhost_user_blk_set_config; vdc->get_features = vhost_user_blk_get_features; vdc->set_status = vhost_user_blk_set_status; + vdc->reset = vhost_user_blk_reset; } static const TypeInfo vhost_user_blk_info = { diff --git a/include/hw/virtio/vhost-user-blk.h b/include/hw/virtio/vhost-user-blk.h index d52944aeeb..445516604a 100644 --- a/include/hw/virtio/vhost-user-blk.h +++ b/include/hw/virtio/vhost-user-blk.h @@ -36,6 +36,7 @@ typedef struct VHostUserBlk { uint32_t queue_size; uint32_t config_wce; struct vhost_dev dev; + struct vhost_inflight *inflight; VhostUserState *vhost_user; } VHostUserBlk; From patchwork Wed Jan 9 11:27:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yongji Xie X-Patchwork-Id: 10753895 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 99FE514E5 for ; Wed, 9 Jan 2019 11:33:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 835E62891A for ; Wed, 9 Jan 2019 11:33:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 72837289B4; Wed, 9 Jan 2019 11:33:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A85412891A for ; Wed, 9 Jan 2019 11:33:55 +0000 (UTC) Received: from localhost ([127.0.0.1]:35988 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghC7O-0005Po-Hc for patchwork-qemu-devel@patchwork.kernel.org; Wed, 09 Jan 2019 06:33:54 -0500 Received: from eggs.gnu.org ([209.51.188.92]:51677) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghC1p-0000xZ-Vn for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:28:11 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ghC1o-0004KJ-K6 for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:28:09 -0500 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:38284) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ghC1o-0004Jz-Bi for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:28:08 -0500 Received: by mail-pl1-x643.google.com with SMTP id e5so3456270plb.5 for ; Wed, 09 Jan 2019 03:28:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=OZGV9ljlbx2vTQObiBxMQUJLeJK6pT30Z24CG6zI6zk=; b=LHKzUXpZYs6C1O8wyi5ELGALStIRoUV9wetZ8xFSxNyLrbB5d7pJUWUjc8iATSODA5 YNnbi4COEN8Cdj7BtJKV2TPFnL6pzVI+WVEIfKaN2kiYKPCK1GUwJthP1Qw4tMbUlH7A /wGOTGW6jfriVfxHjkly+Ti4kVeK+j5joaBcUGef6BTeDiYTb1hmVKagOunG3tws5bKx Vu6zLtROXiMfhXzPXiY9uDFuoxePW3D71T42vkg1Gv+gJhCja6/hVO8xJlnAZEfvMVCO 9Jx0uZe/6aNSb9CXYZLfEwOJZOE5JnecBVdGW99EQTw4UxLcgGUnkUasPX6SCd+ULDMT /s3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=OZGV9ljlbx2vTQObiBxMQUJLeJK6pT30Z24CG6zI6zk=; b=lb7hKWskvfX2LvxthnfoPwN4KruJq36OwjeW6T9aEFYm3kKkGHNevuL58J0l5KTjNs PNlX1XHy/i/B867A1niy67gaHbkEuBClGUvmp21ZN4hF51gWjEQFnO/lzPOz1vwiq/LV 7fvIeP4mdgxUo8cfnkYw457cUIdzMAkmdJ0z0E3rvknH13xzYPbvHEKLekYmUOddhwR2 AXt0aBEOuMW89hKstA6erQz1kKkWo9onLl3uK+a7nyg6t89n1Ia55/ZHAkeRKoJ1i8PS WZIOiB8S8T5Xov1mBPFji8EgMXpYE85MpMu28YRC0moYKWNYxj0GLlcqi3HwfZkHfNLf i/XQ== X-Gm-Message-State: AJcUukfFahdAC64RRD6kC0lSheBIWrUpRGOeEr+2c6K2BNyJqMn1qcNH R+JkCyCnI7psouD62Zhg8yY= X-Google-Smtp-Source: ALg8bN4X0cfyQN6KMg4ZQYKxJy9k5F6a6yY8tlHk896PKNtAnCUlOhMcGnOc7jkh8g5LzqBAsQlfEQ== X-Received: by 2002:a17:902:e085:: with SMTP id cb5mr5604224plb.24.1547033287368; Wed, 09 Jan 2019 03:28:07 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id m11sm97033422pgh.51.2019.01.09.03.28.06 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 09 Jan 2019 03:28:06 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, marcandre.lureau@redhat.com, berrange@redhat.com, jasowang@redhat.com, maxime.coquelin@redhat.com, yury-kotov@yandex-team.ru, wrfsh@yandex-team.ru Date: Wed, 9 Jan 2019 19:27:27 +0800 Message-Id: <20190109112728.9214-7-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190109112728.9214-1-xieyongji@baidu.com> References: <20190109112728.9214-1-xieyongji@baidu.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::643 Subject: [Qemu-devel] [PATCH v4 for-4.0 6/7] vhost-user-blk: Add support to reconnect backend X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Xie Yongji Since we now support the message VHOST_USER_GET_INFLIGHT_FD and VHOST_USER_SET_INFLIGHT_FD. The backend is able to restart safely because it can track inflight I/O in shared memory. This patch allows qemu to reconnect the backend after connection closed. Signed-off-by: Xie Yongji Signed-off-by: Ni Xun Signed-off-by: Zhang Yu --- hw/block/vhost-user-blk.c | 205 +++++++++++++++++++++++------ include/hw/virtio/vhost-user-blk.h | 4 + 2 files changed, 168 insertions(+), 41 deletions(-) diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c index e1c48b938c..a551486151 100644 --- a/hw/block/vhost-user-blk.c +++ b/hw/block/vhost-user-blk.c @@ -101,7 +101,7 @@ const VhostDevConfigOps blk_ops = { .vhost_dev_config_notifier = vhost_user_blk_handle_config_change, }; -static void vhost_user_blk_start(VirtIODevice *vdev) +static int vhost_user_blk_start(VirtIODevice *vdev) { VHostUserBlk *s = VHOST_USER_BLK(vdev); BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(vdev))); @@ -110,13 +110,13 @@ static void vhost_user_blk_start(VirtIODevice *vdev) if (!k->set_guest_notifiers) { error_report("binding does not support guest notifiers"); - return; + return -ENOSYS; } ret = vhost_dev_enable_notifiers(&s->dev, vdev); if (ret < 0) { error_report("Error enabling host notifiers: %d", -ret); - return; + return ret; } ret = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, true); @@ -147,12 +147,13 @@ static void vhost_user_blk_start(VirtIODevice *vdev) vhost_virtqueue_mask(&s->dev, vdev, i, false); } - return; + return ret; err_guest_notifiers: k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false); err_host_notifiers: vhost_dev_disable_notifiers(&s->dev, vdev); + return ret; } static void vhost_user_blk_stop(VirtIODevice *vdev) @@ -171,7 +172,6 @@ static void vhost_user_blk_stop(VirtIODevice *vdev) ret = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false); if (ret < 0) { error_report("vhost guest notifier cleanup failed: %d", ret); - return; } vhost_dev_disable_notifiers(&s->dev, vdev); @@ -181,21 +181,43 @@ static void vhost_user_blk_set_status(VirtIODevice *vdev, uint8_t status) { VHostUserBlk *s = VHOST_USER_BLK(vdev); bool should_start = status & VIRTIO_CONFIG_S_DRIVER_OK; + int ret; if (!vdev->vm_running) { should_start = false; } - if (s->dev.started == should_start) { + if (s->should_start == should_start) { + return; + } + + if (!s->connected || s->dev.started == should_start) { + s->should_start = should_start; return; } if (should_start) { - vhost_user_blk_start(vdev); + s->should_start = true; + /* + * make sure vhost_user_blk_handle_output() ignores fake + * guest kick by vhost_dev_enable_notifiers() + */ + barrier(); + ret = vhost_user_blk_start(vdev); + if (ret < 0) { + error_report("vhost-user-blk: vhost start failed: %s", + strerror(-ret)); + qemu_chr_fe_disconnect(&s->chardev); + } } else { vhost_user_blk_stop(vdev); + /* + * make sure vhost_user_blk_handle_output() ignore fake + * guest kick by vhost_dev_disable_notifiers() + */ + barrier(); + s->should_start = false; } - } static uint64_t vhost_user_blk_get_features(VirtIODevice *vdev, @@ -225,13 +247,22 @@ static uint64_t vhost_user_blk_get_features(VirtIODevice *vdev, static void vhost_user_blk_handle_output(VirtIODevice *vdev, VirtQueue *vq) { VHostUserBlk *s = VHOST_USER_BLK(vdev); - int i; + int i, ret; if (!(virtio_host_has_feature(vdev, VIRTIO_F_VERSION_1) && !virtio_vdev_has_feature(vdev, VIRTIO_F_VERSION_1))) { return; } + if (s->should_start) { + return; + } + s->should_start = true; + + if (!s->connected) { + return; + } + if (s->dev.started) { return; } @@ -239,7 +270,13 @@ static void vhost_user_blk_handle_output(VirtIODevice *vdev, VirtQueue *vq) /* Some guests kick before setting VIRTIO_CONFIG_S_DRIVER_OK so start * vhost here instead of waiting for .set_status(). */ - vhost_user_blk_start(vdev); + ret = vhost_user_blk_start(vdev); + if (ret < 0) { + error_report("vhost-user-blk: vhost start failed: %s", + strerror(-ret)); + qemu_chr_fe_disconnect(&s->chardev); + return; + } /* Kick right away to begin processing requests already in vring */ for (i = 0; i < s->dev.nvqs; i++) { @@ -259,12 +296,105 @@ static void vhost_user_blk_reset(VirtIODevice *vdev) vhost_dev_reset_inflight(s->inflight); } +static int vhost_user_blk_connect(DeviceState *dev) +{ + VirtIODevice *vdev = VIRTIO_DEVICE(dev); + VHostUserBlk *s = VHOST_USER_BLK(vdev); + int ret = 0; + + if (s->connected) { + return 0; + } + s->connected = true; + + s->dev.nvqs = s->num_queues; + s->dev.vqs = s->vqs; + s->dev.vq_index = 0; + s->dev.backend_features = 0; + + vhost_dev_set_config_notifier(&s->dev, &blk_ops); + + ret = vhost_dev_init(&s->dev, s->vhost_user, VHOST_BACKEND_TYPE_USER, 0); + if (ret < 0) { + error_report("vhost-user-blk: vhost initialization failed: %s", + strerror(-ret)); + return ret; + } + + /* restore vhost state */ + if (s->should_start) { + ret = vhost_user_blk_start(vdev); + if (ret < 0) { + error_report("vhost-user-blk: vhost start failed: %s", + strerror(-ret)); + return ret; + } + } + + return 0; +} + +static void vhost_user_blk_disconnect(DeviceState *dev) +{ + VirtIODevice *vdev = VIRTIO_DEVICE(dev); + VHostUserBlk *s = VHOST_USER_BLK(vdev); + + if (!s->connected) { + return; + } + s->connected = false; + + if (s->dev.started) { + vhost_user_blk_stop(vdev); + } + + vhost_dev_cleanup(&s->dev); +} + +static gboolean vhost_user_blk_watch(GIOChannel *chan, GIOCondition cond, + void *opaque) +{ + DeviceState *dev = opaque; + VirtIODevice *vdev = VIRTIO_DEVICE(dev); + VHostUserBlk *s = VHOST_USER_BLK(vdev); + + qemu_chr_fe_disconnect(&s->chardev); + + return true; +} + +static void vhost_user_blk_event(void *opaque, int event) +{ + DeviceState *dev = opaque; + VirtIODevice *vdev = VIRTIO_DEVICE(dev); + VHostUserBlk *s = VHOST_USER_BLK(vdev); + + switch (event) { + case CHR_EVENT_OPENED: + if (vhost_user_blk_connect(dev) < 0) { + qemu_chr_fe_disconnect(&s->chardev); + return; + } + s->watch = qemu_chr_fe_add_watch(&s->chardev, G_IO_HUP, + vhost_user_blk_watch, dev); + break; + case CHR_EVENT_CLOSED: + vhost_user_blk_disconnect(dev); + if (s->watch) { + g_source_remove(s->watch); + s->watch = 0; + } + break; + } +} + static void vhost_user_blk_device_realize(DeviceState *dev, Error **errp) { VirtIODevice *vdev = VIRTIO_DEVICE(dev); VHostUserBlk *s = VHOST_USER_BLK(vdev); VhostUserState *user; int i, ret; + Error *err = NULL; if (!s->chardev.chr) { error_setg(errp, "vhost-user-blk: chardev is mandatory"); @@ -299,26 +429,28 @@ static void vhost_user_blk_device_realize(DeviceState *dev, Error **errp) } s->inflight = g_new0(struct vhost_inflight, 1); - - s->dev.nvqs = s->num_queues; - s->dev.vqs = g_new(struct vhost_virtqueue, s->dev.nvqs); - s->dev.vq_index = 0; - s->dev.backend_features = 0; - - vhost_dev_set_config_notifier(&s->dev, &blk_ops); - - ret = vhost_dev_init(&s->dev, s->vhost_user, VHOST_BACKEND_TYPE_USER, 0); - if (ret < 0) { - error_setg(errp, "vhost-user-blk: vhost initialization failed: %s", - strerror(-ret)); - goto virtio_err; - } + s->vqs = g_new(struct vhost_virtqueue, s->num_queues); + s->watch = 0; + s->should_start = false; + s->connected = false; + + qemu_chr_fe_set_handlers(&s->chardev, NULL, NULL, vhost_user_blk_event, + NULL, (void *)dev, NULL, true); + +reconnect: + do { + if (qemu_chr_fe_wait_connected(&s->chardev, &err) < 0) { + error_report_err(err); + err = NULL; + sleep(1); + } + } while (!s->connected); ret = vhost_dev_get_config(&s->dev, (uint8_t *)&s->blkcfg, - sizeof(struct virtio_blk_config)); + sizeof(struct virtio_blk_config)); if (ret < 0) { - error_setg(errp, "vhost-user-blk: get block config failed"); - goto vhost_err; + error_report("vhost-user-blk: get block config failed"); + goto reconnect; } if (s->blkcfg.num_queues != s->num_queues) { @@ -327,22 +459,11 @@ static void vhost_user_blk_device_realize(DeviceState *dev, Error **errp) ret = vhost_dev_get_inflight(&s->dev, s->inflight); if (ret < 0) { - error_setg(errp, "vhost-user-blk: get inflight failed"); - goto vhost_err; + error_report("vhost-user-blk: get inflight failed"); + goto reconnect; } return; - -vhost_err: - vhost_dev_cleanup(&s->dev); -virtio_err: - g_free(s->dev.vqs); - g_free(s->inflight); - virtio_cleanup(vdev); - - vhost_user_cleanup(user); - g_free(user); - s->vhost_user = NULL; } static void vhost_user_blk_device_unrealize(DeviceState *dev, Error **errp) @@ -351,9 +472,11 @@ static void vhost_user_blk_device_unrealize(DeviceState *dev, Error **errp) VHostUserBlk *s = VHOST_USER_BLK(dev); vhost_user_blk_set_status(vdev, 0); + qemu_chr_fe_set_handlers(&s->chardev, NULL, NULL, NULL, + NULL, NULL, NULL, false); vhost_dev_cleanup(&s->dev); vhost_dev_free_inflight(s->inflight); - g_free(s->dev.vqs); + g_free(s->vqs); g_free(s->inflight); virtio_cleanup(vdev); diff --git a/include/hw/virtio/vhost-user-blk.h b/include/hw/virtio/vhost-user-blk.h index 445516604a..4849aa5eb5 100644 --- a/include/hw/virtio/vhost-user-blk.h +++ b/include/hw/virtio/vhost-user-blk.h @@ -38,6 +38,10 @@ typedef struct VHostUserBlk { struct vhost_dev dev; struct vhost_inflight *inflight; VhostUserState *vhost_user; + struct vhost_virtqueue *vqs; + guint watch; + bool should_start; + bool connected; } VHostUserBlk; #endif From patchwork Wed Jan 9 11:27:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yongji Xie X-Patchwork-Id: 10753891 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 83AE86C5 for ; Wed, 9 Jan 2019 11:31:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 73B0F28F5B for ; Wed, 9 Jan 2019 11:31:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 67D0B28F5C; Wed, 9 Jan 2019 11:31:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 219ED28F4B for ; Wed, 9 Jan 2019 11:31:33 +0000 (UTC) Received: from localhost ([127.0.0.1]:35436 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghC56-0003sd-F2 for patchwork-qemu-devel@patchwork.kernel.org; Wed, 09 Jan 2019 06:31:32 -0500 Received: from eggs.gnu.org ([209.51.188.92]:51707) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ghC1s-0000zg-N5 for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:28:13 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ghC1s-0004Lf-2c for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:28:12 -0500 Received: from mail-pl1-x634.google.com ([2607:f8b0:4864:20::634]:36978) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ghC1r-0004LB-Sq for qemu-devel@nongnu.org; Wed, 09 Jan 2019 06:28:12 -0500 Received: by mail-pl1-x634.google.com with SMTP id b5so3464620plr.4 for ; Wed, 09 Jan 2019 03:28:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uO8ERLhA7hT70caOpcFx0tbdhBCcwIuTDUTqrC/0Y5Y=; b=umunUNOFNuUHFUETbyHksVOrK+iIgAu9/AFsdr8Gw+Fu+sLipnrseMSJEGzputkzuF surw3aPTZY5Lg9aShYCB2OGKa7uN1wsi0nlLYcXLVPtv8hw34fqGsuTuII20x+0HZ7sB 2dwM+zP3G/CrErU3yXSOro1IE3l4W8gFQpCqauOb4ZXnZLC7r5DKQIXP7lMGpzw0/O7/ VWegbBXeFITLi+7YhxJmqP0O34gsdyjjLAfIVfxQLY4ODoyTEhf82UpBb7lJ+4HAFQNA UZd+76m62/OHh+NoJVUTBF7h27x5MSEgElgcgcBf1zxCdZPlbeHOjLNfofg6wxW8K2Z4 c75w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uO8ERLhA7hT70caOpcFx0tbdhBCcwIuTDUTqrC/0Y5Y=; b=QvSucm8SjWxjCeuvj7bk8LiNFr3lZi/ZzpQmkVkLkMZrmdZ0247+UoTtSKmRPwTw1a fAYUZg4HCDKS/gF2+R/xYazfnmvCM3DDIZubqXRDTpdU5ehV6NYU/8DRu06A1WRqhFr/ jG/404irWqWnrgacNroDZPPvEs8dur4BPc+BJmMehRAlE8TYdWM835m6kHs2JeEufD+P 2mR33qTpKOHOGlLZPZXDqZFoAiIaMSsbeHCaoTNX7yefjkVtcxeXHsYE0Gn6lUfpN22N uizfizBekZYQy1DkvA5EDbTEas896NTlqKcVPWLdE3q1Cye0SL5iIXKMnjbObFP2Ox8h qzKw== X-Gm-Message-State: AJcUukdT+LqOfyPS9OxpaHwa8sv3KZPeuRRzUWZSIq4nxsyCIu5ZghgL K3W9uYY9HNE43wnZekekMhI= X-Google-Smtp-Source: ALg8bN5B1hLuVRLvnvwF0LDM1g6t2PZuT2kjPypjhKMsUAn1VZ4Zvsg/utV1hxAZja1A/L1lJf27vQ== X-Received: by 2002:a17:902:784d:: with SMTP id e13mr5749217pln.188.1547033290963; Wed, 09 Jan 2019 03:28:10 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id d25sm118663565pfe.40.2019.01.09.03.28.10 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 09 Jan 2019 03:28:10 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, marcandre.lureau@redhat.com, berrange@redhat.com, jasowang@redhat.com, maxime.coquelin@redhat.com, yury-kotov@yandex-team.ru, wrfsh@yandex-team.ru Date: Wed, 9 Jan 2019 19:27:28 +0800 Message-Id: <20190109112728.9214-8-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190109112728.9214-1-xieyongji@baidu.com> References: <20190109112728.9214-1-xieyongji@baidu.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::634 Subject: [Qemu-devel] [PATCH v4 for-4.0 7/7] contrib/vhost-user-blk: enable inflight I/O tracking X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Xie Yongji This patch enables inflight I/O tracking for vhost-user-blk backend so that we could restart it safely. Signed-off-by: Xie Yongji Signed-off-by: Zhang Yu --- contrib/vhost-user-blk/vhost-user-blk.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/contrib/vhost-user-blk/vhost-user-blk.c b/contrib/vhost-user-blk/vhost-user-blk.c index 858221ad95..8cc033946a 100644 --- a/contrib/vhost-user-blk/vhost-user-blk.c +++ b/contrib/vhost-user-blk/vhost-user-blk.c @@ -327,7 +327,8 @@ vub_get_features(VuDev *dev) static uint64_t vub_get_protocol_features(VuDev *dev) { - return 1ull << VHOST_USER_PROTOCOL_F_CONFIG; + return 1ull << VHOST_USER_PROTOCOL_F_CONFIG | + 1ull << VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD; } static int