From patchwork Thu Aug 16 15:32:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yury Kotov X-Patchwork-Id: 10567733 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0AA2C5A4 for ; Thu, 16 Aug 2018 16:00:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EBDCF2B4E7 for ; Thu, 16 Aug 2018 16:00:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DD5BC2B4EB; Thu, 16 Aug 2018 16:00:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2A81B2B4E7 for ; Thu, 16 Aug 2018 16:00:23 +0000 (UTC) Received: from localhost ([::1]:56545 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fqKhC-0000PN-CR for patchwork-qemu-devel@patchwork.kernel.org; Thu, 16 Aug 2018 12:00:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37999) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fqKGg-00044z-TB for qemu-devel@nongnu.org; Thu, 16 Aug 2018 11:32:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fqKGf-0002nZ-86 for qemu-devel@nongnu.org; Thu, 16 Aug 2018 11:32:58 -0400 Received: from forwardcorp1g.cmail.yandex.net ([2a02:6b8:0:1465::fd]:39407) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fqKGe-0002mG-Nh for qemu-devel@nongnu.org; Thu, 16 Aug 2018 11:32:57 -0400 Received: from mxbackcorp1j.mail.yandex.net (mxbackcorp1j.mail.yandex.net [IPv6:2a02:6b8:0:1619::162]) by forwardcorp1g.cmail.yandex.net (Yandex) with ESMTP id 4A65520B5C; Thu, 16 Aug 2018 18:32:53 +0300 (MSK) Received: from smtpcorp1p.mail.yandex.net (smtpcorp1p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:10]) by mxbackcorp1j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id lzrWao90gV-WrPKhPtD; Thu, 16 Aug 2018 18:32:53 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1534433573; bh=s7Ta67frXAHDa+eUq8IZbwTD5SLzoVQz8VhJdUiFr0c=; h=From:To:Cc:Subject:Date:Message-Id; b=tm6JppZZ4Czq4WelZXYn47QzpQwgA3EzBDX744puFRCkku9zAvkCdO93arUYLhuvg 3QDdH439bKVAccEaarLjhjWucGNGegRpl8SHW6I0RSmcnqMRW+sji4P3QBSxT3dtdt rUN7T4UdFMpRXOC6i0fOSsc5V1XU+p01W2zICb5s= Received: from dynamic-red.dhcp.yndx.net (dynamic-red.dhcp.yndx.net [2a02:6b8:0:40c:e1bb:a1a7:a235:d6b4]) by smtpcorp1p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id 7WDIxKFPmc-WqXSmq0D; Thu, 16 Aug 2018 18:32:53 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1534433573; bh=s7Ta67frXAHDa+eUq8IZbwTD5SLzoVQz8VhJdUiFr0c=; h=From:To:Cc:Subject:Date:Message-Id; b=tm6JppZZ4Czq4WelZXYn47QzpQwgA3EzBDX744puFRCkku9zAvkCdO93arUYLhuvg 3QDdH439bKVAccEaarLjhjWucGNGegRpl8SHW6I0RSmcnqMRW+sji4P3QBSxT3dtdt rUN7T4UdFMpRXOC6i0fOSsc5V1XU+p01W2zICb5s= Authentication-Results: smtpcorp1p.mail.yandex.net; dkim=pass header.i=@yandex-team.ru From: Yury Kotov To: qemu-devel@nongnu.org Date: Thu, 16 Aug 2018 18:32:40 +0300 Message-Id: <1534433563-30865-1-git-send-email-yury-kotov@yandex-team.ru> X-Mailer: git-send-email 2.7.4 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a02:6b8:0:1465::fd Subject: [Qemu-devel] [PATCH 0/3] vhost-user reconnect X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Paolo Bonzini , Evgeny Yakovlev , "Michael S. Tsirkin" Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP We are using QEMU (2.12.0) with SPDK (18.04.1) over vhost-user to emulate block devices. One of our cases it to restart SPDK without restarting VM (in case of some updates or smth like it). We tried to use the 'reconnect' option for the '-chardev' device: -object memory-backend-file,id=mem0,size=1G,mem-path=/dev/hugepages,share=on \ -numa node,memdev=mem0 \ -chardev socket,id=spdk_vhost_blk1,path=/var/tmp/vhost.1,reconnect=10 \ -device vhost-user-blk-pci,chardev=spdk_vhost_blk1,num-queues=4 After this, vhost-user-blk initialization fails with an error below: qemu-system-x86_64: -device ...: Failed to set msg fds. qemu-system-x86_64: -device ...: vhost-user-blk: vhost initialization failed: Operation not permitted We got the same error with the latest QEMU (c542a9f9794ec8e0bc3f). We made some investigations and found out that there are several issues: 1. Reconnect option postpones the first connection till machine init done event. But we need this connection during vhost blk device initialization which happens before the machine init done handling. 2. If the connection is forced, then the reconnection will be successful after SPDK restart. The problem is that virtual queue will not start. The reason for it is that virtual queue initialization commands should be resent: * VHOST_USER_SET_FEATURES * VHOST_USER_SET_MEM_TABLE * VHOST_USER_SET_VRING_NUM * VHOST_USER_SET_VRING_BASE * VHOST_USER_SET_VRING_ADDR * VHOST_USER_SET_VRING_KICK * VHOST_USER_SET_VRING_CALL The patch set resolves both of these issues. Test case: 1. Start fio process (inside VM): fio --name test --ioengine=libaio --iodepth=64 --bs=4096 \ --rw=randrw --direct=1 --sync=1 --verify=md5 \ --size=64M --filename=/dev/vda --loops=100 2. Restart SPDK many times. We are expecting that during SPDK restart fio will pause and fio should continue to work after restart completion. 3. fio process completed successfully without any error. Yury Kotov (3): chardev: prevent extra connection attempt in tcp_chr_machine_done_hook vhost: refactor vhost_dev_start and vhost_virtqueue_start vhost-user: add reconnect support for vhost-user chardev/char-socket.c | 5 +- hw/virtio/vhost-user.c | 65 ++++++++++++-- hw/virtio/vhost.c | 223 +++++++++++++++++++++++++++++++--------------- include/hw/virtio/vhost.h | 2 + 4 files changed, 215 insertions(+), 80 deletions(-) Reviewed-by: Marc-André Lureau