From patchwork Wed Oct 18 15:59:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Michael S. Tsirkin" X-Patchwork-Id: 13427368 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E6E96CDB47E for ; Wed, 18 Oct 2023 16:13:49 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qt8x6-0001gO-5f; Wed, 18 Oct 2023 11:59:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qt8wz-0001Tk-0n for qemu-devel@nongnu.org; Wed, 18 Oct 2023 11:59:13 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qt8ww-0007gL-LF for qemu-devel@nongnu.org; Wed, 18 Oct 2023 11:59:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1697644749; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=IfCJRn8r77VYMrjpWnu9brJ2+zJiZU2dGmOU5Pe2LJc=; b=EjNcaQivuRqs6r+YRdE/1LeiJSgq4vhdw/oSUn/6ufgvVy2RGtLuhujc5rBPN4lvIs1ur3 o3rZBlCHkD6AfyQPlVMniI2rkj7G6QjEQ1eNioITnF/SOpNhpJIlhQu8Dz+C3CC/1ZkKck Nn0po3fhoMiHCz8NTr0Zi1rSJ7VQk3k= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-17--hv9CnSwMXepCrYCJpRtZw-1; Wed, 18 Oct 2023 11:59:08 -0400 X-MC-Unique: -hv9CnSwMXepCrYCJpRtZw-1 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-40570ccc497so54289325e9.3 for ; Wed, 18 Oct 2023 08:59:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697644746; x=1698249546; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=IfCJRn8r77VYMrjpWnu9brJ2+zJiZU2dGmOU5Pe2LJc=; b=MZ28/J8iE2L0Kx2NKn1IhtsMB1m9mzImtCVH4IE+u9ZTrYAhAiL4QpIYbG7vjZ+DmP K1Kf8a7B4ZPnab4vaYjMg5KBA25YStVhUxbWbCe5t0/JJzBKQOgPHOSVBJQm/NP1e0ex 4IOI9Ep4UYbCJM+Zo6ZAFYtC2atJ+XIGTuAnEbl5SBV6zOHLULvLEFacT6QsU9jQuJiM SZ1kPjiz4d0SqDITjtVIC0B6ELPd7zMfArbzD4oCCSvwOsLhQEtI22HtysoT68tQlV0G eLcM4rP+LRPKSj7A5unMWbE6JMYdLL27nxw193pxTXCBXz3gmgjZXuqZqZTB/yXBp4ph 6TeA== X-Gm-Message-State: AOJu0YyVrGXeyQ+MY5Rz31zSZ277WehF/MHezwIa5cAPmhc5h9RNUzaN BYw2zORCHVBHM2xwG3TX5Btnfw+OuEx7OzgQ+Zkqn0/ipkQd8gAohFXI17RJzYBWLzi5NBs7M+Q AtZ3Ir88jwiwKDnsjGjfQS4Voq1hhaek8fYnojIWH1Faw0aftv0+pYgL5ofkB/pGHyVdjRyI= X-Received: by 2002:a05:600c:3ca4:b0:406:592b:e5aa with SMTP id bg36-20020a05600c3ca400b00406592be5aamr4413976wmb.14.1697644746285; Wed, 18 Oct 2023 08:59:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF5lg2V/f6ENa9FGUyk7IZfHf/lNifQieHkJzJKBeFQgseMrr5PZ1Fuz5RTNKlsp1M2ErBWsQ== X-Received: by 2002:a05:600c:3ca4:b0:406:592b:e5aa with SMTP id bg36-20020a05600c3ca400b00406592be5aamr4413952wmb.14.1697644745836; Wed, 18 Oct 2023 08:59:05 -0700 (PDT) Received: from redhat.com ([2a02:14f:1f2:2037:f34:d61b:7da0:a7be]) by smtp.gmail.com with ESMTPSA id x9-20020a05600c420900b004065e235417sm1981386wmh.21.2023.10.18.08.59.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 08:59:05 -0700 (PDT) Date: Wed, 18 Oct 2023 11:59:01 -0400 From: "Michael S. Tsirkin" To: qemu-devel@nongnu.org Cc: Peter Maydell , Li Feng , Manos Pitsidianakis , Paolo Bonzini , Fam Zheng , Raphael Norwitz Subject: [PULL 74/83] vhost-user-scsi: support reconnect to backend Message-ID: <1e78b466dd62c08d8b027119eff37d7d09cd9aa0.1697644299.git.mst@redhat.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Mailer: git-send-email 2.27.0.106.g8ac3dc51b1 X-Mutt-Fcc: =sent Received-SPF: pass client-ip=170.10.129.124; envelope-from=mst@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Li Feng If the backend crashes and restarts, the device is broken. This patch adds reconnect for vhost-user-scsi. This patch also improves the error messages, and reports some silent errors. Tested with spdk backend. Signed-off-by: Li Feng Message-Id: <20231009044735.941655-4-fengli@smartx.com> Reviewed-by: Michael S. Tsirkin Signed-off-by: Michael S. Tsirkin Reviewed-by: Manos Pitsidianakis --- include/hw/virtio/vhost-scsi-common.h | 2 +- include/hw/virtio/vhost-user-scsi.h | 6 + hw/scsi/vhost-scsi-common.c | 16 +- hw/scsi/vhost-scsi.c | 6 +- hw/scsi/vhost-user-scsi.c | 201 +++++++++++++++++++++++--- 5 files changed, 201 insertions(+), 30 deletions(-) diff --git a/include/hw/virtio/vhost-scsi-common.h b/include/hw/virtio/vhost-scsi-common.h index 18f115527c..c5d2c09455 100644 --- a/include/hw/virtio/vhost-scsi-common.h +++ b/include/hw/virtio/vhost-scsi-common.h @@ -39,7 +39,7 @@ struct VHostSCSICommon { struct vhost_inflight *inflight; }; -int vhost_scsi_common_start(VHostSCSICommon *vsc); +int vhost_scsi_common_start(VHostSCSICommon *vsc, Error **errp); void vhost_scsi_common_stop(VHostSCSICommon *vsc); char *vhost_scsi_common_get_fw_dev_path(FWPathProvider *p, BusState *bus, DeviceState *dev); diff --git a/include/hw/virtio/vhost-user-scsi.h b/include/hw/virtio/vhost-user-scsi.h index 521b08e559..78fe616ccb 100644 --- a/include/hw/virtio/vhost-user-scsi.h +++ b/include/hw/virtio/vhost-user-scsi.h @@ -28,7 +28,13 @@ OBJECT_DECLARE_SIMPLE_TYPE(VHostUserSCSI, VHOST_USER_SCSI) struct VHostUserSCSI { VHostSCSICommon parent_obj; + + /* Properties */ + bool connected; + bool started_vu; + VhostUserState vhost_user; + struct vhost_virtqueue *vhost_vqs; }; #endif /* VHOST_USER_SCSI_H */ diff --git a/hw/scsi/vhost-scsi-common.c b/hw/scsi/vhost-scsi-common.c index a61cd0e907..4c8637045d 100644 --- a/hw/scsi/vhost-scsi-common.c +++ b/hw/scsi/vhost-scsi-common.c @@ -16,6 +16,7 @@ */ #include "qemu/osdep.h" +#include "qapi/error.h" #include "qemu/error-report.h" #include "qemu/module.h" #include "hw/virtio/vhost.h" @@ -25,7 +26,7 @@ #include "hw/virtio/virtio-access.h" #include "hw/fw-path-provider.h" -int vhost_scsi_common_start(VHostSCSICommon *vsc) +int vhost_scsi_common_start(VHostSCSICommon *vsc, Error **errp) { int ret, i; VirtIODevice *vdev = VIRTIO_DEVICE(vsc); @@ -35,18 +36,19 @@ int vhost_scsi_common_start(VHostSCSICommon *vsc) VirtIOSCSICommon *vs = (VirtIOSCSICommon *)vsc; if (!k->set_guest_notifiers) { - error_report("binding does not support guest notifiers"); + error_setg(errp, "binding does not support guest notifiers"); return -ENOSYS; } ret = vhost_dev_enable_notifiers(&vsc->dev, vdev); if (ret < 0) { + error_setg_errno(errp, -ret, "Error enabling host notifiers"); return ret; } ret = k->set_guest_notifiers(qbus->parent, vsc->dev.nvqs, true); if (ret < 0) { - error_report("Error binding guest notifier"); + error_setg_errno(errp, -ret, "Error binding guest notifier"); goto err_host_notifiers; } @@ -54,7 +56,7 @@ int vhost_scsi_common_start(VHostSCSICommon *vsc) ret = vhost_dev_prepare_inflight(&vsc->dev, vdev); if (ret < 0) { - error_report("Error setting inflight format: %d", -ret); + error_setg_errno(errp, -ret, "Error setting inflight format"); goto err_guest_notifiers; } @@ -64,21 +66,21 @@ int vhost_scsi_common_start(VHostSCSICommon *vsc) vs->conf.virtqueue_size, vsc->inflight); if (ret < 0) { - error_report("Error getting inflight: %d", -ret); + error_setg_errno(errp, -ret, "Error getting inflight"); goto err_guest_notifiers; } } ret = vhost_dev_set_inflight(&vsc->dev, vsc->inflight); if (ret < 0) { - error_report("Error setting inflight: %d", -ret); + error_setg_errno(errp, -ret, "Error setting inflight"); goto err_guest_notifiers; } } ret = vhost_dev_start(&vsc->dev, vdev, true); if (ret < 0) { - error_report("Error start vhost dev"); + error_setg_errno(errp, -ret, "Error starting vhost dev"); goto err_guest_notifiers; } diff --git a/hw/scsi/vhost-scsi.c b/hw/scsi/vhost-scsi.c index 443f67daa4..95cadb93e7 100644 --- a/hw/scsi/vhost-scsi.c +++ b/hw/scsi/vhost-scsi.c @@ -75,6 +75,7 @@ static int vhost_scsi_start(VHostSCSI *s) int ret, abi_version; VHostSCSICommon *vsc = VHOST_SCSI_COMMON(s); const VhostOps *vhost_ops = vsc->dev.vhost_ops; + Error *local_err = NULL; ret = vhost_ops->vhost_scsi_get_abi_version(&vsc->dev, &abi_version); if (ret < 0) { @@ -88,14 +89,15 @@ static int vhost_scsi_start(VHostSCSI *s) return -ENOSYS; } - ret = vhost_scsi_common_start(vsc); + ret = vhost_scsi_common_start(vsc, &local_err); if (ret < 0) { + error_reportf_err(local_err, "Error starting vhost-scsi"); return ret; } ret = vhost_scsi_set_endpoint(s); if (ret < 0) { - error_report("Error setting vhost-scsi endpoint"); + error_reportf_err(local_err, "Error setting vhost-scsi endpoint"); vhost_scsi_common_stop(vsc); } diff --git a/hw/scsi/vhost-user-scsi.c b/hw/scsi/vhost-user-scsi.c index b7c6100f3e..24c250d3f8 100644 --- a/hw/scsi/vhost-user-scsi.c +++ b/hw/scsi/vhost-user-scsi.c @@ -39,26 +39,56 @@ static const int user_feature_bits[] = { VHOST_INVALID_FEATURE_BIT }; +static int vhost_user_scsi_start(VHostUserSCSI *s, Error **errp) +{ + VHostSCSICommon *vsc = VHOST_SCSI_COMMON(s); + int ret; + + ret = vhost_scsi_common_start(vsc, errp); + s->started_vu = !(ret < 0); + + return ret; +} + +static void vhost_user_scsi_stop(VHostUserSCSI *s) +{ + VHostSCSICommon *vsc = VHOST_SCSI_COMMON(s); + + if (!s->started_vu) { + return; + } + s->started_vu = false; + + vhost_scsi_common_stop(vsc); +} + static void vhost_user_scsi_set_status(VirtIODevice *vdev, uint8_t status) { VHostUserSCSI *s = (VHostUserSCSI *)vdev; + DeviceState *dev = DEVICE(vdev); VHostSCSICommon *vsc = VHOST_SCSI_COMMON(s); - bool start = (status & VIRTIO_CONFIG_S_DRIVER_OK) && vdev->vm_running; + VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(dev); + bool should_start = virtio_device_should_start(vdev, status); + Error *local_err = NULL; + int ret; - if (vhost_dev_is_started(&vsc->dev) == start) { + if (!s->connected) { return; } - if (start) { - int ret; + if (vhost_dev_is_started(&vsc->dev) == should_start) { + return; + } - ret = vhost_scsi_common_start(vsc); + if (should_start) { + ret = vhost_user_scsi_start(s, &local_err); if (ret < 0) { - error_report("unable to start vhost-user-scsi: %s", strerror(-ret)); - exit(1); + error_reportf_err(local_err, "unable to start vhost-user-scsi: %s", + strerror(-ret)); + qemu_chr_fe_disconnect(&vs->conf.chardev); } } else { - vhost_scsi_common_stop(vsc); + vhost_user_scsi_stop(s); } } @@ -66,14 +96,124 @@ static void vhost_dummy_handle_output(VirtIODevice *vdev, VirtQueue *vq) { } +static int vhost_user_scsi_connect(DeviceState *dev, Error **errp) +{ + VirtIODevice *vdev = VIRTIO_DEVICE(dev); + VHostUserSCSI *s = VHOST_USER_SCSI(vdev); + VHostSCSICommon *vsc = VHOST_SCSI_COMMON(s); + VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(dev); + int ret = 0; + + if (s->connected) { + return 0; + } + s->connected = true; + + vsc->dev.num_queues = vs->conf.num_queues; + vsc->dev.nvqs = VIRTIO_SCSI_VQ_NUM_FIXED + vs->conf.num_queues; + vsc->dev.vqs = s->vhost_vqs; + vsc->dev.vq_index = 0; + vsc->dev.backend_features = 0; + + ret = vhost_dev_init(&vsc->dev, &s->vhost_user, VHOST_BACKEND_TYPE_USER, 0, + errp); + if (ret < 0) { + return ret; + } + + /* restore vhost state */ + if (virtio_device_started(vdev, vdev->status)) { + ret = vhost_user_scsi_start(s, errp); + } + + return ret; +} + +static void vhost_user_scsi_event(void *opaque, QEMUChrEvent event); + +static void vhost_user_scsi_disconnect(DeviceState *dev) +{ + VirtIODevice *vdev = VIRTIO_DEVICE(dev); + VHostUserSCSI *s = VHOST_USER_SCSI(vdev); + VHostSCSICommon *vsc = VHOST_SCSI_COMMON(s); + VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(dev); + + if (!s->connected) { + return; + } + s->connected = false; + + vhost_user_scsi_stop(s); + + vhost_dev_cleanup(&vsc->dev); + + /* Re-instate the event handler for new connections */ + qemu_chr_fe_set_handlers(&vs->conf.chardev, NULL, NULL, + vhost_user_scsi_event, NULL, dev, NULL, true); +} + +static void vhost_user_scsi_event(void *opaque, QEMUChrEvent event) +{ + DeviceState *dev = opaque; + VirtIODevice *vdev = VIRTIO_DEVICE(dev); + VHostUserSCSI *s = VHOST_USER_SCSI(vdev); + VHostSCSICommon *vsc = VHOST_SCSI_COMMON(s); + VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(dev); + Error *local_err = NULL; + + switch (event) { + case CHR_EVENT_OPENED: + if (vhost_user_scsi_connect(dev, &local_err) < 0) { + error_report_err(local_err); + qemu_chr_fe_disconnect(&vs->conf.chardev); + return; + } + break; + case CHR_EVENT_CLOSED: + /* defer close until later to avoid circular close */ + vhost_user_async_close(dev, &vs->conf.chardev, &vsc->dev, + vhost_user_scsi_disconnect); + break; + case CHR_EVENT_BREAK: + case CHR_EVENT_MUX_IN: + case CHR_EVENT_MUX_OUT: + /* Ignore */ + break; + } +} + +static int vhost_user_scsi_realize_connect(VHostUserSCSI *s, Error **errp) +{ + DeviceState *dev = DEVICE(s); + VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(dev); + int ret; + + s->connected = false; + + ret = qemu_chr_fe_wait_connected(&vs->conf.chardev, errp); + if (ret < 0) { + return ret; + } + + ret = vhost_user_scsi_connect(dev, errp); + if (ret < 0) { + qemu_chr_fe_disconnect(&vs->conf.chardev); + return ret; + } + assert(s->connected); + + return 0; +} + static void vhost_user_scsi_realize(DeviceState *dev, Error **errp) { + ERRP_GUARD(); VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(dev); VHostUserSCSI *s = VHOST_USER_SCSI(dev); VHostSCSICommon *vsc = VHOST_SCSI_COMMON(s); - struct vhost_virtqueue *vqs = NULL; Error *err = NULL; int ret; + int retries = VU_REALIZE_CONN_RETRIES; if (!vs->conf.chardev.chr) { error_setg(errp, "vhost-user-scsi: missing chardev"); @@ -92,18 +232,28 @@ static void vhost_user_scsi_realize(DeviceState *dev, Error **errp) goto free_virtio; } - vsc->dev.nvqs = VIRTIO_SCSI_VQ_NUM_FIXED + vs->conf.num_queues; - vsc->dev.vqs = g_new0(struct vhost_virtqueue, vsc->dev.nvqs); - vsc->dev.vq_index = 0; - vsc->dev.backend_features = 0; - vqs = vsc->dev.vqs; + vsc->inflight = g_new0(struct vhost_inflight, 1); + s->vhost_vqs = g_new0(struct vhost_virtqueue, + VIRTIO_SCSI_VQ_NUM_FIXED + vs->conf.num_queues); + + assert(!*errp); + do { + if (*errp) { + error_prepend(errp, "Reconnecting after error: "); + error_report_err(*errp); + *errp = NULL; + } + ret = vhost_user_scsi_realize_connect(s, errp); + } while (ret < 0 && retries--); - ret = vhost_dev_init(&vsc->dev, &s->vhost_user, - VHOST_BACKEND_TYPE_USER, 0, errp); if (ret < 0) { goto free_vhost; } + /* we're fully initialized, now we can operate, so add the handler */ + qemu_chr_fe_set_handlers(&vs->conf.chardev, NULL, NULL, + vhost_user_scsi_event, NULL, (void *)dev, + NULL, true); /* Channel and lun both are 0 for bootable vhost-user-scsi disk */ vsc->channel = 0; vsc->lun = 0; @@ -112,8 +262,12 @@ static void vhost_user_scsi_realize(DeviceState *dev, Error **errp) return; free_vhost: + g_free(s->vhost_vqs); + s->vhost_vqs = NULL; + g_free(vsc->inflight); + vsc->inflight = NULL; vhost_user_cleanup(&s->vhost_user); - g_free(vqs); + free_virtio: virtio_scsi_common_unrealize(dev); } @@ -123,16 +277,23 @@ static void vhost_user_scsi_unrealize(DeviceState *dev) VirtIODevice *vdev = VIRTIO_DEVICE(dev); VHostUserSCSI *s = VHOST_USER_SCSI(dev); VHostSCSICommon *vsc = VHOST_SCSI_COMMON(s); - struct vhost_virtqueue *vqs = vsc->dev.vqs; + VirtIOSCSICommon *vs = VIRTIO_SCSI_COMMON(dev); /* This will stop the vhost backend. */ vhost_user_scsi_set_status(vdev, 0); + qemu_chr_fe_set_handlers(&vs->conf.chardev, NULL, NULL, NULL, NULL, NULL, + NULL, false); vhost_dev_cleanup(&vsc->dev); - g_free(vqs); + g_free(s->vhost_vqs); + s->vhost_vqs = NULL; + + vhost_dev_free_inflight(vsc->inflight); + g_free(vsc->inflight); + vsc->inflight = NULL; - virtio_scsi_common_unrealize(dev); vhost_user_cleanup(&s->vhost_user); + virtio_scsi_common_unrealize(dev); } static Property vhost_user_scsi_properties[] = {