From patchwork Wed Mar 30 10:02:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 08005325@163.com X-Patchwork-Id: 12795648 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 893B7C433F5 for ; Wed, 30 Mar 2022 10:13:42 +0000 (UTC) Received: from localhost ([::1]:51366 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZVKf-0001yl-47 for qemu-devel@archiver.kernel.org; Wed, 30 Mar 2022 06:13:41 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37928) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <08005325@163.com>) id 1nZVAc-0002Js-Gw for qemu-devel@nongnu.org; Wed, 30 Mar 2022 06:03:20 -0400 Received: from m12-15.163.com ([220.181.12.15]:55136) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <08005325@163.com>) id 1nZVAW-000690-8h for qemu-devel@nongnu.org; Wed, 30 Mar 2022 06:03:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:Subject:Date:Message-Id; bh=Xx4nHGJS5NY1RwUznW Gf6SvaEsbHwtqKfVJYrvx/akk=; b=TliDBtIxG0j4Vn5qbF9xFPKcsw2We1FJ6K FRYJVECe8yZ62gBZdsQiQAo+pEfq6fM5GFCc0EjK2IhzUVCdr+Trq72XfXfx5eJ1 Kvr4y4+3aOb1KI1YuQGhgK/YgWLmwNxK7VlIISxB74kDq2GzlAG64/aOf7U3IGpO D47DneOsA= Received: from localhost.localdomain.localdomain (unknown [116.228.45.98]) by smtp11 (Coremail) with SMTP id D8CowADXtXLHKkRiJVqqAA--.547S2; Wed, 30 Mar 2022 18:02:48 +0800 (CST) From: 08005325@163.com To: jasowang@redhat.com, mst@redhat.com Subject: [PATCH v2] vdpa: reset the backend device in stage of stop last vhost device Date: Wed, 30 Mar 2022 06:02:41 -0400 Message-Id: <1648634561-12504-1-git-send-email-08005325@163.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648024966-5170-1-git-send-email-08005325@163.com> References: <1648024966-5170-1-git-send-email-08005325@163.com> X-CM-TRANSID: D8CowADXtXLHKkRiJVqqAA--.547S2 X-Coremail-Antispam: 1Uf129KBjvJXoW3GFWDGw1UWrWDJr47KF15urg_yoW7tryDpa s7GF98Ar4UJr1xta1fAF18u3s8G3sYyw4DGFZ29FyYkF1DtrykZrWqga1jyry7CFWrJF13 tF1qgr4Uu345Z3DanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07joa0PUUUUU= X-Originating-IP: [116.228.45.98] X-CM-SenderInfo: qqyqikqtsvqiywtou0bp/1tbivwjTrFWByc-QngAAsU Received-SPF: pass client-ip=220.181.12.15; envelope-from=08005325@163.com; helo=m12-15.163.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lulu@redhat.com, qemu-devel@nongnu.org, eperezma@redhat.com, Michael Qiu , si-wei.liu@oracle.com, lingshan.zhu@intel.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Michael Qiu Currently, when VM poweroff, it will trigger vdpa device(such as mlx bluefield2 VF) reset many times(with 1 datapath queue pair and one control queue, triggered 3 times), this leads to below issue: vhost VQ 2 ring restore failed: -22: Invalid argument (22) This because in vhost_net_stop(), it will stop all vhost device bind to this virtio device, and in vhost_dev_stop(), qemu tries to stop the device , then stop the queue: vhost_virtqueue_stop(). In vhost_dev_stop(), it resets the device, which clear some flags in low level driver, and in next loop(stop other vhost backends), qemu try to stop the queue corresponding to the vhost backend, the driver finds that the VQ is invalied, this is the root cause. To solve the issue, vdpa should set vring unready, and remove reset ops in device stop: vhost_dev_start(hdev, false). and implement a new function vhost_dev_reset, only reset backend device when the last vhost stop triggerd. Signed-off-by: Michael Qiu Acked-by: Jason Wang Signed-off-by: Michael Qiu Acked-by: Jason Wang --- v2 --> v1: implement a new function vhost_dev_reset, reset the backend kernel device at last. --- hw/net/vhost_net.c | 22 +++++++++++++++++++--- hw/virtio/vhost-vdpa.c | 8 ++++---- hw/virtio/vhost.c | 16 +++++++++++++++- include/hw/virtio/vhost.h | 1 + 4 files changed, 39 insertions(+), 8 deletions(-) diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c index 30379d2..3cdf6a4 100644 --- a/hw/net/vhost_net.c +++ b/hw/net/vhost_net.c @@ -299,7 +299,7 @@ fail_notifiers: } static void vhost_net_stop_one(struct vhost_net *net, - VirtIODevice *dev) + VirtIODevice *dev, bool reset) { struct vhost_vring_file file = { .fd = -1 }; @@ -313,6 +313,11 @@ static void vhost_net_stop_one(struct vhost_net *net, net->nc->info->poll(net->nc, true); } vhost_dev_stop(&net->dev, dev); + + if (reset) { + vhost_dev_reset(&net->dev); + } + vhost_dev_disable_notifiers(&net->dev, dev); } @@ -391,7 +396,12 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs, err_start: while (--i >= 0) { peer = qemu_get_peer(ncs , i); - vhost_net_stop_one(get_vhost_net(peer), dev); + + if (i == 0) { + vhost_net_stop_one(get_vhost_net(peer), dev, true); + } else { + vhost_net_stop_one(get_vhost_net(peer), dev, false); + } } e = k->set_guest_notifiers(qbus->parent, total_notifiers, false); if (e < 0) { @@ -420,7 +430,13 @@ void vhost_net_stop(VirtIODevice *dev, NetClientState *ncs, } else { peer = qemu_get_peer(ncs, n->max_queue_pairs); } - vhost_net_stop_one(get_vhost_net(peer), dev); + + /* We only reset backend device during the last vhost */ + if (i == nvhosts - 1) { + vhost_net_stop_one(get_vhost_net(peer), dev, true); + } else { + vhost_net_stop_one(get_vhost_net(peer), dev, false); + } } r = k->set_guest_notifiers(qbus->parent, total_notifiers, false); diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index c5ed7a3..d858b4f 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -719,14 +719,14 @@ static int vhost_vdpa_get_vq_index(struct vhost_dev *dev, int idx) return idx; } -static int vhost_vdpa_set_vring_ready(struct vhost_dev *dev) +static int vhost_vdpa_set_vring_ready(struct vhost_dev *dev, unsigned int ready) { int i; trace_vhost_vdpa_set_vring_ready(dev); for (i = 0; i < dev->nvqs; ++i) { struct vhost_vring_state state = { .index = dev->vq_index + i, - .num = 1, + .num = ready, }; vhost_vdpa_call(dev, VHOST_VDPA_SET_VRING_ENABLE, &state); } @@ -1088,8 +1088,9 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) if (unlikely(!ok)) { return -1; } - vhost_vdpa_set_vring_ready(dev); + vhost_vdpa_set_vring_ready(dev, 1); } else { + vhost_vdpa_set_vring_ready(dev, 0); ok = vhost_vdpa_svqs_stop(dev); if (unlikely(!ok)) { return -1; @@ -1105,7 +1106,6 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) memory_listener_register(&v->listener, &address_space_memory); return vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK); } else { - vhost_vdpa_reset_device(dev); vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE | VIRTIO_CONFIG_S_DRIVER); memory_listener_unregister(&v->listener); diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c index b643f42..6d9b4a3 100644 --- a/hw/virtio/vhost.c +++ b/hw/virtio/vhost.c @@ -1820,7 +1820,7 @@ fail_features: void vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev) { int i; - + printf("vhost_dev_stop test\n"); /* should only be called after backend is connected */ assert(hdev->vhost_ops); @@ -1854,3 +1854,17 @@ int vhost_net_set_backend(struct vhost_dev *hdev, return -ENOSYS; } + +int vhost_dev_reset(struct vhost_dev *hdev) +{ + int ret = 0; + + /* should only be called after backend is connected */ + assert(hdev->vhost_ops); + + if (hdev->vhost_ops->vhost_reset_device) { + ret = hdev->vhost_ops->vhost_reset_device(hdev); + } + + return ret; +} diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h index 58a73e7..b8b7c20 100644 --- a/include/hw/virtio/vhost.h +++ b/include/hw/virtio/vhost.h @@ -114,6 +114,7 @@ int vhost_dev_init(struct vhost_dev *hdev, void *opaque, void vhost_dev_cleanup(struct vhost_dev *hdev); int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev); void vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev); +int vhost_dev_reset(struct vhost_dev *hdev); int vhost_dev_enable_notifiers(struct vhost_dev *hdev, VirtIODevice *vdev); void vhost_dev_disable_notifiers(struct vhost_dev *hdev, VirtIODevice *vdev);