From patchwork Tue Oct 29 11:48:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Freimann X-Patchwork-Id: 11217785 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1EF9F15AB for ; Tue, 29 Oct 2019 11:50:11 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E74DE222C5 for ; Tue, 29 Oct 2019 11:50:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VbfeEKp4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E74DE222C5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:55286 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iPQ0m-0004nC-F1 for patchwork-qemu-devel@patchwork.kernel.org; Tue, 29 Oct 2019 07:50:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40285) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iPQ0B-0003jA-I7 for qemu-devel@nongnu.org; Tue, 29 Oct 2019 07:49:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iPQ08-00021x-0C for qemu-devel@nongnu.org; Tue, 29 Oct 2019 07:49:29 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:21669 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iPQ07-00021Q-Ry for qemu-devel@nongnu.org; Tue, 29 Oct 2019 07:49:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1572349766; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=oPLTXnt6ThbpkBmjJ5seWWrGvU9Hyex+rur7libnsXo=; b=VbfeEKp4CzLiAHT+lAG5Wknw7mjat3jJPKzS04DHsdpIGlAxQ7EI9hUI3KpAxUAYwiJIaq QO51u7wwJBHsYrfxFXBG5LFyuBGJwLofs1DSzr4WiUconVl3ojpcZkXrDjUYTqo0p0cuvb hxAprS2oMVaoeWytkk+AELFkozEV30c= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-378-sg8i_bmGN8ud7-OwgQWExg-1; Tue, 29 Oct 2019 07:49:22 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 53F4D81A334; Tue, 29 Oct 2019 11:49:21 +0000 (UTC) Received: from localhost (ovpn-116-231.ams2.redhat.com [10.36.116.231]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D01A7600F0; Tue, 29 Oct 2019 11:49:09 +0000 (UTC) From: Jens Freimann To: qemu-devel@nongnu.org Subject: [PATCH v7 0/11] add failover feature for assigned network devices Date: Tue, 29 Oct 2019 12:48:54 +0100 Message-Id: <20191029114905.6856-1-jfreimann@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: sg8i_bmGN8ud7-OwgQWExg-1 X-Mimecast-Spam-Score: 0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 205.139.110.61 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: berrange@redhat.com, ehabkost@redhat.com, mst@redhat.com, aadam@redhat.com, jasowang@redhat.com, quintela@redhat.com, dgilbert@redhat.com, armbru@redhat.com, alex.williamson@redhat.com, laine@redhat.com, pbonzini@redhat.com, ailan@redhat.com, parav@mellanox.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" This is implementing the host side of the net_failover concept (https://www.kernel.org/doc/html/latest/networking/net_failover.html) Changes since v6: * reword patch description of 06/11 (Markus) * have qemu_savevm_state_guest_unplug_pending() return true when at least one device is still not unplugged (Dave) The general idea is that we have a pair of devices, a vfio-pci and a virtio-net device. Before migration the vfio device is unplugged and data flows to the virtio-net device, on the target side another vfio-pci device is plugged in to take over the data-path. In the guest the net_failover module will pair net devices with the same MAC address. * Patch 1 adds the infrastructure to hide the device for the qbus and qdev APIs * Patch 2 adds checks to PCIDevice for only allowing ethernet devices as failover primary and only PCIExpress capable devices * Patch 3 sets a new flag for PCIDevice 'partially_hotplugged' which we use to skip the unrealize code path when doing a unplug of the primary device * Patch 4 sets the pending_deleted_event before triggering the guest unplug request * Patch 5 and 6 add new qmp events, one sends the device id of a device that was just requested to be unplugged from the guest and another one to let libvirt know if VIRTIO_NET_F_STANDBY was negotiated * Patch 7 make sure that we can unplug the vfio-device before migration starts * Patch 8 adds a new migration state that is entered while we wait for devices to be unplugged by guest OS * Patch 9 just adds the new migration state to a check in libqos code * Patch 10 In the second patch the virtio-net uses the API to defer adding the vfio device until the VIRTIO_NET_F_STANDBY feature is acked. It also implements the migration handler to unplug the device from the guest and re-plug in case of migration failure * Patch 11 allows migration for failover vfio-pci devices Previous discussion: RFC v1 https://patchwork.ozlabs.org/cover/989098/ RFC v2 https://www.mail-archive.com/qemu-devel@nongnu.org/msg606906.html v1: https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg03968.html v2: https://www.mail-archive.com/qemu-devel@nongnu.org/msg635214.html v3: https://patchew.org/QEMU/20191011112015.11785-1-jfreimann@redhat.com/ v4: https://patchew.org/QEMU/20191018202040.30349-1-jfreimann@redhat.com/ v5: https://patchew.org/QEMU/20191023082711.16694-1-jfreimann@redhat.com/ v6: https://patchew.org/QEMU/20191025121930.6855-1-jfreimann@redhat.com/ To summarize concerns/feedback from previous discussion: 1.- guest OS can reject or worse _delay_ unplug by any amount of time. Migration might get stuck for unpredictable time with unclear reason. This approach combines two tricky things, hot/unplug and migration. -> We need to let libvirt know what's happening. Add new qmp events and a new migration state. When a primary device is (partially) unplugged (only from guest) we send a qmp event with the device id. When it is unplugged from the guest the DEVICE_DELETED event is sent. Migration will enter the wait-unplug state while waiting for the guest os to unplug all primary devices and then move on with migration. 2. PCI devices are a precious ressource. The primary device should never be added to QEMU if it won't be used by guest instead of hiding it in QEMU. -> We only hotplug the device when the standby feature bit was negotiated. We save the device cmdline options until we need it for qdev_device_add() Hiding a device can be a useful concept to model. For example a pci device in a powered-off slot could be marked as hidden until the slot is powered on (mst). 3. Management layer software should handle this. Open Stack already has components/code to handle unplug/replug VFIO devices and metadata to provide to the guest for detecting which devices should be paired. -> An approach that includes all software from firmware to higher-level management software wasn't tried in the last years. This is an attempt to keep it simple and contained in QEMU as much as possible. One of the problems that stopped management software and libvirt from implementing this idea is that it can't be sure that it's possible to re-plug the primary device. By not freeing the devices resources in QEMU and only asking the guest OS to unplug it is possible to re-plug the device in case of a migration failure. 4. Hotplugging a device and then making it part of a failover setup is not possible -> addressed by extending qdev hotplug functions to check for hidden attribute, so e.g. device_add can be used to plug a device. I have tested this with a mlx5 and igb NIC and was able to migrate the VM. Command line example: qemu-system-x86_64 -enable-kvm -m 3072 -smp 3 \ -machine q35,kernel-irqchip=split -cpu host \ -serial stdio \ -net none \ -qmp unix:/tmp/qmp.socket,server,nowait \ -monitor telnet:127.0.0.1:5555,server,nowait \ -device pcie-root-port,id=root0,multifunction=on,chassis=0,addr=0xa \ -device pcie-root-port,id=root1,bus=pcie.0,chassis=1 \ -device pcie-root-port,id=root2,bus=pcie.0,chassis=2 \ -netdev tap,script=/root/bin/bridge.sh,downscript=no,id=hostnet1,vhost=on \ -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:6f:55:cc,bus=root2,failover=on \ -device vfio-pci,host=5e:00.2,id=hostdev0,bus=root1,failover_pair_id =net1 \ /root/rhel-guest-image-8.0-1781.x86_64.qcow2 I'm grateful for any remarks or ideas! Thanks! regards, Jens Jens Freimann (11): qdev/qbus: add hidden device support pci: add option for net failover pci: mark devices partially unplugged pci: mark device having guest unplug request pending qapi: add unplug primary event qapi: add failover negotiated event migration: allow unplug during migration for failover devices migration: add new migration state wait-unplug libqos: tolerate wait-unplug migration state net/virtio: add failover support vfio: unplug failover primary device before migration MAINTAINERS | 1 + docs/virtio-net-failover.rst | 68 ++++++++ hw/core/qdev.c | 25 +++ hw/net/virtio-net.c | 302 +++++++++++++++++++++++++++++++++ hw/pci/pci.c | 32 ++++ hw/pci/pcie.c | 6 + hw/vfio/pci.c | 26 ++- hw/vfio/pci.h | 1 + include/hw/pci/pci.h | 4 + include/hw/qdev-core.h | 30 ++++ include/hw/virtio/virtio-net.h | 12 ++ include/hw/virtio/virtio.h | 1 + include/migration/vmstate.h | 2 + migration/migration.c | 21 +++ migration/migration.h | 3 + migration/savevm.c | 33 ++++ migration/savevm.h | 2 + qapi/migration.json | 24 ++- qapi/net.json | 19 +++ qdev-monitor.c | 43 ++++- tests/libqos/libqos.c | 3 +- vl.c | 6 +- 22 files changed, 649 insertions(+), 15 deletions(-) create mode 100644 docs/virtio-net-failover.rst