From patchwork Tue Nov 20 20:39:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 10691199 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 899461709 for ; Tue, 20 Nov 2018 20:44:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 766BD2AE29 for ; Tue, 20 Nov 2018 20:44:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6A9252AE50; Tue, 20 Nov 2018 20:44:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B62892AE6A for ; Tue, 20 Nov 2018 20:44:04 +0000 (UTC) Received: from localhost ([::1]:35825 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCsO-0008HL-0c for patchwork-qemu-devel@patchwork.kernel.org; Tue, 20 Nov 2018 15:44:04 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52452) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCqE-0005r1-RX for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:55 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gPCq9-0003L0-3o for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:50 -0500 Received: from hqemgate16.nvidia.com ([216.228.121.65]:10891) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gPCq7-0003ID-8O for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:45 -0500 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate16.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Tue, 20 Nov 2018 12:41:49 -0800 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Tue, 20 Nov 2018 12:41:40 -0800 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Tue, 20 Nov 2018 12:41:40 -0800 Received: from HQMAIL102.nvidia.com (172.18.146.10) by HQMAIL103.nvidia.com (172.20.187.11) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:41:39 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL102.nvidia.com (172.18.146.10) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:41:39 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4 via Frontend Transport; Tue, 20 Nov 2018 20:41:32 +0000 From: Kirti Wankhede To: , Date: Wed, 21 Nov 2018 02:09:39 +0530 Message-ID: <1542746383-18288-2-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> References: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1542746509; bh=+760c3xMZ5cmBUggliF6j0hmN+CTRFRqNQY7cEt2zaE=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=Bmv/ULA8Ou5mBSKJmWzqIYZHTo45aNbwoZyVlWMrW1WZYmI9dVIxNX1SxdydUtEbb e9O2gLsMQbHBe4GX1K/JkEVaQIQKwSlIiVPcbdgrwgMTIPmJ1PxEz//Lf48CV9h8ci PnAYEATF/QsRubBkm3iT3Jy1YD8aZwsmNbZHxElkmo7N7TAMwYWAVLgILb3KJgsrqA qO+0BzvgR8g5eEN2XosEKdM8hCkjGmT1Gf15sl/J4BEYJnCLZ0j+bFe912UsBSR7MS tvEI2Z1Ht5vUCqfbzFLerfctZZ5YRmUYPAvO6hkZuO79F4CBS7lRdijdCAEUiT0iJy yf2G0Rf4JopZw== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.65 Subject: [Qemu-devel] [PATCH 1/5] VFIO KABI for migration interface X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, Kirti Wankhede , eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP - Defined MIGRATION region type and sub-type. - Defined VFIO device states during migration process. - Defined vfio_device_migration_info structure which will be placed at 0th offset of migration region to get/set VFIO device related information. Defined actions and members of structure usage for each action: * To convey VFIO device state to be transitioned to. * To get pending bytes yet to be migrated for VFIO device * To ask driver to write data to migration region and return number of bytes written in the region * In migration resume path, user space app writes to migration region and communicates it to vendor driver. * Get bitmap of dirty pages from vendor driver from given start address Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- linux-headers/linux/vfio.h | 130 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) diff --git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h index 3615a269d378..a6e45cb2cae2 100644 --- a/linux-headers/linux/vfio.h +++ b/linux-headers/linux/vfio.h @@ -301,6 +301,10 @@ struct vfio_region_info_cap_type { #define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2) #define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3) +/* Migration region type and sub-type */ +#define VFIO_REGION_TYPE_MIGRATION (1 << 30) +#define VFIO_REGION_SUBTYPE_MIGRATION (1) + /* * The MSIX mappable capability informs that MSIX data of a BAR can be mmapped * which allows direct access to non-MSIX registers which happened to be within @@ -602,6 +606,132 @@ struct vfio_device_ioeventfd { #define VFIO_DEVICE_IOEVENTFD _IO(VFIO_TYPE, VFIO_BASE + 16) +/** + * VFIO device states : + * VFIO User space application should set the device state to indicate vendor + * driver in which state the VFIO device should transitioned. + * - VFIO_DEVICE_STATE_NONE: + * State when VFIO device is initialized but not yet running. + * - VFIO_DEVICE_STATE_RUNNING: + * Transition VFIO device in running state, that is, user space application or + * VM is active. + * - VFIO_DEVICE_STATE_MIGRATION_SETUP: + * Transition VFIO device in migration setup state. This is used to prepare + * VFIO device for migration while application or VM and vCPUs are still in + * running state. + * - VFIO_DEVICE_STATE_MIGRATION_PRECOPY: + * When VFIO user space application or VM is active and vCPUs are running, + * transition VFIO device in pre-copy state. + * - VFIO_DEVICE_STATE_MIGRATION_STOPNCOPY: + * When VFIO user space application or VM is stopped and vCPUs are halted, + * transition VFIO device in stop-and-copy state. + * - VFIO_DEVICE_STATE_MIGRATION_SAVE_COMPLETED: + * When VFIO user space application has copied data provided by vendor driver. + * This state is used by vendor driver to clean up all software state that was + * setup during MIGRATION_SETUP state. + * - VFIO_DEVICE_STATE_MIGRATION_RESUME: + * Transition VFIO device to resume state, that is, start resuming VFIO device + * when user space application or VM is not running and vCPUs are halted. + * - VFIO_DEVICE_STATE_MIGRATION_RESUME_COMPLETED: + * When user space application completes iterations of providing device state + * data, transition device in resume completed state. + * - VFIO_DEVICE_STATE_MIGRATION_FAILED: + * Migration process failed due to some reason, transition device to failed + * state. If migration process fails while saving at source, resume device at + * source. If migration process fails while resuming application or VM at + * destination, stop restoration at destination and resume at source. + * - VFIO_DEVICE_STATE_MIGRATION_CANCELLED: + * User space application has cancelled migration process either for some + * known reason or due to user's intervention. Transition device to Cancelled + * state, that is, resume device state as it was during running state at + * source. + */ + +enum { + VFIO_DEVICE_STATE_NONE, + VFIO_DEVICE_STATE_RUNNING, + VFIO_DEVICE_STATE_MIGRATION_SETUP, + VFIO_DEVICE_STATE_MIGRATION_PRECOPY, + VFIO_DEVICE_STATE_MIGRATION_STOPNCOPY, + VFIO_DEVICE_STATE_MIGRATION_SAVE_COMPLETED, + VFIO_DEVICE_STATE_MIGRATION_RESUME, + VFIO_DEVICE_STATE_MIGRATION_RESUME_COMPLETED, + VFIO_DEVICE_STATE_MIGRATION_FAILED, + VFIO_DEVICE_STATE_MIGRATION_CANCELLED, +}; + +/** + * Structure vfio_device_migration_info is placed at 0th offset of + * VFIO_REGION_SUBTYPE_MIGRATION region to get/set VFIO device related migration + * information. + * + * Action Set state: + * To tell vendor driver the state VFIO device should be transitioned to. + * device_state [input] : User space app sends device state to vendor + * driver on state change, the state to which VFIO device should be + * transitioned to. + * + * Action Get pending bytes: + * To get pending bytes yet to be migrated from vendor driver + * pending.threshold_size [Input] : threshold of buffer in User space app. + * pending.precopy_only [output] : pending data which must be migrated in + * precopy phase or in stopped state, in other words - before target + * user space application or VM start. In case of migration, this + * indicates pending bytes to be transfered while application or VM or + * vCPUs are active and running. + * pending.compatible [output] : pending data which may be migrated any + * time , either when application or VM is active and vCPUs are active + * or when application or VM is halted and vCPUs are halted. + * pending.postcopy_only [output] : pending data which must be migrated in + * postcopy phase or in stopped state, in other words - after source + * application or VM stopped and vCPUs are halted. + * Sum of pending.precopy_only, pending.compatible and + * pending.postcopy_only is the whole amount of pending data. + * + * Action Get buffer: + * On this action, vendor driver should write data to migration region and + * return number of bytes written in the region. + * data.offset [output] : offset in the region from where data is written. + * data.size [output] : number of bytes written in migration buffer by + * vendor driver. + * + * Action Set buffer: + * In migration resume path, user space app writes to migration region and + * communicates it to vendor driver with this action. + * data.offset [Input] : offset in the region from where data is written. + * data.size [Input] : number of bytes written in migration buffer by + * user space app. + * + * Action Get dirty pages bitmap: + * Get bitmap of dirty pages from vendor driver from given start address. + * dirty_pfns.start_addr [Input] : start address + * dirty_pfns.total [Input] : Total pfn count from start_addr for which + * dirty bitmap is requested + * dirty_pfns.copied [Output] : pfn count for which dirty bitmap is copied + * to migration region. + * Vendor driver should copy the bitmap with bits set only for pages to be + * marked dirty in migration region. + */ + +struct vfio_device_migration_info { + __u32 device_state; /* VFIO device state */ + struct { + __u64 precopy_only; + __u64 compatible; + __u64 postcopy_only; + __u64 threshold_size; + } pending; + struct { + __u64 offset; /* offset */ + __u64 size; /* size */ + } data; + struct { + __u64 start_addr; + __u64 total; + __u64 copied; + } dirty_pfns; +} __attribute__((packed)); + /* -------- API for Type1 VFIO IOMMU -------- */ /** From patchwork Tue Nov 20 20:39:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 10691193 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DB4A21709 for ; Tue, 20 Nov 2018 20:43:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C875B2AE78 for ; Tue, 20 Nov 2018 20:43:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BBAD82AE80; Tue, 20 Nov 2018 20:43:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 0DAA02AE78 for ; Tue, 20 Nov 2018 20:43:23 +0000 (UTC) Received: from localhost ([::1]:35820 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCri-0007lt-Jg for patchwork-qemu-devel@patchwork.kernel.org; Tue, 20 Nov 2018 15:43:22 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52501) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCqJ-0005tb-6b for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:59 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gPCqE-0003SW-TJ for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:55 -0500 Received: from hqemgate16.nvidia.com ([216.228.121.65]:10903) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gPCqD-0003Q7-3q for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:50 -0500 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate16.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Tue, 20 Nov 2018 12:41:56 -0800 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Tue, 20 Nov 2018 12:41:47 -0800 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Tue, 20 Nov 2018 12:41:47 -0800 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL106.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:41:47 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4 via Frontend Transport; Tue, 20 Nov 2018 20:41:39 +0000 From: Kirti Wankhede To: , Date: Wed, 21 Nov 2018 02:09:40 +0530 Message-ID: <1542746383-18288-3-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> References: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1542746516; bh=Fsa9hp8Frjv38+rK7idpKFnTlEMNyVBZ02YCObJA0sc=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=q9mnt2cQh1vLb7R8ZxRmvA2ltvh+rI3q6LyumG7vwPLb0U6ioRvgDME3W9s6eGFFK 3CARMmefyQO+0iFqSNNO5U3XbOsbD5RAq8316AZGmLlqfcs5HQp88Y1m7DZIVFB4ta MEvkAv8ejviB1b9IYkEsDDV0z1U1qZhdwfJHWT5JOkhdANhoYFyWb1+nGUcjh8+5xD P/gGRVwuJy5Ww5fkgZCiD3jRl5bBl6YdEyjN4efgzohDlKxj9wRQ7hLVnaiNnb+tFL uqqrIo59o4DZLvsFNCYPzwaGrgKhQ+jGnexwTzdqvOR/BS8w9PJkojBfP7IHDt6QDn ZADlDZB9HdygQ== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.65 Subject: [Qemu-devel] [PATCH 2/5] Add save and load functions for VFIO PCI devices X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, Kirti Wankhede , eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Save and restore with MSIX type is not tested. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/pci.c | 95 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.h | 29 ++++++++++++++++++ 2 files changed, 124 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 6cbb8fa0549d..72daf1a358a0 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -1234,6 +1234,101 @@ void vfio_pci_write_config(PCIDevice *pdev, } } +void vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f) +{ + VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev); + PCIDevice *pdev = &vdev->pdev; + int i; + + for (i = 0; i < PCI_ROM_SLOT; i++) { + uint32_t bar; + + bar = pci_default_read_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, 4); + qemu_put_be32(f, bar); + } + + qemu_put_be32(f, vdev->interrupt); + if (vdev->interrupt == VFIO_INT_MSI) { + uint32_t msi_flags, msi_addr_lo, msi_addr_hi = 0, msi_data; + bool msi_64bit; + + msi_flags = pci_default_read_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, + 2); + msi_64bit = (msi_flags & PCI_MSI_FLAGS_64BIT); + + msi_addr_lo = pci_default_read_config(pdev, + pdev->msi_cap + PCI_MSI_ADDRESS_LO, 4); + qemu_put_be32(f, msi_addr_lo); + + if (msi_64bit) { + msi_addr_hi = pci_default_read_config(pdev, + pdev->msi_cap + PCI_MSI_ADDRESS_HI, + 4); + } + qemu_put_be32(f, msi_addr_hi); + + msi_data = pci_default_read_config(pdev, + pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32), + 2); + qemu_put_be32(f, msi_data); + } else if (vdev->interrupt == VFIO_INT_MSIX) { + msix_save(pdev, f); + } +} + +void vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f) +{ + VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev); + PCIDevice *pdev = &vdev->pdev; + uint32_t pci_cmd, interrupt_type; + uint32_t msi_flags, msi_addr_lo, msi_addr_hi = 0, msi_data; + bool msi_64bit; + int i; + + /* retore pci bar configuration */ + pci_cmd = pci_default_read_config(pdev, PCI_COMMAND, 2); + vfio_pci_write_config(pdev, PCI_COMMAND, + pci_cmd & (!(PCI_COMMAND_IO | PCI_COMMAND_MEMORY)), 2); + for (i = 0; i < PCI_ROM_SLOT; i++) { + uint32_t bar = qemu_get_be32(f); + + vfio_pci_write_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, bar, 4); + } + vfio_pci_write_config(pdev, PCI_COMMAND, + pci_cmd | PCI_COMMAND_IO | PCI_COMMAND_MEMORY, 2); + + interrupt_type = qemu_get_be32(f); + + if (interrupt_type == VFIO_INT_MSI) { + /* restore msi configuration */ + msi_flags = pci_default_read_config(pdev, + pdev->msi_cap + PCI_MSI_FLAGS, 2); + msi_64bit = (msi_flags & PCI_MSI_FLAGS_64BIT); + + vfio_pci_write_config(&vdev->pdev, pdev->msi_cap + PCI_MSI_FLAGS, + msi_flags & (!PCI_MSI_FLAGS_ENABLE), 2); + + msi_addr_lo = qemu_get_be32(f); + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_LO, + msi_addr_lo, 4); + + msi_addr_hi = qemu_get_be32(f); + if (msi_64bit) { + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_HI, + msi_addr_hi, 4); + } + msi_data = qemu_get_be32(f); + vfio_pci_write_config(pdev, + pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32), + msi_data, 2); + + vfio_pci_write_config(&vdev->pdev, pdev->msi_cap + PCI_MSI_FLAGS, + msi_flags | PCI_MSI_FLAGS_ENABLE, 2); + } else if (vdev->interrupt == VFIO_INT_MSIX) { + msix_load(pdev, f); + } +} + /* * Interrupt setup */ diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 52b065421a68..890d77d66a6b 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -20,6 +20,7 @@ #include "qemu/queue.h" #include "qemu/timer.h" +#ifdef CONFIG_LINUX #define PCI_ANY_ID (~0) struct VFIOPCIDevice; @@ -198,4 +199,32 @@ void vfio_display_reset(VFIOPCIDevice *vdev); int vfio_display_probe(VFIOPCIDevice *vdev, Error **errp); void vfio_display_finalize(VFIOPCIDevice *vdev); +void vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f); +void vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f); + +static inline Object *vfio_pci_get_object(VFIODevice *vbasedev) +{ + VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev); + + return OBJECT(vdev); +} + +#else +static inline void vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f) +{ + g_assert(false); +} + +static inline void vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f) +{ + g_assert(false); +} + +static inline Object *vfio_pci_get_object(VFIODevice *vbasedev) +{ + return NULL; +} + +#endif + #endif /* HW_VFIO_VFIO_PCI_H */ From patchwork Tue Nov 20 20:39:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 10691197 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 844E51709 for ; Tue, 20 Nov 2018 20:43:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D97A286F6 for ; Tue, 20 Nov 2018 20:43:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 618AC2893C; Tue, 20 Nov 2018 20:43:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F1F88286F6 for ; Tue, 20 Nov 2018 20:43:40 +0000 (UTC) Received: from localhost ([::1]:35823 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCs0-0007xa-3k for patchwork-qemu-devel@patchwork.kernel.org; Tue, 20 Nov 2018 15:43:40 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52638) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCqQ-0005zK-Mw for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gPCqM-0003c7-9I for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:02 -0500 Received: from hqemgate14.nvidia.com ([216.228.121.143]:16421) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gPCqL-0003ZV-Oh for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:58 -0500 Received: from hqpgpgate102.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Tue, 20 Nov 2018 12:41:56 -0800 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate102.nvidia.com (PGP Universal service); Tue, 20 Nov 2018 12:41:56 -0800 X-PGP-Universal: processed; by hqpgpgate102.nvidia.com on Tue, 20 Nov 2018 12:41:56 -0800 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:41:55 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4 via Frontend Transport; Tue, 20 Nov 2018 20:41:47 +0000 From: Kirti Wankhede To: , Date: Wed, 21 Nov 2018 02:09:41 +0530 Message-ID: <1542746383-18288-4-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> References: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1542746516; bh=7wbLhfIH2Wr/3Kis5uhHsHAcHZiDPSdTau7uvr9wiyc=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=f2tXFCBmmf72y9FybBteES0aoZn32rZbnR7c717xU7PACK9A4/WoAaX0lflxWvKli N8MiGZDmOdGBw3dxc16FN9N5sS9PBTQ0p13JwiEHU9KouBO5hF069h+WQ6BuKkm5MR ULBw9QsCk72aobdMLYexqUU9vIK49qo3zGVxjLXrba4U6lnXbLX3hCukhlWiXZFqw9 WRi1ah0Iotnx3WpgZ/o8af+vJ5Fiaztsx3iL3/0FI0LfSMgLvk79vjcn8zZWsF+6oL 1WVmk0nivHJa89vDdDoj1vocHHbzOYsOLSYpqEgE0Peu6yedzEJHALqdeorRq9i/+j uA9Brd2jbEHYA== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.143 Subject: [Qemu-devel] [PATCH 3/5] Add migration functions for VFIO devices X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, Kirti Wankhede , eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP - Migration function are implemented for VFIO_DEVICE_TYPE_PCI device. - Added SaveVMHandlers and implemented all basic functions required for live migration. - Added VM state change handler to know running or stopped state of VM. - Added migration state change notifier to get notification on migration state change. This state is translated to VFIO device state and conveyed to vendor driver. - VFIO device supportd migration or not is decided based of migration region query. If migration region query is successful then migration is supported else migration is blocked. - Structure vfio_device_migration_info is mapped at 0th offset of migration region and should always trapped by VFIO device's driver. Added both type of access support, trapped or mmapped, for data section of the region. - To save device state, read data offset and size using structure vfio_device_migration_info.data, accordingly copy data from the region. - To restore device state, write data offset and size in the structure and write data in the region. - To get dirty page bitmap, write start address and pfn count then read count of pfns copied and accordingly read those from the rest of the region or mmaped part of the region. This copy is iterated till page bitmap for all requested pfns are copied. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/Makefile.objs | 2 +- hw/vfio/migration.c | 729 ++++++++++++++++++++++++++++++++++++++++++ include/hw/vfio/vfio-common.h | 23 ++ 3 files changed, 753 insertions(+), 1 deletion(-) create mode 100644 hw/vfio/migration.c diff --git a/hw/vfio/Makefile.objs b/hw/vfio/Makefile.objs index a2e7a0a7cf02..2cf2ba1440f2 100644 --- a/hw/vfio/Makefile.objs +++ b/hw/vfio/Makefile.objs @@ -1,5 +1,5 @@ ifeq ($(CONFIG_LINUX), y) -obj-$(CONFIG_SOFTMMU) += common.o +obj-$(CONFIG_SOFTMMU) += common.o migration.o obj-$(CONFIG_PCI) += pci.o pci-quirks.o display.o obj-$(CONFIG_VFIO_CCW) += ccw.o obj-$(CONFIG_SOFTMMU) += platform.o diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c new file mode 100644 index 000000000000..717fb63e4f43 --- /dev/null +++ b/hw/vfio/migration.c @@ -0,0 +1,729 @@ +/* + * Migration support for VFIO devices + * + * Copyright NVIDIA, Inc. 2018 + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include + +#include "hw/vfio/vfio-common.h" +#include "cpu.h" +#include "migration/migration.h" +#include "migration/qemu-file.h" +#include "migration/register.h" +#include "migration/blocker.h" +#include "migration/misc.h" +#include "qapi/error.h" +#include "exec/ramlist.h" +#include "exec/ram_addr.h" +#include "pci.h" + +/* + * Flags used as delimiter: + * 0xffffffff => MSB 32-bit all 1s + * 0xef10 => emulated (virtual) function IO + * 0x0000 => 16-bits reserved for flags + */ +#define VFIO_MIG_FLAG_END_OF_STATE (0xffffffffef100001ULL) +#define VFIO_MIG_FLAG_DEV_CONFIG_STATE (0xffffffffef100002ULL) +#define VFIO_MIG_FLAG_DEV_SETUP_STATE (0xffffffffef100003ULL) + +static void vfio_migration_region_exit(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + + if (!migration) { + return; + } + + if (migration->region.buffer.size) { + vfio_region_exit(&migration->region.buffer); + vfio_region_finalize(&migration->region.buffer); + } +} + +static int vfio_migration_region_init(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + Object *obj = NULL; + int ret = -EINVAL; + + if (!migration) { + return ret; + } + + /* Migration support added for PCI device only */ + if (vbasedev->type == VFIO_DEVICE_TYPE_PCI) { + obj = vfio_pci_get_object(vbasedev); + } + + if (!obj) { + return ret; + } + + ret = vfio_region_setup(obj, vbasedev, &migration->region.buffer, + migration->region.index, "migration"); + if (ret) { + error_report("Failed to setup VFIO migration region %d: %s", + migration->region.index, strerror(-ret)); + goto err; + } + + if (!migration->region.buffer.size) { + ret = -EINVAL; + error_report("Invalid region size of VFIO migration region %d: %s", + migration->region.index, strerror(-ret)); + goto err; + } + + if (migration->region.buffer.mmaps) { + ret = vfio_region_mmap(&migration->region.buffer); + if (ret) { + error_report("Failed to mmap VFIO migration region %d: %s", + migration->region.index, strerror(-ret)); + goto err; + } + } + + return 0; + +err: + vfio_migration_region_exit(vbasedev); + return ret; +} + +static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t state) +{ + VFIOMigration *migration = vbasedev->migration; + VFIORegion *region = &migration->region.buffer; + int ret = 0; + + if (vbasedev->device_state == state) { + return ret; + } + + ret = pwrite(vbasedev->fd, &state, sizeof(state), + region->fd_offset + offsetof(struct vfio_device_migration_info, + device_state)); + if (ret < 0) { + error_report("Failed to set migration state %d %s", + ret, strerror(errno)); + return ret; + } + + vbasedev->device_state = state; + return ret; +} + +void vfio_get_dirty_page_list(VFIODevice *vbasedev, + uint64_t start_addr, + uint64_t pfn_count) +{ + VFIOMigration *migration = vbasedev->migration; + VFIORegion *region = &migration->region.buffer; + struct vfio_device_migration_info migration_info; + uint64_t count = 0; + int ret; + + migration_info.dirty_pfns.start_addr = start_addr; + migration_info.dirty_pfns.total = pfn_count; + + ret = pwrite(vbasedev->fd, &migration_info.dirty_pfns, + sizeof(migration_info.dirty_pfns), + region->fd_offset + offsetof(struct vfio_device_migration_info, + dirty_pfns)); + if (ret < 0) { + error_report("Failed to set dirty pages start address %d %s", + ret, strerror(errno)); + return; + } + + do { + /* Read dirty_pfns.copied */ + ret = pread(vbasedev->fd, &migration_info.dirty_pfns, + sizeof(migration_info.dirty_pfns), + region->fd_offset + offsetof(struct vfio_device_migration_info, + dirty_pfns)); + if (ret < 0) { + error_report("Failed to get dirty pages bitmap count %d %s", + ret, strerror(errno)); + return; + } + + if (migration_info.dirty_pfns.copied) { + uint64_t bitmap_size; + void *buf = NULL; + bool buffer_mmaped = false; + + bitmap_size = (BITS_TO_LONGS(migration_info.dirty_pfns.copied) + 1) + * sizeof(unsigned long); + + if (region->mmaps) { + int i; + + for (i = 0; i < region->nr_mmaps; i++) { + if (region->mmaps[i].size >= bitmap_size) { + buf = region->mmaps[i].mmap; + buffer_mmaped = true; + break; + } + } + } + + if (!buffer_mmaped) { + buf = g_malloc0(bitmap_size); + + ret = pread(vbasedev->fd, buf, bitmap_size, + region->fd_offset + sizeof(migration_info) + 1); + if (ret != bitmap_size) { + error_report("Failed to get migration data %d", ret); + g_free(buf); + return; + } + } + + cpu_physical_memory_set_dirty_lebitmap((unsigned long *)buf, + start_addr + (count * TARGET_PAGE_SIZE), + migration_info.dirty_pfns.copied); + count += migration_info.dirty_pfns.copied; + + if (!buffer_mmaped) { + g_free(buf); + } + } + } while (count < migration_info.dirty_pfns.total); +} + +static int vfio_save_device_config_state(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_CONFIG_STATE); + + if (vbasedev->type == VFIO_DEVICE_TYPE_PCI) { + vfio_pci_save_config(vbasedev, f); + } + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + return qemu_file_get_error(f); +} + +static int vfio_load_device_config_state(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + + if (vbasedev->type == VFIO_DEVICE_TYPE_PCI) { + vfio_pci_load_config(vbasedev, f); + } + + if (qemu_get_be64(f) != VFIO_MIG_FLAG_END_OF_STATE) { + error_report("Wrong end of block while loading device config space"); + return -EINVAL; + } + + return qemu_file_get_error(f); +} + +/* ---------------------------------------------------------------------- */ + +static bool vfio_is_active_iterate(void *opaque) +{ + VFIODevice *vbasedev = opaque; + + if (vbasedev->vm_running && vbasedev->migration && + (vbasedev->migration->pending_precopy_only != 0)) + return true; + + if (!vbasedev->vm_running && vbasedev->migration && + (vbasedev->migration->pending_postcopy != 0)) + return true; + + return false; +} + +static int vfio_save_setup(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + int ret; + + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_SETUP_STATE); + + qemu_mutex_lock_iothread(); + ret = vfio_migration_region_init(vbasedev); + qemu_mutex_unlock_iothread(); + if (ret) { + return ret; + } + + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + ret = qemu_file_get_error(f); + if (ret) { + return ret; + } + + return 0; +} + +static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + VFIORegion *region = &migration->region.buffer; + struct vfio_device_migration_info migration_info; + int ret; + + ret = pread(vbasedev->fd, &migration_info.data, + sizeof(migration_info.data), + region->fd_offset + offsetof(struct vfio_device_migration_info, + data)); + if (ret != sizeof(migration_info.data)) { + error_report("Failed to get migration buffer information %d", + ret); + return -EINVAL; + } + + if (migration_info.data.size) { + void *buf = NULL; + bool buffer_mmaped = false; + + if (region->mmaps) { + int i; + + for (i = 0; i < region->nr_mmaps; i++) { + if (region->mmaps[i].offset == migration_info.data.offset) { + buf = region->mmaps[i].mmap; + buffer_mmaped = true; + break; + } + } + } + + if (!buffer_mmaped) { + buf = g_malloc0(migration_info.data.size); + ret = pread(vbasedev->fd, buf, migration_info.data.size, + region->fd_offset + migration_info.data.offset); + if (ret != migration_info.data.size) { + error_report("Failed to get migration data %d", ret); + return -EINVAL; + } + } + + qemu_put_be64(f, migration_info.data.size); + qemu_put_buffer(f, buf, migration_info.data.size); + + if (!buffer_mmaped) { + g_free(buf); + } + + } else { + qemu_put_be64(f, migration_info.data.size); + } + + ret = qemu_file_get_error(f); + if (ret) { + return ret; + } + + return migration_info.data.size; +} + +static int vfio_save_iterate(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + int ret; + + ret = vfio_save_buffer(f, vbasedev); + if (ret < 0) { + error_report("vfio_save_buffer failed %s", + strerror(errno)); + return ret; + } + + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + ret = qemu_file_get_error(f); + if (ret) { + return ret; + } + + return ret; +} + +static void vfio_update_pending(VFIODevice *vbasedev, uint64_t threshold_size) +{ + VFIOMigration *migration = vbasedev->migration; + VFIORegion *region = &migration->region.buffer; + struct vfio_device_migration_info migration_info; + int ret; + + ret = pwrite(vbasedev->fd, &threshold_size, sizeof(threshold_size), + region->fd_offset + offsetof(struct vfio_device_migration_info, + pending.threshold_size)); + if (ret < 0) { + error_report("Failed to set threshold size %d %s", + ret, strerror(errno)); + return; + } + + ret = pread(vbasedev->fd, &migration_info.pending, + sizeof(migration_info.pending), + region->fd_offset + offsetof(struct vfio_device_migration_info, + pending)); + if (ret != sizeof(migration_info.pending)) { + error_report("Failed to get pending bytes %d", ret); + return; + } + + migration->pending_precopy_only = migration_info.pending.precopy_only; + migration->pending_compatible = migration_info.pending.compatible; + migration->pending_postcopy = migration_info.pending.postcopy_only; + + return; +} + +static void vfio_save_pending(QEMUFile *f, void *opaque, + uint64_t threshold_size, + uint64_t *res_precopy_only, + uint64_t *res_compatible, + uint64_t *res_postcopy_only) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + + vfio_update_pending(vbasedev, threshold_size); + + *res_precopy_only += migration->pending_precopy_only; + *res_compatible += migration->pending_compatible; + *res_postcopy_only += migration->pending_postcopy; +} + +static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + MigrationState *ms = migrate_get_current(); + int ret; + + if (vbasedev->vm_running) { + vbasedev->vm_running = 0; + } + + ret = vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_STOPNCOPY); + if (ret) { + error_report("Failed to set state STOPNCOPY_ACTIVE"); + return ret; + } + + ret = vfio_save_device_config_state(f, opaque); + if (ret) { + return ret; + } + + do { + vfio_update_pending(vbasedev, ms->threshold_size); + + if (vfio_is_active_iterate(opaque)) { + ret = vfio_save_buffer(f, vbasedev); + if (ret < 0) { + error_report("Failed to save buffer"); + break; + } else if (ret == 0) { + break; + } + } + } while ((migration->pending_compatible + migration->pending_postcopy) > 0); + + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + ret = qemu_file_get_error(f); + if (ret) { + return ret; + } + + ret = vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_SAVE_COMPLETED); + if (ret) { + error_report("Failed to set state SAVE_COMPLETED"); + return ret; + } + return ret; +} + +static void vfio_save_cleanup(void *opaque) +{ + VFIODevice *vbasedev = opaque; + + vfio_migration_region_exit(vbasedev); +} + +static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) +{ + VFIODevice *vbasedev = opaque; + int ret; + uint64_t data; + + data = qemu_get_be64(f); + while (data != VFIO_MIG_FLAG_END_OF_STATE) { + if (data == VFIO_MIG_FLAG_DEV_CONFIG_STATE) { + ret = vfio_load_device_config_state(f, opaque); + if (ret) { + return ret; + } + } else if (data == VFIO_MIG_FLAG_DEV_SETUP_STATE) { + data = qemu_get_be64(f); + if (data == VFIO_MIG_FLAG_END_OF_STATE) { + return 0; + } else { + error_report("SETUP STATE: EOS not found 0x%lx", data); + return -EINVAL; + } + } else if (data != 0) { + VFIOMigration *migration = vbasedev->migration; + VFIORegion *region = &migration->region.buffer; + struct vfio_device_migration_info migration_info; + void *buf = NULL; + bool buffer_mmaped = false; + + migration_info.data.size = data; + + if (region->mmaps) { + int i; + + for (i = 0; i < region->nr_mmaps; i++) { + if (region->mmaps[i].mmap && + (region->mmaps[i].size >= data)) { + buf = region->mmaps[i].mmap; + migration_info.data.offset = region->mmaps[i].offset; + buffer_mmaped = true; + break; + } + } + } + + if (!buffer_mmaped) { + buf = g_malloc0(migration_info.data.size); + migration_info.data.offset = sizeof(migration_info) + 1; + } + + qemu_get_buffer(f, buf, data); + + ret = pwrite(vbasedev->fd, &migration_info.data, + sizeof(migration_info.data), + region->fd_offset + + offsetof(struct vfio_device_migration_info, data)); + if (ret != sizeof(migration_info.data)) { + error_report("Failed to set migration buffer information %d", + ret); + return -EINVAL; + } + + if (!buffer_mmaped) { + ret = pwrite(vbasedev->fd, buf, migration_info.data.size, + region->fd_offset + migration_info.data.offset); + g_free(buf); + + if (ret != migration_info.data.size) { + error_report("Failed to set migration buffer %d", ret); + return -EINVAL; + } + } + } + + ret = qemu_file_get_error(f); + if (ret) { + return ret; + } + data = qemu_get_be64(f); + } + + return 0; +} + +static int vfio_load_setup(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + int ret; + + ret = vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_RESUME); + if (ret) { + error_report("Failed to set state RESUME"); + } + + ret = vfio_migration_region_init(vbasedev); + if (ret) { + error_report("Failed to initialise migration region"); + return ret; + } + + return 0; +} + +static int vfio_load_cleanup(void *opaque) +{ + VFIODevice *vbasedev = opaque; + int ret = 0; + + ret = vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_RESUME_COMPLETED); + if (ret) { + error_report("Failed to set state RESUME_COMPLETED"); + } + + vfio_migration_region_exit(vbasedev); + return ret; +} + +static SaveVMHandlers savevm_vfio_handlers = { + .save_setup = vfio_save_setup, + .save_live_iterate = vfio_save_iterate, + .save_live_complete_precopy = vfio_save_complete_precopy, + .save_live_pending = vfio_save_pending, + .save_cleanup = vfio_save_cleanup, + .load_state = vfio_load_state, + .load_setup = vfio_load_setup, + .load_cleanup = vfio_load_cleanup, + .is_active_iterate = vfio_is_active_iterate, +}; + +static void vfio_vmstate_change(void *opaque, int running, RunState state) +{ + VFIODevice *vbasedev = opaque; + + if ((vbasedev->vm_running != running) && running) { + int ret; + + ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_RUNNING); + if (ret) { + error_report("Failed to set state RUNNING"); + } + } + + vbasedev->vm_running = running; +} + +static void vfio_migration_state_notifier(Notifier *notifier, void *data) +{ + MigrationState *s = data; + VFIODevice *vbasedev = container_of(notifier, VFIODevice, migration_state); + int ret; + + switch (s->state) { + case MIGRATION_STATUS_SETUP: + ret = vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_SETUP); + if (ret) { + error_report("Failed to set state SETUP"); + } + return; + + case MIGRATION_STATUS_ACTIVE: + if (vbasedev->device_state == VFIO_DEVICE_STATE_MIGRATION_SETUP) { + if (vbasedev->vm_running) { + ret = vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_PRECOPY); + if (ret) { + error_report("Failed to set state PRECOPY_ACTIVE"); + } + } else { + ret = vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_STOPNCOPY); + if (ret) { + error_report("Failed to set state STOPNCOPY_ACTIVE"); + } + } + } else { + ret = vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_RESUME); + if (ret) { + error_report("Failed to set state RESUME"); + } + } + return; + + case MIGRATION_STATUS_CANCELLING: + case MIGRATION_STATUS_CANCELLED: + ret = vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_CANCELLED); + if (ret) { + error_report("Failed to set state CANCELLED"); + } + return; + + case MIGRATION_STATUS_FAILED: + ret = vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_FAILED); + if (ret) { + error_report("Failed to set state FAILED"); + } + return; + } +} + +static int vfio_migration_init(VFIODevice *vbasedev, + struct vfio_region_info *info) +{ + vbasedev->migration = g_new0(VFIOMigration, 1); + vbasedev->migration->region.index = info->index; + + register_savevm_live(NULL, "vfio", -1, 1, &savevm_vfio_handlers, vbasedev); + vbasedev->vm_state = qemu_add_vm_change_state_handler(vfio_vmstate_change, + vbasedev); + + vbasedev->migration_state.notify = vfio_migration_state_notifier; + add_migration_state_change_notifier(&vbasedev->migration_state); + + return 0; +} + + +/* ---------------------------------------------------------------------- */ + +int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) +{ + struct vfio_region_info *info; + int ret; + + ret = vfio_get_dev_region_info(vbasedev, VFIO_REGION_TYPE_MIGRATION, + VFIO_REGION_SUBTYPE_MIGRATION, &info); + if (ret) { + Error *local_err = NULL; + + error_setg(&vbasedev->migration_blocker, + "VFIO device doesn't support migration"); + ret = migrate_add_blocker(vbasedev->migration_blocker, &local_err); + if (local_err) { + error_propagate(errp, local_err); + error_free(vbasedev->migration_blocker); + return ret; + } + } else { + return vfio_migration_init(vbasedev, info); + } + + return 0; +} + +void vfio_migration_finalize(VFIODevice *vbasedev) +{ + if (!vbasedev->migration) { + return; + } + + if (vbasedev->vm_state) { + qemu_del_vm_change_state_handler(vbasedev->vm_state); + remove_migration_state_change_notifier(&vbasedev->migration_state); + } + + if (vbasedev->migration_blocker) { + migrate_del_blocker(vbasedev->migration_blocker); + error_free(vbasedev->migration_blocker); + } + + g_free(vbasedev->migration); +} diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index a9036929b220..ab8217c9e249 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -30,6 +30,8 @@ #include #endif +#include "sysemu/sysemu.h" + #define ERR_PREFIX "vfio error: %s: " #define WARN_PREFIX "vfio warning: %s: " @@ -57,6 +59,16 @@ typedef struct VFIORegion { uint8_t nr; /* cache the region number for debug */ } VFIORegion; +typedef struct VFIOMigration { + struct { + VFIORegion buffer; + uint32_t index; + } region; + uint64_t pending_precopy_only; + uint64_t pending_compatible; + uint64_t pending_postcopy; +} VFIOMigration; + typedef struct VFIOAddressSpace { AddressSpace *as; QLIST_HEAD(, VFIOContainer) containers; @@ -116,6 +128,12 @@ typedef struct VFIODevice { unsigned int num_irqs; unsigned int num_regions; unsigned int flags; + uint32_t device_state; + VMChangeStateEntry *vm_state; + int vm_running; + Notifier migration_state; + VFIOMigration *migration; + Error *migration_blocker; } VFIODevice; struct VFIODeviceOps { @@ -193,4 +211,9 @@ int vfio_spapr_create_window(VFIOContainer *container, int vfio_spapr_remove_window(VFIOContainer *container, hwaddr offset_within_address_space); +int vfio_migration_probe(VFIODevice *vbasedev, Error **errp); +void vfio_migration_finalize(VFIODevice *vbasedev); +void vfio_get_dirty_page_list(VFIODevice *vbasedev, uint64_t start_addr, + uint64_t pfn_count); + #endif /* HW_VFIO_VFIO_COMMON_H */ From patchwork Tue Nov 20 20:39:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 10691203 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 823511709 for ; Tue, 20 Nov 2018 20:46:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6EDFA2AE44 for ; Tue, 20 Nov 2018 20:46:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6326C2AE50; Tue, 20 Nov 2018 20:46:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 0F9572AE44 for ; Tue, 20 Nov 2018 20:46:10 +0000 (UTC) Received: from localhost ([::1]:35828 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCuQ-0000Xm-6Y for patchwork-qemu-devel@patchwork.kernel.org; Tue, 20 Nov 2018 15:46:10 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52807) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCqX-000653-KF for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gPCqT-0003pY-4M for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:09 -0500 Received: from hqemgate15.nvidia.com ([216.228.121.64]:4274) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gPCqS-0003my-A6 for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:04 -0500 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate15.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Tue, 20 Nov 2018 12:41:49 -0800 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Tue, 20 Nov 2018 12:42:02 -0800 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Tue, 20 Nov 2018 12:42:02 -0800 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:42:02 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4 via Frontend Transport; Tue, 20 Nov 2018 20:41:56 +0000 From: Kirti Wankhede To: , Date: Wed, 21 Nov 2018 02:09:42 +0530 Message-ID: <1542746383-18288-5-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> References: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1542746510; bh=gyn4bhWH/PoYVA0P8KcuSgdTA0KWVgAOZrqTd5xTVyk=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=FmHq3aHCvoChx1s8UT30gBmL96gg58C6RifrKyIF9Hhicg7+oyjbWSQP8aZW6hU7M jwSdxNyorezGInDIo96l3WqPDdGJTBjWS4tBhSiE+FsaXbOh9Cv755YK3bPy+EZ7P0 q2PBBAx/48QDoOdYt/XycG/bbdrNGFqQmMuWDHwaw4hLZLQ2lSIlg+NyYwO1DVdowr W+EnjuEznKGUAmiNZ3RuzSR9xb58uEW+i6oPdhpXu9rDL9BW4wN8JIxPx149NgDCrt JSqf+/DT7iZr1DkWv1lbZyfw4eCv03oPs/UdJGo0+HKhRlQenHpF0lZObwg3ZVSK0o CuwCY5iBdqfkw== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.64 Subject: [Qemu-devel] [PATCH 4/5] Add vfio_listerner_log_sync to mark dirty pages X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, Kirti Wankhede , eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP vfio_listerner_log_sync gets list of dirty pages from vendor driver and mark those pages dirty. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/common.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index fb396cf00ac4..338aad7426f0 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -697,9 +697,41 @@ static void vfio_listener_region_del(MemoryListener *listener, } } +static void vfio_listerner_log_sync(MemoryListener *listener, + MemoryRegionSection *section) +{ + uint64_t start_addr, size, pfn_count; + VFIOGroup *group; + VFIODevice *vbasedev; + + QLIST_FOREACH(group, &vfio_group_list, next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + switch (vbasedev->device_state) { + case VFIO_DEVICE_STATE_MIGRATION_PRECOPY: + case VFIO_DEVICE_STATE_MIGRATION_STOPNCOPY: + continue; + + default: + return; + } + } + } + + start_addr = TARGET_PAGE_ALIGN(section->offset_within_address_space); + size = int128_get64(section->size); + pfn_count = size >> TARGET_PAGE_BITS; + + QLIST_FOREACH(group, &vfio_group_list, next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + vfio_get_dirty_page_list(vbasedev, start_addr, pfn_count); + } + } +} + static const MemoryListener vfio_memory_listener = { .region_add = vfio_listener_region_add, .region_del = vfio_listener_region_del, + .log_sync = vfio_listerner_log_sync, }; static void vfio_listener_release(VFIOContainer *container) From patchwork Tue Nov 20 20:39:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 10691205 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A92DB1709 for ; Tue, 20 Nov 2018 20:46:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 956C12AE44 for ; Tue, 20 Nov 2018 20:46:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 892BE2AE50; Tue, 20 Nov 2018 20:46:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2B96F2AE44 for ; Tue, 20 Nov 2018 20:46:46 +0000 (UTC) Received: from localhost ([::1]:35848 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCuz-0001vU-Gb for patchwork-qemu-devel@patchwork.kernel.org; Tue, 20 Nov 2018 15:46:45 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52908) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCqq-0006Lh-Ux for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:33 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gPCqc-00044a-Rn for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:23 -0500 Received: from hqemgate15.nvidia.com ([216.228.121.64]:4291) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gPCqZ-0003yY-2D for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:12 -0500 Received: from hqpgpgate102.nvidia.com (Not Verified[216.228.121.13]) by hqemgate15.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Tue, 20 Nov 2018 12:41:56 -0800 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate102.nvidia.com (PGP Universal service); Tue, 20 Nov 2018 12:42:09 -0800 X-PGP-Universal: processed; by hqpgpgate102.nvidia.com on Tue, 20 Nov 2018 12:42:09 -0800 Received: from HQMAIL102.nvidia.com (172.18.146.10) by HQMAIL103.nvidia.com (172.20.187.11) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:42:09 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL102.nvidia.com (172.18.146.10) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:42:08 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4 via Frontend Transport; Tue, 20 Nov 2018 20:42:02 +0000 From: Kirti Wankhede To: , Date: Wed, 21 Nov 2018 02:09:43 +0530 Message-ID: <1542746383-18288-6-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> References: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1542746516; bh=uJUAloTLG8+CnoIbbmdia4XyNi6ws0pOtAQSOcwm+AI=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=mADwhIuOci+YtlNmmsSsYKlPzObhqUWxiY8ut5aYQrrZvK+3sRpgraZOnzUqLYgkB GjhKkEO7pK+0OCwu9t1p5Pku06z9lLUllz9NZbSVLVDjh2tZTp/3EZKMouIdGGTOCT X3Ar3gck0Z9+F3x4kJOMZUYuvUlK/LY8RQgVGyd7LtGidT3oXO0YqXN6BkBpovJW/T S/VYfUcAbcwprXqUbdZ3J4zuCNMJZgVagtSHzvb4lxi62oyAhhJC7mAiQCeOdffYHR OJs0I28iOhbeHvgrdy2xu/p1G9dzerJFbmteWYDvO73Da1MDdtVm/2NkEqEma15HPR KMvJkKKalaWXw== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.64 Subject: [Qemu-devel] [PATCH 5/5] Make vfio-pci device migration capable. X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, Kirti Wankhede , eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Call vfio_migration_probe() and vfio_migration_finalize() functions for vfio-pci device to enable migration for vfio PCI device. Removed vfio_pci_vmstate structure. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/pci.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 72daf1a358a0..0f9d06981b1b 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2930,6 +2930,7 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) vdev->vbasedev.ops = &vfio_pci_ops; vdev->vbasedev.type = VFIO_DEVICE_TYPE_PCI; vdev->vbasedev.dev = &vdev->pdev.qdev; + vdev->vbasedev.device_state = VFIO_DEVICE_STATE_NONE; tmp = g_strdup_printf("%s/iommu_group", vdev->vbasedev.sysfsdev); len = readlink(tmp, group_path, sizeof(group_path)); @@ -3141,10 +3142,11 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) } } + ret = vfio_migration_probe(&vdev->vbasedev, errp); + vfio_register_err_notifier(vdev); vfio_register_req_notifier(vdev); vfio_setup_resetfn_quirk(vdev); - return; out_teardown: @@ -3180,6 +3182,8 @@ static void vfio_exitfn(PCIDevice *pdev) { VFIOPCIDevice *vdev = DO_UPCAST(VFIOPCIDevice, pdev, pdev); + vdev->vbasedev.device_state = VFIO_DEVICE_STATE_NONE; + vfio_unregister_req_notifier(vdev); vfio_unregister_err_notifier(vdev); pci_device_set_intx_routing_notifier(&vdev->pdev, NULL); @@ -3189,6 +3193,7 @@ static void vfio_exitfn(PCIDevice *pdev) } vfio_teardown_msi(vdev); vfio_bars_exit(vdev); + vfio_migration_finalize(&vdev->vbasedev); } static void vfio_pci_reset(DeviceState *dev) @@ -3294,11 +3299,6 @@ static Property vfio_pci_dev_properties[] = { DEFINE_PROP_END_OF_LIST(), }; -static const VMStateDescription vfio_pci_vmstate = { - .name = "vfio-pci", - .unmigratable = 1, -}; - static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) { DeviceClass *dc = DEVICE_CLASS(klass); @@ -3306,7 +3306,6 @@ static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) dc->reset = vfio_pci_reset; dc->props = vfio_pci_dev_properties; - dc->vmsd = &vfio_pci_vmstate; dc->desc = "VFIO-based PCI device assignment"; set_bit(DEVICE_CATEGORY_MISC, dc->categories); pdc->realize = vfio_realize;