From patchwork Thu Jun 20 14:37:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 11007087 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CF13314DB for ; Thu, 20 Jun 2019 15:06:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BF0A5286EC for ; Thu, 20 Jun 2019 15:06:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B2C54287B5; Thu, 20 Jun 2019 15:06:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 162E6286EC for ; Thu, 20 Jun 2019 15:06:24 +0000 (UTC) Received: from localhost ([::1]:48868 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdydr-0005Co-VB for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Jun 2019 11:06:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37820) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyJm-0005qQ-JG for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdyJk-0005KP-VK for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:38 -0400 Received: from hqemgate16.nvidia.com ([216.228.121.65]:1930) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdyJk-0005Fs-LE for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:36 -0400 Received: from hqpgpgate102.nvidia.com (Not Verified[216.228.121.13]) by hqemgate16.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 20 Jun 2019 07:40:02 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate102.nvidia.com (PGP Universal service); Thu, 20 Jun 2019 07:40:03 -0700 X-PGP-Universal: processed; by hqpgpgate102.nvidia.com on Thu, 20 Jun 2019 07:40:03 -0700 Received: from HQMAIL106.nvidia.com (172.18.146.12) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:40:03 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 20 Jun 2019 14:39:56 +0000 From: Kirti Wankhede To: , Date: Thu, 20 Jun 2019 20:07:29 +0530 Message-ID: <1561041461-22326-2-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> References: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1561041603; bh=THnmNciup8K6MltIgZk/d5O+MlQPVi/MbEN8A0oJa+Y=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=hS0yLmphazasVj+UCwipyFec04P+70ew/U7JKL6aJg1/09jkYH5YNjLwftmSDUt3P JAGrtLPSnGCMhE0WurRwdOQeKsrTyaz23zzkhp2V2ZhYFyMTVLVU8FBsDXhSJI+0S4 917TII1ZZOzH1/wD34ExlpbrFxgvEPKJ4yNqNcUD763bSFAT0IFgjzlsWEs3iX2LD8 CUZX+M+elFzztc/hS/bbPqovVHnWlglm2i2JpjmwWyvvVNOTKUcAq9OvJ6+eMxI6gE tfjVkzuawjk5boFEr0+dx4zFoDrh6X+fOKsBT/vovfUr6U6ghIsuBs+6qF8ASWmCEJ 4luoR0uXpmHZQ== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.65 Subject: [Qemu-devel] [PATCH v4 01/13] vfio: KABI for migration interface X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, yan.y.zhao@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, yulei.zhang@intel.com, eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP - Defined MIGRATION region type and sub-type. - Used 3 bits to define VFIO device states. Bit 0 => _RUNNING Bit 1 => _SAVING Bit 2 => _RESUMING Combination of these bits defines VFIO device's state during migration _STOPPED => All bits 0 indicates VFIO device stopped. _RUNNING => Normal VFIO device running state. _SAVING | _RUNNING => vCPUs are running, VFIO device is running but start saving state of device i.e. pre-copy state _SAVING => vCPUs are stoppped, VFIO device should be stopped, and save device state,i.e. stop-n-copy state _RESUMING => VFIO device resuming state. _SAVING | _RESUMING => Invalid state if _SAVING and _RESUMING bits are set - Defined vfio_device_migration_info structure which will be placed at 0th offset of migration region to get/set VFIO device related information. Defined members of structure and usage on read/write access: * device_state: (read/write) To convey VFIO device state to be transitioned to. Only 3 bits are used as of now. * pending bytes: (read only) To get pending bytes yet to be migrated for VFIO device. * data_offset: (read only) To get data offset in migration from where data exist during _SAVING and from where data should be written by user space application during _RESUMING state * data_size: (read/write) To get and set size of data copied in migration region during _SAVING and _RESUMING state. * start_pfn, page_size, total_pfns: (write only) To get bitmap of dirty pages from vendor driver from given start address for total_pfns. * copied_pfns: (read only) To get number of pfns bitmap copied in migration region. Vendor driver should copy the bitmap with bits set only for pages to be marked dirty in migration region. Vendor driver should return 0 if there are 0 pages dirty in requested range. Vendor driver should return -1 to mark all pages in the section as dirty Migration region looks like: ------------------------------------------------------------------ |vfio_device_migration_info| data section | | | /////////////////////////////// | ------------------------------------------------------------------ ^ ^ ^ offset 0-trapped part data_offset data_size Data section is always followed by vfio_device_migration_info structure in the region, so data_offset will always be none-0. Offset from where data is copied is decided by kernel driver, data section can be trapped or mapped depending on how kernel driver defines data section. If mmapped, then data_offset should be page aligned, where as initial section which contain vfio_device_migration_info structure might not end at offset which is page aligned. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- linux-headers/linux/vfio.h | 71 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h index 24f505199f83..274ec477eb82 100644 --- a/linux-headers/linux/vfio.h +++ b/linux-headers/linux/vfio.h @@ -372,6 +372,77 @@ struct vfio_region_gfx_edid { */ #define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD (1) +/* Migration region type and sub-type */ +#define VFIO_REGION_TYPE_MIGRATION (2) +#define VFIO_REGION_SUBTYPE_MIGRATION (1) + +/** + * Structure vfio_device_migration_info is placed at 0th offset of + * VFIO_REGION_SUBTYPE_MIGRATION region to get/set VFIO device related migration + * information. Field accesses from this structure are only supported at their + * native width and alignment, otherwise should return error. + * + * device_state: (read/write) + * To indicate vendor driver the state VFIO device should be transitioned + * to. If device state transition fails, write to this field return error. + * It consists of 3 bits: + * - If bit 0 set, indicates _RUNNING state. When its reset, that indicates + * _STOPPED state. When device is changed to _STOPPED, driver should stop + * device before write returns. + * - If bit 1 set, indicates _SAVING state. + * - If bit 2 set, indicates _RESUMING state. + * + * pending bytes: (read only) + * Read pending bytes yet to be migrated from vendor driver + * + * data_offset: (read only) + * User application should read data_offset in migration region from where + * user application should read data during _SAVING state or write data + * during _RESUMING state. + * + * data_size: (read/write) + * User application should read data_size to know data copied in migration + * region during _SAVING state and write size of data copied in migration + * region during _RESUMING state. + * + * start_pfn: (write only) + * Start address pfn to get bitmap of dirty pages from vendor driver duing + * _SAVING state. + * + * page_size: (write only) + * User application should write the page_size of pfn. + * + * total_pfns: (write only) + * Total pfn count from start_pfn for which dirty bitmap is requested. + * + * copied_pfns: (read only) + * pfn count for which dirty bitmap is copied to migration region. + * Vendor driver should copy the bitmap with bits set only for pages to be + * marked dirty in migration region. + * Vendor driver should return 0 if there are 0 pages dirty in requested + * range. + * Vendor driver should return -1 to mark all pages in the section as + * dirty. + */ + +struct vfio_device_migration_info { + __u32 device_state; /* VFIO device state */ +#define VFIO_DEVICE_STATE_STOPPED (0) +#define VFIO_DEVICE_STATE_RUNNING (1 << 0) +#define VFIO_DEVICE_STATE_SAVING (1 << 1) +#define VFIO_DEVICE_STATE_RESUMING (1 << 2) +#define VFIO_DEVICE_STATE_INVALID (VFIO_DEVICE_STATE_SAVING | \ + VFIO_DEVICE_STATE_RESUMING) + __u32 reserved; + __u64 pending_bytes; + __u64 data_offset; + __u64 data_size; + __u64 start_pfn; + __u64 page_size; + __u64 total_pfns; + __s64 copied_pfns; +} __attribute__((packed)); + /* * The MSIX mappable capability informs that MSIX data of a BAR can be mmapped * which allows direct access to non-MSIX registers which happened to be within From patchwork Thu Jun 20 14:37:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 11007113 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1897E14DB for ; Thu, 20 Jun 2019 15:10:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 096AD28784 for ; Thu, 20 Jun 2019 15:10:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F05E6287A5; Thu, 20 Jun 2019 15:10:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 85F2528784 for ; Thu, 20 Jun 2019 15:10:53 +0000 (UTC) Received: from localhost ([::1]:48999 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyiC-0001j2-PC for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Jun 2019 11:10:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37769) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyJk-0005nL-An for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdyJi-0005IF-UH for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:36 -0400 Received: from hqemgate14.nvidia.com ([216.228.121.143]:13217) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdyJi-0005GM-KA for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:34 -0400 Received: from hqpgpgate102.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 20 Jun 2019 07:40:09 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate102.nvidia.com (PGP Universal service); Thu, 20 Jun 2019 07:40:10 -0700 X-PGP-Universal: processed; by hqpgpgate102.nvidia.com on Thu, 20 Jun 2019 07:40:10 -0700 Received: from HQMAIL106.nvidia.com (172.18.146.12) by HQMAIL108.nvidia.com (172.18.146.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:40:09 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 20 Jun 2019 14:40:03 +0000 From: Kirti Wankhede To: , Date: Thu, 20 Jun 2019 20:07:30 +0530 Message-ID: <1561041461-22326-3-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> References: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1561041609; bh=m/Xr+/jSSGEXPbqE+0Lnd8Arpqs+tjXrFmsmcvJshkU=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=o5Fqc/voyhtaonHTHtratAMQAB8UO2ku/IGqZK+C1UOGl7oFx7JyPYHummjlkNmUT LcWUoFfflkHGKvrFvdcvHdTXuzP9kT7kJZj4lYCxnGdFfc2etrWbFGlYbx+npkNUom bF0e8M/Q+oq4N76nGgutcBkQe+76NgxJ4UnDVdI0BWP8mNmZgqYIhkV2J1dH8/ppK3 PII7eTouwfvUDK8LV3LAQXtbfy+Jq0/TgQjig+hUNOHZiolNoZsB0cIbqViCEM8vR6 K0ve3Xw4cDoTE/0YPDbr/l+OK9uSySkozHUeHrMXuJI4nP1Pnz8pYZhTjmatl/SK9F 9WtmOUJQlXhLQ== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.143 Subject: [Qemu-devel] [PATCH v4 02/13] vfio: Add function to unmap VFIO region X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, yan.y.zhao@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, yulei.zhang@intel.com, eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP This function is used in follwing patch in this series. Migration region is mmaped when migration starts and will be unmapped when migration is complete. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/common.c | 20 ++++++++++++++++++++ hw/vfio/trace-events | 1 + include/hw/vfio/vfio-common.h | 1 + 3 files changed, 22 insertions(+) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index a859298fdad9..de74dae8d6a6 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -964,6 +964,26 @@ int vfio_region_mmap(VFIORegion *region) return 0; } +void vfio_region_unmap(VFIORegion *region) +{ + int i; + + if (!region->mem) { + return; + } + + for (i = 0; i < region->nr_mmaps; i++) { + trace_vfio_region_unmap(memory_region_name(®ion->mmaps[i].mem), + region->mmaps[i].offset, + region->mmaps[i].offset + + region->mmaps[i].size - 1); + memory_region_del_subregion(region->mem, ®ion->mmaps[i].mem); + munmap(region->mmaps[i].mmap, region->mmaps[i].size); + object_unparent(OBJECT(®ion->mmaps[i].mem)); + region->mmaps[i].mmap = NULL; + } +} + void vfio_region_exit(VFIORegion *region) { int i; diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index b1ef55a33ffd..8cdc27946cb8 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -111,6 +111,7 @@ vfio_region_mmap(const char *name, unsigned long offset, unsigned long end) "Reg vfio_region_exit(const char *name, int index) "Device %s, region %d" vfio_region_finalize(const char *name, int index) "Device %s, region %d" vfio_region_mmaps_set_enabled(const char *name, bool enabled) "Region %s mmaps enabled: %d" +vfio_region_unmap(const char *name, unsigned long offset, unsigned long end) "Region %s unmap [0x%lx - 0x%lx]" vfio_region_sparse_mmap_header(const char *name, int index, int nr_areas) "Device %s region %d: %d sparse mmap entries" vfio_region_sparse_mmap_entry(int i, unsigned long start, unsigned long end) "sparse entry %d [0x%lx - 0x%lx]" vfio_get_dev_region(const char *name, int index, uint32_t type, uint32_t subtype) "%s index %d, %08x/%0x8" diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index a88b69b6750e..ef078cf60ef9 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -176,6 +176,7 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region, int index, const char *name); int vfio_region_mmap(VFIORegion *region); void vfio_region_mmaps_set_enabled(VFIORegion *region, bool enabled); +void vfio_region_unmap(VFIORegion *region); void vfio_region_exit(VFIORegion *region); void vfio_region_finalize(VFIORegion *region); void vfio_reset_handler(void *opaque); From patchwork Thu Jun 20 14:37:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 11006895 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 15E56112C for ; Thu, 20 Jun 2019 14:58:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0405920502 for ; Thu, 20 Jun 2019 14:58:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EB67722701; Thu, 20 Jun 2019 14:58:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6AD7D20502 for ; Thu, 20 Jun 2019 14:58:00 +0000 (UTC) Received: from localhost ([::1]:48800 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyVj-0006qY-Cm for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Jun 2019 10:57:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37813) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyJm-0005ps-8G for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdyJk-0005K6-My for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:38 -0400 Received: from hqemgate15.nvidia.com ([216.228.121.64]:6435) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdyJk-0005IY-Do for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:36 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate15.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 20 Jun 2019 07:40:18 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Thu, 20 Jun 2019 07:40:17 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Thu, 20 Jun 2019 07:40:17 -0700 Received: from HQMAIL106.nvidia.com (172.18.146.12) by HQMAIL104.nvidia.com (172.18.146.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:40:16 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 20 Jun 2019 14:40:10 +0000 From: Kirti Wankhede To: , Date: Thu, 20 Jun 2019 20:07:31 +0530 Message-ID: <1561041461-22326-4-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> References: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1561041618; bh=820//1adwBrsybd0iJJD10rOM2cUvJ+T90dz3F2qvpc=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=osgsy+Tm16I7Tq/PIriD2F7zqY4SQUtbCutHyactcaKGyXR9CvxmzWeLla5Q3BHiW RQ7ocfkGCMqfj6vYo2LJ4XmVoIXBLgfBQwOipiFavdwL9ErBIdy+3vHuHRpWO9/9qy MD+/B3Knw8IF7sLDSLVkhxRY42Z+NZj9QKqU44Yw4YgkDvO7abtIglcHbJhQMdZYIl DGgqq42eG4sv9N4MuPMEy/311zQHvYZ6PShY6eJgdOh5agtuledZDTIuziAx4j3A8j 50TqyUvLI/kHjHpAlv1W8B6mh/NttGn9endBr+5lm7VjwoN0XYpkFhScL88DE7ecT3 ctqYKRbVp9Heg== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.64 Subject: [Qemu-devel] [PATCH v4 03/13] vfio: Add save and load functions for VFIO PCI devices X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, yan.y.zhao@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, yulei.zhang@intel.com, eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP These functions save and restore PCI device specific data - config space of PCI device. Tested save and restore with MSI and MSIX type. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/pci.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.h | 29 +++++++++++++++ 2 files changed, 141 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index ce3fe96efe2c..09a0821a5b1c 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -1187,6 +1187,118 @@ void vfio_pci_write_config(PCIDevice *pdev, } } +void vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f) +{ + VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev); + PCIDevice *pdev = &vdev->pdev; + uint16_t pci_cmd; + int i; + + for (i = 0; i < PCI_ROM_SLOT; i++) { + uint32_t bar; + + bar = pci_default_read_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, 4); + qemu_put_be32(f, bar); + } + + qemu_put_be32(f, vdev->interrupt); + if (vdev->interrupt == VFIO_INT_MSI) { + uint32_t msi_flags, msi_addr_lo, msi_addr_hi = 0, msi_data; + bool msi_64bit; + + msi_flags = pci_default_read_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, + 2); + msi_64bit = (msi_flags & PCI_MSI_FLAGS_64BIT); + + msi_addr_lo = pci_default_read_config(pdev, + pdev->msi_cap + PCI_MSI_ADDRESS_LO, 4); + qemu_put_be32(f, msi_addr_lo); + + if (msi_64bit) { + msi_addr_hi = pci_default_read_config(pdev, + pdev->msi_cap + PCI_MSI_ADDRESS_HI, + 4); + } + qemu_put_be32(f, msi_addr_hi); + + msi_data = pci_default_read_config(pdev, + pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32), + 2); + qemu_put_be32(f, msi_data); + } else if (vdev->interrupt == VFIO_INT_MSIX) { + uint16_t offset; + + /* save enable bit and maskall bit */ + offset = pci_default_read_config(pdev, + pdev->msix_cap + PCI_MSIX_FLAGS + 1, 2); + qemu_put_be16(f, offset); + msix_save(pdev, f); + } + pci_cmd = pci_default_read_config(pdev, PCI_COMMAND, 2); + qemu_put_be16(f, pci_cmd); +} + +void vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f) +{ + VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev); + PCIDevice *pdev = &vdev->pdev; + uint32_t interrupt_type; + uint32_t msi_flags, msi_addr_lo, msi_addr_hi = 0, msi_data; + uint16_t pci_cmd; + bool msi_64bit; + int i; + + /* retore pci bar configuration */ + pci_cmd = pci_default_read_config(pdev, PCI_COMMAND, 2); + vfio_pci_write_config(pdev, PCI_COMMAND, + pci_cmd & (!(PCI_COMMAND_IO | PCI_COMMAND_MEMORY)), 2); + for (i = 0; i < PCI_ROM_SLOT; i++) { + uint32_t bar = qemu_get_be32(f); + + vfio_pci_write_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, bar, 4); + } + vfio_pci_write_config(pdev, PCI_COMMAND, + pci_cmd | PCI_COMMAND_IO | PCI_COMMAND_MEMORY, 2); + + interrupt_type = qemu_get_be32(f); + + if (interrupt_type == VFIO_INT_MSI) { + /* restore msi configuration */ + msi_flags = pci_default_read_config(pdev, + pdev->msi_cap + PCI_MSI_FLAGS, 2); + msi_64bit = (msi_flags & PCI_MSI_FLAGS_64BIT); + + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, + msi_flags & (!PCI_MSI_FLAGS_ENABLE), 2); + + msi_addr_lo = qemu_get_be32(f); + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_LO, + msi_addr_lo, 4); + + msi_addr_hi = qemu_get_be32(f); + if (msi_64bit) { + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_HI, + msi_addr_hi, 4); + } + msi_data = qemu_get_be32(f); + vfio_pci_write_config(pdev, + pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DATA_32), + msi_data, 2); + + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, + msi_flags | PCI_MSI_FLAGS_ENABLE, 2); + } else if (interrupt_type == VFIO_INT_MSIX) { + uint16_t offset = qemu_get_be16(f); + + /* load enable bit and maskall bit */ + vfio_pci_write_config(pdev, pdev->msix_cap + PCI_MSIX_FLAGS + 1, + offset, 2); + msix_load(pdev, f); + } + pci_cmd = qemu_get_be16(f); + vfio_pci_write_config(pdev, PCI_COMMAND, pci_cmd, 2); +} + /* * Interrupt setup */ diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 834a90d64686..847be5f56478 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -19,6 +19,7 @@ #include "qemu/queue.h" #include "qemu/timer.h" +#ifdef CONFIG_LINUX #define PCI_ANY_ID (~0) struct VFIOPCIDevice; @@ -202,4 +203,32 @@ void vfio_display_reset(VFIOPCIDevice *vdev); int vfio_display_probe(VFIOPCIDevice *vdev, Error **errp); void vfio_display_finalize(VFIOPCIDevice *vdev); +void vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f); +void vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f); + +static inline Object *vfio_pci_get_object(VFIODevice *vbasedev) +{ + VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev); + + return OBJECT(vdev); +} + +#else +static inline void vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f) +{ + g_assert(false); +} + +static inline void vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f) +{ + g_assert(false); +} + +static inline Object *vfio_pci_get_object(VFIODevice *vbasedev) +{ + return NULL; +} + +#endif + #endif /* HW_VFIO_VFIO_PCI_H */ From patchwork Thu Jun 20 14:37:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 11007103 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8365014E5 for ; Thu, 20 Jun 2019 15:09:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 72584201B0 for ; Thu, 20 Jun 2019 15:09:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 651DC20502; Thu, 20 Jun 2019 15:09:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A2A1128774 for ; Thu, 20 Jun 2019 15:09:00 +0000 (UTC) Received: from localhost ([::1]:48976 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdygN-0007rs-Uz for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Jun 2019 11:08:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37905) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyKA-0005zd-3x for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdyK8-0005hA-Lr for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:02 -0400 Received: from hqemgate14.nvidia.com ([216.228.121.143]:13242) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdyK5-0005bs-OC for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:59 -0400 Received: from hqpgpgate102.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 20 Jun 2019 07:40:23 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate102.nvidia.com (PGP Universal service); Thu, 20 Jun 2019 07:40:24 -0700 X-PGP-Universal: processed; by hqpgpgate102.nvidia.com on Thu, 20 Jun 2019 07:40:24 -0700 Received: from HQMAIL106.nvidia.com (172.18.146.12) by HQMAIL104.nvidia.com (172.18.146.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:40:23 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 20 Jun 2019 14:40:17 +0000 From: Kirti Wankhede To: , Date: Thu, 20 Jun 2019 20:07:32 +0530 Message-ID: <1561041461-22326-5-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> References: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1561041623; bh=A8LxyFw0N3pgqPM1TG7/DMPRsfTFzYb8JI28vCOEjWk=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=Pji7itl/N1RPi9Vcs5C8LP1afeymQaCPZ4X68x22EL4IYb+WSD07yN4AcBG6gGxYw gXXqtOKz68HZzA7CLARYNmE+0t1tuyqrNolu1yNhTFdPGW4DtjgZ3EmfgjncBINxHu oYv1FXDZEz9/lv+MB26EU3xm0AIOK9plJ3lPK/en3pzYuYBnaoSBMpTalzex1I2Sb1 1mlWcKaPg0K1+ao/w9/WcOIPKZUe+5J9jFDmWaDhn7X0dC47lFArWD5e5yXCcUyVj1 vmi4SflTq7pKSovYmiTVg/zdRPOGpZkVGw7zwLndpTTVt9eevVXJqDs4JgUeyU68Ap aIGbqiPa+o3tQ== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.143 Subject: [Qemu-devel] [PATCH v4 04/13] vfio: Add migration region initialization and finalize function X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, yan.y.zhao@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, yulei.zhang@intel.com, eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP - Migration functions are implemented for VFIO_DEVICE_TYPE_PCI device in this patch series. - VFIO device supports migration or not is decided based of migration region query. If migration region query is successful then migration is supported else migration is blocked. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/Makefile.objs | 2 +- hw/vfio/migration.c | 137 ++++++++++++++++++++++++++++++++++++++++++ include/hw/vfio/vfio-common.h | 14 +++++ 3 files changed, 152 insertions(+), 1 deletion(-) create mode 100644 hw/vfio/migration.c diff --git a/hw/vfio/Makefile.objs b/hw/vfio/Makefile.objs index abad8b818c9b..36033d1437c5 100644 --- a/hw/vfio/Makefile.objs +++ b/hw/vfio/Makefile.objs @@ -1,4 +1,4 @@ -obj-y += common.o spapr.o +obj-y += common.o spapr.o migration.o obj-$(CONFIG_VFIO_PCI) += pci.o pci-quirks.o display.o obj-$(CONFIG_VFIO_CCW) += ccw.o obj-$(CONFIG_VFIO_PLATFORM) += platform.o diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c new file mode 100644 index 000000000000..ba58d9253d26 --- /dev/null +++ b/hw/vfio/migration.c @@ -0,0 +1,137 @@ +/* + * Migration support for VFIO devices + * + * Copyright NVIDIA, Inc. 2019 + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include + +#include "hw/vfio/vfio-common.h" +#include "cpu.h" +#include "migration/migration.h" +#include "migration/qemu-file.h" +#include "migration/register.h" +#include "migration/blocker.h" +#include "migration/misc.h" +#include "qapi/error.h" +#include "exec/ramlist.h" +#include "exec/ram_addr.h" +#include "pci.h" + +static void vfio_migration_region_exit(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + + if (!migration) { + return; + } + + if (migration->region.buffer.size) { + vfio_region_exit(&migration->region.buffer); + vfio_region_finalize(&migration->region.buffer); + } +} + +static int vfio_migration_region_init(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + Object *obj = NULL; + int ret = -EINVAL; + + if (!migration) { + return ret; + } + + /* Migration support added for PCI device only */ + if (vbasedev->type == VFIO_DEVICE_TYPE_PCI) { + obj = vfio_pci_get_object(vbasedev); + } + + if (!obj) { + return ret; + } + + ret = vfio_region_setup(obj, vbasedev, &migration->region.buffer, + migration->region.index, "migration"); + if (ret) { + error_report("Failed to setup VFIO migration region %d: %s", + migration->region.index, strerror(-ret)); + goto err; + } + + if (!migration->region.buffer.size) { + ret = -EINVAL; + error_report("Invalid region size of VFIO migration region %d: %s", + migration->region.index, strerror(-ret)); + goto err; + } + + return 0; + +err: + vfio_migration_region_exit(vbasedev); + return ret; +} + +static int vfio_migration_init(VFIODevice *vbasedev, + struct vfio_region_info *info) +{ + int ret; + + vbasedev->migration = g_new0(VFIOMigration, 1); + vbasedev->migration->region.index = info->index; + + ret = vfio_migration_region_init(vbasedev); + if (ret) { + error_report("Failed to initialise migration region"); + return ret; + } + + return 0; +} + +/* ---------------------------------------------------------------------- */ + +int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) +{ + struct vfio_region_info *info; + int ret; + + ret = vfio_get_dev_region_info(vbasedev, VFIO_REGION_TYPE_MIGRATION, + VFIO_REGION_SUBTYPE_MIGRATION, &info); + if (ret) { + Error *local_err = NULL; + + error_setg(&vbasedev->migration_blocker, + "VFIO device doesn't support migration"); + ret = migrate_add_blocker(vbasedev->migration_blocker, &local_err); + if (local_err) { + error_propagate(errp, local_err); + error_free(vbasedev->migration_blocker); + return ret; + } + } else { + return vfio_migration_init(vbasedev, info); + } + + return 0; +} + +void vfio_migration_finalize(VFIODevice *vbasedev) +{ + if (!vbasedev->migration) { + return; + } + + if (vbasedev->migration_blocker) { + migrate_del_blocker(vbasedev->migration_blocker); + error_free(vbasedev->migration_blocker); + } + + vfio_migration_region_exit(vbasedev); + g_free(vbasedev->migration); +} diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index ef078cf60ef9..1374a03470d8 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -57,6 +57,15 @@ typedef struct VFIORegion { uint8_t nr; /* cache the region number for debug */ } VFIORegion; +typedef struct VFIOMigration { + struct { + VFIORegion buffer; + uint32_t index; + } region; + uint64_t pending_bytes; + QemuMutex lock; +} VFIOMigration; + typedef struct VFIOAddressSpace { AddressSpace *as; QLIST_HEAD(, VFIOContainer) containers; @@ -118,6 +127,8 @@ typedef struct VFIODevice { unsigned int num_irqs; unsigned int num_regions; unsigned int flags; + VFIOMigration *migration; + Error *migration_blocker; } VFIODevice; struct VFIODeviceOps { @@ -206,4 +217,7 @@ int vfio_spapr_create_window(VFIOContainer *container, int vfio_spapr_remove_window(VFIOContainer *container, hwaddr offset_within_address_space); +int vfio_migration_probe(VFIODevice *vbasedev, Error **errp); +void vfio_migration_finalize(VFIODevice *vbasedev); + #endif /* HW_VFIO_VFIO_COMMON_H */ From patchwork Thu Jun 20 14:37:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 11006897 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 48F84112C for ; Thu, 20 Jun 2019 14:59:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 37C5D20502 for ; Thu, 20 Jun 2019 14:59:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2C0F922ADC; Thu, 20 Jun 2019 14:59:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B977320502 for ; Thu, 20 Jun 2019 14:59:30 +0000 (UTC) Received: from localhost ([::1]:48806 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyXB-0008Aq-OO for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Jun 2019 10:59:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37791) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyJl-0005oP-DO for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdyJk-0005J9-0m for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:37 -0400 Received: from hqemgate14.nvidia.com ([216.228.121.143]:13220) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdyJj-0005I3-OL for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:35 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 20 Jun 2019 07:40:30 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Thu, 20 Jun 2019 07:40:31 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Thu, 20 Jun 2019 07:40:31 -0700 Received: from HQMAIL109.nvidia.com (172.20.187.15) by HQMAIL103.nvidia.com (172.20.187.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:40:30 +0000 Received: from HQMAIL106.nvidia.com (172.18.146.12) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:40:30 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 20 Jun 2019 14:40:24 +0000 From: Kirti Wankhede To: , Date: Thu, 20 Jun 2019 20:07:33 +0530 Message-ID: <1561041461-22326-6-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> References: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1561041630; bh=5AbXL8+lzah5/nkUztRbkxhK9HCmKouonbFzeonusws=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=ZRqoPgDuK0UAG/MXfOHYysJ4IRK6JpC/kfn2fRvjQ1uSlJ9+fHFAi+MUNWHbxac1R 95Zn5ZUHJR3ajVswaA9uzQ7eBeAzXkNN2nD5fWPh4ka860ZNg4SRvvZ5XFLAHRjtYc jzSQnS16b1lgVbMnmeGTJ8MrJlPjLZWj9UsejAHuD5iLDKoV6834jZHTiEkpZPAFGT ZMz5bzbrrGhjlvqclnnR0IDTR6LhFipJjDhe1XeH17QIUBgoAzXK1jTUAWKlfN41AQ QHYKg3S2gWGpOO7VNvaLoGZdibEAxLmkcXxn6DlB4ipoAMLxQz3b0lfVDsPXA0z0id hkbmuyolS+EkA== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.143 Subject: [Qemu-devel] [PATCH v4 05/13] vfio: Add VM state change handler to know state of VM X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, yan.y.zhao@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, yulei.zhang@intel.com, eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP VM state change handler gets called on change in VM's state. This is used to set VFIO device state to _RUNNING. VM state change handler, migration state change handler and log_sync listener are called asynchronously, which sometimes lead to data corruption in migration region. Initialised mutex that is used to serialize operations on migration data region during saving state. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia Reviewed-by: Dr. David Alan Gilbert --- hw/vfio/migration.c | 45 +++++++++++++++++++++++++++++++++++++++++++ include/hw/vfio/vfio-common.h | 4 ++++ 2 files changed, 49 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index ba58d9253d26..15af218c23d1 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -77,6 +77,41 @@ err: return ret; } +static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t state) +{ + VFIOMigration *migration = vbasedev->migration; + VFIORegion *region = &migration->region.buffer; + int ret = 0; + + ret = pwrite(vbasedev->fd, &state, sizeof(state), + region->fd_offset + offsetof(struct vfio_device_migration_info, + device_state)); + if (ret < 0) { + error_report("Failed to set migration state %d %s", + ret, strerror(errno)); + return ret; + } + + vbasedev->device_state = state; + return 0; +} + +static void vfio_vmstate_change(void *opaque, int running, RunState state) +{ + VFIODevice *vbasedev = opaque; + + if ((vbasedev->vm_running != running) && running) { + int ret; + + ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_RUNNING); + if (ret) { + error_report("Failed to set state RUNNING"); + } + } + + vbasedev->vm_running = running; +} + static int vfio_migration_init(VFIODevice *vbasedev, struct vfio_region_info *info) { @@ -91,6 +126,11 @@ static int vfio_migration_init(VFIODevice *vbasedev, return ret; } + qemu_mutex_init(&vbasedev->migration->lock); + + vbasedev->vm_state = qemu_add_vm_change_state_handler(vfio_vmstate_change, + vbasedev); + return 0; } @@ -127,11 +167,16 @@ void vfio_migration_finalize(VFIODevice *vbasedev) return; } + if (vbasedev->vm_state) { + qemu_del_vm_change_state_handler(vbasedev->vm_state); + } + if (vbasedev->migration_blocker) { migrate_del_blocker(vbasedev->migration_blocker); error_free(vbasedev->migration_blocker); } + qemu_mutex_destroy(&vbasedev->migration->lock); vfio_migration_region_exit(vbasedev); g_free(vbasedev->migration); } diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 1374a03470d8..f2392e97fa57 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -29,6 +29,7 @@ #ifdef CONFIG_LINUX #include #endif +#include "sysemu/sysemu.h" #define VFIO_MSG_PREFIX "vfio %s: " @@ -129,6 +130,9 @@ typedef struct VFIODevice { unsigned int flags; VFIOMigration *migration; Error *migration_blocker; + uint32_t device_state; + VMChangeStateEntry *vm_state; + int vm_running; } VFIODevice; struct VFIODeviceOps { From patchwork Thu Jun 20 14:37:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 11006949 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6632D14BB for ; Thu, 20 Jun 2019 15:03:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 57FB7286E4 for ; Thu, 20 Jun 2019 15:03:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4BCFC2885E; Thu, 20 Jun 2019 15:03:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id CC637286E4 for ; Thu, 20 Jun 2019 15:03:30 +0000 (UTC) Received: from localhost ([::1]:48858 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyb3-00033e-Nq for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Jun 2019 11:03:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37863) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyK1-0005xA-6v for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdyJz-0005YR-R5 for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:53 -0400 Received: from hqemgate14.nvidia.com ([216.228.121.143]:13233) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdyJz-0005VS-IR for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:51 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 20 Jun 2019 07:40:37 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Thu, 20 Jun 2019 07:40:37 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Thu, 20 Jun 2019 07:40:37 -0700 Received: from HQMAIL112.nvidia.com (172.18.146.18) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:40:37 +0000 Received: from HQMAIL106.nvidia.com (172.18.146.12) by HQMAIL112.nvidia.com (172.18.146.18) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:40:37 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 20 Jun 2019 14:40:31 +0000 From: Kirti Wankhede To: , Date: Thu, 20 Jun 2019 20:07:34 +0530 Message-ID: <1561041461-22326-7-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> References: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1561041637; bh=lBhaXThY2cPoc8ul1w3nQGKRFmg6IBN6bpYoTDrFtTE=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=Yqb4z967z8eaCKiK5yVu2pIZwmPoANmrsU0R3/ErqOosLfWdFewX510sgQpw0AF4R fz2NQK9z18FmWb3M+C1HUZJBtvm8nVP3UGrEbh8IwHRxKi7szt8D5s5akcp9+VnrX/ EiC1/n8QIT1tMM9vBliwGkXjAAYFMUY7AvjVozjzDCnfrqUiH1J1+biJ5tFU2Mwq42 cTISqzixXSaOpQvK5TLFRaEnrEsAUJlmy/5slrjVqJ6BYWfBnkNoxjEt/hHTbJ5X73 6yqvjL2fTvZyNm0h5xzqHaQ0N95ijyuSeIt950P+zyKjVw18danBhl57NMJ4cdkBGP QOk3B8OipHIkw== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.143 Subject: [Qemu-devel] [PATCH v4 06/13] vfio: Add migration state change notifier X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, yan.y.zhao@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, yulei.zhang@intel.com, eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Added migration state change notifier to get notification on migration state change. These states are translated to VFIO device state and conveyed to vendor driver. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/migration.c | 49 +++++++++++++++++++++++++++++++++++++++++++ include/hw/vfio/vfio-common.h | 1 + 2 files changed, 50 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 15af218c23d1..7f9858e6c995 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -112,6 +112,48 @@ static void vfio_vmstate_change(void *opaque, int running, RunState state) vbasedev->vm_running = running; } +static void vfio_migration_state_notifier(Notifier *notifier, void *data) +{ + MigrationState *s = data; + VFIODevice *vbasedev = container_of(notifier, VFIODevice, migration_state); + int ret; + + switch (s->state) { + case MIGRATION_STATUS_ACTIVE: + if (vbasedev->device_state & VFIO_DEVICE_STATE_RUNNING) { + if (vbasedev->vm_running) { + ret = vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_RUNNING | VFIO_DEVICE_STATE_SAVING); + if (ret) { + error_report("Failed to set state RUNNING and SAVING"); + } + } else { + ret = vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_SAVING); + if (ret) { + error_report("Failed to set state STOP and SAVING"); + } + } + } else { + ret = vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_RESUMING); + if (ret) { + error_report("Failed to set state RESUMING"); + } + } + return; + + case MIGRATION_STATUS_CANCELLING: + case MIGRATION_STATUS_CANCELLED: + case MIGRATION_STATUS_FAILED: + ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_RUNNING); + if (ret) { + error_report("Failed to set state RUNNING"); + } + return; + } +} + static int vfio_migration_init(VFIODevice *vbasedev, struct vfio_region_info *info) { @@ -131,6 +173,9 @@ static int vfio_migration_init(VFIODevice *vbasedev, vbasedev->vm_state = qemu_add_vm_change_state_handler(vfio_vmstate_change, vbasedev); + vbasedev->migration_state.notify = vfio_migration_state_notifier; + add_migration_state_change_notifier(&vbasedev->migration_state); + return 0; } @@ -167,6 +212,10 @@ void vfio_migration_finalize(VFIODevice *vbasedev) return; } + if (vbasedev->migration_state.notify) { + remove_migration_state_change_notifier(&vbasedev->migration_state); + } + if (vbasedev->vm_state) { qemu_del_vm_change_state_handler(vbasedev->vm_state); } diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index f2392e97fa57..1d26e6be8d48 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -133,6 +133,7 @@ typedef struct VFIODevice { uint32_t device_state; VMChangeStateEntry *vm_state; int vm_running; + Notifier migration_state; } VFIODevice; struct VFIODeviceOps { From patchwork Thu Jun 20 14:37:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 11007111 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0EE0A1580 for ; Thu, 20 Jun 2019 15:10:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0032427E01 for ; Thu, 20 Jun 2019 15:10:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E56B528707; Thu, 20 Jun 2019 15:10:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id DB1AD27E01 for ; Thu, 20 Jun 2019 15:10:03 +0000 (UTC) Received: from localhost ([::1]:48990 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyhP-0000UC-0q for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Jun 2019 11:10:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37866) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyK1-0005xC-Bg for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdyK0-0005Yp-63 for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:53 -0400 Received: from hqemgate14.nvidia.com ([216.228.121.143]:13232) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdyJz-0005VR-Ii for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:51 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 20 Jun 2019 07:40:43 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Thu, 20 Jun 2019 07:40:44 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Thu, 20 Jun 2019 07:40:44 -0700 Received: from HQMAIL106.nvidia.com (172.18.146.12) by HQMAIL104.nvidia.com (172.18.146.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:40:44 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 20 Jun 2019 14:40:37 +0000 From: Kirti Wankhede To: , Date: Thu, 20 Jun 2019 20:07:35 +0530 Message-ID: <1561041461-22326-8-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> References: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1561041644; bh=0qYiyk3jmnAKM28P+YXeeo6B7Cpq1ljyPZK55UqIrn8=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=ms/G2SMpmzMM3t45JGrNC2wvIKoSW6WHUQCmGrU9J4NbiuAJqyE7Tej2PwuAHUc9h EcOLFpJ7yzzfiHQoOwVKyKqKyzm/549FZiMAOEu/sJUdsFSrJe95kE3gFzt+aDsE5N o4hRmqJAxnmEsWUYEwzDB6FZX3oJKSD9bGlcQuyUwNR95LcLDpk1snzS9pQA0Bze0K yM3yFW4Y61GzltY5+rJdmemItt9arXc55cXsUN5DanAXY/McJvmi6ljrWo5+gy+hop 5iqymSbT2NEONyWv8L65oEbK0SkGXkYFz3r1ai22aRJl63serd6l3R+whgNUh+OeWL d1GAZXMW4GyfQ== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.143 Subject: [Qemu-devel] [PATCH v4 07/13] vfio: Register SaveVMHandlers for VFIO device X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, yan.y.zhao@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, yulei.zhang@intel.com, eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Define flags to be used as delimeter in migration file stream. Added .save_setup and .save_cleanup functions. Mapped & unmapped migration region from these functions at source during saving or pre-copy phase. Set VFIO device state depending on VM's state. During live migration, VM is running when .save_setup is called, _SAVING | _RUNNING state is set for VFIO device. During save-restore, VM is paused, _SAVING state is set for VFIO device. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/migration.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 75 insertions(+), 1 deletion(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 7f9858e6c995..fe0887c27664 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -22,6 +22,17 @@ #include "exec/ram_addr.h" #include "pci.h" +/* + * Flags used as delimiter: + * 0xffffffff => MSB 32-bit all 1s + * 0xef10 => emulated (virtual) function IO + * 0x0000 => 16-bits reserved for flags + */ +#define VFIO_MIG_FLAG_END_OF_STATE (0xffffffffef100001ULL) +#define VFIO_MIG_FLAG_DEV_CONFIG_STATE (0xffffffffef100002ULL) +#define VFIO_MIG_FLAG_DEV_SETUP_STATE (0xffffffffef100003ULL) +#define VFIO_MIG_FLAG_DEV_DATA_STATE (0xffffffffef100004ULL) + static void vfio_migration_region_exit(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; @@ -96,6 +107,69 @@ static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t state) return 0; } +/* ---------------------------------------------------------------------- */ + +static int vfio_save_setup(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + int ret; + + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_SETUP_STATE); + + if (migration->region.buffer.mmaps) { + qemu_mutex_lock_iothread(); + ret = vfio_region_mmap(&migration->region.buffer); + qemu_mutex_unlock_iothread(); + if (ret) { + error_report("Failed to mmap VFIO migration region %d: %s", + migration->region.index, strerror(-ret)); + return ret; + } + } + + if (vbasedev->vm_running) { + ret = vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_RUNNING | VFIO_DEVICE_STATE_SAVING); + if (ret) { + error_report("Failed to set state RUNNING and SAVING"); + return ret; + } + } else { + ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_SAVING); + if (ret) { + error_report("Failed to set state STOP and SAVING"); + return ret; + } + } + + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + ret = qemu_file_get_error(f); + if (ret) { + return ret; + } + + return 0; +} + +static void vfio_save_cleanup(void *opaque) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + + if (migration->region.buffer.mmaps) { + vfio_region_unmap(&migration->region.buffer); + } +} + +static SaveVMHandlers savevm_vfio_handlers = { + .save_setup = vfio_save_setup, + .save_cleanup = vfio_save_cleanup, +}; + +/* ---------------------------------------------------------------------- */ + static void vfio_vmstate_change(void *opaque, int running, RunState state) { VFIODevice *vbasedev = opaque; @@ -169,7 +243,7 @@ static int vfio_migration_init(VFIODevice *vbasedev, } qemu_mutex_init(&vbasedev->migration->lock); - + register_savevm_live(NULL, "vfio", -1, 1, &savevm_vfio_handlers, vbasedev); vbasedev->vm_state = qemu_add_vm_change_state_handler(vfio_vmstate_change, vbasedev); From patchwork Thu Jun 20 14:37:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 11007109 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8315914DB for ; Thu, 20 Jun 2019 15:10:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D98F20502 for ; Thu, 20 Jun 2019 15:10:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5C9EB286EC; Thu, 20 Jun 2019 15:10:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9E5D720502 for ; Thu, 20 Jun 2019 15:10:03 +0000 (UTC) Received: from localhost ([::1]:48988 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyhO-0000Td-Vn for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Jun 2019 11:10:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37894) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyK9-0005yv-DD for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdyK7-0005fs-CW for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:01 -0400 Received: from hqemgate14.nvidia.com ([216.228.121.143]:13243) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdyK5-0005bt-I7 for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:45:58 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 20 Jun 2019 07:40:51 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Thu, 20 Jun 2019 07:40:51 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Thu, 20 Jun 2019 07:40:51 -0700 Received: from HQMAIL106.nvidia.com (172.18.146.12) by HQMAIL106.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:40:51 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 20 Jun 2019 14:40:44 +0000 From: Kirti Wankhede To: , Date: Thu, 20 Jun 2019 20:07:36 +0530 Message-ID: <1561041461-22326-9-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> References: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1561041651; bh=AYjCKCG6U7oS8eRuxBjCX/xkqGGrD9U9l/a9tXx+r8g=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=bwVnklTu0d7EkQ8qJUwAHNY0JL9mULgsOl+Hq6/iAkXw04X07Ki4XNPhL1HAbFoEk 1LmcWThRVtA5se4QB2oZtqK2aSdbQIDk4SCv6PtXWFwuHUmDJB1FXCQQ6XL93yXhCR ux15CVt3F4gpUinTNsoQ8NOM3QsVXRK8YiWtPy4WhVJw4+MFBHg93SC40cFQfTitGx XKdYSqxAr/OfcCGV/xVcnNXYnBi07HX0DuuuG8GURhMfrLIM2aXodyGkHnQnP2EvfU Wd6vVeDvPNCyXT7rmfNEwfdjKk4vJYCYwd1O9TVuagEKmW8t9jgwWJfErGCRqhHkvc /ZB4r9p5v6gTw== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.143 Subject: [Qemu-devel] [PATCH v4 08/13] vfio: Add save state functions to SaveVMHandlers X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, yan.y.zhao@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, yulei.zhang@intel.com, eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Added .save_live_pending, .save_live_iterate and .save_live_complete_precopy functions. These functions handles pre-copy and stop-and-copy phase. In _SAVING|_RUNNING device state or pre-copy phase: - read pending_bytes - read data_offset - indicates kernel driver to write data to staging buffer which is mmapped. - read data_size - amount of data in bytes written by vendor driver in migration region. - if data section is trapped, pread() number of bytes in data_size, from data_offset. - if data section is mmaped, read mmaped buffer of size data_size. - Write data packet to file stream as below: {VFIO_MIG_FLAG_DEV_DATA_STATE, data_size, actual data, VFIO_MIG_FLAG_END_OF_STATE } In _SAVING device state or stop-and-copy phase a. read config space of device and save to migration file stream. This doesn't need to be from vendor driver. Any other special config state from driver can be saved as data in following iteration. b. read pending_bytes - indicates kernel driver to write data to staging buffer which is mmapped. c. read data_size - amount of data in bytes written by vendor driver in migration region. d. if data section is trapped, pread() from data_offset of size data_size. e. if data section is mmaped, read mmaped buffer of size data_size. f. Write data packet as below: {VFIO_MIG_FLAG_DEV_DATA_STATE, data_size, actual data} g. iterate through steps b to f until (pending_bytes > 0) h. Write {VFIO_MIG_FLAG_END_OF_STATE} .save_live_iterate runs outside the iothread lock in the migration case, which could race with asynchronous call to get dirty page list causing data corruption in mapped migration region. Mutex added here to serial migration buffer read operation. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/migration.c | 212 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 212 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index fe0887c27664..0a2f30872316 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -107,6 +107,111 @@ static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t state) return 0; } +static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + VFIORegion *region = &migration->region.buffer; + uint64_t data_offset = 0, data_size = 0; + int ret; + + ret = pread(vbasedev->fd, &data_offset, sizeof(data_offset), + region->fd_offset + offsetof(struct vfio_device_migration_info, + data_offset)); + if (ret != sizeof(data_offset)) { + error_report("Failed to get migration buffer data offset %d", + ret); + return -EINVAL; + } + + ret = pread(vbasedev->fd, &data_size, sizeof(data_size), + region->fd_offset + offsetof(struct vfio_device_migration_info, + data_size)); + if (ret != sizeof(data_size)) { + error_report("Failed to get migration buffer data size %d", + ret); + return -EINVAL; + } + + if (data_size > 0) { + void *buf = NULL; + bool buffer_mmaped = false; + + if (region->mmaps) { + int i; + + for (i = 0; i < region->nr_mmaps; i++) { + if ((data_offset >= region->mmaps[i].offset) && + (data_offset < region->mmaps[i].offset + + region->mmaps[i].size)) { + buf = region->mmaps[i].mmap + (data_offset - + region->mmaps[i].offset); + buffer_mmaped = true; + break; + } + } + } + + if (!buffer_mmaped) { + buf = g_malloc0(data_size); + ret = pread(vbasedev->fd, buf, data_size, + region->fd_offset + data_offset); + if (ret != data_size) { + error_report("Failed to get migration data %d", ret); + g_free(buf); + return -EINVAL; + } + } + + qemu_put_be64(f, data_size); + qemu_put_buffer(f, buf, data_size); + + if (!buffer_mmaped) { + g_free(buf); + } + migration->pending_bytes -= data_size; + } else { + qemu_put_be64(f, data_size); + } + + ret = qemu_file_get_error(f); + + return data_size; +} + +static int vfio_update_pending(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + VFIORegion *region = &migration->region.buffer; + uint64_t pending_bytes = 0; + int ret; + + ret = pread(vbasedev->fd, &pending_bytes, sizeof(pending_bytes), + region->fd_offset + offsetof(struct vfio_device_migration_info, + pending_bytes)); + if ((ret < 0) || (ret != sizeof(pending_bytes))) { + error_report("Failed to get pending bytes %d", ret); + migration->pending_bytes = 0; + return (ret < 0) ? ret : -EINVAL; + } + + migration->pending_bytes = pending_bytes; + return 0; +} + +static int vfio_save_device_config_state(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_CONFIG_STATE); + + if (vbasedev->type == VFIO_DEVICE_TYPE_PCI) { + vfio_pci_save_config(vbasedev, f); + } + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + return qemu_file_get_error(f); +} + /* ---------------------------------------------------------------------- */ static int vfio_save_setup(QEMUFile *f, void *opaque) @@ -163,9 +268,116 @@ static void vfio_save_cleanup(void *opaque) } } +static void vfio_save_pending(QEMUFile *f, void *opaque, + uint64_t threshold_size, + uint64_t *res_precopy_only, + uint64_t *res_compatible, + uint64_t *res_postcopy_only) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + int ret; + + ret = vfio_update_pending(vbasedev); + if (ret) { + return; + } + + if (vbasedev->device_state & VFIO_DEVICE_STATE_RUNNING) { + *res_precopy_only += migration->pending_bytes; + } else { + *res_postcopy_only += migration->pending_bytes; + } + *res_compatible += 0; +} + +static int vfio_save_iterate(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + int ret; + + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); + + qemu_mutex_lock(&migration->lock); + ret = vfio_save_buffer(f, vbasedev); + qemu_mutex_unlock(&migration->lock); + + if (ret < 0) { + error_report("vfio_save_buffer failed %s", + strerror(errno)); + return ret; + } + + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + ret = qemu_file_get_error(f); + if (ret) { + return ret; + } + + return ret; +} + +static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + int ret; + + ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_SAVING); + if (ret) { + error_report("Failed to set state STOP and SAVING"); + return ret; + } + + ret = vfio_save_device_config_state(f, opaque); + if (ret) { + return ret; + } + + ret = vfio_update_pending(vbasedev); + if (ret) { + return ret; + } + + while (migration->pending_bytes > 0) { + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); + ret = vfio_save_buffer(f, vbasedev); + if (ret < 0) { + error_report("Failed to save buffer"); + return ret; + } else if (ret == 0) { + break; + } + + ret = vfio_update_pending(vbasedev); + if (ret) { + return ret; + } + } + + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + ret = qemu_file_get_error(f); + if (ret) { + return ret; + } + + ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOPPED); + if (ret) { + error_report("Failed to set state STOPPED"); + return ret; + } + return ret; +} + static SaveVMHandlers savevm_vfio_handlers = { .save_setup = vfio_save_setup, .save_cleanup = vfio_save_cleanup, + .save_live_pending = vfio_save_pending, + .save_live_iterate = vfio_save_iterate, + .save_live_complete_precopy = vfio_save_complete_precopy, }; /* ---------------------------------------------------------------------- */ From patchwork Thu Jun 20 14:37:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 11007123 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2272C112C for ; Thu, 20 Jun 2019 15:16:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0F39D1FE95 for ; Thu, 20 Jun 2019 15:16:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F414D205F8; Thu, 20 Jun 2019 15:16:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E86111FE95 for ; Thu, 20 Jun 2019 15:16:51 +0000 (UTC) Received: from localhost ([::1]:49078 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyny-0005ib-81 for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Jun 2019 11:16:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37934) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyKF-00061c-GQ for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdyKD-0005mK-5C for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:07 -0400 Received: from hqemgate14.nvidia.com ([216.228.121.143]:13252) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdyKC-0005kJ-S4 for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:05 -0400 Received: from hqpgpgate102.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 20 Jun 2019 07:40:58 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate102.nvidia.com (PGP Universal service); Thu, 20 Jun 2019 07:40:58 -0700 X-PGP-Universal: processed; by hqpgpgate102.nvidia.com on Thu, 20 Jun 2019 07:40:58 -0700 Received: from HQMAIL112.nvidia.com (172.18.146.18) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:40:58 +0000 Received: from HQMAIL106.nvidia.com (172.18.146.12) by HQMAIL112.nvidia.com (172.18.146.18) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:40:58 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 20 Jun 2019 14:40:51 +0000 From: Kirti Wankhede To: , Date: Thu, 20 Jun 2019 20:07:37 +0530 Message-ID: <1561041461-22326-10-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> References: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1561041658; bh=8hz/KBF+f4hKDwTOMIRwStk2ND+CbU1c/DAfrF19Fl8=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=GbAk5a3UrJmRWROOARo2hYK1i+DawdbEQUX/IGs4veiavd7k9qvkUWTb/3HzlMV5F 4zgqcCptbqOW8vBh/gIt27eCcllwQuv0ExiDW//qrQgtSJ87Sp5iF2eZksFwam00Mp jHxWKAu/G8AeZTt4FYy+yVFGsA/vQHIDHDEPekT1LWJY9+C3VkOwI8j5bLj6NFfPwG 27qN60TV+bBfhV/SqcfRMCHrXKaYnqNUnhzth/fmiYkISjdm7qaC6FJnEUVZbR2FrK clqpYHnLQHtERfhTC5Yglrb4ytjbb1Q31Nb1xQblDP0TfH7I4xWmOQY4RO620hrXp0 1/5QrzhoeuHGQ== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.143 Subject: [Qemu-devel] [PATCH v4 09/13] vfio: Add load state functions to SaveVMHandlers X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, yan.y.zhao@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, yulei.zhang@intel.com, eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP During _RESUMING device state: - If Vendor driver defines mappable region, mmap migration region. - Load config state. - For data packet, till VFIO_MIG_FLAG_END_OF_STATE is not reached - read data_size from packet, read buffer of data_size - read data_offset from where QEMU should write data. if region is mmaped, write data of data_size to mmaped region. - write data_size. In case of mmapped region, write to data_size indicates kernel driver that data is written in staging buffer. - if region is trapped, pwrite() data of data_size from data_offset. - Repeat above until VFIO_MIG_FLAG_END_OF_STATE. - Unmap migration region. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/migration.c | 153 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 0a2f30872316..e4895f91761d 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -212,6 +212,22 @@ static int vfio_save_device_config_state(QEMUFile *f, void *opaque) return qemu_file_get_error(f); } +static int vfio_load_device_config_state(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + + if (vbasedev->type == VFIO_DEVICE_TYPE_PCI) { + vfio_pci_load_config(vbasedev, f); + } + + if (qemu_get_be64(f) != VFIO_MIG_FLAG_END_OF_STATE) { + error_report("Wrong end of block while loading device config space"); + return -EINVAL; + } + + return qemu_file_get_error(f); +} + /* ---------------------------------------------------------------------- */ static int vfio_save_setup(QEMUFile *f, void *opaque) @@ -372,12 +388,149 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) return ret; } +static int vfio_load_setup(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + int ret = 0; + + if (migration->region.buffer.mmaps) { + ret = vfio_region_mmap(&migration->region.buffer); + if (ret) { + error_report("Failed to mmap VFIO migration region %d: %s", + migration->region.index, strerror(-ret)); + return ret; + } + } + + ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_RESUMING); + if (ret) { + error_report("Failed to set state RESUMING"); + } + return ret; +} + +static int vfio_load_cleanup(void *opaque) +{ + vfio_save_cleanup(opaque); + return 0; +} + +static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + int ret = 0; + uint64_t data, data_size; + + data = qemu_get_be64(f); + while (data != VFIO_MIG_FLAG_END_OF_STATE) { + switch (data) { + case VFIO_MIG_FLAG_DEV_CONFIG_STATE: + { + ret = vfio_load_device_config_state(f, opaque); + if (ret) { + return ret; + } + break; + } + case VFIO_MIG_FLAG_DEV_SETUP_STATE: + { + data = qemu_get_be64(f); + if (data == VFIO_MIG_FLAG_END_OF_STATE) { + return ret; + } else { + error_report("SETUP STATE: EOS not found 0x%lx", data); + return -EINVAL; + } + break; + } + case VFIO_MIG_FLAG_DEV_DATA_STATE: + { + VFIORegion *region = &migration->region.buffer; + void *buf = NULL; + bool buffer_mmaped = false; + uint64_t data_offset = 0; + + data_size = qemu_get_be64(f); + if (data_size == 0) { + break; + } + + ret = pread(vbasedev->fd, &data_offset, sizeof(data_offset), + region->fd_offset + + offsetof(struct vfio_device_migration_info, + data_offset)); + if (ret != sizeof(data_offset)) { + error_report("Failed to get migration buffer data offset %d", + ret); + return -EINVAL; + } + + if (region->mmaps) { + int i; + + for (i = 0; i < region->nr_mmaps; i++) { + if (region->mmaps[i].mmap && + (data_offset >= region->mmaps[i].offset) && + (data_offset < region->mmaps[i].offset + + region->mmaps[i].size)) { + buf = region->mmaps[i].mmap + (data_offset - + region->mmaps[i].offset); + buffer_mmaped = true; + break; + } + } + } + + if (!buffer_mmaped) { + buf = g_malloc0(data_size); + } + + qemu_get_buffer(f, buf, data_size); + + ret = pwrite(vbasedev->fd, &data_size, sizeof(data_size), + region->fd_offset + + offsetof(struct vfio_device_migration_info, data_size)); + if (ret != sizeof(data_size)) { + error_report("Failed to set migration buffer data size %d", + ret); + return -EINVAL; + } + + if (!buffer_mmaped) { + ret = pwrite(vbasedev->fd, buf, data_size, + region->fd_offset + data_offset); + g_free(buf); + + if (ret != data_size) { + error_report("Failed to set migration buffer %d", ret); + return -EINVAL; + } + } + break; + } + } + + ret = qemu_file_get_error(f); + if (ret) { + return ret; + } + data = qemu_get_be64(f); + } + + return ret; +} + static SaveVMHandlers savevm_vfio_handlers = { .save_setup = vfio_save_setup, .save_cleanup = vfio_save_cleanup, .save_live_pending = vfio_save_pending, .save_live_iterate = vfio_save_iterate, .save_live_complete_precopy = vfio_save_complete_precopy, + .load_setup = vfio_load_setup, + .load_cleanup = vfio_load_cleanup, + .load_state = vfio_load_state, }; /* ---------------------------------------------------------------------- */ From patchwork Thu Jun 20 14:37:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 11007115 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9C10514DB for ; Thu, 20 Jun 2019 15:12:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 86B8D2888D for ; Thu, 20 Jun 2019 15:12:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 774DB2888C; Thu, 20 Jun 2019 15:12:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id EBCB72887E for ; Thu, 20 Jun 2019 15:12:40 +0000 (UTC) Received: from localhost ([::1]:49016 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyjv-00032i-VW for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Jun 2019 11:12:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37949) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyKI-000635-59 for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdyKF-0005p4-OT for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:09 -0400 Received: from hqemgate16.nvidia.com ([216.228.121.65]:1966) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdyKF-0005ng-G6 for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:07 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate16.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 20 Jun 2019 07:41:05 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Thu, 20 Jun 2019 07:41:05 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Thu, 20 Jun 2019 07:41:05 -0700 Received: from HQMAIL106.nvidia.com (172.18.146.12) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:41:05 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 20 Jun 2019 14:40:58 +0000 From: Kirti Wankhede To: , Date: Thu, 20 Jun 2019 20:07:38 +0530 Message-ID: <1561041461-22326-11-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> References: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1561041665; bh=zwCotcuQgQ/QY2vqSK3NjO/F2occ1fuAnSjeBVHdgh0=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=gX4/ipn+7skEaxCM2cfAoHylULu5+SmwkI8AM1dWf+yozVhi85VM1FR5RdMJ20JM9 h4eLr+8Sxg3K1c3WjuIp7kF1XsKSkZkp2IoxWiU8AEmr/YRFNkuz9rohr8JExu/vWJ 65T0E1rnFLqk/57Eb4rfsxEni2V6535QD5kP7EoPUpNTRVcwEfMGStCr7aX82ZFjxG xBeww31nSE7dYgx2EnAByLltUXOta8lxLDFz6gl8Ey+z5N8fDvcQuUMXSAvolVcLDd 2z+NeWqtf8Gwr3ITMTgq0Z6hvvrdL17BYXbGkvjxh5x63dlkfTEvZL5YsQWvQ9AJKL HTmkA/txpXOvQ== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.65 Subject: [Qemu-devel] [PATCH v4 10/13] vfio: Add function to get dirty page list X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, yan.y.zhao@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, yulei.zhang@intel.com, eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Dirty page tracking (.log_sync) is part of RAM copying state, where vendor driver provides the bitmap of pages which are dirtied by vendor driver through migration region and as part of RAM copy, those pages gets copied to file stream. To get dirty page bitmap: - write start address, page_size and pfn count. - read count of pfns copied. - Vendor driver should return 0 if driver doesn't have any page to report dirty in given range. - Vendor driver should return -1 to mark all pages dirty for given range. - read data_offset, where vendor driver has written bitmap. - read bitmap from the region or mmaped part of the region. This copy is iterated till page bitmap for all requested pfns are copied. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/migration.c | 119 ++++++++++++++++++++++++++++++++++++++++++ include/hw/vfio/vfio-common.h | 2 + 2 files changed, 121 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index e4895f91761d..68775b5dec11 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -228,6 +228,125 @@ static int vfio_load_device_config_state(QEMUFile *f, void *opaque) return qemu_file_get_error(f); } +void vfio_get_dirty_page_list(VFIODevice *vbasedev, + uint64_t start_pfn, + uint64_t pfn_count, + uint64_t page_size) +{ + VFIOMigration *migration = vbasedev->migration; + VFIORegion *region = &migration->region.buffer; + uint64_t count = 0; + int64_t copied_pfns = 0; + int ret; + + qemu_mutex_lock(&migration->lock); + ret = pwrite(vbasedev->fd, &start_pfn, sizeof(start_pfn), + region->fd_offset + offsetof(struct vfio_device_migration_info, + start_pfn)); + if (ret < 0) { + error_report("Failed to set dirty pages start address %d %s", + ret, strerror(errno)); + goto dpl_unlock; + } + + ret = pwrite(vbasedev->fd, &page_size, sizeof(page_size), + region->fd_offset + offsetof(struct vfio_device_migration_info, + page_size)); + if (ret < 0) { + error_report("Failed to set dirty page size %d %s", + ret, strerror(errno)); + goto dpl_unlock; + } + + ret = pwrite(vbasedev->fd, &pfn_count, sizeof(pfn_count), + region->fd_offset + offsetof(struct vfio_device_migration_info, + total_pfns)); + if (ret < 0) { + error_report("Failed to set dirty page total pfns %d %s", + ret, strerror(errno)); + goto dpl_unlock; + } + + do { + uint64_t bitmap_size, data_offset = 0; + void *buf = NULL; + bool buffer_mmaped = false; + + /* Read copied dirty pfns */ + ret = pread(vbasedev->fd, &copied_pfns, sizeof(copied_pfns), + region->fd_offset + offsetof(struct vfio_device_migration_info, + copied_pfns)); + if (ret < 0) { + error_report("Failed to get dirty pages bitmap count %d %s", + ret, strerror(errno)); + goto dpl_unlock; + } + + if (copied_pfns == 0) { + /* + * copied_pfns could be 0 if driver doesn't have any page to + * report dirty in given range + */ + break; + } else if (copied_pfns == -1) { + /* Mark all pages dirty for this range */ + cpu_physical_memory_set_dirty_range(start_pfn * page_size, + pfn_count * page_size, + DIRTY_MEMORY_MIGRATION); + break; + } + + bitmap_size = (BITS_TO_LONGS(copied_pfns) + 1) * sizeof(unsigned long); + + ret = pread(vbasedev->fd, &data_offset, sizeof(data_offset), + region->fd_offset + offsetof(struct vfio_device_migration_info, + data_offset)); + if (ret != sizeof(data_offset)) { + error_report("Failed to get migration buffer data offset %d", + ret); + goto dpl_unlock; + } + + if (region->mmaps) { + int i; + for (i = 0; i < region->nr_mmaps; i++) { + if ((region->mmaps[i].offset >= data_offset) && + (data_offset < region->mmaps[i].offset + + region->mmaps[i].size)) { + buf = region->mmaps[i].mmap + (data_offset - + region->mmaps[i].offset); + buffer_mmaped = true; + break; + } + } + } + + if (!buffer_mmaped) { + buf = g_malloc0(bitmap_size); + + ret = pread(vbasedev->fd, buf, bitmap_size, + region->fd_offset + data_offset); + if (ret != bitmap_size) { + error_report("Failed to get dirty pages bitmap %d", ret); + g_free(buf); + goto dpl_unlock; + } + } + + cpu_physical_memory_set_dirty_lebitmap((unsigned long *)buf, + (start_pfn + count) * page_size, + copied_pfns); + count += copied_pfns; + + if (!buffer_mmaped) { + g_free(buf); + } + } while (count < pfn_count); + +dpl_unlock: + qemu_mutex_unlock(&migration->lock); +} + /* ---------------------------------------------------------------------- */ static int vfio_save_setup(QEMUFile *f, void *opaque) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 1d26e6be8d48..423d6dbccace 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -224,5 +224,7 @@ int vfio_spapr_remove_window(VFIOContainer *container, int vfio_migration_probe(VFIODevice *vbasedev, Error **errp); void vfio_migration_finalize(VFIODevice *vbasedev); +void vfio_get_dirty_page_list(VFIODevice *vbasedev, uint64_t start_pfn, + uint64_t pfn_count, uint64_t page_size); #endif /* HW_VFIO_VFIO_COMMON_H */ From patchwork Thu Jun 20 14:37:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 11007141 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1636E112C for ; Thu, 20 Jun 2019 15:19:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0538927861 for ; Thu, 20 Jun 2019 15:19:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ED2D328306; Thu, 20 Jun 2019 15:19:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9571727861 for ; Thu, 20 Jun 2019 15:19:22 +0000 (UTC) Received: from localhost ([::1]:49088 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyqP-0007Ed-GJ for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Jun 2019 11:19:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38014) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyKS-00068E-9H for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdyKQ-00060Q-0k for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:20 -0400 Received: from hqemgate14.nvidia.com ([216.228.121.143]:13269) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdyKP-0005xN-NA for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:17 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 20 Jun 2019 07:41:11 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Thu, 20 Jun 2019 07:41:12 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Thu, 20 Jun 2019 07:41:12 -0700 Received: from HQMAIL102.nvidia.com (172.18.146.10) by HQMAIL108.nvidia.com (172.18.146.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:41:11 +0000 Received: from HQMAIL106.nvidia.com (172.18.146.12) by HQMAIL102.nvidia.com (172.18.146.10) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:41:11 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 20 Jun 2019 14:41:05 +0000 From: Kirti Wankhede To: , Date: Thu, 20 Jun 2019 20:07:39 +0530 Message-ID: <1561041461-22326-12-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> References: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1561041672; bh=gFFxatm5vEcsYi2oQ1vg+wARzP2JUq9a8FiNQRij5Bw=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=SnnMH2YglZRzMIBCVrjeNElZ3SC/hz3GOfTNyRuNuZHa0blzth1Mvio0QAjvOo4sP TUIJoSPJ540jXjGawSwFsqDqk1uiaokAox37wBeb/oA9HEG8MAl4EleXY8lNqd1Cqd wjTaWmoA8AYljNeqhmlKczNPp3PDxKxmeS+baDP4xgV32+ZWd6uBl1CZhkl7G5a6Rl doZ3bJEj+q3HQD5aP97ZtIRoWn2rrNyZuansccRfnBkNzcj4kVfdSRYW74e4dF13Fn fCk4jvZ/nqpNHMw8lSAjIZgIWlCDTk4T1lf5yXxg+m90pzhy0Gzi7O/ZbxI35awTr/ nW1OncRMPwPXg== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.143 Subject: [Qemu-devel] [PATCH v4 11/13] vfio: Add vfio_listerner_log_sync to mark dirty pages X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, yan.y.zhao@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, yulei.zhang@intel.com, eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP vfio_listerner_log_sync gets list of dirty pages from vendor driver and mark those pages dirty when in _SAVING state. Return early for the RAM block section of mapped MMIO region. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/common.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index de74dae8d6a6..d5ee35c95e76 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -36,6 +36,7 @@ #include "sysemu/kvm.h" #include "trace.h" #include "qapi/error.h" +#include "migration/migration.h" VFIOGroupList vfio_group_list = QLIST_HEAD_INITIALIZER(vfio_group_list); @@ -794,9 +795,43 @@ static void vfio_listener_region_del(MemoryListener *listener, } } +static void vfio_listerner_log_sync(MemoryListener *listener, + MemoryRegionSection *section) +{ + uint64_t start_addr, size, pfn_count; + VFIOGroup *group; + VFIODevice *vbasedev; + + if (memory_region_is_ram_device(section->mr)) { + return; + } + + QLIST_FOREACH(group, &vfio_group_list, next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + if (vbasedev->device_state & VFIO_DEVICE_STATE_SAVING) { + continue; + } else { + return; + } + } + } + + start_addr = TARGET_PAGE_ALIGN(section->offset_within_address_space); + size = int128_get64(section->size); + pfn_count = size >> TARGET_PAGE_BITS; + + QLIST_FOREACH(group, &vfio_group_list, next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + vfio_get_dirty_page_list(vbasedev, start_addr >> TARGET_PAGE_BITS, + pfn_count, TARGET_PAGE_SIZE); + } + } +} + static const MemoryListener vfio_memory_listener = { .region_add = vfio_listener_region_add, .region_del = vfio_listener_region_del, + .log_sync = vfio_listerner_log_sync, }; static void vfio_listener_release(VFIOContainer *container) From patchwork Thu Jun 20 14:37:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 11007117 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3CE1614E5 for ; Thu, 20 Jun 2019 15:14:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2171B2876D for ; Thu, 20 Jun 2019 15:14:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 15D262887E; Thu, 20 Jun 2019 15:14:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A64422876D for ; Thu, 20 Jun 2019 15:14:20 +0000 (UTC) Received: from localhost ([::1]:49064 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdylX-0004Bj-Sc for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Jun 2019 11:14:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38066) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyKa-0006DL-2x for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdyKY-0006DK-SZ for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:27 -0400 Received: from hqemgate15.nvidia.com ([216.228.121.64]:6456) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdyKY-0006Al-J7 for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:26 -0400 Received: from hqpgpgate102.nvidia.com (Not Verified[216.228.121.13]) by hqemgate15.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 20 Jun 2019 07:41:20 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate102.nvidia.com (PGP Universal service); Thu, 20 Jun 2019 07:41:18 -0700 X-PGP-Universal: processed; by hqpgpgate102.nvidia.com on Thu, 20 Jun 2019 07:41:18 -0700 Received: from HQMAIL106.nvidia.com (172.18.146.12) by HQMAIL103.nvidia.com (172.20.187.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:41:18 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 20 Jun 2019 14:41:12 +0000 From: Kirti Wankhede To: , Date: Thu, 20 Jun 2019 20:07:40 +0530 Message-ID: <1561041461-22326-13-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> References: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1561041680; bh=ppqMB6UfbPIBKPvs9O0vDjVXpwPZivlPc7fIHiWjiVM=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=X8Z09RjKoRqGKkpN+PGynLe4AKAuPaFecLTTuLg5XWFivMArbRnWqAA0RJJ8UHJob AF+jFtTPQbjxwYrWwFMResg0puXyZwRXDRP8l5LUv3uYjhmfhf6TI52Xs5ebzQ9gJ5 prhmkMfl9ApQqfZWZjlJh0D2EvBLDP5V3R6BUOD7iiUVLDiyI8r4SK+tzyebDXRkKK 9oWf/COY8lS8+6SEcz4bkTW+NlYT27WVMPL6A75uR0N/Hw+kITKz5RwahOVGttxhbZ boVEppw654g5R9kyw2BpIjH6tXbTyTg9/nxKLjQCb14q4yYsVX8iU0vDo+J1iUY4IG yQHRt+AICsBaQ== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.64 Subject: [Qemu-devel] [PATCH v4 12/13] vfio: Make vfio-pci device migration capable. X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, yan.y.zhao@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, yulei.zhang@intel.com, eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Call vfio_migration_probe() and vfio_migration_finalize() functions for vfio-pci device to enable migration for vfio PCI device. Removed vfio_pci_vmstate structure. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/pci.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 09a0821a5b1c..b230b0ab9282 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2839,6 +2839,7 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) vdev->vbasedev.ops = &vfio_pci_ops; vdev->vbasedev.type = VFIO_DEVICE_TYPE_PCI; vdev->vbasedev.dev = DEVICE(vdev); + vdev->vbasedev.device_state = VFIO_DEVICE_STATE_STOPPED; tmp = g_strdup_printf("%s/iommu_group", vdev->vbasedev.sysfsdev); len = readlink(tmp, group_path, sizeof(group_path)); @@ -3099,6 +3100,11 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) } } + ret = vfio_migration_probe(&vdev->vbasedev, errp); + if (ret) { + error_report("Failed to setup migration region"); + } + vfio_register_err_notifier(vdev); vfio_register_req_notifier(vdev); vfio_setup_resetfn_quirk(vdev); @@ -3118,6 +3124,7 @@ static void vfio_instance_finalize(Object *obj) VFIOPCIDevice *vdev = PCI_VFIO(obj); VFIOGroup *group = vdev->vbasedev.group; + vdev->vbasedev.device_state = VFIO_DEVICE_STATE_STOPPED; vfio_display_finalize(vdev); vfio_bars_finalize(vdev); g_free(vdev->emulated_config_bits); @@ -3146,6 +3153,7 @@ static void vfio_exitfn(PCIDevice *pdev) } vfio_teardown_msi(vdev); vfio_bars_exit(vdev); + vfio_migration_finalize(&vdev->vbasedev); } static void vfio_pci_reset(DeviceState *dev) @@ -3254,11 +3262,6 @@ static Property vfio_pci_dev_properties[] = { DEFINE_PROP_END_OF_LIST(), }; -static const VMStateDescription vfio_pci_vmstate = { - .name = "vfio-pci", - .unmigratable = 1, -}; - static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) { DeviceClass *dc = DEVICE_CLASS(klass); @@ -3266,7 +3269,6 @@ static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) dc->reset = vfio_pci_reset; dc->props = vfio_pci_dev_properties; - dc->vmsd = &vfio_pci_vmstate; dc->desc = "VFIO-based PCI device assignment"; set_bit(DEVICE_CATEGORY_MISC, dc->categories); pdc->realize = vfio_realize; From patchwork Thu Jun 20 14:37:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 11007143 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AAD041398 for ; Thu, 20 Jun 2019 15:22:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9B2022838F for ; Thu, 20 Jun 2019 15:22:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8F43B28880; Thu, 20 Jun 2019 15:22:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id EB49828861 for ; Thu, 20 Jun 2019 15:22:38 +0000 (UTC) Received: from localhost ([::1]:49164 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyta-0001fH-3f for patchwork-qemu-devel@patchwork.kernel.org; Thu, 20 Jun 2019 11:22:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38088) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hdyKd-0006FL-F2 for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hdyKa-0006FE-8N for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:31 -0400 Received: from hqemgate15.nvidia.com ([216.228.121.64]:6458) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hdyKZ-0006DY-U4 for qemu-devel@nongnu.org; Thu, 20 Jun 2019 10:46:28 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate15.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 20 Jun 2019 07:41:27 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Thu, 20 Jun 2019 07:41:25 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Thu, 20 Jun 2019 07:41:25 -0700 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL108.nvidia.com (172.18.146.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:41:24 +0000 Received: from HQMAIL106.nvidia.com (172.18.146.12) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 20 Jun 2019 14:41:25 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 20 Jun 2019 14:41:18 +0000 From: Kirti Wankhede To: , Date: Thu, 20 Jun 2019 20:07:41 +0530 Message-ID: <1561041461-22326-14-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> References: <1561041461-22326-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1561041687; bh=1S/KxQ69YuW1Nvqhm0/2EWNCTQLOARisCkaV+xZdDpg=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=LyUsCduPrLIevNwCWZHM9TO9p32gf7IwKza1iyy35H/vBGZVnSVk4+ePtwmHOVF9O UAA2v0rac2DH/Zc7ua1tOJgiqT8voZFSokIwwwHvy65feGKY4C9LUMB7e6o8P9HmD4 EjWcgjfeLlZRV2RepIL+J/iwvSqJvXo+uCkviRb8RcD5QBrNS2hk2HqnZR+azWdpOw 6SaF1y22bIf0yK2ehY+5UePPviNsltuJtW5MrEGOmL4YrqUUeKWWfNlaAugorh7BR8 kH7yTYoUUnpmsBhHF2+re/7G2m4Hb8cq1tpnk3s1zzQUYK23uIrRklHK6ASji4eGoQ 7kF5BFetqWL6Q== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.64 Subject: [Qemu-devel] [PATCH v4 13/13] vfio: Add trace events in migration code path X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, yan.y.zhao@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, yulei.zhang@intel.com, eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/migration.c | 26 ++++++++++++++++++++++++++ hw/vfio/trace-events | 18 ++++++++++++++++++ 2 files changed, 44 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 68775b5dec11..70c03f1a969f 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -21,6 +21,7 @@ #include "exec/ramlist.h" #include "exec/ram_addr.h" #include "pci.h" +#include "trace.h" /* * Flags used as delimiter: @@ -104,6 +105,7 @@ static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t state) } vbasedev->device_state = state; + trace_vfio_migration_set_state(vbasedev->name, state); return 0; } @@ -173,6 +175,8 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev) qemu_put_be64(f, data_size); } + trace_vfio_save_buffer(vbasedev->name, data_offset, data_size, + migration->pending_bytes); ret = qemu_file_get_error(f); return data_size; @@ -195,6 +199,7 @@ static int vfio_update_pending(VFIODevice *vbasedev) } migration->pending_bytes = pending_bytes; + trace_vfio_update_pending(vbasedev->name, pending_bytes); return 0; } @@ -209,6 +214,8 @@ static int vfio_save_device_config_state(QEMUFile *f, void *opaque) } qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + trace_vfio_save_device_config_state(vbasedev->name); + return qemu_file_get_error(f); } @@ -225,6 +232,7 @@ static int vfio_load_device_config_state(QEMUFile *f, void *opaque) return -EINVAL; } + trace_vfio_load_device_config_state(vbasedev->name); return qemu_file_get_error(f); } @@ -343,6 +351,9 @@ void vfio_get_dirty_page_list(VFIODevice *vbasedev, } } while (count < pfn_count); + trace_vfio_get_dirty_page_list(vbasedev->name, start_pfn, pfn_count, + page_size); + dpl_unlock: qemu_mutex_unlock(&migration->lock); } @@ -390,6 +401,7 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) return ret; } + trace_vfio_save_setup(vbasedev->name); return 0; } @@ -401,6 +413,7 @@ static void vfio_save_cleanup(void *opaque) if (migration->region.buffer.mmaps) { vfio_region_unmap(&migration->region.buffer); } + trace_vfio_cleanup(vbasedev->name); } static void vfio_save_pending(QEMUFile *f, void *opaque, @@ -424,6 +437,7 @@ static void vfio_save_pending(QEMUFile *f, void *opaque, *res_postcopy_only += migration->pending_bytes; } *res_compatible += 0; + trace_vfio_save_pending(vbasedev->name); } static int vfio_save_iterate(QEMUFile *f, void *opaque) @@ -451,6 +465,7 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque) return ret; } + trace_vfio_save_iterate(vbasedev->name); return ret; } @@ -504,6 +519,8 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) error_report("Failed to set state STOPPED"); return ret; } + + trace_vfio_save_complete_precopy(vbasedev->name); return ret; } @@ -544,6 +561,9 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) data = qemu_get_be64(f); while (data != VFIO_MIG_FLAG_END_OF_STATE) { + + trace_vfio_load_state(vbasedev->name, data); + switch (data) { case VFIO_MIG_FLAG_DEV_CONFIG_STATE: { @@ -627,6 +647,8 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) return -EINVAL; } } + trace_vfio_load_state_device_data(vbasedev->name, data_offset, + data_size); break; } } @@ -668,6 +690,7 @@ static void vfio_vmstate_change(void *opaque, int running, RunState state) } vbasedev->vm_running = running; + trace_vfio_vmstate_change(vbasedev->name, running); } static void vfio_migration_state_notifier(Notifier *notifier, void *data) @@ -676,6 +699,8 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data) VFIODevice *vbasedev = container_of(notifier, VFIODevice, migration_state); int ret; + trace_vfio_migration_state_notifier(vbasedev->name, s->state); + switch (s->state) { case MIGRATION_STATUS_ACTIVE: if (vbasedev->device_state & VFIO_DEVICE_STATE_RUNNING) { @@ -758,6 +783,7 @@ int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) return ret; } } else { + trace_vfio_migration_probe(vbasedev->name, info->index); return vfio_migration_init(vbasedev, info); } diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 8cdc27946cb8..b1f19ae7a806 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -143,3 +143,21 @@ vfio_display_edid_link_up(void) "" vfio_display_edid_link_down(void) "" vfio_display_edid_update(uint32_t prefx, uint32_t prefy) "%ux%u" vfio_display_edid_write_error(void) "" + +# migration.c +vfio_migration_set_state(char *name, uint32_t state) " (%s) state %d" +vfio_save_buffer(char *name, uint64_t data_offset, uint64_t data_size, uint64_t pending) " (%s), Offset 0x%"PRIx64" size 0x%"PRIx64" pending 0x%"PRIx64 +vfio_update_pending(char *name, uint64_t pending) " (%s), pending 0x%"PRIx64 +vfio_save_device_config_state(char *name) " (%s)" +vfio_load_device_config_state(char *name) " (%s)" +vfio_get_dirty_page_list(char *name, uint64_t start, uint64_t pfn_count, uint64_t page_size) " (%s) start 0x%"PRIx64" pfn_count 0x%"PRIx64 " page size 0x%"PRIx64 +vfio_save_setup(char *name) " (%s)" +vfio_cleanup(char *name) " (%s)" +vfio_save_pending(char *name) " (%s)" +vfio_save_iterate(char *name) " (%s)" +vfio_save_complete_precopy(char *name) " (%s)" +vfio_load_state(char *name, uint64_t data) " (%s) data 0x%"PRIx64 +vfio_load_state_device_data(char *name, uint64_t data_offset, uint64_t data_size) " (%s), Offset 0x%"PRIx64" size 0x%"PRIx64 +vfio_vmstate_change(char *name, int running) " (%s) running %d" +vfio_migration_state_notifier(char *name, int state) " (%s) state %d" +vfio_migration_probe(char *name, uint32_t index) " (%s) Region %d"