From patchwork Tue May 24 06:18:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rao, Lei" X-Patchwork-Id: 12859703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8007DC433F5 for ; Tue, 24 May 2022 06:22:51 +0000 (UTC) Received: from localhost ([::1]:40586 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ntNwQ-0002Od-Jx for qemu-devel@archiver.kernel.org; Tue, 24 May 2022 02:22:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38406) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNt6-0007ti-Qh for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:24 -0400 Received: from mga17.intel.com ([192.55.52.151]:41948) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNsr-0000Fd-JQ for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653373149; x=1684909149; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=uvCzKu9Uu5PXiNmmvrkHXtmIZIyrXKybleaiUQQ6ZLg=; b=QW4MnK2B1BBp659TlpNtHszUkdeSDejc3bDzygW6F52Dy+hB1Mxv+/1p kNoTnzaTYCFq3URasF530H/PIL4l5muVyH6Y+PN/ADTvJ+i0sVyVVI9w+ tLePe3QtdqVYJZd1wKvYx52vPD4/3USvkoxyWRndr+q1sgEOwn+jKbj9I wtElu34TuT/kPfhmnP6ClhJODE/qXnYCRrq8W3wj2k2sI+MTiSs6ww1i6 YfpZ+17nA+amzKaennHzlYPbwRWwr8+Ch68cP2DG6GveSSgGHwDwV+HAQ QFvzv6R+diHNrKMp1FaHxGwOqeyZB4307L8pqdDvJSLCj9URxWbEZPS7P A==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="253943141" X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="253943141" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 23:19:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="601059609" Received: from leirao-pc.bj.intel.com ([10.238.156.102]) by orsmga008.jf.intel.com with ESMTP; 23 May 2022 23:19:02 -0700 From: Lei Rao To: alex.williamson@redhat.com, kevin.tian@intel.com, eddie.dong@intel.com, jason.zeng@intel.com, quintela@redhat.com, dgilbert@redhat.com, yadong.li@intel.com, yi.l.liu@intel.com Cc: qemu-devel@nongnu.org, Lei Rao Subject: [RFC PATCH 01/13] vfio/migration: put together checks of migration initialization conditions Date: Tue, 24 May 2022 14:18:36 +0800 Message-Id: <20220524061848.1615706-2-lei.rao@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220524061848.1615706-1-lei.rao@intel.com> References: <20220524061848.1615706-1-lei.rao@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=lei.rao@intel.com; helo=mga17.intel.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Current VFIO live migration initialization code is tightly coupled with local migration region handling. It is necessary to decouple it to facilitate the introduction of a generic VFIO live migration framework so that other approaches can be possible besides the In-Band approach. This patch puts various checks of migration initialization conditions into one function vfio_migration_check(). Signed-off-by: Lei Rao Reviewed-by: Eddie Dong --- hw/vfio/migration.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index a6ad1f8945..770f535e81 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -787,6 +787,21 @@ static void vfio_migration_exit(VFIODevice *vbasedev) vbasedev->migration = NULL; } +static int vfio_migration_check(VFIODevice *vbasedev) +{ + VFIOContainer *container = vbasedev->group->container; + + if (!vbasedev->enable_migration || !container->dirty_pages_supported) { + return -EINVAL; + } + + if (!vbasedev->ops->vfio_get_object) { + return -EINVAL; + } + + return 0; +} + static int vfio_migration_init(VFIODevice *vbasedev, struct vfio_region_info *info) { @@ -796,10 +811,6 @@ static int vfio_migration_init(VFIODevice *vbasedev, char id[256] = ""; g_autofree char *path = NULL, *oid = NULL; - if (!vbasedev->ops->vfio_get_object) { - return -EINVAL; - } - obj = vbasedev->ops->vfio_get_object(vbasedev); if (!obj) { return -EINVAL; @@ -857,11 +868,11 @@ int64_t vfio_mig_bytes_transferred(void) int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) { - VFIOContainer *container = vbasedev->group->container; struct vfio_region_info *info = NULL; int ret = -ENOTSUP; - if (!vbasedev->enable_migration || !container->dirty_pages_supported) { + ret = vfio_migration_check(vbasedev); + if (ret) { goto add_blocker; } From patchwork Tue May 24 06:18:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rao, Lei" X-Patchwork-Id: 12859702 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B5F82C433FE for ; Tue, 24 May 2022 06:22:46 +0000 (UTC) Received: from localhost ([::1]:40354 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ntNwL-0002ED-Tn for qemu-devel@archiver.kernel.org; Tue, 24 May 2022 02:22:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38352) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNsz-0007jE-S7 for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:17 -0400 Received: from mga17.intel.com ([192.55.52.151]:41958) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNst-0000Fo-9f for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653373151; x=1684909151; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ExdLP7W0V6Rn/8DcMpfXQ8tmFgDyolan+OBovJ9wppc=; b=DvNoS1Xl06TASXaxSc0Ni3Xw1LQZ54crQV5r1vQGTWNLK/BqR0EGZIPM svM5kfBjFRgx7ms0utyp5aCJ8SejKCcLFvxRiXFaowHbOYucA0Q19AW2p il73yhTcoxdEkgfNI4T8VSOhwhMAiqfiDAIeN+PZt38zocfjfhD2e4Jxu 7Voxa7bZ6LhYib/VffNolcomTj/8Jv7Ic9mSjXn/jyTWw2kColxHW0Dhi lGrzTJMW1tU5ZdgkUpb5vxg+IJHO3b78n26i8CmPHmKVevXAYR8XgysQt XIaHHrwChUWP5InmrOVIyucPymWiiqZZG9a7720xNYBW1P6Fk65vW7Aqp A==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="253943163" X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="253943163" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 23:19:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="601059640" Received: from leirao-pc.bj.intel.com ([10.238.156.102]) by orsmga008.jf.intel.com with ESMTP; 23 May 2022 23:19:06 -0700 From: Lei Rao To: alex.williamson@redhat.com, kevin.tian@intel.com, eddie.dong@intel.com, jason.zeng@intel.com, quintela@redhat.com, dgilbert@redhat.com, yadong.li@intel.com, yi.l.liu@intel.com Cc: qemu-devel@nongnu.org, Lei Rao Subject: [RFC PATCH 02/13] vfio/migration: move migration struct allocation out of vfio_migration_init Date: Tue, 24 May 2022 14:18:37 +0800 Message-Id: <20220524061848.1615706-3-lei.rao@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220524061848.1615706-1-lei.rao@intel.com> References: <20220524061848.1615706-1-lei.rao@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=lei.rao@intel.com; helo=mga17.intel.com X-Spam_score_int: -44 X-Spam_score: -4.5 X-Spam_bar: ---- X-Spam_report: (-4.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Migration struct is a common data structure. Memory allocation of migration struct is not unique to In-Band approach. So, move it from vfio_migration_init() to vfio_migration_probe(). Signed-off-by: Lei Rao --- hw/vfio/migration.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 770f535e81..11ce87bb1a 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -807,17 +807,15 @@ static int vfio_migration_init(VFIODevice *vbasedev, { int ret; Object *obj; - VFIOMigration *migration; char id[256] = ""; g_autofree char *path = NULL, *oid = NULL; + VFIOMigration *migration = vbasedev->migration; obj = vbasedev->ops->vfio_get_object(vbasedev); if (!obj) { return -EINVAL; } - vbasedev->migration = g_new0(VFIOMigration, 1); - ret = vfio_region_setup(obj, vbasedev, &vbasedev->migration->region, info->index, "migration"); if (ret) { @@ -833,9 +831,6 @@ static int vfio_migration_init(VFIODevice *vbasedev, goto err; } - migration = vbasedev->migration; - migration->vbasedev = vbasedev; - oid = vmstate_if_get_id(VMSTATE_IF(DEVICE(obj))); if (oid) { path = g_strdup_printf("%s/vfio", oid); @@ -876,6 +871,9 @@ int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) goto add_blocker; } + vbasedev->migration = g_new0(VFIOMigration, 1); + vbasedev->migration->vbasedev = vbasedev; + ret = vfio_get_dev_region_info(vbasedev, VFIO_REGION_TYPE_MIGRATION_DEPRECATED, VFIO_REGION_SUBTYPE_MIGRATION_DEPRECATED, @@ -903,6 +901,8 @@ add_blocker: error_free(vbasedev->migration_blocker); vbasedev->migration_blocker = NULL; } + g_free(vbasedev->migration); + vbasedev->migration = NULL; return ret; } From patchwork Tue May 24 06:18:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rao, Lei" X-Patchwork-Id: 12859701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DD301C433F5 for ; Tue, 24 May 2022 06:22:44 +0000 (UTC) Received: from localhost ([::1]:40152 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ntNwJ-00025Z-Jk for qemu-devel@archiver.kernel.org; Tue, 24 May 2022 02:22:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38368) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNt0-0007jR-BE for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:18 -0400 Received: from mga17.intel.com ([192.55.52.151]:41945) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNsy-0000EA-Fv for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653373156; x=1684909156; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Vv7gnOasp6/ZZp3HaPei/aQKyYTdtUk1origL7k3ekM=; b=Mt1Yiby3fSnbReYn9ky7DEKB8Rs2lkh3QFA6p2HVcFPZP228V2JqElxJ 0m2jGh5J9bUQbAuy8evckEj3z+h2F56GgxxLjqqPM3vXWkSriJkpreqCs goPTA+RvtFOowaUG9et8KyImJCamT3Bww+zgDz3T/DT0RRrKGY2/NvaW4 BmgQLWFvRcr5lCC06zVXcpcvYuKuF47lwq5DfFRsHmN8bMsooz1cSORgy OvbwrSwPKljcTb4N5Rl3eQnzfcxC5K8lP98mF3y4i9t/vVVCoNBVxSsTH sJDPSgPVuOKO5YdyLeWahXeIK8SkBWub5e0SDVimzC/eFlMNb91xskKf+ A==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="253943179" X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="253943179" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 23:19:14 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="601059662" Received: from leirao-pc.bj.intel.com ([10.238.156.102]) by orsmga008.jf.intel.com with ESMTP; 23 May 2022 23:19:11 -0700 From: Lei Rao To: alex.williamson@redhat.com, kevin.tian@intel.com, eddie.dong@intel.com, jason.zeng@intel.com, quintela@redhat.com, dgilbert@redhat.com, yadong.li@intel.com, yi.l.liu@intel.com Cc: qemu-devel@nongnu.org, Lei Rao Subject: [RFC PATCH 03/13] vfio/migration: move vfio_get_dev_region_info out of vfio_migration_probe Date: Tue, 24 May 2022 14:18:38 +0800 Message-Id: <20220524061848.1615706-4-lei.rao@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220524061848.1615706-1-lei.rao@intel.com> References: <20220524061848.1615706-1-lei.rao@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=lei.rao@intel.com; helo=mga17.intel.com X-Spam_score_int: -44 X-Spam_score: -4.5 X-Spam_bar: ---- X-Spam_report: (-4.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" vfio_get_dev_region_info() in vfio_migration_probe() is a specific operation of In-Band approach. So, it's better to put it in vfio_migration_init() because most of the setup of In-Band approach are handled there. The vfio_migration_init will be rename to vfio_migration_probe_local(). Signed-off-by: Lei Rao --- hw/vfio/migration.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 11ce87bb1a..e61c19171a 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -802,12 +802,12 @@ static int vfio_migration_check(VFIODevice *vbasedev) return 0; } -static int vfio_migration_init(VFIODevice *vbasedev, - struct vfio_region_info *info) +static int vfio_migration_init(VFIODevice *vbasedev) { int ret; Object *obj; char id[256] = ""; + struct vfio_region_info *info = NULL; g_autofree char *path = NULL, *oid = NULL; VFIOMigration *migration = vbasedev->migration; @@ -816,6 +816,14 @@ static int vfio_migration_init(VFIODevice *vbasedev, return -EINVAL; } + ret = vfio_get_dev_region_info(vbasedev, + VFIO_REGION_TYPE_MIGRATION_DEPRECATED, + VFIO_REGION_SUBTYPE_MIGRATION_DEPRECATED, + &info); + if (ret) { + return -EINVAL; + } + ret = vfio_region_setup(obj, vbasedev, &vbasedev->migration->region, info->index, "migration"); if (ret) { @@ -847,10 +855,14 @@ static int vfio_migration_init(VFIODevice *vbasedev, vbasedev); migration->migration_state.notify = vfio_migration_state_notifier; add_migration_state_change_notifier(&migration->migration_state); + + trace_vfio_migration_probe(vbasedev->name, info->index); + g_free(info); return 0; err: vfio_migration_exit(vbasedev); + g_free(info); return ret; } @@ -863,7 +875,6 @@ int64_t vfio_mig_bytes_transferred(void) int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) { - struct vfio_region_info *info = NULL; int ret = -ENOTSUP; ret = vfio_migration_check(vbasedev); @@ -874,27 +885,16 @@ int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) vbasedev->migration = g_new0(VFIOMigration, 1); vbasedev->migration->vbasedev = vbasedev; - ret = vfio_get_dev_region_info(vbasedev, - VFIO_REGION_TYPE_MIGRATION_DEPRECATED, - VFIO_REGION_SUBTYPE_MIGRATION_DEPRECATED, - &info); - if (ret) { - goto add_blocker; - } - - ret = vfio_migration_init(vbasedev, info); + ret = vfio_migration_init(vbasedev); if (ret) { goto add_blocker; } - trace_vfio_migration_probe(vbasedev->name, info->index); - g_free(info); return 0; add_blocker: error_setg(&vbasedev->migration_blocker, "VFIO device doesn't support migration"); - g_free(info); ret = migrate_add_blocker(vbasedev->migration_blocker, errp); if (ret < 0) { From patchwork Tue May 24 06:18:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rao, Lei" X-Patchwork-Id: 12859717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C4D7CC433EF for ; Tue, 24 May 2022 06:32:19 +0000 (UTC) Received: from localhost ([::1]:48492 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ntO5a-0007x0-Cn for qemu-devel@archiver.kernel.org; Tue, 24 May 2022 02:32:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38386) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNt3-0007nN-Hx for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:21 -0400 Received: from mga17.intel.com ([192.55.52.151]:41972) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNt1-0000GZ-Q6 for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653373159; x=1684909159; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=4mb5xQjmgPF5412Sdh4ynXjM0cMFDDt2pd56fHCzg4U=; b=X8dUCAA7SBOCJdN75jEejjxeTDaisah6W2vJDfAtfkrmZAAzqutA5oMe /VMxaFOe1lBnlGtps4UaIGMlL3BW7ELGbN8wdBxrKr0aeh1uU4LOiG3uU Md5pwvl/N3qm1N12mMnO6hVDaARAwh1N1XOEjYeBWxXBuHjTQKdtgdPwR wqk6OBcEQ5XJmxdHyjnznlfhv1OC0Z1M+Nf0vKzFA9y1X86JAh7uofE0x XffjpzSciQXVuwRl3EphfSekTv7LrAtConpzDn01+iqdK+lMkj+riv5W8 M5MBxpa3PKY23DVmtPToAwHc8NA3nyxDzafNU9kN6l7JlbZuzbpOKiyFg Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="253943190" X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="253943190" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 23:19:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="601059692" Received: from leirao-pc.bj.intel.com ([10.238.156.102]) by orsmga008.jf.intel.com with ESMTP; 23 May 2022 23:19:15 -0700 From: Lei Rao To: alex.williamson@redhat.com, kevin.tian@intel.com, eddie.dong@intel.com, jason.zeng@intel.com, quintela@redhat.com, dgilbert@redhat.com, yadong.li@intel.com, yi.l.liu@intel.com Cc: qemu-devel@nongnu.org, Lei Rao Subject: [RFC PATCH 04/13] vfio/migration: Separated functions that relate to the In-Band approach Date: Tue, 24 May 2022 14:18:39 +0800 Message-Id: <20220524061848.1615706-5-lei.rao@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220524061848.1615706-1-lei.rao@intel.com> References: <20220524061848.1615706-1-lei.rao@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=lei.rao@intel.com; helo=mga17.intel.com X-Spam_score_int: -44 X-Spam_score: -4.5 X-Spam_bar: ---- X-Spam_report: (-4.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Split functions of In-Band approach from common function, to prepare for the introduction of generic VFIO live migration layer and another Sub-Ops. Signed-off-by: Lei Rao Reviewed-by: Eddie Dong --- hw/vfio/migration.c | 64 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 49 insertions(+), 15 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index e61c19171a..c2df2caae6 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -394,7 +394,7 @@ static int vfio_load_device_config_state(QEMUFile *f, void *opaque) return qemu_file_get_error(f); } -static void vfio_migration_cleanup(VFIODevice *vbasedev) +static void vfio_migration_cleanup_local(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; @@ -403,17 +403,17 @@ static void vfio_migration_cleanup(VFIODevice *vbasedev) } } +static void vfio_migration_cleanup(VFIODevice *vbasedev) +{ + vfio_migration_cleanup_local(vbasedev); +} + /* ---------------------------------------------------------------------- */ -static int vfio_save_setup(QEMUFile *f, void *opaque) +static int vfio_migration_save_setup_local(VFIODevice *vbasedev) { - VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; - int ret; - - trace_vfio_save_setup(vbasedev->name); - - qemu_put_be64(f, VFIO_MIG_FLAG_DEV_SETUP_STATE); + int ret = -1; if (migration->region.mmaps) { /* @@ -430,6 +430,24 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) error_report("%s: Falling back to slow path", vbasedev->name); } } + return ret; +} + +static int vfio_save_setup(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + int ret; + + trace_vfio_save_setup(vbasedev->name); + + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_SETUP_STATE); + + ret = vfio_migration_save_setup_local(vbasedev); + if (ret) { + error_report("%s: Failed to vfio lm save setup:%s", + vbasedev->name, strerror(-ret)); + return ret; + } ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_MASK, VFIO_DEVICE_STATE_V1_SAVING); @@ -592,11 +610,10 @@ static void vfio_save_state(QEMUFile *f, void *opaque) } } -static int vfio_load_setup(QEMUFile *f, void *opaque) +static int vfio_migration_load_setup_local(VFIODevice *vbasedev) { - VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; - int ret = 0; + int ret = -1; if (migration->region.mmaps) { ret = vfio_region_mmap(&migration->region); @@ -607,14 +624,26 @@ static int vfio_load_setup(QEMUFile *f, void *opaque) error_report("%s: Falling back to slow path", vbasedev->name); } } + return ret; +} + +static int vfio_load_setup(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + int ret = 0; + + ret = vfio_migration_load_setup_local(vbasedev); + if (ret < 0) { + error_report("%s: Failed to migration load setup", vbasedev->name); + return ret; + } ret = vfio_migration_set_state(vbasedev, ~VFIO_DEVICE_STATE_MASK, VFIO_DEVICE_STATE_V1_RESUMING); if (ret) { error_report("%s: Failed to set state RESUMING", vbasedev->name); - if (migration->region.mmaps) { - vfio_region_unmap(&migration->region); - } + vfio_migration_cleanup(vbasedev); + return ret; } return ret; } @@ -777,12 +806,17 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data) } } -static void vfio_migration_exit(VFIODevice *vbasedev) +static void vfio_migration_exit_local(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; vfio_region_exit(&migration->region); vfio_region_finalize(&migration->region); +} + +static void vfio_migration_exit(VFIODevice *vbasedev) +{ + vfio_migration_exit_local(vbasedev); g_free(vbasedev->migration); vbasedev->migration = NULL; } From patchwork Tue May 24 06:18:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rao, Lei" X-Patchwork-Id: 12859719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BD1A3C433F5 for ; Tue, 24 May 2022 06:33:09 +0000 (UTC) Received: from localhost ([::1]:49208 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ntO6O-0008VL-NK for qemu-devel@archiver.kernel.org; Tue, 24 May 2022 02:33:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38416) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNt7-0007vq-U0 for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:25 -0400 Received: from mga17.intel.com ([192.55.52.151]:41972) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNt5-0000GZ-AC for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653373163; x=1684909163; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MTVZIHubx9UoalfP6fFQLeCYUN615ZIuuSDRJi+Qj+s=; b=CF+zIiZD9ktLBOqMwV4tYKqXDH3q//qLXJ+QS9yBwNqLsflyzhc2B+t7 iKABtNtukJt1/pHqFkxOyBajYef1Wfk+CeW/fNzDEn56JEOt9xIMmh1Nl To+c8kN6DeMCT1N1Zid+6ZJqForA0UqGKyxIDG2leVSdeH+gzUAuTiQRQ ZOfrc4ls04Fiq4bkNc3e4XdgSB5epyHBAzWbQ/Sal5F3vfFGfcZATmkIN BukRr1PiWvnj0ZztIQ8MU0nwEZLJ7MF/IWP1R+pjKBpKVe3A/8OExwetM wRWwSnyg64sWXg3873PaqL0s9U9VNY7bfoN2aUuNrNPO6rVCHRDftAO+b w==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="253943192" X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="253943192" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 23:19:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="601059714" Received: from leirao-pc.bj.intel.com ([10.238.156.102]) by orsmga008.jf.intel.com with ESMTP; 23 May 2022 23:19:19 -0700 From: Lei Rao To: alex.williamson@redhat.com, kevin.tian@intel.com, eddie.dong@intel.com, jason.zeng@intel.com, quintela@redhat.com, dgilbert@redhat.com, yadong.li@intel.com, yi.l.liu@intel.com Cc: qemu-devel@nongnu.org, Lei Rao Subject: [RFC PATCH 05/13] vfio/migration: rename functions that relate to the In-Band approach Date: Tue, 24 May 2022 14:18:40 +0800 Message-Id: <20220524061848.1615706-6-lei.rao@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220524061848.1615706-1-lei.rao@intel.com> References: <20220524061848.1615706-1-lei.rao@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=lei.rao@intel.com; helo=mga17.intel.com X-Spam_score_int: -44 X-Spam_score: -4.5 X-Spam_bar: ---- X-Spam_report: (-4.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Rename some functions that are related to the In-Band approach to facilitate introducing generic vfio live migration layer. Rename vfio_migration_set_state to vfio_migration_set_state_local, vfio_save_buffer to vfio_migration_save_buffer_local, vfio_load_buffer to vfio_migration_load_buffer_local, vfio_update_pending to vfio_migration_update_pending_local, vfio_migration_init to vfio_migration_probe_local, vfio_migration_exit to vfio_migration_exit_local. Signed-off-by: Lei Rao Reviewed-by: Eddie Dong --- hw/vfio/migration.c | 74 +++++++++++++++++++++++--------------------- hw/vfio/trace-events | 6 ++-- 2 files changed, 42 insertions(+), 38 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index c2df2caae6..04360e1f17 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -107,8 +107,8 @@ static int vfio_mig_rw(VFIODevice *vbasedev, __u8 *buf, size_t count, * an error is returned. */ -static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t mask, - uint32_t value) +static int vfio_migration_set_state_local(VFIODevice *vbasedev, uint32_t mask, + uint32_t value) { VFIOMigration *migration = vbasedev->migration; VFIORegion *region = &migration->region; @@ -193,7 +193,8 @@ static void *get_data_section_size(VFIORegion *region, uint64_t data_offset, return ptr; } -static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size) +static int vfio_migration_save_buffer_local(QEMUFile *f, VFIODevice *vbasedev, + uint64_t *size) { VFIOMigration *migration = vbasedev->migration; VFIORegion *region = &migration->region; @@ -212,8 +213,8 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size) return ret; } - trace_vfio_save_buffer(vbasedev->name, data_offset, data_size, - migration->pending_bytes); + trace_vfio_save_buffer_local(vbasedev->name, data_offset, data_size, + migration->pending_bytes); qemu_put_be64(f, data_size); sz = data_size; @@ -260,8 +261,8 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size) return ret; } -static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, - uint64_t data_size) +static int vfio_migration_load_buffer_local(QEMUFile *f, VFIODevice *vbasedev, + uint64_t data_size) { VFIORegion *region = &vbasedev->migration->region; uint64_t data_offset = 0, size, report_size; @@ -288,7 +289,8 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, data_size = 0; } - trace_vfio_load_state_device_data(vbasedev->name, data_offset, size); + trace_vfio_load_state_device_data_local(vbasedev->name, data_offset, + size); while (size) { void *buf; @@ -331,7 +333,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, return 0; } -static int vfio_update_pending(VFIODevice *vbasedev) +static int vfio_migration_update_pending_local(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; VFIORegion *region = &migration->region; @@ -449,8 +451,8 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) return ret; } - ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_MASK, - VFIO_DEVICE_STATE_V1_SAVING); + ret = vfio_migration_set_state_local(vbasedev, VFIO_DEVICE_STATE_MASK, + VFIO_DEVICE_STATE_V1_SAVING); if (ret) { error_report("%s: Failed to set state SAVING", vbasedev->name); return ret; @@ -484,7 +486,7 @@ static void vfio_save_pending(QEMUFile *f, void *opaque, VFIOMigration *migration = vbasedev->migration; int ret; - ret = vfio_update_pending(vbasedev); + ret = vfio_migration_update_pending_local(vbasedev); if (ret) { return; } @@ -505,7 +507,7 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque) qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); if (migration->pending_bytes == 0) { - ret = vfio_update_pending(vbasedev); + ret = vfio_migration_update_pending_local(vbasedev); if (ret) { return ret; } @@ -518,10 +520,10 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque) } } - ret = vfio_save_buffer(f, vbasedev, &data_size); + ret = vfio_migration_save_buffer_local(f, vbasedev, &data_size); if (ret) { - error_report("%s: vfio_save_buffer failed %s", vbasedev->name, - strerror(errno)); + error_report("%s: vfio_miragion_save_buffer_local failed %s", + vbasedev->name, strerror(errno)); return ret; } @@ -534,8 +536,8 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque) /* * Reset pending_bytes as .save_live_pending is not called during savevm or - * snapshot case, in such case vfio_update_pending() at the start of this - * function updates pending_bytes. + * snapshot case, in such case vfio_migration_update_pending_local() at the + * start of this function updates pending_bytes. */ migration->pending_bytes = 0; trace_vfio_save_iterate(vbasedev->name, data_size); @@ -549,22 +551,23 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) uint64_t data_size; int ret; - ret = vfio_migration_set_state(vbasedev, ~VFIO_DEVICE_STATE_V1_RUNNING, - VFIO_DEVICE_STATE_V1_SAVING); + ret = vfio_migration_set_state_local(vbasedev, + ~VFIO_DEVICE_STATE_V1_RUNNING, + VFIO_DEVICE_STATE_V1_SAVING); if (ret) { error_report("%s: Failed to set state STOP and SAVING", vbasedev->name); return ret; } - ret = vfio_update_pending(vbasedev); + ret = vfio_migration_update_pending_local(vbasedev); if (ret) { return ret; } while (migration->pending_bytes > 0) { qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); - ret = vfio_save_buffer(f, vbasedev, &data_size); + ret = vfio_migration_save_buffer_local(f, vbasedev, &data_size); if (ret < 0) { error_report("%s: Failed to save buffer", vbasedev->name); return ret; @@ -574,7 +577,7 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) break; } - ret = vfio_update_pending(vbasedev); + ret = vfio_migration_update_pending_local(vbasedev); if (ret) { return ret; } @@ -587,7 +590,8 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) return ret; } - ret = vfio_migration_set_state(vbasedev, ~VFIO_DEVICE_STATE_V1_SAVING, 0); + ret = vfio_migration_set_state_local(vbasedev, ~VFIO_DEVICE_STATE_V1_SAVING, + 0); if (ret) { error_report("%s: Failed to set state STOPPED", vbasedev->name); return ret; @@ -638,7 +642,7 @@ static int vfio_load_setup(QEMUFile *f, void *opaque) return ret; } - ret = vfio_migration_set_state(vbasedev, ~VFIO_DEVICE_STATE_MASK, + ret = vfio_migration_set_state_local(vbasedev, ~VFIO_DEVICE_STATE_MASK, VFIO_DEVICE_STATE_V1_RESUMING); if (ret) { error_report("%s: Failed to set state RESUMING", vbasedev->name); @@ -690,7 +694,7 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) uint64_t data_size = qemu_get_be64(f); if (data_size) { - ret = vfio_load_buffer(f, vbasedev, data_size); + ret = vfio_migration_load_buffer_local(f, vbasedev, data_size); if (ret < 0) { return ret; } @@ -765,7 +769,7 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) } } - ret = vfio_migration_set_state(vbasedev, mask, value); + ret = vfio_migration_set_state_local(vbasedev, mask, value); if (ret) { /* * Migration should be aborted in this case, but vm_state_notify() @@ -796,10 +800,10 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data) case MIGRATION_STATUS_CANCELLED: case MIGRATION_STATUS_FAILED: bytes_transferred = 0; - ret = vfio_migration_set_state(vbasedev, - ~(VFIO_DEVICE_STATE_V1_SAVING | - VFIO_DEVICE_STATE_V1_RESUMING), - VFIO_DEVICE_STATE_V1_RUNNING); + ret = vfio_migration_set_state_local(vbasedev, + ~(VFIO_DEVICE_STATE_V1_SAVING | + VFIO_DEVICE_STATE_V1_RESUMING), + VFIO_DEVICE_STATE_V1_RUNNING); if (ret) { error_report("%s: Failed to set state RUNNING", vbasedev->name); } @@ -836,7 +840,7 @@ static int vfio_migration_check(VFIODevice *vbasedev) return 0; } -static int vfio_migration_init(VFIODevice *vbasedev) +static int vfio_migration_probe_local(VFIODevice *vbasedev) { int ret; Object *obj; @@ -890,12 +894,12 @@ static int vfio_migration_init(VFIODevice *vbasedev) migration->migration_state.notify = vfio_migration_state_notifier; add_migration_state_change_notifier(&migration->migration_state); - trace_vfio_migration_probe(vbasedev->name, info->index); + trace_vfio_migration_probe_local(vbasedev->name, info->index); g_free(info); return 0; err: - vfio_migration_exit(vbasedev); + vfio_migration_exit_local(vbasedev); g_free(info); return ret; } @@ -919,7 +923,7 @@ int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) vbasedev->migration = g_new0(VFIOMigration, 1); vbasedev->migration->vbasedev = vbasedev; - ret = vfio_migration_init(vbasedev); + ret = vfio_migration_probe_local(vbasedev); if (ret) { goto add_blocker; } diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 582882db91..ca85edeb11 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -148,13 +148,13 @@ vfio_display_edid_update(uint32_t prefx, uint32_t prefy) "%ux%u" vfio_display_edid_write_error(void) "" # migration.c -vfio_migration_probe(const char *name, uint32_t index) " (%s) Region %d" +vfio_migration_probe_local(const char *name, uint32_t index) " (%s) Region %d" vfio_migration_set_state(const char *name, uint32_t state) " (%s) state %d" vfio_vmstate_change(const char *name, int running, const char *reason, uint32_t dev_state) " (%s) running %d reason %s device state %d" vfio_migration_state_notifier(const char *name, const char *state) " (%s) state %s" vfio_save_setup(const char *name) " (%s)" vfio_save_cleanup(const char *name) " (%s)" -vfio_save_buffer(const char *name, uint64_t data_offset, uint64_t data_size, uint64_t pending) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64" pending 0x%"PRIx64 +vfio_save_buffer_local(const char *name, uint64_t data_offset, uint64_t data_size, uint64_t pending) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64" pending 0x%"PRIx64 vfio_update_pending(const char *name, uint64_t pending) " (%s) pending 0x%"PRIx64 vfio_save_device_config_state(const char *name) " (%s)" vfio_save_pending(const char *name, uint64_t precopy, uint64_t postcopy, uint64_t compatible) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" compatible 0x%"PRIx64 @@ -162,7 +162,7 @@ vfio_save_iterate(const char *name, int data_size) " (%s) data_size %d" vfio_save_complete_precopy(const char *name) " (%s)" vfio_load_device_config_state(const char *name) " (%s)" vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64 -vfio_load_state_device_data(const char *name, uint64_t data_offset, uint64_t data_size) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64 +vfio_load_state_device_data_local(const char *name, uint64_t data_offset, uint64_t data_size) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64 vfio_load_cleanup(const char *name) " (%s)" vfio_get_dirty_bitmap(int fd, uint64_t iova, uint64_t size, uint64_t bitmap_size, uint64_t start) "container fd=%d, iova=0x%"PRIx64" size= 0x%"PRIx64" bitmap_size=0x%"PRIx64" start=0x%"PRIx64 vfio_iommu_map_dirty_notify(uint64_t iova_start, uint64_t iova_end) "iommu dirty @ 0x%"PRIx64" - 0x%"PRIx64 From patchwork Tue May 24 06:18:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rao, Lei" X-Patchwork-Id: 12859720 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B8E9DC433F5 for ; Tue, 24 May 2022 06:34:54 +0000 (UTC) Received: from localhost ([::1]:51652 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ntO85-0002BX-II for qemu-devel@archiver.kernel.org; Tue, 24 May 2022 02:34:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38434) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtC-00084r-Kz for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:30 -0400 Received: from mga17.intel.com ([192.55.52.151]:41983) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtA-0000HO-8s for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653373168; x=1684909168; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yFksj+5b6JElUXpSeZB4v3E5u0VKDCgdJjHfiu8AXMQ=; b=gneoSlny8W14l7ApHF2/AdFGeQ1QT2lIQaCWPf3iTEL8cjMndb0Dl720 dasEQGrLoDXIEZ0odWv0GguwJvp8UL5sczVI4651us+hOP2b6UnShll1P zHggGUTkCMEDtUwQ0E4rClrrHnHrz3N0BbGYl3RQwdFiL3hKI0OcNw6Ad 08xipmx1Y4RHZ9lhM9vSLjamm6yS5z1QatGEeq2uV7v395plyMu1mNabB W37dR8L+BI8IdZgIW68o0kIrVo/lQI8QLt1ImUspkjxmwglAVIEI+7au6 LIF8PVjHAUpOLbY4aMUdJI0KAIX96iNgoFCALv0nNt0lIMD/6LlJaKMnk Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="253943203" X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="253943203" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 23:19:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="601059733" Received: from leirao-pc.bj.intel.com ([10.238.156.102]) by orsmga008.jf.intel.com with ESMTP; 23 May 2022 23:19:23 -0700 From: Lei Rao To: alex.williamson@redhat.com, kevin.tian@intel.com, eddie.dong@intel.com, jason.zeng@intel.com, quintela@redhat.com, dgilbert@redhat.com, yadong.li@intel.com, yi.l.liu@intel.com Cc: qemu-devel@nongnu.org, Lei Rao Subject: [RFC PATCH 06/13] vfio/migration: introduce VFIOMigrationOps layer in VFIO live migration framework Date: Tue, 24 May 2022 14:18:41 +0800 Message-Id: <20220524061848.1615706-7-lei.rao@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220524061848.1615706-1-lei.rao@intel.com> References: <20220524061848.1615706-1-lei.rao@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=lei.rao@intel.com; helo=mga17.intel.com X-Spam_score_int: -44 X-Spam_score: -4.5 X-Spam_bar: ---- X-Spam_report: (-4.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add an abstraction layer, VFIOMigrationOps, to the VFIO live migration framework. Also adapt the In-Band approach to this abstraction layer by defining its own VFIOMigrationOps callbacks. Signed-off-by: Lei Rao Reviewed-by: Eddie Dong --- hw/vfio/migration.c | 203 +++++++++++++++++++++------------- include/hw/vfio/vfio-common.h | 14 +++ 2 files changed, 142 insertions(+), 75 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 04360e1f17..4736af90e7 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -407,7 +407,11 @@ static void vfio_migration_cleanup_local(VFIODevice *vbasedev) static void vfio_migration_cleanup(VFIODevice *vbasedev) { - vfio_migration_cleanup_local(vbasedev); + VFIOMigration *migration = vbasedev->migration; + + if (migration->ops->cleanup) { + migration->ops->cleanup(vbasedev); + } } /* ---------------------------------------------------------------------- */ @@ -438,24 +442,29 @@ static int vfio_migration_save_setup_local(VFIODevice *vbasedev) static int vfio_save_setup(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; int ret; trace_vfio_save_setup(vbasedev->name); qemu_put_be64(f, VFIO_MIG_FLAG_DEV_SETUP_STATE); - ret = vfio_migration_save_setup_local(vbasedev); - if (ret) { - error_report("%s: Failed to vfio lm save setup:%s", - vbasedev->name, strerror(-ret)); - return ret; + if (migration->ops->save_setup) { + ret = migration->ops->save_setup(vbasedev); + if (ret) { + error_report("%s: Failed to vfio lm save setup:%s", + vbasedev->name, strerror(-ret)); + return ret; + } } - ret = vfio_migration_set_state_local(vbasedev, VFIO_DEVICE_STATE_MASK, - VFIO_DEVICE_STATE_V1_SAVING); - if (ret) { - error_report("%s: Failed to set state SAVING", vbasedev->name); - return ret; + if (migration->ops->set_state) { + ret = migration->ops->set_state(vbasedev, VFIO_DEVICE_STATE_MASK, + VFIO_DEVICE_STATE_V1_SAVING); + if (ret) { + error_report("%s: Failed to set state SAVING", vbasedev->name); + return ret; + } } qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); @@ -486,9 +495,11 @@ static void vfio_save_pending(QEMUFile *f, void *opaque, VFIOMigration *migration = vbasedev->migration; int ret; - ret = vfio_migration_update_pending_local(vbasedev); - if (ret) { - return; + if (migration->ops->update_pending) { + ret = migration->ops->update_pending(vbasedev); + if (ret) { + return; + } } *res_precopy_only += migration->pending_bytes; @@ -507,9 +518,11 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque) qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); if (migration->pending_bytes == 0) { - ret = vfio_migration_update_pending_local(vbasedev); - if (ret) { - return ret; + if (migration->ops->update_pending) { + ret = migration->ops->update_pending(vbasedev); + if (ret) { + return ret; + } } if (migration->pending_bytes == 0) { @@ -520,11 +533,13 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque) } } - ret = vfio_migration_save_buffer_local(f, vbasedev, &data_size); - if (ret) { - error_report("%s: vfio_miragion_save_buffer_local failed %s", - vbasedev->name, strerror(errno)); - return ret; + if (migration->ops->save_buffer) { + ret = migration->ops->save_buffer(f, vbasedev, &data_size); + if (ret) { + error_report("%s: vfio_miragion_save_buffer_local failed %s", + vbasedev->name, strerror(errno)); + return ret; + } } qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); @@ -551,35 +566,43 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) uint64_t data_size; int ret; - ret = vfio_migration_set_state_local(vbasedev, - ~VFIO_DEVICE_STATE_V1_RUNNING, - VFIO_DEVICE_STATE_V1_SAVING); - if (ret) { - error_report("%s: Failed to set state STOP and SAVING", - vbasedev->name); - return ret; + if (migration->ops->set_state) { + ret = migration->ops->set_state(vbasedev, + ~VFIO_DEVICE_STATE_V1_RUNNING, + VFIO_DEVICE_STATE_V1_SAVING); + if (ret) { + error_report("%s: Failed to set state STOP and SAVING", + vbasedev->name); + return ret; + } } - ret = vfio_migration_update_pending_local(vbasedev); - if (ret) { - return ret; + if (migration->ops->update_pending) { + ret = migration->ops->update_pending(vbasedev); + if (ret) { + return ret; + } } while (migration->pending_bytes > 0) { qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); - ret = vfio_migration_save_buffer_local(f, vbasedev, &data_size); - if (ret < 0) { - error_report("%s: Failed to save buffer", vbasedev->name); - return ret; + if (migration->ops->save_buffer) { + ret = migration->ops->save_buffer(f, vbasedev, &data_size); + if (ret < 0) { + error_report("%s: Failed to save buffer", vbasedev->name); + return ret; + } } if (data_size == 0) { break; } - ret = vfio_migration_update_pending_local(vbasedev); - if (ret) { - return ret; + if (migration->ops->update_pending) { + ret = migration->ops->update_pending(vbasedev); + if (ret) { + return ret; + } } } @@ -590,11 +613,13 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) return ret; } - ret = vfio_migration_set_state_local(vbasedev, ~VFIO_DEVICE_STATE_V1_SAVING, - 0); - if (ret) { - error_report("%s: Failed to set state STOPPED", vbasedev->name); - return ret; + if (migration->ops->set_state) { + ret = migration->ops->set_state(vbasedev, ~VFIO_DEVICE_STATE_V1_SAVING, + 0); + if (ret) { + error_report("%s: Failed to set state STOPPED", vbasedev->name); + return ret; + } } trace_vfio_save_complete_precopy(vbasedev->name); @@ -634,20 +659,25 @@ static int vfio_migration_load_setup_local(VFIODevice *vbasedev) static int vfio_load_setup(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; int ret = 0; - ret = vfio_migration_load_setup_local(vbasedev); - if (ret < 0) { - error_report("%s: Failed to migration load setup", vbasedev->name); - return ret; + if (migration->ops->load_setup) { + ret = migration->ops->load_setup(vbasedev); + if (ret < 0) { + error_report("%s: Failed to migration load setup", vbasedev->name); + return ret; + } } - ret = vfio_migration_set_state_local(vbasedev, ~VFIO_DEVICE_STATE_MASK, - VFIO_DEVICE_STATE_V1_RESUMING); - if (ret) { - error_report("%s: Failed to set state RESUMING", vbasedev->name); - vfio_migration_cleanup(vbasedev); - return ret; + if (migration->ops->set_state) { + ret = migration->ops->set_state(vbasedev, ~VFIO_DEVICE_STATE_MASK, + VFIO_DEVICE_STATE_V1_RESUMING); + if (ret) { + error_report("%s: Failed to set state RESUMING", vbasedev->name); + vfio_migration_cleanup(vbasedev); + return ret; + } } return ret; } @@ -692,11 +722,14 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) case VFIO_MIG_FLAG_DEV_DATA_STATE: { uint64_t data_size = qemu_get_be64(f); + VFIOMigration *migration = vbasedev->migration; if (data_size) { - ret = vfio_migration_load_buffer_local(f, vbasedev, data_size); - if (ret < 0) { - return ret; + if (migration->ops->load_buffer) { + ret = migration->ops->load_buffer(f, vbasedev, data_size); + if (ret < 0) { + return ret; + } } } break; @@ -736,7 +769,7 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) uint32_t value, mask; int ret; - if (vbasedev->migration->vm_running == running) { + if (migration->vm_running == running) { return; } @@ -769,17 +802,19 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) } } - ret = vfio_migration_set_state_local(vbasedev, mask, value); - if (ret) { - /* - * Migration should be aborted in this case, but vm_state_notify() - * currently does not support reporting failures. - */ - error_report("%s: Failed to set device state 0x%x", vbasedev->name, - (migration->device_state & mask) | value); - qemu_file_set_error(migrate_get_current()->to_dst_file, ret); + if (migration->ops->set_state) { + ret = migration->ops->set_state(vbasedev, mask, value); + if (ret) { + /* + * Migration should be aborted in this case, but vm_state_notify() + * currently does not support reporting failures. + */ + error_report("%s: Failed to set device state 0x%x", vbasedev->name, + (migration->device_state & mask) | value); + qemu_file_set_error(migrate_get_current()->to_dst_file, ret); + } } - vbasedev->migration->vm_running = running; + migration->vm_running = running; trace_vfio_vmstate_change(vbasedev->name, running, RunState_str(state), (migration->device_state & mask) | value); } @@ -800,12 +835,14 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data) case MIGRATION_STATUS_CANCELLED: case MIGRATION_STATUS_FAILED: bytes_transferred = 0; - ret = vfio_migration_set_state_local(vbasedev, - ~(VFIO_DEVICE_STATE_V1_SAVING | - VFIO_DEVICE_STATE_V1_RESUMING), - VFIO_DEVICE_STATE_V1_RUNNING); - if (ret) { - error_report("%s: Failed to set state RUNNING", vbasedev->name); + if (migration->ops->set_state) { + ret = migration->ops->set_state(vbasedev, + ~(VFIO_DEVICE_STATE_V1_SAVING | + VFIO_DEVICE_STATE_V1_RESUMING), + VFIO_DEVICE_STATE_V1_RUNNING); + if (ret) { + error_report("%s: Failed to set state RUNNING", vbasedev->name); + } } } } @@ -820,7 +857,11 @@ static void vfio_migration_exit_local(VFIODevice *vbasedev) static void vfio_migration_exit(VFIODevice *vbasedev) { - vfio_migration_exit_local(vbasedev); + VFIOMigration *migration = vbasedev->migration; + + if (migration->ops->exit) { + migration->ops->exit(vbasedev); + } g_free(vbasedev->migration); vbasedev->migration = NULL; } @@ -840,6 +881,17 @@ static int vfio_migration_check(VFIODevice *vbasedev) return 0; } +static VFIOMigrationOps vfio_local_method = { + .save_setup = vfio_migration_save_setup_local, + .load_setup = vfio_migration_load_setup_local, + .update_pending = vfio_migration_update_pending_local, + .save_buffer = vfio_migration_save_buffer_local, + .load_buffer = vfio_migration_load_buffer_local, + .set_state = vfio_migration_set_state_local, + .cleanup = vfio_migration_cleanup_local, + .exit = vfio_migration_exit_local, +}; + static int vfio_migration_probe_local(VFIODevice *vbasedev) { int ret; @@ -895,6 +947,7 @@ static int vfio_migration_probe_local(VFIODevice *vbasedev) add_migration_state_change_notifier(&migration->migration_state); trace_vfio_migration_probe_local(vbasedev->name, info->index); + migration->ops = &vfio_local_method; g_free(info); return 0; diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index e573f5a9f1..8ef85a871c 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -58,10 +58,13 @@ typedef struct VFIORegion { uint8_t nr; /* cache the region number for debug */ } VFIORegion; +typedef struct VFIOMigrationOps VFIOMigrationOps; + typedef struct VFIOMigration { struct VFIODevice *vbasedev; VMChangeStateEntry *vm_state; VFIORegion region; + VFIOMigrationOps *ops; uint32_t device_state; int vm_running; Notifier migration_state; @@ -154,6 +157,17 @@ struct VFIODeviceOps { int (*vfio_load_config)(VFIODevice *vdev, QEMUFile *f); }; +typedef struct VFIOMigrationOps { + int (*save_setup)(VFIODevice *vbasedev); + int (*load_setup)(VFIODevice *vbasedev); + int (*update_pending)(VFIODevice *vbasedev); + int (*save_buffer)(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size); + int (*load_buffer)(QEMUFile *f, VFIODevice *vbasedev, uint64_t data_size); + int (*set_state)(VFIODevice *vbasedev, uint32_t mask, uint32_t value); + void (*cleanup)(VFIODevice *vbasedev); + void (*exit)(VFIODevice *vbasedev); +} VFIOMigrationOps; + typedef struct VFIOGroup { int fd; int groupid; From patchwork Tue May 24 06:18:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rao, Lei" X-Patchwork-Id: 12859724 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BEE43C433EF for ; Tue, 24 May 2022 06:41:52 +0000 (UTC) Received: from localhost ([::1]:57896 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ntOEp-00074B-JI for qemu-devel@archiver.kernel.org; Tue, 24 May 2022 02:41:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38448) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtG-0008D1-5I for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:34 -0400 Received: from mga17.intel.com ([192.55.52.151]:41983) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtE-0000HO-F5 for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653373172; x=1684909172; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pfQR45LZch1v1vRjXbsnVrV5Nq/XR+Tvb/OTap5qN9o=; b=Ks7j8/WT6b/LaSoeWE8mgCW517oFqKcDQJgbYDRyWztxjGYKN0lrGLvb Nu48B6BctitBFM1H4Hw36kd83l0d8OCrOC+XRiE3tL7SYQxM5xTw1ZsVT YUCgdHhn1XpxZcU/599VrUGSXbeNTsgUdmb1gVwjn8CTu/vEEblNNE86p 7Io2wZxYZ8pULrJRvwWosmn0sbYV3ts05rHudHkahSJA5r6+ugvdlD4sn jVewAhBYsGESJ1gBgawdI94xWTJfen8h2JpQGlaVtmvDMcPLoE2oaNMOF 1GQCoapF5O5eLXVpCfvnKodxxEXNoIpYVesYE1g4vGVK8N81pVyCfbrAt A==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="253943221" X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="253943221" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 23:19:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="601059758" Received: from leirao-pc.bj.intel.com ([10.238.156.102]) by orsmga008.jf.intel.com with ESMTP; 23 May 2022 23:19:27 -0700 From: Lei Rao To: alex.williamson@redhat.com, kevin.tian@intel.com, eddie.dong@intel.com, jason.zeng@intel.com, quintela@redhat.com, dgilbert@redhat.com, yadong.li@intel.com, yi.l.liu@intel.com Cc: qemu-devel@nongnu.org, Lei Rao Subject: [RFC PATCH 07/13] vfio/migration: move the statistics of bytes_transferred to generic VFIO migration layer Date: Tue, 24 May 2022 14:18:42 +0800 Message-Id: <20220524061848.1615706-8-lei.rao@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220524061848.1615706-1-lei.rao@intel.com> References: <20220524061848.1615706-1-lei.rao@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=lei.rao@intel.com; helo=mga17.intel.com X-Spam_score_int: -44 X-Spam_score: -4.5 X-Spam_bar: ---- X-Spam_report: (-4.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The statistics of bytes transferred conceptually belong to The VFIO live migration framework, and should not belong to any specific implementation such In-Band approach, so move it out from vfio_migration_region_save_buffer(), which makes it easier to add other implementations. Signed-off-by: Lei Rao Reviewed-by: Eddie Dong --- hw/vfio/migration.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 4736af90e7..c114fab3a2 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -257,7 +257,6 @@ static int vfio_migration_save_buffer_local(QEMUFile *f, VFIODevice *vbasedev, *size = data_size; } - bytes_transferred += data_size; return ret; } @@ -540,6 +539,7 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque) vbasedev->name, strerror(errno)); return ret; } + bytes_transferred += data_size; } qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); @@ -592,6 +592,7 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) error_report("%s: Failed to save buffer", vbasedev->name); return ret; } + bytes_transferred += data_size; } if (data_size == 0) { From patchwork Tue May 24 06:18:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rao, Lei" X-Patchwork-Id: 12859718 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A35ADC433EF for ; Tue, 24 May 2022 06:32:41 +0000 (UTC) Received: from localhost ([::1]:48788 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ntO5w-0008BO-Dz for qemu-devel@archiver.kernel.org; Tue, 24 May 2022 02:32:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38468) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtK-0008LJ-0J for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:38 -0400 Received: from mga17.intel.com ([192.55.52.151]:41983) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtI-0000HO-4N for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653373176; x=1684909176; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8/TnV9NVIWyHgmix5BRtBbQKn+ZIngPWJh2/EHtAMo8=; b=NtXogOPG71EF9CBqEu/6jaV47498YKIenYGPkalpUV0bwTi9XrB+rPpT FYv6x6vqPuWsNh2NsYQTkRKAwi2c5+Bk4O9lVuecJrPRxLb0KZCBBMWeR ohjkjdtmJRHITp37BR+dHnNtBgw2nQVZ2K2TlJOkskbb+6j5k7Lv2bvn5 cWUqw0LfNvhA24n3nLeWZBoT7fGPtIbDwqFmcbta/A1Z0HG/FhPdXKLWK vKXr5F082yG1b7EGUl06QGUlkiUpEZxpL+rdrCbFfD6MCx0Vk5K1vnpUs CKvKdQFrbGdsR5bqPKIqED738ba3ecx7Ekl/3M5XXOuWMUmLEXvD+Fph7 g==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="253943235" X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="253943235" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 23:19:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="601059780" Received: from leirao-pc.bj.intel.com ([10.238.156.102]) by orsmga008.jf.intel.com with ESMTP; 23 May 2022 23:19:32 -0700 From: Lei Rao To: alex.williamson@redhat.com, kevin.tian@intel.com, eddie.dong@intel.com, jason.zeng@intel.com, quintela@redhat.com, dgilbert@redhat.com, yadong.li@intel.com, yi.l.liu@intel.com Cc: qemu-devel@nongnu.org, Lei Rao Subject: [RFC PATCH 08/13] vfio/migration: split migration handler registering from vfio_migration_init Date: Tue, 24 May 2022 14:18:43 +0800 Message-Id: <20220524061848.1615706-9-lei.rao@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220524061848.1615706-1-lei.rao@intel.com> References: <20220524061848.1615706-1-lei.rao@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=lei.rao@intel.com; helo=mga17.intel.com X-Spam_score_int: -44 X-Spam_score: -4.5 X-Spam_bar: ---- X-Spam_report: (-4.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" vfio_migration_init() is mainly related to initialization of In-Band approach. Migration handler registering may also be used by other approaches. so split it from vfio_migration_init() and move it to vfio_migration_probe(). Signed-off-by: Lei Rao Reviewed-by: Eddie Dong --- hw/vfio/migration.c | 56 ++++++++++++++++++++++++++++++--------------- 1 file changed, 37 insertions(+), 19 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index c114fab3a2..0c67ed85f3 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -882,6 +882,38 @@ static int vfio_migration_check(VFIODevice *vbasedev) return 0; } +static int vfio_migration_register_handlers(VFIODevice *vbasedev) +{ + Object *obj; + char id[256] = ""; + g_autofree char *path = NULL, *oid = NULL; + VFIOMigration *migration = vbasedev->migration; + + obj = vbasedev->ops->vfio_get_object(vbasedev); + if (!obj) { + return -EINVAL; + } + + oid = vmstate_if_get_id(VMSTATE_IF(DEVICE(obj))); + if (oid) { + path = g_strdup_printf("%s/vfio", oid); + } else { + path = g_strdup("vfio"); + } + strpadcpy(id, sizeof(id), path, '\0'); + + register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, &savevm_vfio_handlers, + vbasedev); + + migration->vm_state = qdev_add_vm_change_state_handler(vbasedev->dev, + vfio_vmstate_change, + vbasedev); + migration->migration_state.notify = vfio_migration_state_notifier; + add_migration_state_change_notifier(&migration->migration_state); + + return 0; +} + static VFIOMigrationOps vfio_local_method = { .save_setup = vfio_migration_save_setup_local, .load_setup = vfio_migration_load_setup_local, @@ -897,9 +929,7 @@ static int vfio_migration_probe_local(VFIODevice *vbasedev) { int ret; Object *obj; - char id[256] = ""; struct vfio_region_info *info = NULL; - g_autofree char *path = NULL, *oid = NULL; VFIOMigration *migration = vbasedev->migration; obj = vbasedev->ops->vfio_get_object(vbasedev); @@ -930,23 +960,6 @@ static int vfio_migration_probe_local(VFIODevice *vbasedev) goto err; } - oid = vmstate_if_get_id(VMSTATE_IF(DEVICE(obj))); - if (oid) { - path = g_strdup_printf("%s/vfio", oid); - } else { - path = g_strdup("vfio"); - } - strpadcpy(id, sizeof(id), path, '\0'); - - register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, &savevm_vfio_handlers, - vbasedev); - - migration->vm_state = qdev_add_vm_change_state_handler(vbasedev->dev, - vfio_vmstate_change, - vbasedev); - migration->migration_state.notify = vfio_migration_state_notifier; - add_migration_state_change_notifier(&migration->migration_state); - trace_vfio_migration_probe_local(vbasedev->name, info->index); migration->ops = &vfio_local_method; g_free(info); @@ -982,6 +995,11 @@ int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) goto add_blocker; } + ret = vfio_migration_register_handlers(vbasedev); + if (ret) { + goto add_blocker; + } + return 0; add_blocker: From patchwork Tue May 24 06:18:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rao, Lei" X-Patchwork-Id: 12859723 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 791D8C433EF for ; Tue, 24 May 2022 06:41:12 +0000 (UTC) Received: from localhost ([::1]:57452 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ntOEB-0006k2-D8 for qemu-devel@archiver.kernel.org; Tue, 24 May 2022 02:41:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38504) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtX-0008V7-RR for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:52 -0400 Received: from mga17.intel.com ([192.55.52.151]:42009) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtU-0000LD-KQ for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653373188; x=1684909188; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MmRu2cF6faa9WhIABdn4utJiy2fIgs4rcGrOPQjsIR4=; b=L4ujBkxO3isNoko0EJ5O7LazPInr1XHCJXzP/nQe/Ts6VO2q7k7ubX+g ga5osH3K1+g2w/OPNH40Zim65mWRSsSwzNgb+iBraP1iBOE75ui0KN8H2 6GWNSnj8Ny50OhMGEOlAlaY+gf70hiD4uAgtz828C6jWS3u5Zo/bJBht/ HkzOqNW0AAXORgPw4iMVrtZ1eQ6h3onahqTVameWonwlrhV2H4V0/Oztl 2Y6yBUbqZ/bfWLI0fdAuMydji/UxIOQi8JZj8R6t1vFB619nozZjLEUZ1 DnGMy4ikf3Iak/DthCESQi6NRK6Vik2ml2Yw89AsnL+s1IIxoq72XX1bp g==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="253943267" X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="253943267" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 23:19:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="601059813" Received: from leirao-pc.bj.intel.com ([10.238.156.102]) by orsmga008.jf.intel.com with ESMTP; 23 May 2022 23:19:36 -0700 From: Lei Rao To: alex.williamson@redhat.com, kevin.tian@intel.com, eddie.dong@intel.com, jason.zeng@intel.com, quintela@redhat.com, dgilbert@redhat.com, yadong.li@intel.com, yi.l.liu@intel.com Cc: qemu-devel@nongnu.org, Lei Rao Subject: [RFC PATCH 09/13] vfio/migration: move the functions of In-Band approach to a new file Date: Tue, 24 May 2022 14:18:44 +0800 Message-Id: <20220524061848.1615706-10-lei.rao@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220524061848.1615706-1-lei.rao@intel.com> References: <20220524061848.1615706-1-lei.rao@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=lei.rao@intel.com; helo=mga17.intel.com X-Spam_score_int: -44 X-Spam_score: -4.5 X-Spam_bar: ---- X-Spam_report: (-4.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Moving the functions of In-Band approach into a new file to match with the new abstraction layer of migration ops. Signed-off-by: Lei Rao Reviewed-by: Eddie Dong --- hw/vfio/meson.build | 1 + hw/vfio/migration-local.c | 453 ++++++++++++++++++++++++++++++++++ hw/vfio/migration.c | 421 ------------------------------- include/hw/vfio/vfio-common.h | 1 + 4 files changed, 455 insertions(+), 421 deletions(-) create mode 100644 hw/vfio/migration-local.c diff --git a/hw/vfio/meson.build b/hw/vfio/meson.build index da9af297a0..5a72b8c349 100644 --- a/hw/vfio/meson.build +++ b/hw/vfio/meson.build @@ -3,6 +3,7 @@ vfio_ss.add(files( 'common.c', 'spapr.c', 'migration.c', + 'migration-local.c', )) vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( 'display.c', diff --git a/hw/vfio/migration-local.c b/hw/vfio/migration-local.c new file mode 100644 index 0000000000..46c8baed50 --- /dev/null +++ b/hw/vfio/migration-local.c @@ -0,0 +1,453 @@ +/* + * QEMU VFIO Migration Support + * + * Copyright NVIDIA, Inc. 2020 + * + * This software is licensed under the terms of the GNU General Public + * License version 2, as published by the Free Software Foundation, and + * may be copied, distributed, and modified under those terms. + * + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include "qemu/main-loop.h" +#include "qemu/cutils.h" +#include +#include + +#include "sysemu/runstate.h" +#include "hw/vfio/vfio-common.h" +#include "migration/migration.h" +#include "migration/vmstate.h" +#include "migration/qemu-file.h" +#include "migration/register.h" +#include "migration/blocker.h" +#include "migration/misc.h" +#include "qapi/error.h" +#include "exec/ramlist.h" +#include "exec/ram_addr.h" +#include "pci.h" +#include "trace.h" +#include "hw/hw.h" +#include "ui/console.h" + +static inline int vfio_mig_access(VFIODevice *vbasedev, void *val, int count, + off_t off, bool iswrite) +{ + int ret; + + ret = iswrite ? pwrite(vbasedev->fd, val, count, off) : + pread(vbasedev->fd, val, count, off); + if (ret < count) { + error_report("vfio_mig_%s %d byte %s: failed at offset 0x%" + HWADDR_PRIx", err: %s", iswrite ? "write" : "read", count, + vbasedev->name, off, strerror(errno)); + return (ret < 0) ? ret : -EINVAL; + } + return 0; +} + +static int vfio_mig_rw(VFIODevice *vbasedev, __u8 *buf, size_t count, + off_t off, bool iswrite) +{ + int ret, done = 0; + __u8 *tbuf = buf; + + while (count) { + int bytes = 0; + + if (count >= 8 && !(off % 8)) { + bytes = 8; + } else if (count >= 4 && !(off % 4)) { + bytes = 4; + } else if (count >= 2 && !(off % 2)) { + bytes = 2; + } else { + bytes = 1; + } + + ret = vfio_mig_access(vbasedev, tbuf, bytes, off, iswrite); + if (ret) { + return ret; + } + + count -= bytes; + done += bytes; + off += bytes; + tbuf += bytes; + } + return done; +} + +#define vfio_mig_read(f, v, c, o) vfio_mig_rw(f, (__u8 *)v, c, o, false) +#define vfio_mig_write(f, v, c, o) vfio_mig_rw(f, (__u8 *)v, c, o, true) + +#define VFIO_MIG_STRUCT_OFFSET(f) \ + offsetof(struct vfio_device_migration_info, f) +/* + * Change the device_state register for device @vbasedev. Bits set in @mask + * are preserved, bits set in @value are set, and bits not set in either @mask + * or @value are cleared in device_state. If the register cannot be accessed, + * the resulting state would be invalid, or the device enters an error state, + * an error is returned. + */ + +static int vfio_migration_set_state_local(VFIODevice *vbasedev, uint32_t mask, + uint32_t value) +{ + VFIOMigration *migration = vbasedev->migration; + VFIORegion *region = &migration->region; + off_t dev_state_off = region->fd_offset + + VFIO_MIG_STRUCT_OFFSET(device_state); + uint32_t device_state; + int ret; + + ret = vfio_mig_read(vbasedev, &device_state, sizeof(device_state), + dev_state_off); + if (ret < 0) { + return ret; + } + + device_state = (device_state & mask) | value; + + if (!VFIO_DEVICE_STATE_VALID(device_state)) { + return -EINVAL; + } + + ret = vfio_mig_write(vbasedev, &device_state, sizeof(device_state), + dev_state_off); + if (ret < 0) { + int rret; + + rret = vfio_mig_read(vbasedev, &device_state, sizeof(device_state), + dev_state_off); + + if ((rret < 0) || (VFIO_DEVICE_STATE_IS_ERROR(device_state))) { + hw_error("%s: Device in error state 0x%x", vbasedev->name, + device_state); + return rret ? rret : -EIO; + } + return ret; + } + + migration->device_state = device_state; + trace_vfio_migration_set_state(vbasedev->name, device_state); + return 0; +} + +static void *get_data_section_size(VFIORegion *region, uint64_t data_offset, + uint64_t data_size, uint64_t *size) +{ + void *ptr = NULL; + uint64_t limit = 0; + int i; + + if (!region->mmaps) { + if (size) { + *size = MIN(data_size, region->size - data_offset); + } + return ptr; + } + + for (i = 0; i < region->nr_mmaps; i++) { + VFIOMmap *map = region->mmaps + i; + + if ((data_offset >= map->offset) && + (data_offset < map->offset + map->size)) { + + /* check if data_offset is within sparse mmap areas */ + ptr = map->mmap + data_offset - map->offset; + if (size) { + *size = MIN(data_size, map->offset + map->size - data_offset); + } + break; + } else if ((data_offset < map->offset) && + (!limit || limit > map->offset)) { + /* + * data_offset is not within sparse mmap areas, find size of + * non-mapped area. Check through all list since region->mmaps list + * is not sorted. + */ + limit = map->offset; + } + } + + if (!ptr && size) { + *size = limit ? MIN(data_size, limit - data_offset) : data_size; + } + return ptr; +} + +static int vfio_migration_save_buffer_local(QEMUFile *f, VFIODevice *vbasedev, + uint64_t *size) +{ + VFIOMigration *migration = vbasedev->migration; + VFIORegion *region = &migration->region; + uint64_t data_offset = 0, data_size = 0, sz; + int ret; + + ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset), + region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset)); + if (ret < 0) { + return ret; + } + + ret = vfio_mig_read(vbasedev, &data_size, sizeof(data_size), + region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_size)); + if (ret < 0) { + return ret; + } + + trace_vfio_save_buffer_local(vbasedev->name, data_offset, data_size, + migration->pending_bytes); + + qemu_put_be64(f, data_size); + sz = data_size; + + while (sz) { + void *buf; + uint64_t sec_size; + bool buf_allocated = false; + + buf = get_data_section_size(region, data_offset, sz, &sec_size); + + if (!buf) { + buf = g_try_malloc(sec_size); + if (!buf) { + error_report("%s: Error allocating buffer ", __func__); + return -ENOMEM; + } + buf_allocated = true; + + ret = vfio_mig_read(vbasedev, buf, sec_size, + region->fd_offset + data_offset); + if (ret < 0) { + g_free(buf); + return ret; + } + } + + qemu_put_buffer(f, buf, sec_size); + + if (buf_allocated) { + g_free(buf); + } + sz -= sec_size; + data_offset += sec_size; + } + + ret = qemu_file_get_error(f); + + if (!ret && size) { + *size = data_size; + } + + return ret; +} + +static int vfio_migration_load_buffer_local(QEMUFile *f, VFIODevice *vbasedev, + uint64_t data_size) +{ + VFIORegion *region = &vbasedev->migration->region; + uint64_t data_offset = 0, size, report_size; + int ret; + + do { + ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset), + region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset)); + if (ret < 0) { + return ret; + } + + if (data_offset + data_size > region->size) { + /* + * If data_size is greater than the data section of migration region + * then iterate the write buffer operation. This case can occur if + * size of migration region at destination is smaller than size of + * migration region at source. + */ + report_size = size = region->size - data_offset; + data_size -= size; + } else { + report_size = size = data_size; + data_size = 0; + } + + trace_vfio_load_state_device_data_local(vbasedev->name, data_offset, + size); + + while (size) { + void *buf; + uint64_t sec_size; + bool buf_alloc = false; + + buf = get_data_section_size(region, data_offset, size, &sec_size); + + if (!buf) { + buf = g_try_malloc(sec_size); + if (!buf) { + error_report("%s: Error allocating buffer ", __func__); + return -ENOMEM; + } + buf_alloc = true; + } + + qemu_get_buffer(f, buf, sec_size); + + if (buf_alloc) { + ret = vfio_mig_write(vbasedev, buf, sec_size, + region->fd_offset + data_offset); + g_free(buf); + + if (ret < 0) { + return ret; + } + } + size -= sec_size; + data_offset += sec_size; + } + + ret = vfio_mig_write(vbasedev, &report_size, sizeof(report_size), + region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_size)); + if (ret < 0) { + return ret; + } + } while (data_size); + + return 0; +} + +static int vfio_migration_update_pending_local(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + VFIORegion *region = &migration->region; + uint64_t pending_bytes = 0; + int ret; + + ret = vfio_mig_read(vbasedev, &pending_bytes, sizeof(pending_bytes), + region->fd_offset + VFIO_MIG_STRUCT_OFFSET(pending_bytes)); + if (ret < 0) { + migration->pending_bytes = 0; + return ret; + } + + migration->pending_bytes = pending_bytes; + trace_vfio_update_pending(vbasedev->name, pending_bytes); + return 0; +} + +static void vfio_migration_cleanup_local(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + + if (migration->region.mmaps) { + vfio_region_unmap(&migration->region); + } +} + +static int vfio_migration_save_setup_local(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + int ret = -1; + + if (migration->region.mmaps) { + /* + * Calling vfio_region_mmap() from migration thread. Memory API called + * from this function require locking the iothread when called from + * outside the main loop thread. + */ + qemu_mutex_lock_iothread(); + ret = vfio_region_mmap(&migration->region); + qemu_mutex_unlock_iothread(); + if (ret) { + error_report("%s: Failed to mmap VFIO migration region: %s", + vbasedev->name, strerror(-ret)); + error_report("%s: Falling back to slow path", vbasedev->name); + } + } + return ret; +} + +static int vfio_migration_load_setup_local(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + int ret = -1; + + if (migration->region.mmaps) { + ret = vfio_region_mmap(&migration->region); + if (ret) { + error_report("%s: Failed to mmap VFIO migration region %d: %s", + vbasedev->name, migration->region.nr, + strerror(-ret)); + error_report("%s: Falling back to slow path", vbasedev->name); + } + } + return ret; +} + +static void vfio_migration_exit_local(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + + vfio_region_exit(&migration->region); + vfio_region_finalize(&migration->region); +} + +static VFIOMigrationOps vfio_local_method = { + .save_setup = vfio_migration_save_setup_local, + .load_setup = vfio_migration_load_setup_local, + .update_pending = vfio_migration_update_pending_local, + .save_buffer = vfio_migration_save_buffer_local, + .load_buffer = vfio_migration_load_buffer_local, + .set_state = vfio_migration_set_state_local, + .cleanup = vfio_migration_cleanup_local, + .exit = vfio_migration_exit_local, +}; + +int vfio_migration_probe_local(VFIODevice *vbasedev) +{ + int ret; + Object *obj; + struct vfio_region_info *info = NULL; + VFIOMigration *migration = vbasedev->migration; + + obj = vbasedev->ops->vfio_get_object(vbasedev); + if (!obj) { + return -EINVAL; + } + + ret = vfio_get_dev_region_info(vbasedev, + VFIO_REGION_TYPE_MIGRATION_DEPRECATED, + VFIO_REGION_SUBTYPE_MIGRATION_DEPRECATED, + &info); + if (ret) { + return -EINVAL; + } + + ret = vfio_region_setup(obj, vbasedev, &vbasedev->migration->region, + info->index, "migration"); + if (ret) { + error_report("%s: Failed to setup VFIO migration region %d: %s", + vbasedev->name, info->index, strerror(-ret)); + goto err; + } + + if (!vbasedev->migration->region.size) { + error_report("%s: Invalid zero-sized VFIO migration region %d", + vbasedev->name, info->index); + ret = -EINVAL; + goto err; + } + + trace_vfio_migration_probe_local(vbasedev->name, info->index); + migration->ops = &vfio_local_method; + g_free(info); + return 0; + +err: + vfio_migration_exit_local(vbasedev); + g_free(info); + return ret; +} diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 0c67ed85f3..bb62e1ca0e 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -46,311 +46,6 @@ static int64_t bytes_transferred; -static inline int vfio_mig_access(VFIODevice *vbasedev, void *val, int count, - off_t off, bool iswrite) -{ - int ret; - - ret = iswrite ? pwrite(vbasedev->fd, val, count, off) : - pread(vbasedev->fd, val, count, off); - if (ret < count) { - error_report("vfio_mig_%s %d byte %s: failed at offset 0x%" - HWADDR_PRIx", err: %s", iswrite ? "write" : "read", count, - vbasedev->name, off, strerror(errno)); - return (ret < 0) ? ret : -EINVAL; - } - return 0; -} - -static int vfio_mig_rw(VFIODevice *vbasedev, __u8 *buf, size_t count, - off_t off, bool iswrite) -{ - int ret, done = 0; - __u8 *tbuf = buf; - - while (count) { - int bytes = 0; - - if (count >= 8 && !(off % 8)) { - bytes = 8; - } else if (count >= 4 && !(off % 4)) { - bytes = 4; - } else if (count >= 2 && !(off % 2)) { - bytes = 2; - } else { - bytes = 1; - } - - ret = vfio_mig_access(vbasedev, tbuf, bytes, off, iswrite); - if (ret) { - return ret; - } - - count -= bytes; - done += bytes; - off += bytes; - tbuf += bytes; - } - return done; -} - -#define vfio_mig_read(f, v, c, o) vfio_mig_rw(f, (__u8 *)v, c, o, false) -#define vfio_mig_write(f, v, c, o) vfio_mig_rw(f, (__u8 *)v, c, o, true) - -#define VFIO_MIG_STRUCT_OFFSET(f) \ - offsetof(struct vfio_device_migration_info, f) -/* - * Change the device_state register for device @vbasedev. Bits set in @mask - * are preserved, bits set in @value are set, and bits not set in either @mask - * or @value are cleared in device_state. If the register cannot be accessed, - * the resulting state would be invalid, or the device enters an error state, - * an error is returned. - */ - -static int vfio_migration_set_state_local(VFIODevice *vbasedev, uint32_t mask, - uint32_t value) -{ - VFIOMigration *migration = vbasedev->migration; - VFIORegion *region = &migration->region; - off_t dev_state_off = region->fd_offset + - VFIO_MIG_STRUCT_OFFSET(device_state); - uint32_t device_state; - int ret; - - ret = vfio_mig_read(vbasedev, &device_state, sizeof(device_state), - dev_state_off); - if (ret < 0) { - return ret; - } - - device_state = (device_state & mask) | value; - - if (!VFIO_DEVICE_STATE_VALID(device_state)) { - return -EINVAL; - } - - ret = vfio_mig_write(vbasedev, &device_state, sizeof(device_state), - dev_state_off); - if (ret < 0) { - int rret; - - rret = vfio_mig_read(vbasedev, &device_state, sizeof(device_state), - dev_state_off); - - if ((rret < 0) || (VFIO_DEVICE_STATE_IS_ERROR(device_state))) { - hw_error("%s: Device in error state 0x%x", vbasedev->name, - device_state); - return rret ? rret : -EIO; - } - return ret; - } - - migration->device_state = device_state; - trace_vfio_migration_set_state(vbasedev->name, device_state); - return 0; -} - -static void *get_data_section_size(VFIORegion *region, uint64_t data_offset, - uint64_t data_size, uint64_t *size) -{ - void *ptr = NULL; - uint64_t limit = 0; - int i; - - if (!region->mmaps) { - if (size) { - *size = MIN(data_size, region->size - data_offset); - } - return ptr; - } - - for (i = 0; i < region->nr_mmaps; i++) { - VFIOMmap *map = region->mmaps + i; - - if ((data_offset >= map->offset) && - (data_offset < map->offset + map->size)) { - - /* check if data_offset is within sparse mmap areas */ - ptr = map->mmap + data_offset - map->offset; - if (size) { - *size = MIN(data_size, map->offset + map->size - data_offset); - } - break; - } else if ((data_offset < map->offset) && - (!limit || limit > map->offset)) { - /* - * data_offset is not within sparse mmap areas, find size of - * non-mapped area. Check through all list since region->mmaps list - * is not sorted. - */ - limit = map->offset; - } - } - - if (!ptr && size) { - *size = limit ? MIN(data_size, limit - data_offset) : data_size; - } - return ptr; -} - -static int vfio_migration_save_buffer_local(QEMUFile *f, VFIODevice *vbasedev, - uint64_t *size) -{ - VFIOMigration *migration = vbasedev->migration; - VFIORegion *region = &migration->region; - uint64_t data_offset = 0, data_size = 0, sz; - int ret; - - ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset)); - if (ret < 0) { - return ret; - } - - ret = vfio_mig_read(vbasedev, &data_size, sizeof(data_size), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_size)); - if (ret < 0) { - return ret; - } - - trace_vfio_save_buffer_local(vbasedev->name, data_offset, data_size, - migration->pending_bytes); - - qemu_put_be64(f, data_size); - sz = data_size; - - while (sz) { - void *buf; - uint64_t sec_size; - bool buf_allocated = false; - - buf = get_data_section_size(region, data_offset, sz, &sec_size); - - if (!buf) { - buf = g_try_malloc(sec_size); - if (!buf) { - error_report("%s: Error allocating buffer ", __func__); - return -ENOMEM; - } - buf_allocated = true; - - ret = vfio_mig_read(vbasedev, buf, sec_size, - region->fd_offset + data_offset); - if (ret < 0) { - g_free(buf); - return ret; - } - } - - qemu_put_buffer(f, buf, sec_size); - - if (buf_allocated) { - g_free(buf); - } - sz -= sec_size; - data_offset += sec_size; - } - - ret = qemu_file_get_error(f); - - if (!ret && size) { - *size = data_size; - } - - return ret; -} - -static int vfio_migration_load_buffer_local(QEMUFile *f, VFIODevice *vbasedev, - uint64_t data_size) -{ - VFIORegion *region = &vbasedev->migration->region; - uint64_t data_offset = 0, size, report_size; - int ret; - - do { - ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset)); - if (ret < 0) { - return ret; - } - - if (data_offset + data_size > region->size) { - /* - * If data_size is greater than the data section of migration region - * then iterate the write buffer operation. This case can occur if - * size of migration region at destination is smaller than size of - * migration region at source. - */ - report_size = size = region->size - data_offset; - data_size -= size; - } else { - report_size = size = data_size; - data_size = 0; - } - - trace_vfio_load_state_device_data_local(vbasedev->name, data_offset, - size); - - while (size) { - void *buf; - uint64_t sec_size; - bool buf_alloc = false; - - buf = get_data_section_size(region, data_offset, size, &sec_size); - - if (!buf) { - buf = g_try_malloc(sec_size); - if (!buf) { - error_report("%s: Error allocating buffer ", __func__); - return -ENOMEM; - } - buf_alloc = true; - } - - qemu_get_buffer(f, buf, sec_size); - - if (buf_alloc) { - ret = vfio_mig_write(vbasedev, buf, sec_size, - region->fd_offset + data_offset); - g_free(buf); - - if (ret < 0) { - return ret; - } - } - size -= sec_size; - data_offset += sec_size; - } - - ret = vfio_mig_write(vbasedev, &report_size, sizeof(report_size), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_size)); - if (ret < 0) { - return ret; - } - } while (data_size); - - return 0; -} - -static int vfio_migration_update_pending_local(VFIODevice *vbasedev) -{ - VFIOMigration *migration = vbasedev->migration; - VFIORegion *region = &migration->region; - uint64_t pending_bytes = 0; - int ret; - - ret = vfio_mig_read(vbasedev, &pending_bytes, sizeof(pending_bytes), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(pending_bytes)); - if (ret < 0) { - migration->pending_bytes = 0; - return ret; - } - - migration->pending_bytes = pending_bytes; - trace_vfio_update_pending(vbasedev->name, pending_bytes); - return 0; -} - static int vfio_save_device_config_state(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque; @@ -395,15 +90,6 @@ static int vfio_load_device_config_state(QEMUFile *f, void *opaque) return qemu_file_get_error(f); } -static void vfio_migration_cleanup_local(VFIODevice *vbasedev) -{ - VFIOMigration *migration = vbasedev->migration; - - if (migration->region.mmaps) { - vfio_region_unmap(&migration->region); - } -} - static void vfio_migration_cleanup(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; @@ -413,31 +99,6 @@ static void vfio_migration_cleanup(VFIODevice *vbasedev) } } -/* ---------------------------------------------------------------------- */ - -static int vfio_migration_save_setup_local(VFIODevice *vbasedev) -{ - VFIOMigration *migration = vbasedev->migration; - int ret = -1; - - if (migration->region.mmaps) { - /* - * Calling vfio_region_mmap() from migration thread. Memory API called - * from this function require locking the iothread when called from - * outside the main loop thread. - */ - qemu_mutex_lock_iothread(); - ret = vfio_region_mmap(&migration->region); - qemu_mutex_unlock_iothread(); - if (ret) { - error_report("%s: Failed to mmap VFIO migration region: %s", - vbasedev->name, strerror(-ret)); - error_report("%s: Falling back to slow path", vbasedev->name); - } - } - return ret; -} - static int vfio_save_setup(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque; @@ -640,23 +301,6 @@ static void vfio_save_state(QEMUFile *f, void *opaque) } } -static int vfio_migration_load_setup_local(VFIODevice *vbasedev) -{ - VFIOMigration *migration = vbasedev->migration; - int ret = -1; - - if (migration->region.mmaps) { - ret = vfio_region_mmap(&migration->region); - if (ret) { - error_report("%s: Failed to mmap VFIO migration region %d: %s", - vbasedev->name, migration->region.nr, - strerror(-ret)); - error_report("%s: Falling back to slow path", vbasedev->name); - } - } - return ret; -} - static int vfio_load_setup(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque; @@ -848,14 +492,6 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data) } } -static void vfio_migration_exit_local(VFIODevice *vbasedev) -{ - VFIOMigration *migration = vbasedev->migration; - - vfio_region_exit(&migration->region); - vfio_region_finalize(&migration->region); -} - static void vfio_migration_exit(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; @@ -914,63 +550,6 @@ static int vfio_migration_register_handlers(VFIODevice *vbasedev) return 0; } -static VFIOMigrationOps vfio_local_method = { - .save_setup = vfio_migration_save_setup_local, - .load_setup = vfio_migration_load_setup_local, - .update_pending = vfio_migration_update_pending_local, - .save_buffer = vfio_migration_save_buffer_local, - .load_buffer = vfio_migration_load_buffer_local, - .set_state = vfio_migration_set_state_local, - .cleanup = vfio_migration_cleanup_local, - .exit = vfio_migration_exit_local, -}; - -static int vfio_migration_probe_local(VFIODevice *vbasedev) -{ - int ret; - Object *obj; - struct vfio_region_info *info = NULL; - VFIOMigration *migration = vbasedev->migration; - - obj = vbasedev->ops->vfio_get_object(vbasedev); - if (!obj) { - return -EINVAL; - } - - ret = vfio_get_dev_region_info(vbasedev, - VFIO_REGION_TYPE_MIGRATION_DEPRECATED, - VFIO_REGION_SUBTYPE_MIGRATION_DEPRECATED, - &info); - if (ret) { - return -EINVAL; - } - - ret = vfio_region_setup(obj, vbasedev, &vbasedev->migration->region, - info->index, "migration"); - if (ret) { - error_report("%s: Failed to setup VFIO migration region %d: %s", - vbasedev->name, info->index, strerror(-ret)); - goto err; - } - - if (!vbasedev->migration->region.size) { - error_report("%s: Invalid zero-sized VFIO migration region %d", - vbasedev->name, info->index); - ret = -EINVAL; - goto err; - } - - trace_vfio_migration_probe_local(vbasedev->name, info->index); - migration->ops = &vfio_local_method; - g_free(info); - return 0; - -err: - vfio_migration_exit_local(vbasedev); - g_free(info); - return ret; -} - /* ---------------------------------------------------------------------- */ int64_t vfio_mig_bytes_transferred(void) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 8ef85a871c..be8adf890f 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -256,6 +256,7 @@ int vfio_spapr_remove_window(VFIOContainer *container, hwaddr offset_within_address_space); int vfio_migration_probe(VFIODevice *vbasedev, Error **errp); +int vfio_migration_probe_local(VFIODevice *vbasedev); void vfio_migration_finalize(VFIODevice *vbasedev); #endif /* HW_VFIO_VFIO_COMMON_H */ From patchwork Tue May 24 06:18:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rao, Lei" X-Patchwork-Id: 12859732 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7484AC433EF for ; Tue, 24 May 2022 06:47:53 +0000 (UTC) Received: from localhost ([::1]:37820 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ntOKe-0004Sf-B0 for qemu-devel@archiver.kernel.org; Tue, 24 May 2022 02:47:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38522) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtZ-0008WA-3w for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:53 -0400 Received: from mga17.intel.com ([192.55.52.151]:42013) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtW-0000MN-1L for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653373190; x=1684909190; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LeOqX7h2dQ6e5+PbZ6OqSdcvsKT7TR0ojCs3G20PDkk=; b=Lf5E59ej40ia2bF8WFafXbakvVtoHZ0z4/9g7J+vCEPSdWv8MkGFa7Ac pmS1YCIQlV55C077uT7CSvjkxt77DZ6WEFqwcvCbqhbk/nJpyuvFJw4Kk 0BrdvF0bytI75XhqdybuIE+ONs5GD4VafV8vXUXVle27gS0DkkzFWk//W Mk0SWun0I6FSBtLIwccRXAfTfypFtxvAiPbTy/Zd7ixuYFF/n8wPudMmc VO84ZLXal2LvVJoPrOaxYo4ud65Zw3+KMP1kUsu9QBOEmsjeKNC9lt3pm NHTGgnIBBN/HM1nS0n3WoWKVf1Yqyj0P8Jom2ucL41E4XOAD94NAFUp8p w==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="253943269" X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="253943269" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 23:19:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="601059869" Received: from leirao-pc.bj.intel.com ([10.238.156.102]) by orsmga008.jf.intel.com with ESMTP; 23 May 2022 23:19:41 -0700 From: Lei Rao To: alex.williamson@redhat.com, kevin.tian@intel.com, eddie.dong@intel.com, jason.zeng@intel.com, quintela@redhat.com, dgilbert@redhat.com, yadong.li@intel.com, yi.l.liu@intel.com Cc: qemu-devel@nongnu.org, Lei Rao Subject: [RFC PATCH 10/13] vfio/pci: introduce command-line parameters to specify migration method Date: Tue, 24 May 2022 14:18:45 +0800 Message-Id: <20220524061848.1615706-11-lei.rao@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220524061848.1615706-1-lei.rao@intel.com> References: <20220524061848.1615706-1-lei.rao@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=lei.rao@intel.com; helo=mga17.intel.com X-Spam_score_int: -44 X-Spam_score: -4.5 X-Spam_bar: ---- X-Spam_report: (-4.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add command-line parameters (x-plugin-path and x-plugin-arg) of migration plugin for VFIO PCI functions. x-plugin-path indicates the path of a dynamic load library and x-plugin-arg is the necessary parameter to load and use it. A typical example is, if the plugin communicates with the agent running on IPU/DPU backend SOC through network, the argument should be the IP and Port of agent. The usage as follows: -device vfio-pci,id=$ID,host=$bdf,x-enable-migration,\ x-plugin-path=$plugin_path,x-plugin-arg= Signed-off-by: Lei Rao Reviewed-by: Eddie Dong --- hw/vfio/pci.c | 2 ++ include/hw/vfio/vfio-common.h | 6 ++++++ 2 files changed, 8 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 939dcc3d4a..1553ba7116 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3309,6 +3309,8 @@ static Property vfio_pci_dev_properties[] = { qdev_prop_nv_gpudirect_clique, uint8_t), DEFINE_PROP_OFF_AUTO_PCIBAR("x-msix-relocation", VFIOPCIDevice, msix_relo, OFF_AUTOPCIBAR_OFF), + DEFINE_PROP_STRING("x-plugin-path", VFIOPCIDevice, vbasedev.desc.path), + DEFINE_PROP_STRING("x-plugin-arg", VFIOPCIDevice, vbasedev.desc.arg), /* * TODO - support passed fds... is this necessary? * DEFINE_PROP_STRING("vfiofd", VFIOPCIDevice, vfiofd_name), diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index be8adf890f..45d6d75284 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -58,6 +58,11 @@ typedef struct VFIORegion { uint8_t nr; /* cache the region number for debug */ } VFIORegion; +struct vfio_migration_plugin_desc { + char *path; + char *arg; +}; + typedef struct VFIOMigrationOps VFIOMigrationOps; typedef struct VFIOMigration { @@ -144,6 +149,7 @@ typedef struct VFIODevice { unsigned int num_regions; unsigned int flags; VFIOMigration *migration; + struct vfio_migration_plugin_desc desc; Error *migration_blocker; OnOffAuto pre_copy_dirty_page_tracking; } VFIODevice; From patchwork Tue May 24 06:18:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rao, Lei" X-Patchwork-Id: 12859731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CE35FC433EF for ; Tue, 24 May 2022 06:42:58 +0000 (UTC) Received: from localhost ([::1]:60134 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ntOFt-0000LS-Mi for qemu-devel@archiver.kernel.org; Tue, 24 May 2022 02:42:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38544) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtb-00005L-0H for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:55 -0400 Received: from mga17.intel.com ([192.55.52.151]:42009) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtY-0000LD-43 for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653373192; x=1684909192; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jBV91yYHhKnXobqOAYSB7Y2hYo/KgNApTDMAh7AUjek=; b=EHuDBt0le/nDRIdNnCEmk4nB3FmfFRvp4CM0UiYAFfdlKv4mNPG/Co7W GqEy3WpCHaEMeHyqIYlz7Z8kbE0CCo+1LF4HXp8u+vSx+idpCBrimxysb nojKv9P3nfd6KbZvu+t7ekJB+r3U1NpGu6QVFkwpkOy/g3VZQdDv1MWSZ RHAqOIcnXRI5g4vZaN6LJBd/NteROyYkzDqiiHhk+Toh7oZeMrXOOLfVa R5nQ9YAhiREqoxh7x/ncLOW2iYqDNoZ4M7dFDVF/NN9T+IlDGVAG3C2LS YqlW1BxvpOhQjn9sVEVPQOMyDKpxASPIChyV2OIs64Uewd+/Tr3//Uy/b A==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="253943273" X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="253943273" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 23:19:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="601059877" Received: from leirao-pc.bj.intel.com ([10.238.156.102]) by orsmga008.jf.intel.com with ESMTP; 23 May 2022 23:19:45 -0700 From: Lei Rao To: alex.williamson@redhat.com, kevin.tian@intel.com, eddie.dong@intel.com, jason.zeng@intel.com, quintela@redhat.com, dgilbert@redhat.com, yadong.li@intel.com, yi.l.liu@intel.com Cc: qemu-devel@nongnu.org, Lei Rao Subject: [RFC PATCH 11/13] vfio/migration: add a plugin layer to support out-of-band live migration Date: Tue, 24 May 2022 14:18:46 +0800 Message-Id: <20220524061848.1615706-12-lei.rao@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220524061848.1615706-1-lei.rao@intel.com> References: <20220524061848.1615706-1-lei.rao@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=lei.rao@intel.com; helo=mga17.intel.com X-Spam_score_int: -44 X-Spam_score: -4.5 X-Spam_bar: ---- X-Spam_report: (-4.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Introduce a plugin mechanism under VFIOMigrationOps layer. Each vendor can provide a dynamic load library that implements the communication driver to talk with IPU/DPU backend agent for saving and restoring device state during live migration. There are three interfaces between QEMU VFIO and a migration plugin: - VFIOLMPluginGetVersion: This is a function type. Plugin must expose a function symbol named "vfio_lm_get_plugin_version" with this function type to return the interface version supported by the plugin. - VFIOLMPluginGetOps: This is a function type. Plugin must expose a function symbol named "vfio_lm_get_plugin_ops" with this function type to return a pointer to VFIOMigrationPluginOps struct. - VFIOMigrationPluginOps: This is a struct type containing a set of callbacks that plugin exposes. The callbacks will be invoked by QEMU VFIO during live migration for saving and restoring device states. The interfaces are defined in include/hw/vfio/vfio-migration-plugin.h. When QEMU loads a migration plugin, it will first find and invoke function symbol named "vfio_lm_get_plugin_version" to check the interface version that plugin supports. And then find and invoke function symbol named "vfio_lm_get_plugin_ops" to get vendor device specific VFIOMigrationPluginOps which will be used for saving/restoring device states during live migration. Signed-off-by: Lei Rao Reviewed-by: Eddie Dong --- docs/devel/vfio-migration-plugin.rst | 165 +++++++++++++++ hw/vfio/meson.build | 1 + hw/vfio/migration-plugin.c | 262 ++++++++++++++++++++++++ hw/vfio/migration.c | 13 +- include/hw/vfio/vfio-common.h | 12 ++ include/hw/vfio/vfio-migration-plugin.h | 21 ++ 6 files changed, 471 insertions(+), 3 deletions(-) create mode 100644 docs/devel/vfio-migration-plugin.rst create mode 100644 hw/vfio/migration-plugin.c create mode 100644 include/hw/vfio/vfio-migration-plugin.h diff --git a/docs/devel/vfio-migration-plugin.rst b/docs/devel/vfio-migration-plugin.rst new file mode 100644 index 0000000000..800d1bac0a --- /dev/null +++ b/docs/devel/vfio-migration-plugin.rst @@ -0,0 +1,165 @@ +============================ +VFIO Device Migration Plugins +============================ + +Contents: +========= +* Introduction +* Usage +* Plugin based VFIO Live Migration Flow +* Interface Description between QEMU and Plugins + +Introduction: +============ + +Plugin based VFIO live migration is an extension to VFIO live migration +mechanism, which is described in ``docs/devel/vfio-migration.rst``. It provides +an out-of-band migration solution for PCIe functions exposed by Infrastructure +Processing Units (IPU) and Data Processing Units (DPU). + +IPU/DPU usually has an SoC in the backend where a Linux system usually runs +out-of-band agents to provision and configure the interfaces and communicate +with a host management stack such as gRPC or JSON-RPC. Plugin based VFIO live +migration leverage the agents in the Soc to save/restore PCIe device states. + +This is a new feature for VFIO live migration and it allows device vendors to +develop out-of-tree plugins that can be dynamically loaded into a running QEMU +process during VFIO passthrough devices live migration. + +This document describes the interfaces between QEMU VFIO live migration +framework and the plugins. + +Usage: +====== + +An example to use VFIO migration plugin is as the following command line: + +-device vfio-pci-emu,x-enable-migration=on,x-plugin-path=$plugin_path,x-plugin-arg=$plugin_arg + +Where, + +- the 'x-enable-migration' controls whether the VFIO device supports live + migration (Not supported by default). + +- 'x-plugin-path' indicates the path of the plugin on the host. + +- 'x-plugin-arg' is a parameter required by QEMU to load and use the out-of-tree + plugin, if the plugin communicates with the backend on IPU/DPU by network, + this parameter should be . + +Plugin based VFIO Live Migration Flow: +====================================== + +The following ASCII graph describes the overall component relationship: + + +----------------------------------------------------+ + | QEMU | + | +------------------------------------------------+ | + | | VFIO Live Migration Framework | | + | | +--------------------------------------+ | | + | | | VFIOMigrationOps | | | + | | +-------^---------------------^--------+ | | + | | | | | | + | | +-------v-------+ +-------v--------+ | | + | | | VFIO LM Based | | VFIO LM Based | | | + | | |On Local Region| | On Plugin | | | + | | +-------^-------+ | +----------+ | | + | | | | |Plugin Ops+----+-+------------+ + | | | +-----+----------+ | | | + | | | | | +---------v----------+ + | +------------+-----------------------------------+ | | Vendor Specific | + | | | | Plugins(.so) | + +--------------+-------------------------------------+ +----------+---------+ + UserSpace | | +----------------+--------------------------------------------- | + Kernel | | + | | + +----------v----------------------+ | + | Kernel VFIO Driver | | + | +-------------------------+ | | + | | | | | Network + | | Vendor-Specific Driver | | | + | | | | | + | +----------^--------------+ | | + | | | | + +---------------+-----------------+ | + | | + | | +---------------------+----------------------------------------- | + Hardware | | + | +-----+-----+-----+----+-----+ | + +----------v------+ | VF0 | VF1 | VF2 | ...| VFn | | + | Traditional | +-----+-----+-----+----+-----+ | + | PCIe Devices | | | | + +-----------------+ | +--------+------------+ | | + | | | Agent |<-+----+ + | | +------------+ | + | | | | + | | SOC | | + | +---------------------+ | + | IPU/DPU | + +----------------------------+ + +Two QEMU command line options (x-plugin-path and x-plugin-arg) are introduced to +specify the corresponding plugin and its parameters for a passthrough device. +If they are specified, the plugin will be loaded in vfio_migration_probe(), +which will check the plugin version and get the pointer to the plugin's +VFIOMigrationPluginOps. If any failure during the probing, the plugin will not +be loaded, and this PCIe device will be marked as no supporting of live +migration. + +When live migration happens, VFIO live migration framework will invoke the +callbacks defined in VFIOMigrationPluginOps to save/restore the device states, +as described in the following section. + +Interface Description between QEMU and Plugins: +============================================= + +The interfaces between QEMU VFIO live migration framework and vendor-specific +plugin are defined as follows: + + - VFIOLMPluginGetVersion: + This is a function type. Plugins must expose a function symbol named + ``vfio_lm_get_plugin_version`` with this function type to return the + interface version supported by the plugin. + - VFIOLMPluginGetOps: + This is a function type. Plugins must expose a function symbol named + ``vfio_lm_get_plugin_ops`` with this function type to return a pointer + to VFIOMigrationPluginOps struct. + - VFIOMigrationPluginOps: + This is a struct type containing a set of callbacks that plugin + exposes. The callbacks will be invoked by QEMU VFIO during live + migration for saving and restoring device states. + +The interfaces are defined in include/hw/vfio/vfio-migration-plugin.h. + +When QEMU loads a migration plugin, it will first find and invoke a function +symbol named ``vfio_lm_get_plugin_version`` to check the interface version that +plugin supports. The core code will refuse to load a plugin if it doesn't export +the symbol or the version doesn't match the one QEMU supports. + +Then QEMU finds and invokes function symbol named ``vfio_lm_get_plugin_ops`` to +get vendor device-specific VFIOMigrationPluginOps which will be used for +saving/restoring device states. + +VFIOMigrationPluginOps is defined as follows: + +typedef struct VFIOMigrationPluginOps { + void *(*init)(char *devid, char *arg); + int (*save)(void *handle, uint8_t *state, uint64_t len); + int (*load)(void *handle, uint8_t *state, uint64_t len); + int (*update_pending)(void *handle, uint64_t *pending_bytes); + int (*set_state)(void *handle, uint32_t value); + int (*get_state)(void *handle, uint32_t *value); + int (*cleanup)(void *handle); +} VFIOMigrationPluginOps; + +Here: + - init(): set the PCIe device BDF and args, and get the plugin handle. + - save(): save the VFIO passthrough device states on the source. + - load(): restore the VFIO passthrough device states on the destination. + - set_state(): set the PCIe device states including SAVING, RUNNING, + STOP, and RESUMING. + - get_state(): get the PCIe device states. + - update_pending(): get the remaining bytes during data transfer. + - cleanup(): unload the plugin and release some resources. diff --git a/hw/vfio/meson.build b/hw/vfio/meson.build index 5a72b8c349..592d56536e 100644 --- a/hw/vfio/meson.build +++ b/hw/vfio/meson.build @@ -4,6 +4,7 @@ vfio_ss.add(files( 'spapr.c', 'migration.c', 'migration-local.c', + 'migration-plugin.c', )) vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( 'display.c', diff --git a/hw/vfio/migration-plugin.c b/hw/vfio/migration-plugin.c new file mode 100644 index 0000000000..63124e1571 --- /dev/null +++ b/hw/vfio/migration-plugin.c @@ -0,0 +1,262 @@ +/* + * QEMU VFIO Migration Support + * + * Copyright Intel Corporation, 2022 + * + * This software is licensed under the terms of the GNU General Public + * License version 2, as published by the Free Software Foundation, and + * may be copied, distributed, and modified under those terms. + * + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include "qemu/cutils.h" + +#include "hw/vfio/vfio-common.h" +#include "migration/qemu-file.h" +#include "qapi/error.h" +#include "hw/vfio/vfio-migration-plugin.h" +#include "sysemu/sysemu.h" + +#define CHUNK_SIZE (1024 * 1024) + +static int vfio_migration_load_plugin(VFIODevice *vbasedev) +{ + char *path = vbasedev->desc.path; + VFIOMigration *migration = vbasedev->migration; + VFIOMigrationPlugin *plugin = NULL; + VFIOLMPluginGetVersion vfio_lm_get_plugin_version = NULL; + VFIOLMPluginGetOps vfio_lm_get_plugin_ops = NULL; + + plugin = g_malloc0(sizeof(VFIOMigrationPlugin)); + if (!plugin) { + error_report("%s: Error allocating buffer", __func__); + return -ENOMEM; + } + + plugin->module = g_module_open(path, G_MODULE_BIND_LOCAL); + if (!plugin->module) { + error_report("Failed to load VFIO migration plugin:%s", path); + g_free(plugin); + return -1; + } + + if (!g_module_symbol(plugin->module, "vfio_lm_get_plugin_version", + (void *)&vfio_lm_get_plugin_version)) { + error_report("Failed to load plugin ops %s: %s", path, + g_module_error()); + goto err; + } + + if (vfio_lm_get_plugin_version() != VFIO_LM_PLUGIN_API_VERSION) { + error_report("Invalid VFIO Plugin API Version %s : %s", path, + g_module_error()); + goto err; + } + + if (!g_module_symbol(plugin->module, "vfio_lm_get_plugin_ops", + (void *)&vfio_lm_get_plugin_ops)) { + error_report("Failed to load plugin ops %s: %s", path, + g_module_error()); + goto err; + } + + plugin->ops = vfio_lm_get_plugin_ops(); + if (!plugin->ops) { + error_report("Failed to Get Plugin Ops: %s", path); + goto err; + } + + migration->plugin = plugin; + + return 0; + +err: + g_module_close(plugin->module); + g_free(plugin); + plugin = NULL; + return -1; +} + +static int vfio_migration_save_load_setup_plugin(VFIODevice *vbasedev) +{ + char *arg = vbasedev->desc.arg; + VFIOMigrationPlugin *plugin = vbasedev->migration->plugin; + + /* The name is BDF for PCIe device */ + plugin->handle = plugin->ops->init(vbasedev->name, arg); + if (!plugin->handle) { + error_report("Failed to init: %s", vbasedev->desc.path); + return -1; + } + + return 0; +} + +static void vfio_migration_cleanup_plugin(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + VFIOMigrationPlugin *plugin = migration->plugin; + + if (plugin->ops->cleanup) { + plugin->ops->cleanup(plugin->handle); + plugin->handle = NULL; + } + + if (migration->plugin->module) { + g_module_close(migration->plugin->module); + migration->plugin->module = NULL; + } + + g_free(migration->plugin); + migration->plugin = NULL; +} + +static int vfio_migration_update_pending_plugin(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + VFIOMigrationPlugin *plugin = migration->plugin; + uint64_t pending_bytes = 0; + int ret = -1; + + ret = plugin->ops->update_pending(plugin->handle, &pending_bytes); + if (ret) { + migration->pending_bytes = 0; + error_report("%s: Failed to get pending size", __func__); + return ret; + } + migration->pending_bytes = pending_bytes; + + return 0; +} + +static int vfio_migration_set_state_plugin(VFIODevice *vbasedev, uint32_t mask, + uint32_t value) +{ + int ret = -1; + uint32_t device_state = 0; + VFIOMigrationPlugin *plugin = vbasedev->migration->plugin; + + ret = plugin->ops->get_state(plugin->handle, &device_state); + if (ret) { + error_report("%s: Get device state error", vbasedev->name); + return ret; + } + + device_state = (device_state & mask) | value; + + if (!VFIO_DEVICE_STATE_VALID(device_state)) { + return -EINVAL; + } + + ret = plugin->ops->set_state(plugin->handle, device_state); + if (ret) { + error_report("%s: Device in error state 0x%x", vbasedev->name, + value); + return ret; + } + + vbasedev->migration->device_state = device_state; + + return 0; +} + +static int vfio_migration_save_buffer_plugin(QEMUFile *f, VFIODevice *vbasedev, + uint64_t *size) +{ + int ret = 0; + VFIOMigrationPlugin *plugin = vbasedev->migration->plugin; + uint64_t data_size, tmp_size; + + ret = plugin->ops->update_pending(plugin->handle, &data_size); + if (ret < 0) { + error_report("%s: Failed to get pending size", __func__); + return ret; + } + + qemu_put_be64(f, data_size); + tmp_size = data_size; + + while (tmp_size) { + uint64_t sz = tmp_size <= CHUNK_SIZE ? tmp_size : CHUNK_SIZE; + void *buf = g_try_malloc(sz); + + if (!buf) { + error_report("%s: Error allocating buffer", __func__); + return -ENOMEM; + } + + ret = plugin->ops->save(plugin->handle, buf, sz); + if (ret) { + error_report("%s:Failed saving device state", __func__); + g_free(buf); + return ret; + } + + qemu_put_buffer(f, buf, sz); + g_free(buf); + tmp_size -= sz; + } + + ret = qemu_file_get_error(f); + if (!ret && size) { + *size = data_size; + } + + return ret; +} + +static int vfio_migration_load_buffer_plugin(QEMUFile *f, VFIODevice *vbasedev, + uint64_t data_size) +{ + int ret = 0; + VFIOMigrationPlugin *plugin = vbasedev->migration->plugin; + + while (data_size) { + uint64_t sz = data_size <= CHUNK_SIZE ? data_size : CHUNK_SIZE; + void *buf = g_try_malloc(sz); + + if (!buf) { + error_report("%s: Error allocating buffer", __func__); + return -ENOMEM; + } + + qemu_get_buffer(f, buf, sz); + ret = plugin->ops->load(plugin->handle, buf, sz); + g_free(buf); + if (ret < 0) { + error_report("%s: Error loading device state", vbasedev->name); + return ret; + } + + data_size -= sz; + } + + return ret; +} + +static VFIOMigrationOps vfio_plugin_method = { + .save_setup = vfio_migration_save_load_setup_plugin, + .load_setup = vfio_migration_save_load_setup_plugin, + .update_pending = vfio_migration_update_pending_plugin, + .save_buffer = vfio_migration_save_buffer_plugin, + .load_buffer = vfio_migration_load_buffer_plugin, + .set_state = vfio_migration_set_state_plugin, + .cleanup = vfio_migration_cleanup_plugin +}; + +int vfio_migration_probe_plugin(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + + if (vfio_migration_load_plugin(vbasedev)) { + error_report("vfio migration plugin probe failed"); + return -1; + } + + migration->ops = &vfio_plugin_method; + + return 0; +} diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index bb62e1ca0e..24a3126a56 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -569,9 +569,16 @@ int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) vbasedev->migration = g_new0(VFIOMigration, 1); vbasedev->migration->vbasedev = vbasedev; - ret = vfio_migration_probe_local(vbasedev); - if (ret) { - goto add_blocker; + if (vbasedev->desc.arg != NULL && vbasedev->desc.path != NULL) { + ret = vfio_migration_probe_plugin(vbasedev); + if (ret) { + goto add_blocker; + } + } else { + ret = vfio_migration_probe_local(vbasedev); + if (ret) { + goto add_blocker; + } } ret = vfio_migration_register_handlers(vbasedev); diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 45d6d75284..2ea016a894 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -22,6 +22,7 @@ #define HW_VFIO_VFIO_COMMON_H #include "exec/memory.h" +#include "qemu/iov.h" #include "qemu/queue.h" #include "qemu/notify.h" #include "ui/console.h" @@ -30,6 +31,9 @@ #include #endif #include "sysemu/sysemu.h" +#include "vfio-migration-plugin.h" +#include +#include #define VFIO_MSG_PREFIX "vfio %s: " @@ -58,6 +62,12 @@ typedef struct VFIORegion { uint8_t nr; /* cache the region number for debug */ } VFIORegion; +typedef struct VFIOMigrationPlugin { + GModule *module; + VFIOMigrationPluginOps *ops; + void *handle; +} VFIOMigrationPlugin; + struct vfio_migration_plugin_desc { char *path; char *arg; @@ -70,6 +80,7 @@ typedef struct VFIOMigration { VMChangeStateEntry *vm_state; VFIORegion region; VFIOMigrationOps *ops; + VFIOMigrationPlugin *plugin; uint32_t device_state; int vm_running; Notifier migration_state; @@ -263,6 +274,7 @@ int vfio_spapr_remove_window(VFIOContainer *container, int vfio_migration_probe(VFIODevice *vbasedev, Error **errp); int vfio_migration_probe_local(VFIODevice *vbasedev); +int vfio_migration_probe_plugin(VFIODevice *vbasedev); void vfio_migration_finalize(VFIODevice *vbasedev); #endif /* HW_VFIO_VFIO_COMMON_H */ diff --git a/include/hw/vfio/vfio-migration-plugin.h b/include/hw/vfio/vfio-migration-plugin.h new file mode 100644 index 0000000000..02f6cc4608 --- /dev/null +++ b/include/hw/vfio/vfio-migration-plugin.h @@ -0,0 +1,21 @@ +#ifndef HW_VFIO_PLUGIN_MIGRATION_H +#define HW_VFIO_PLUGIN_MIGRATION_H + +#include + +#define VFIO_LM_PLUGIN_API_VERSION 0 + +typedef struct VFIOMigrationPluginOps { + void *(*init)(char *devid, char *arg); + int (*save)(void *handle, uint8_t *state, uint64_t len); + int (*load)(void *handle, uint8_t *state, uint64_t len); + int (*update_pending)(void *handle, uint64_t *pending_bytes); + int (*set_state)(void *handle, uint32_t value); + int (*get_state)(void *handle, uint32_t *value); + int (*cleanup)(void *handle); +} VFIOMigrationPluginOps; + +typedef int (*VFIOLMPluginGetVersion)(void); +typedef VFIOMigrationPluginOps* (*VFIOLMPluginGetOps)(void); + +#endif From patchwork Tue May 24 06:18:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rao, Lei" X-Patchwork-Id: 12859733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C6896C433EF for ; Tue, 24 May 2022 06:47:59 +0000 (UTC) Received: from localhost ([::1]:38020 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ntOKk-0004cm-It for qemu-devel@archiver.kernel.org; Tue, 24 May 2022 02:47:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38546) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtb-00005M-9i for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:55 -0400 Received: from mga17.intel.com ([192.55.52.151]:42013) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtZ-0000MN-Hi for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653373193; x=1684909193; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=BidQ/Lj4AzL1srh2FSCncmHpFcQXw1WY/Lep5pe1iY4=; b=V6dcbX6q1+r0g2WLfnZYyW3JiVduC/fshsd3yqLxLtudN3ozKHg1oJo5 Qo5w/oBHKn0JI3Mc99Fps6nKV6jHEKGp0Pn2rcgoslZsB1tskGNq/BtZT wSBFcsbuTXTB5s/rbYStAgRFDhTJ1+D6OVBy4b9QNvU41x+NQCW2VBs3v DdOYRjioXMPrf5Z4clm7m40ZSMU/O/CgkYxRAIZuYL9lEe5cpxjoq0W0t FX6FEYwwJWnVOuIz7NrkB3aEqHKsyT3fIQwBon1y5fzNMaZFkt9aoWTFk qbzY/9F2JUTMkpdo9De13micMCV5aj99ncn5gOoW7DIeiaye2w8fGCN2D Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="253943289" X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="253943289" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 23:19:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="601059892" Received: from leirao-pc.bj.intel.com ([10.238.156.102]) by orsmga008.jf.intel.com with ESMTP; 23 May 2022 23:19:49 -0700 From: Lei Rao To: alex.williamson@redhat.com, kevin.tian@intel.com, eddie.dong@intel.com, jason.zeng@intel.com, quintela@redhat.com, dgilbert@redhat.com, yadong.li@intel.com, yi.l.liu@intel.com Cc: qemu-devel@nongnu.org, Lei Rao , Eddie Dong Subject: [RFC PATCH 12/13] vfio/migration: add some trace-events for vfio migration plugin Date: Tue, 24 May 2022 14:18:47 +0800 Message-Id: <20220524061848.1615706-13-lei.rao@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220524061848.1615706-1-lei.rao@intel.com> References: <20220524061848.1615706-1-lei.rao@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=lei.rao@intel.com; helo=mga17.intel.com X-Spam_score_int: -44 X-Spam_score: -4.5 X-Spam_bar: ---- X-Spam_report: (-4.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add some trace-events including trace_vfio_migration_plugin_probe trace_vfio_plugin_save_buffer, trace_vfio_plugin_load_state_device_data trace_vfio_update_pending, and trace_vfio_migration_set_state to make debugging easier. Signed-off-by: Lei Rao Reviewed-by: Eddie Dong --- hw/vfio/migration-plugin.c | 10 +++++++--- hw/vfio/trace-events | 3 +++ 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/hw/vfio/migration-plugin.c b/hw/vfio/migration-plugin.c index 63124e1571..c545cbe334 100644 --- a/hw/vfio/migration-plugin.c +++ b/hw/vfio/migration-plugin.c @@ -19,6 +19,7 @@ #include "qapi/error.h" #include "hw/vfio/vfio-migration-plugin.h" #include "sysemu/sysemu.h" +#include "trace.h" #define CHUNK_SIZE (1024 * 1024) @@ -128,7 +129,7 @@ static int vfio_migration_update_pending_plugin(VFIODevice *vbasedev) return ret; } migration->pending_bytes = pending_bytes; - + trace_vfio_update_pending(vbasedev->name, pending_bytes); return 0; } @@ -159,7 +160,7 @@ static int vfio_migration_set_state_plugin(VFIODevice *vbasedev, uint32_t mask, } vbasedev->migration->device_state = device_state; - + trace_vfio_migration_set_state(vbasedev->name, device_state); return 0; } @@ -179,6 +180,7 @@ static int vfio_migration_save_buffer_plugin(QEMUFile *f, VFIODevice *vbasedev, qemu_put_be64(f, data_size); tmp_size = data_size; + trace_vfio_save_buffer_plugin(vbasedev->name, data_size); while (tmp_size) { uint64_t sz = tmp_size <= CHUNK_SIZE ? tmp_size : CHUNK_SIZE; void *buf = g_try_malloc(sz); @@ -214,6 +216,7 @@ static int vfio_migration_load_buffer_plugin(QEMUFile *f, VFIODevice *vbasedev, int ret = 0; VFIOMigrationPlugin *plugin = vbasedev->migration->plugin; + trace_vfio_load_state_device_data_plugin(vbasedev->name, data_size); while (data_size) { uint64_t sz = data_size <= CHUNK_SIZE ? data_size : CHUNK_SIZE; void *buf = g_try_malloc(sz); @@ -257,6 +260,7 @@ int vfio_migration_probe_plugin(VFIODevice *vbasedev) } migration->ops = &vfio_plugin_method; - + trace_vfio_migration_probe_plugin(vbasedev->name, vbasedev->desc.path, + vbasedev->desc.arg); return 0; } diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index ca85edeb11..6c2cba29fd 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -149,12 +149,14 @@ vfio_display_edid_write_error(void) "" # migration.c vfio_migration_probe_local(const char *name, uint32_t index) " (%s) Region %d" +vfio_migration_probe_plugin(const char *name, const char *path, const char *arg) " (%s) Plugin path=%s arg=%s" vfio_migration_set_state(const char *name, uint32_t state) " (%s) state %d" vfio_vmstate_change(const char *name, int running, const char *reason, uint32_t dev_state) " (%s) running %d reason %s device state %d" vfio_migration_state_notifier(const char *name, const char *state) " (%s) state %s" vfio_save_setup(const char *name) " (%s)" vfio_save_cleanup(const char *name) " (%s)" vfio_save_buffer_local(const char *name, uint64_t data_offset, uint64_t data_size, uint64_t pending) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64" pending 0x%"PRIx64 +vfio_save_buffer_plugin(const char *name, uint64_t data_size) " (%s) data size 0x%"PRIx64 vfio_update_pending(const char *name, uint64_t pending) " (%s) pending 0x%"PRIx64 vfio_save_device_config_state(const char *name) " (%s)" vfio_save_pending(const char *name, uint64_t precopy, uint64_t postcopy, uint64_t compatible) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" compatible 0x%"PRIx64 @@ -163,6 +165,7 @@ vfio_save_complete_precopy(const char *name) " (%s)" vfio_load_device_config_state(const char *name) " (%s)" vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64 vfio_load_state_device_data_local(const char *name, uint64_t data_offset, uint64_t data_size) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64 +vfio_load_state_device_data_plugin(const char *name, uint64_t data_size) " (%s) data size 0x%"PRIx64 vfio_load_cleanup(const char *name) " (%s)" vfio_get_dirty_bitmap(int fd, uint64_t iova, uint64_t size, uint64_t bitmap_size, uint64_t start) "container fd=%d, iova=0x%"PRIx64" size= 0x%"PRIx64" bitmap_size=0x%"PRIx64" start=0x%"PRIx64 vfio_iommu_map_dirty_notify(uint64_t iova_start, uint64_t iova_end) "iommu dirty @ 0x%"PRIx64" - 0x%"PRIx64 From patchwork Tue May 24 06:18:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rao, Lei" X-Patchwork-Id: 12859722 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7FD18C433EF for ; Tue, 24 May 2022 06:39:50 +0000 (UTC) Received: from localhost ([::1]:56898 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ntOCr-0006Ik-Bv for qemu-devel@archiver.kernel.org; Tue, 24 May 2022 02:39:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38576) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtf-0000A2-Ob for qemu-devel@nongnu.org; Tue, 24 May 2022 02:20:00 -0400 Received: from mga17.intel.com ([192.55.52.151]:42016) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntNtd-0000NA-Sl for qemu-devel@nongnu.org; Tue, 24 May 2022 02:19:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653373197; x=1684909197; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=cE6E4Jc2xo4pfVcaXlXSw4DTF9+/qGTREWZwt/36f30=; b=OUlCTIhqYyPo4xxCB+FUgnQ3RAwYycPc5rLwWQenEK8TZtq+uu4I16tt z7YqTyT7YxzGJW1wAIi+9ThB2V1mfPaZZWctm7BrAATWR0umi/SJbvVgX yeMAeLjgPkAePW3a1ud01vQv79p4HEIbb2a3wDeIyUzbkdXR+FP5UYDTW c+2ymtk2TtYKV0LARGR/g0L1Eog93Otu/SErvyhWOSHmq5F1ccEkwl+o9 5zhSC21qN+VcpM4Uh8O/Ftkod8PZdJjxoq2M0/1PNVMsVijtn6UjmYNOZ 7B8V7nxtY04F8mKuFK9lZbPcDjaSgVduSqBh77FnMio4dKi6evCuC1lRx Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="253943301" X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="253943301" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 23:19:56 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="601059911" Received: from leirao-pc.bj.intel.com ([10.238.156.102]) by orsmga008.jf.intel.com with ESMTP; 23 May 2022 23:19:53 -0700 From: Lei Rao To: alex.williamson@redhat.com, kevin.tian@intel.com, eddie.dong@intel.com, jason.zeng@intel.com, quintela@redhat.com, dgilbert@redhat.com, yadong.li@intel.com, yi.l.liu@intel.com Cc: qemu-devel@nongnu.org, Lei Rao Subject: [RFC PATCH 13/13] vfio/migration: make the region and plugin member of struct VFIOMigration to be a union Date: Tue, 24 May 2022 14:18:48 +0800 Message-Id: <20220524061848.1615706-14-lei.rao@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220524061848.1615706-1-lei.rao@intel.com> References: <20220524061848.1615706-1-lei.rao@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=lei.rao@intel.com; helo=mga17.intel.com X-Spam_score_int: -44 X-Spam_score: -4.5 X-Spam_bar: ---- X-Spam_report: (-4.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Since a VFIO device either uses In-Band or Out-of-Band live migration. So, the region and plugin in VFIOMigration can be put into a union. Signed-off-by: Lei Rao Reviewed-by: Eddie Dong --- hw/vfio/migration-local.c | 33 ++++++++++++++++++--------------- include/hw/vfio/vfio-common.h | 6 ++++-- 2 files changed, 22 insertions(+), 17 deletions(-) diff --git a/hw/vfio/migration-local.c b/hw/vfio/migration-local.c index 46c8baed50..13d1abee5a 100644 --- a/hw/vfio/migration-local.c +++ b/hw/vfio/migration-local.c @@ -98,7 +98,7 @@ static int vfio_migration_set_state_local(VFIODevice *vbasedev, uint32_t mask, uint32_t value) { VFIOMigration *migration = vbasedev->migration; - VFIORegion *region = &migration->region; + VFIORegion *region = migration->region; off_t dev_state_off = region->fd_offset + VFIO_MIG_STRUCT_OFFSET(device_state); uint32_t device_state; @@ -184,7 +184,7 @@ static int vfio_migration_save_buffer_local(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size) { VFIOMigration *migration = vbasedev->migration; - VFIORegion *region = &migration->region; + VFIORegion *region = migration->region; uint64_t data_offset = 0, data_size = 0, sz; int ret; @@ -250,7 +250,7 @@ static int vfio_migration_save_buffer_local(QEMUFile *f, VFIODevice *vbasedev, static int vfio_migration_load_buffer_local(QEMUFile *f, VFIODevice *vbasedev, uint64_t data_size) { - VFIORegion *region = &vbasedev->migration->region; + VFIORegion *region = vbasedev->migration->region; uint64_t data_offset = 0, size, report_size; int ret; @@ -322,7 +322,7 @@ static int vfio_migration_load_buffer_local(QEMUFile *f, VFIODevice *vbasedev, static int vfio_migration_update_pending_local(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; - VFIORegion *region = &migration->region; + VFIORegion *region = migration->region; uint64_t pending_bytes = 0; int ret; @@ -342,8 +342,8 @@ static void vfio_migration_cleanup_local(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; - if (migration->region.mmaps) { - vfio_region_unmap(&migration->region); + if (migration->region->mmaps) { + vfio_region_unmap(migration->region); } } @@ -352,14 +352,14 @@ static int vfio_migration_save_setup_local(VFIODevice *vbasedev) VFIOMigration *migration = vbasedev->migration; int ret = -1; - if (migration->region.mmaps) { + if (migration->region->mmaps) { /* * Calling vfio_region_mmap() from migration thread. Memory API called * from this function require locking the iothread when called from * outside the main loop thread. */ qemu_mutex_lock_iothread(); - ret = vfio_region_mmap(&migration->region); + ret = vfio_region_mmap(migration->region); qemu_mutex_unlock_iothread(); if (ret) { error_report("%s: Failed to mmap VFIO migration region: %s", @@ -375,11 +375,11 @@ static int vfio_migration_load_setup_local(VFIODevice *vbasedev) VFIOMigration *migration = vbasedev->migration; int ret = -1; - if (migration->region.mmaps) { - ret = vfio_region_mmap(&migration->region); + if (migration->region->mmaps) { + ret = vfio_region_mmap(migration->region); if (ret) { error_report("%s: Failed to mmap VFIO migration region %d: %s", - vbasedev->name, migration->region.nr, + vbasedev->name, migration->region->nr, strerror(-ret)); error_report("%s: Falling back to slow path", vbasedev->name); } @@ -391,8 +391,10 @@ static void vfio_migration_exit_local(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; - vfio_region_exit(&migration->region); - vfio_region_finalize(&migration->region); + vfio_region_exit(migration->region); + vfio_region_finalize(migration->region); + g_free(migration->region); + migration->region = NULL; } static VFIOMigrationOps vfio_local_method = { @@ -426,7 +428,8 @@ int vfio_migration_probe_local(VFIODevice *vbasedev) return -EINVAL; } - ret = vfio_region_setup(obj, vbasedev, &vbasedev->migration->region, + migration->region = g_new0(VFIORegion, 1); + ret = vfio_region_setup(obj, vbasedev, vbasedev->migration->region, info->index, "migration"); if (ret) { error_report("%s: Failed to setup VFIO migration region %d: %s", @@ -434,7 +437,7 @@ int vfio_migration_probe_local(VFIODevice *vbasedev) goto err; } - if (!vbasedev->migration->region.size) { + if (!vbasedev->migration->region->size) { error_report("%s: Invalid zero-sized VFIO migration region %d", vbasedev->name, info->index); ret = -EINVAL; diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 2ea016a894..bded2b4908 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -78,9 +78,11 @@ typedef struct VFIOMigrationOps VFIOMigrationOps; typedef struct VFIOMigration { struct VFIODevice *vbasedev; VMChangeStateEntry *vm_state; - VFIORegion region; VFIOMigrationOps *ops; - VFIOMigrationPlugin *plugin; + union { + VFIORegion *region; + VFIOMigrationPlugin *plugin; + }; uint32_t device_state; int vm_running; Notifier migration_state;