From patchwork Fri Jun 30 07:36:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Duan, Zhenzhong" X-Patchwork-Id: 13297697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5F1B9EB64D7 for ; Fri, 30 Jun 2023 07:51:10 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qF8tG-0001tD-C3; Fri, 30 Jun 2023 03:50:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qF8tE-0001sB-6P for qemu-devel@nongnu.org; Fri, 30 Jun 2023 03:50:00 -0400 Received: from mga03.intel.com ([134.134.136.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qF8tC-0002be-4F for qemu-devel@nongnu.org; Fri, 30 Jun 2023 03:49:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688111398; x=1719647398; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=20hMq7y7c4qZBJSiY004C/qGpgUHmVDDFDGqjY1tRBw=; b=FhtHnsc5iSPkXIf2xSqobFQmbGDrRQ44JjJ3TabDPX+tKvKyTvB0ZyvH GKAvPQRCReDQHspFbEzu8gTbrHA2S4NUATTuD08cN4VVwig2EE19J6NsZ d/grblQGdhuaCzJGbFd6S5R9Q+E4VWQmZSHd20xBDxTlXl7avl4UlTDU1 21IFPY79aQZNE8Ao+6xP6V3IuVCV5rMehr/LjeYmUKGzAu/xlYA+yIcWr et+H72+rRjwBJHzVojOvzimxJZz4103oaBVktYOARiQauqviKj9zGBvyt XJ+QA2SIOdNUN+1A9lNw1UW/ucBKe1Q+WgN+BPqQ7e0CZsPyuA66mnZEo A==; X-IronPort-AV: E=McAfee;i="6600,9927,10756"; a="365824443" X-IronPort-AV: E=Sophos;i="6.01,170,1684825200"; d="scan'208";a="365824443" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Jun 2023 00:49:55 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10756"; a="841770774" X-IronPort-AV: E=Sophos;i="6.01,170,1684825200"; d="scan'208";a="841770774" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Jun 2023 00:49:53 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, chao.p.peng@intel.com Subject: [PATCH v5 3/7] vfio/pci: Disable INTx in vfio_realize error path Date: Fri, 30 Jun 2023 15:36:33 +0800 Message-Id: <20230630073637.124234-2-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230630073637.124234-1-zhenzhong.duan@intel.com> References: <20230630073637.124234-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=134.134.136.65; envelope-from=zhenzhong.duan@intel.com; helo=mga03.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org When vfio realize fails, INTx isn't disabled if it has been enabled. This may confuse host side with unhandled interrupt report. Fixes: c5478fea27ac ("vfio/pci: Respond to KVM irqchip change notifier") Signed-off-by: Zhenzhong Duan Reviewed-by: Joao Martins Reviewed-by: Cédric Le Goater --- hw/vfio/pci.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index ab6645ba60af..31c4ab250fbe 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3220,6 +3220,9 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) return; out_deregister: + if (vdev->interrupt == VFIO_INT_INTx) { + vfio_intx_disable(vdev); + } pci_device_set_intx_routing_notifier(&vdev->pdev, NULL); if (vdev->irqchip_change_notifier.notify) { kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier); From patchwork Fri Jun 30 07:36:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Duan, Zhenzhong" X-Patchwork-Id: 13297701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E3EBDC001B3 for ; Fri, 30 Jun 2023 07:51:12 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qF8tH-0001tP-B5; Fri, 30 Jun 2023 03:50:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qF8tF-0001sk-UD for qemu-devel@nongnu.org; Fri, 30 Jun 2023 03:50:01 -0400 Received: from mga03.intel.com ([134.134.136.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qF8tD-0002bq-Uf for qemu-devel@nongnu.org; Fri, 30 Jun 2023 03:50:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688111399; x=1719647399; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=W7DbU87/2q+anUUQGx9Q18uIWNKsxzWKBvXvAYBRIv8=; b=hK0zxY7XonZm2ONMPVWBTvwOOaR3H4XgO8WyFj7FwzadBGMXmoVbI8KS zgLH9k5b6r2QgTC1g7kTMQK32l1sL3+iyqo8GfuFCo2DEDFZtQBHzj1C0 eVWZZ4dlgDAhAXTJ9v5hTl7nm3ztF2DL/PwiA5/0X/DzQSyZIpkpJRhh6 ejCK4dfjQjwCPkDQY3iMjEZ87JmAmdEvu9XH9kwQERSEk9FdHk5c3GBCB NDjHeFdefYqxg/l0l2e1VTIFsPrd5kv+Y7IiuddiPEBMIWpaO61CB0H3y Poh6sWWMwSfsRgzbEjJWXbJjOGZAsipkrEq3Qt1UrZYPbn9mFXi9qQo1O g==; X-IronPort-AV: E=McAfee;i="6600,9927,10756"; a="365824464" X-IronPort-AV: E=Sophos;i="6.01,170,1684825200"; d="scan'208";a="365824464" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Jun 2023 00:49:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10756"; a="841770790" X-IronPort-AV: E=Sophos;i="6.01,170,1684825200"; d="scan'208";a="841770790" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Jun 2023 00:49:55 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, chao.p.peng@intel.com Subject: [PATCH v5 4/7] vfio/migration: Return bool type for some vfio migration related functions Date: Fri, 30 Jun 2023 15:36:34 +0800 Message-Id: <20230630073637.124234-3-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230630073637.124234-1-zhenzhong.duan@intel.com> References: <20230630073637.124234-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=134.134.136.65; envelope-from=zhenzhong.duan@intel.com; helo=mga03.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Include: vfio_block_multiple_devices_migration(), vfio_block_giommu_migration(), vfio_block_migration(), vfio_migration_realize(). Suggested-by: Cédric Le Goater Signed-off-by: Zhenzhong Duan --- hw/vfio/common.c | 16 ++++++++-------- hw/vfio/migration.c | 19 ++++++++++++------- include/hw/vfio/vfio-common.h | 6 +++--- 3 files changed, 23 insertions(+), 18 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 77e2ee0e5c6e..00fef5aa08be 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -381,19 +381,19 @@ static unsigned int vfio_migratable_device_num(void) return device_num; } -int vfio_block_multiple_devices_migration(VFIODevice *vbasedev, Error **errp) +bool vfio_block_multiple_devices_migration(VFIODevice *vbasedev, Error **errp) { int ret; if (multiple_devices_migration_blocker || vfio_migratable_device_num() <= 1) { - return 0; + return true; } if (vbasedev->enable_migration == ON_OFF_AUTO_ON) { error_setg(errp, "Migration is currently not supported with multiple " "VFIO devices"); - return -EINVAL; + return false; } error_setg(&multiple_devices_migration_blocker, @@ -405,7 +405,7 @@ int vfio_block_multiple_devices_migration(VFIODevice *vbasedev, Error **errp) multiple_devices_migration_blocker = NULL; } - return ret; + return !ret; } void vfio_unblock_multiple_devices_migration(void) @@ -433,19 +433,19 @@ static bool vfio_viommu_preset(void) return false; } -int vfio_block_giommu_migration(VFIODevice *vbasedev, Error **errp) +bool vfio_block_giommu_migration(VFIODevice *vbasedev, Error **errp) { int ret; if (giommu_migration_blocker || !vfio_viommu_preset()) { - return 0; + return true; } if (vbasedev->enable_migration == ON_OFF_AUTO_ON) { error_setg(errp, "Migration is currently not supported with vIOMMU enabled"); - return -EINVAL; + return false; } error_setg(&giommu_migration_blocker, @@ -456,7 +456,7 @@ int vfio_block_giommu_migration(VFIODevice *vbasedev, Error **errp) giommu_migration_blocker = NULL; } - return ret; + return !ret; } void vfio_migration_finalize(void) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 1db7d52ab2c1..09fa4714a085 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -802,13 +802,13 @@ static int vfio_migration_init(VFIODevice *vbasedev) return 0; } -static int vfio_block_migration(VFIODevice *vbasedev, Error *err, Error **errp) +static bool vfio_block_migration(VFIODevice *vbasedev, Error *err, Error **errp) { int ret; if (vbasedev->enable_migration == ON_OFF_AUTO_ON) { error_propagate(errp, err); - return -EINVAL; + return false; } vbasedev->migration_blocker = error_copy(err); @@ -820,7 +820,7 @@ static int vfio_block_migration(VFIODevice *vbasedev, Error *err, Error **errp) vbasedev->migration_blocker = NULL; } - return ret; + return !ret; } /* ---------------------------------------------------------------------- */ @@ -835,7 +835,12 @@ void vfio_reset_bytes_transferred(void) bytes_transferred = 0; } -int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) +/* + * Return true when either migration initialized or blocker registered. + * Currently only return false when adding blocker fails which will + * de-register vfio device. + */ +bool vfio_migration_realize(VFIODevice *vbasedev, Error **errp) { Error *err = NULL; int ret; @@ -874,17 +879,17 @@ int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) } ret = vfio_block_multiple_devices_migration(vbasedev, errp); - if (ret) { + if (!ret) { return ret; } ret = vfio_block_giommu_migration(vbasedev, errp); - if (ret) { + if (!ret) { return ret; } trace_vfio_migration_realize(vbasedev->name); - return 0; + return true; } void vfio_migration_exit(VFIODevice *vbasedev) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 93429b9abba0..b3014ece2edf 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -225,9 +225,9 @@ typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList; extern VFIOGroupList vfio_group_list; bool vfio_mig_active(void); -int vfio_block_multiple_devices_migration(VFIODevice *vbasedev, Error **errp); +bool vfio_block_multiple_devices_migration(VFIODevice *vbasedev, Error **errp); void vfio_unblock_multiple_devices_migration(void); -int vfio_block_giommu_migration(VFIODevice *vbasedev, Error **errp); +bool vfio_block_giommu_migration(VFIODevice *vbasedev, Error **errp); int64_t vfio_mig_bytes_transferred(void); void vfio_reset_bytes_transferred(void); @@ -252,7 +252,7 @@ int vfio_spapr_create_window(VFIOContainer *container, int vfio_spapr_remove_window(VFIOContainer *container, hwaddr offset_within_address_space); -int vfio_migration_realize(VFIODevice *vbasedev, Error **errp); +bool vfio_migration_realize(VFIODevice *vbasedev, Error **errp); void vfio_migration_exit(VFIODevice *vbasedev); void vfio_migration_finalize(void); From patchwork Fri Jun 30 07:36:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Duan, Zhenzhong" X-Patchwork-Id: 13297699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7BB0BC0015E for ; Fri, 30 Jun 2023 07:51:12 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qF8tJ-0001tw-UB; Fri, 30 Jun 2023 03:50:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qF8tI-0001ti-5q for qemu-devel@nongnu.org; Fri, 30 Jun 2023 03:50:04 -0400 Received: from mga03.intel.com ([134.134.136.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qF8tG-0002bq-8H for qemu-devel@nongnu.org; Fri, 30 Jun 2023 03:50:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688111402; x=1719647402; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dX0W6Lwgs02fXVLOQjBdVVT7wZQlUQbNlFD/jzw8V5s=; b=GX0MWGJiNvB9QOMVECK9xMgMJY6I25ipcq5N7LAg3MBVUuIxlfsYC+64 Ytyf5dy6jzRmY35/GgAQC6ud6bXPX0rXaDaWlb4rqGjdHlrYFqH9xQq4/ swOuBigDX0BE8tnPoFw4CnydJe/9PjdIDedMihTd1wqaSuA6a+PW5N+U7 huvW6rvVMMD+/i/a2yyZ20jQObfFFVzjij37/zyUP9WVeNqdIyq0x45fR 3OAHcDv6HM8H3brawcdfdOttdq+IccZERe7aoSrIXcDymSMEKcn0amuqC EnPtfCwaz1Job8SQqnHhZLDZNMMH3hFgzno+OKcgXKNc3WKq1e49XGheh g==; X-IronPort-AV: E=McAfee;i="6600,9927,10756"; a="365824475" X-IronPort-AV: E=Sophos;i="6.01,170,1684825200"; d="scan'208";a="365824475" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Jun 2023 00:50:01 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10756"; a="841770807" X-IronPort-AV: E=Sophos;i="6.01,170,1684825200"; d="scan'208";a="841770807" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Jun 2023 00:49:58 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, chao.p.peng@intel.com Subject: [PATCH v5 5/7] vfio/migration: Change vIOMMU blocker from global to per device Date: Fri, 30 Jun 2023 15:36:35 +0800 Message-Id: <20230630073637.124234-4-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230630073637.124234-1-zhenzhong.duan@intel.com> References: <20230630073637.124234-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=134.134.136.65; envelope-from=zhenzhong.duan@intel.com; helo=mga03.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org vIOMMU migration blocker can be consolidated to per device default blocker, then some vIOMMU global blocker related functions and data could be removed. This change also makes vfio_mig_active() more accurate as it doesn't check for global blocker. Signed-off-by: Zhenzhong Duan Reviewed-by: Joao Martins --- hw/vfio/common.c | 51 ++--------------------------------- hw/vfio/migration.c | 7 ++--- hw/vfio/pci.c | 1 - include/hw/vfio/vfio-common.h | 3 +-- 4 files changed, 7 insertions(+), 55 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 00fef5aa08be..a8439447b990 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -362,7 +362,6 @@ bool vfio_mig_active(void) } static Error *multiple_devices_migration_blocker; -static Error *giommu_migration_blocker; static unsigned int vfio_migratable_device_num(void) { @@ -420,55 +419,9 @@ void vfio_unblock_multiple_devices_migration(void) multiple_devices_migration_blocker = NULL; } -static bool vfio_viommu_preset(void) +bool vfio_viommu_preset(VFIODevice *vbasedev) { - VFIOAddressSpace *space; - - QLIST_FOREACH(space, &vfio_address_spaces, list) { - if (space->as != &address_space_memory) { - return true; - } - } - - return false; -} - -bool vfio_block_giommu_migration(VFIODevice *vbasedev, Error **errp) -{ - int ret; - - if (giommu_migration_blocker || - !vfio_viommu_preset()) { - return true; - } - - if (vbasedev->enable_migration == ON_OFF_AUTO_ON) { - error_setg(errp, - "Migration is currently not supported with vIOMMU enabled"); - return false; - } - - error_setg(&giommu_migration_blocker, - "Migration is currently not supported with vIOMMU enabled"); - ret = migrate_add_blocker(giommu_migration_blocker, errp); - if (ret < 0) { - error_free(giommu_migration_blocker); - giommu_migration_blocker = NULL; - } - - return !ret; -} - -void vfio_migration_finalize(void) -{ - if (!giommu_migration_blocker || - vfio_viommu_preset()) { - return; - } - - migrate_del_blocker(giommu_migration_blocker); - error_free(giommu_migration_blocker); - giommu_migration_blocker = NULL; + return vbasedev->group->container->space->as != &address_space_memory; } static void vfio_set_migration_error(int err) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 09fa4714a085..80509958f0d3 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -883,9 +883,10 @@ bool vfio_migration_realize(VFIODevice *vbasedev, Error **errp) return ret; } - ret = vfio_block_giommu_migration(vbasedev, errp); - if (!ret) { - return ret; + if (vfio_viommu_preset(vbasedev)) { + error_setg(&err, "%s: Migration is currently not supported " + "with vIOMMU enabled", vbasedev->name); + return vfio_block_migration(vbasedev, err, errp); } trace_vfio_migration_realize(vbasedev->name); diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 31c4ab250fbe..c2cf7454ece6 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3255,7 +3255,6 @@ static void vfio_instance_finalize(Object *obj) */ vfio_put_device(vdev); vfio_put_group(group); - vfio_migration_finalize(); } static void vfio_exitfn(PCIDevice *pdev) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index b3014ece2edf..3c18572322fc 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -227,7 +227,7 @@ extern VFIOGroupList vfio_group_list; bool vfio_mig_active(void); bool vfio_block_multiple_devices_migration(VFIODevice *vbasedev, Error **errp); void vfio_unblock_multiple_devices_migration(void); -bool vfio_block_giommu_migration(VFIODevice *vbasedev, Error **errp); +bool vfio_viommu_preset(VFIODevice *vbasedev); int64_t vfio_mig_bytes_transferred(void); void vfio_reset_bytes_transferred(void); @@ -254,6 +254,5 @@ int vfio_spapr_remove_window(VFIOContainer *container, bool vfio_migration_realize(VFIODevice *vbasedev, Error **errp); void vfio_migration_exit(VFIODevice *vbasedev); -void vfio_migration_finalize(void); #endif /* HW_VFIO_VFIO_COMMON_H */ From patchwork Fri Jun 30 07:36:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Duan, Zhenzhong" X-Patchwork-Id: 13297696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9E7D1EB64DA for ; Fri, 30 Jun 2023 07:50:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qF8tL-0001uQ-Pz; Fri, 30 Jun 2023 03:50:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qF8tK-0001u9-J8 for qemu-devel@nongnu.org; Fri, 30 Jun 2023 03:50:06 -0400 Received: from mga03.intel.com ([134.134.136.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qF8tI-0002cC-EN for qemu-devel@nongnu.org; Fri, 30 Jun 2023 03:50:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688111404; x=1719647404; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=0ow06bSHCmm8G35/UJT+C3SMQ0vZMLKNuD9EGNXd0l0=; b=Al0haCXCbkjRJGa8VFJ/8ePCj2rotB2/X0MzaURtNd5AJ1QwXHt9Z20i R9TAXaxRCikyAl7717c6gSgsJ6ZzvMezoa7nw9SOUp5Od3ymM8R+atxWU nhof5w9c01RI0xl0gJtOItKqekspHPpsNXsrxAvf/0yh2ImQIoi/OdLZI orBtAqI6Ix6uLof/GykK2TyVudMq23SkaMASJFbLM/dm6IHxWM13YiZYK lLQJexoq9qmKqtKqFKQVqAs5Ym6izBjunILf/+dmDhQ+ixCpDPPW3KnMr KGSQldKfEwWGaWKfwokxSgcYPCWIEt89PueBoOr9eveu1O1q4G+xkPc5z Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10756"; a="365824484" X-IronPort-AV: E=Sophos;i="6.01,170,1684825200"; d="scan'208";a="365824484" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Jun 2023 00:50:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10756"; a="841770826" X-IronPort-AV: E=Sophos;i="6.01,170,1684825200"; d="scan'208";a="841770826" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Jun 2023 00:50:01 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, chao.p.peng@intel.com Subject: [PATCH v5 6/7] vfio/migration: Free resources when vfio_migration_realize fails Date: Fri, 30 Jun 2023 15:36:36 +0800 Message-Id: <20230630073637.124234-5-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230630073637.124234-1-zhenzhong.duan@intel.com> References: <20230630073637.124234-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=134.134.136.65; envelope-from=zhenzhong.duan@intel.com; helo=mga03.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org When vfio_realize() succeeds, hot unplug will call vfio_exitfn() to free resources allocated in vfio_realize(); when vfio_realize() fails, vfio_exitfn() is never called and we need to free resources in vfio_realize(). In the case that vfio_migration_realize() fails, e.g: with -only-migratable & enable-migration=off, we see below: (qemu) device_add vfio-pci,host=81:11.1,id=vfio1,bus=root1,enable-migration=off 0000:81:11.1: Migration disabled Error: disallowing migration blocker (--only-migratable) for: 0000:81:11.1: Migration is disabled for VFIO device If we hotplug again we should see same log as above, but we see: (qemu) device_add vfio-pci,host=81:11.1,id=vfio1,bus=root1,enable-migration=off Error: vfio 0000:81:11.1: device is already attached That's because some references to VFIO device isn't released. For resources allocated in vfio_migration_realize(), free them by jumping to out_deinit path with calling a new function vfio_migration_deinit(). For resources allocated in vfio_realize(), free them by jumping to de-register path in vfio_realize(). Signed-off-by: Zhenzhong Duan --- hw/vfio/migration.c | 37 +++++++++++++++++++++++++------------ hw/vfio/pci.c | 3 ++- 2 files changed, 27 insertions(+), 13 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 80509958f0d3..2110c4dc21e2 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -802,6 +802,19 @@ static int vfio_migration_init(VFIODevice *vbasedev) return 0; } +static void vfio_migration_deinit(VFIODevice *vbasedev) +{ + if (vbasedev->migration) { + VFIOMigration *migration = vbasedev->migration; + + remove_migration_state_change_notifier(&migration->migration_state); + qemu_del_vm_change_state_handler(migration->vm_state); + unregister_savevm(VMSTATE_IF(vbasedev->dev), "vfio", vbasedev); + vfio_migration_free(vbasedev); + vfio_unblock_multiple_devices_migration(); + } +} + static bool vfio_block_migration(VFIODevice *vbasedev, Error *err, Error **errp) { int ret; @@ -871,7 +884,7 @@ bool vfio_migration_realize(VFIODevice *vbasedev, Error **errp) error_setg(&err, "%s: VFIO device doesn't support device dirty tracking", vbasedev->name); - return vfio_block_migration(vbasedev, err, errp); + goto add_blocker; } warn_report("%s: VFIO device doesn't support device dirty tracking", @@ -880,30 +893,30 @@ bool vfio_migration_realize(VFIODevice *vbasedev, Error **errp) ret = vfio_block_multiple_devices_migration(vbasedev, errp); if (!ret) { - return ret; + goto out_deinit; } if (vfio_viommu_preset(vbasedev)) { error_setg(&err, "%s: Migration is currently not supported " "with vIOMMU enabled", vbasedev->name); - return vfio_block_migration(vbasedev, err, errp); + goto add_blocker; } trace_vfio_migration_realize(vbasedev->name); return true; + +add_blocker: + ret = vfio_block_migration(vbasedev, err, errp); +out_deinit: + if (!ret) { + vfio_migration_deinit(vbasedev); + } + return ret; } void vfio_migration_exit(VFIODevice *vbasedev) { - if (vbasedev->migration) { - VFIOMigration *migration = vbasedev->migration; - - remove_migration_state_change_notifier(&migration->migration_state); - qemu_del_vm_change_state_handler(migration->vm_state); - unregister_savevm(VMSTATE_IF(vbasedev->dev), "vfio", vbasedev); - vfio_migration_free(vbasedev); - vfio_unblock_multiple_devices_migration(); - } + vfio_migration_deinit(vbasedev); if (vbasedev->migration_blocker) { migrate_del_blocker(vbasedev->migration_blocker); diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index c2cf7454ece6..9cf70c1965d3 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3208,8 +3208,9 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) if (!pdev->failover_pair_id) { ret = vfio_migration_realize(vbasedev, errp); - if (ret) { + if (!ret) { error_report("%s: Migration disabled", vbasedev->name); + goto out_deregister; } } From patchwork Fri Jun 30 07:36:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Duan, Zhenzhong" X-Patchwork-Id: 13297698 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AE5C9EB64DA for ; Fri, 30 Jun 2023 07:51:10 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qF8tQ-0001ur-1U; Fri, 30 Jun 2023 03:50:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qF8tO-0001ui-0z for qemu-devel@nongnu.org; Fri, 30 Jun 2023 03:50:10 -0400 Received: from mga03.intel.com ([134.134.136.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qF8tM-0002oc-4X for qemu-devel@nongnu.org; Fri, 30 Jun 2023 03:50:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688111408; x=1719647408; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jxpKTAInzfx8UIcmYnemexQBA/d+xvputn2hB27MCd8=; b=Ejz+VvwWO2F32a1XByZ6uSIwMpGDXP2xtabJTRcKbJDk+nZDgmB0S+XI mjAmkuN8Tjd0uCVsepCp6axt89+z8SsPpRZYfaSvbqRa48g7Bl6csYkMR 65oibojQjWRVxVFXslq1fV5omReBODPYyJWwcd72r1XvREreMGjpy4Cqq wXkhcKZZBW7DxM9YEzD9S3bEFmOe4vCxsHFZ+YX6zv1NoT33FjRJKS0iB mXwRHXLcsRjJ4yFNALWKVcSTWYaKXCW9ueFvPdY/kLS/1y/x2T+kNPVxs HWNzQikT71mmO01evotJxzDom+JhrBGhUBaNTOz0n5INnk1JoSvlgJ5ld w==; X-IronPort-AV: E=McAfee;i="6600,9927,10756"; a="365824497" X-IronPort-AV: E=Sophos;i="6.01,170,1684825200"; d="scan'208";a="365824497" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Jun 2023 00:50:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10756"; a="841770837" X-IronPort-AV: E=Sophos;i="6.01,170,1684825200"; d="scan'208";a="841770837" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Jun 2023 00:50:03 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, chao.p.peng@intel.com Subject: [PATCH v5 7/7] vfio/migration: Remove print of "Migration disabled" Date: Fri, 30 Jun 2023 15:36:37 +0800 Message-Id: <20230630073637.124234-6-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230630073637.124234-1-zhenzhong.duan@intel.com> References: <20230630073637.124234-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=134.134.136.65; envelope-from=zhenzhong.duan@intel.com; helo=mga03.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Property enable_migration supports [on/off/auto]. In ON mode, error pointer is passed to errp and logged. In OFF mode, we doesn't need to log "Migration disabled" as it's intentional. In AUTO mode, we should only ever see errors or warnings if the device supports migration and an error or incompatibility occurs while further probing or configuring it. Lack of support for migration shoundn't generate an error or warning. Signed-off-by: Zhenzhong Duan Reviewed-by: Joao Martins --- hw/vfio/pci.c | 1 - 1 file changed, 1 deletion(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 9cf70c1965d3..1c969fccc543 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3209,7 +3209,6 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) if (!pdev->failover_pair_id) { ret = vfio_migration_realize(vbasedev, errp); if (!ret) { - error_report("%s: Migration disabled", vbasedev->name); goto out_deregister; } }