From patchwork Mon May 30 17:07:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 12864839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A93B8C433F5 for ; Mon, 30 May 2022 17:55:31 +0000 (UTC) Received: from localhost ([::1]:48108 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nvjc2-00069q-KJ for qemu-devel@archiver.kernel.org; Mon, 30 May 2022 13:55:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54230) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvis1-0003GR-Dr for qemu-devel@nongnu.org; Mon, 30 May 2022 13:07:57 -0400 Received: from mail-bn8nam12on2061d.outbound.protection.outlook.com ([2a01:111:f400:fe5b::61d]:16096 helo=NAM12-BN8-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvirz-0008LP-9Z for qemu-devel@nongnu.org; Mon, 30 May 2022 13:07:56 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OoaoAEgILqiPyShX8ux3Bt4r7d6wc6FGCtqIyPR5vbOKVoSeCV5DTHpdO+vPPai43nEGrYRUq6e0fsLt8ACDAK2MCX+rMKYx8SDmQPwqnHKIdKE8k8dfBZ77v4EboaU1H9hotR6+wpGU8axX9/LBeAbQnTN70Ss+EcCQakYzBPd3RtpqJtyGKnH1p60XHIvEBE2hyUjC8Tu2dSHPMkwgVpvjM7i9i6Irs/zTub/k1gmdwOKKVQrrHNrIhkm3uHNBc2EY3aw60tiRIBRhVVpYlz3V5lHXb9XJ+c+Za+B1DGugnMn6nHIfnZeO8gDIKTMI4gSaQwykIGf8H/NPxJ360A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kO1Q0d7wykn/1VkH5MNBlO0UmPGgbg874YdG/5D9cHw=; b=TrDmoCMYD/j+z/sazvdmXK4/VZj/HmjvSOuLHGT2RHpkyRpwL41Pqrdhn1obRciPBrjAF0mAhOFDmpISVeL8CQPS42xEFfXDrk9gwYFyGJipN+l4HcMVtG+Vw6CUTZtSyiESPCbHLXgUHG+9f4L9a8J+pISr315Yy2AwHBwuXi2RK6zXW/FrO5wbZ2Gk9abj2bvIVlCcQf03ZrSkE3nmB6xrH4F9Tg4tfSugpyrcuwO2cPSCATk3WWxrk3eevSo7AY6Vy4kDVw7bze/LFf7P5PDplPuNR9TdrwGcSPt1sWPlNMEMh9p0MpenoxoeuSwNs12NekE0xb3iAmAGrNSaUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.234) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kO1Q0d7wykn/1VkH5MNBlO0UmPGgbg874YdG/5D9cHw=; b=PhggqPBKtKXIq9cWfkTZcrj3wNhjPeDkWE0Uf2z8yz6Zpcj/todUEQmXQp9TvoNowjr8a/IaZRRzWwZSSdSgWGvvN+BBykbW1O6XKqL7XUjGSKZIQUbfr5jTA+B9VGUDzEt2Lg6HmgcOuSPZLvhCOdb0Wn1KGGTuQPrduOAgLiFxpYDD9HxYZ2vqu+En/6j9fKlOuewoc9C2BhpQ93IW8OvbPF1BEiglTzRFIeGnXqmeXk85SyWFA3LEsdymfs82jKIl7c6mTPkr2jGNaPlXrgoK09M7S1ljRJcu+8h+IJvduAhcCV9fOqm7B7+I/c+A/Z1SOl/H4AyOYRmZQr1M7Q== Received: from MW4PR04CA0040.namprd04.prod.outlook.com (2603:10b6:303:6a::15) by MN2PR12MB4535.namprd12.prod.outlook.com (2603:10b6:208:267::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Mon, 30 May 2022 17:07:48 +0000 Received: from CO1NAM11FT058.eop-nam11.prod.protection.outlook.com (2603:10b6:303:6a:cafe::5a) by MW4PR04CA0040.outlook.office365.com (2603:10b6:303:6a::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:07:48 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.234) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.234 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.234; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.234) by CO1NAM11FT058.mail.protection.outlook.com (10.13.174.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:07:47 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by DRHQMAIL101.nvidia.com (10.27.9.10) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 30 May 2022 17:07:47 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 30 May 2022 10:07:46 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.22 via Frontend Transport; Mon, 30 May 2022 10:07:43 -0700 From: Avihai Horon To: , Cornelia Huck , "Alex Williamson" , Juan Quintela , "Dr . David Alan Gilbert" CC: Joao Martins , Yishai Hadas , Jason Gunthorpe , Mark Bloch , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH v2 01/11] vfio/migration: Fix NULL pointer dereference bug Date: Mon, 30 May 2022 20:07:29 +0300 Message-ID: <20220530170739.19072-2-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20220530170739.19072-1-avihaih@nvidia.com> References: <20220530170739.19072-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 759bf2b6-9c08-459c-5040-08da425eebf0 X-MS-TrafficTypeDiagnostic: MN2PR12MB4535:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4GsBO3Z6XPGkkcmnpjRudBR9IGO3hM0SdMtgqNZyl5Y1Pc8fT87IOv/yYwz8tNhNZEd8tUJWKvkfiCZ5ZAroH+7MGdcdwO01aL5y1ey8lNCD37ytdsCCwYnqSH6/6u/E54hkQnJLkTTj61u796m35ePhlIm49yD0Q5EGF07wdJmZT8P51g/Ag5CCDJv7mQFuBEjRR49sJ+RxIYj95NxY7u8L858g3JogJF1QKpC+hFO+kEMraz8xFJHFmJ5Q9j/y8bHEurmDPUxmYCgM3KBa+Ldj5+NG01jnU8q1hXr0KQydUGQW1lT8cEzJc79hDiVe+XmOqMkFQJIwKzOPX+ehjSbbE3tlrgTUFyIWOgILu6jIIEH8waER9qAqsTNES6EykplQUVXXcYdpa6+XbyfYbR4UYApv/hf3XRGIFS1s1C6szvChh9A/ZUE3mYGKQ9ZVY1f8E17lXFYFKJEpjQVl+fRTTpk1jPTm98WR5ryvv6w2nKonRZu7VVkJPpvtuLxLEGHM2cK05hJrVkOcNkz9Ig9k5MlINZhkXlGSIy5PfhV4HPhU1pklyMthugtzz1t/ArYZopLpbGsrssPb8wc3MldEgunHCDmt7bNNDmozGT45sXqZRAYFDUQSBROymz1xatIuu8Vw8USzWc8FwvJzHFTdEsE10BeqD586sHoA6nAtjsdX0x4eu9U3fGfvz3ZJp8hhMpWCFWJEnUMRX05log== X-Forefront-Antispam-Report: CIP:12.22.5.234; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(36840700001)(46966006)(40470700004)(336012)(186003)(40460700003)(6666004)(86362001)(81166007)(70206006)(107886003)(356005)(5660300002)(4326008)(70586007)(1076003)(36756003)(8936002)(426003)(83380400001)(7696005)(2616005)(8676002)(2906002)(110136005)(316002)(54906003)(26005)(508600001)(47076005)(36860700001)(82310400005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2022 17:07:47.8006 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 759bf2b6-9c08-459c-5040-08da425eebf0 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.234]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT058.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4535 Received-SPF: softfail client-ip=2a01:111:f400:fe5b::61d; envelope-from=avihaih@nvidia.com; helo=NAM12-BN8-obe.outbound.protection.outlook.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" As part of its error flow, vfio_vmstate_change() accesses MigrationState->to_dst_file without any checks. This can cause a NULL pointer dereference if the error flow is taken and MigrationState->to_dst_file is not set. For example, this can happen if VM is started or stopped not during migration and vfio_vmstate_change() error flow is taken, as MigrationState->to_dst_file is not set at that time. Fix it by checking that MigrationState->to_dst_file is set before using it. Fixes: 02a7e71b1e5b ("vfio: Add VM state change handler to know state of VM") Signed-off-by: Avihai Horon Reviewed-by: Juan Quintela --- hw/vfio/migration.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index a6ad1f8945..34f9f894ed 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -744,7 +744,9 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) */ error_report("%s: Failed to set device state 0x%x", vbasedev->name, (migration->device_state & mask) | value); - qemu_file_set_error(migrate_get_current()->to_dst_file, ret); + if (migrate_get_current()->to_dst_file) { + qemu_file_set_error(migrate_get_current()->to_dst_file, ret); + } } vbasedev->migration->vm_running = running; trace_vfio_vmstate_change(vbasedev->name, running, RunState_str(state), From patchwork Mon May 30 17:07:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 12864831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7302CC433EF for ; Mon, 30 May 2022 17:46:17 +0000 (UTC) Received: from localhost ([::1]:59992 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nvjT6-00038x-5k for qemu-devel@archiver.kernel.org; Mon, 30 May 2022 13:46:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54228) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvis1-0003G3-Di for qemu-devel@nongnu.org; Mon, 30 May 2022 13:07:57 -0400 Received: from mail-bn8nam08on20620.outbound.protection.outlook.com ([2a01:111:f400:7e8d::620]:49761 helo=NAM04-BN8-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nviry-0008LQ-KA for qemu-devel@nongnu.org; Mon, 30 May 2022 13:07:56 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ynt4aRVr/xS5Uv+RCqcZ3C37AEs2ZJG9kGoKuYHEJFi46SFaQfZjYojjsDp1bZrtb49ARIDMD9cxnxkCl3GS+2yi/9UBCHmF1KLTGzHB2PuVG9pQ50ybkp9l9GX8oqJ1bhSpil7NRfbSUEoJn+we4ObEJ6A++sqpcDLpMIvXT4q3b1P6aTb7M/MXE9NIkA8pdU5EVDt3AXrhrcd1cFr0rU+YHXgsx61FAsXAo2NY9YhA3hlSKekT4F38KjJ0KvrIFFep6IIxJU+/TV+8i/1VqjAZmOiprZobTKmyB5bBjWQ3lroWrJi3KQXLWb3v3qPZbzBFtrcHoONdNbjfUzANQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DWmmJp9DZzsJirVqwnDaGWW1CwYh74LsXTFYv6GpP9U=; b=OQ7jeucBrTH0VWn/9InGZLQNHJ4SKGMDleH0Q8s54Zmzhy7iW55lbvLHAawKTfjl7SbIDQHm8v0IPBqWepnuleoPF4IKnc1uiPytt4YlGM4mMsxd2RcOxT1KW7N2KoeOa+D3HrzKr/ZM7y2qEDvnLSH/ExmUWuI8UTlPocKdilC4RjpzUhabpLYZCuK9QXIbimJbBJoNIo48COoyu2zTX+vCkHmtK3417Jb72uH8N9QDZOjkYTKggpWrazdSPsn3HvdAGIsJW2Nl4s8QMhj3pOnw81U9jQ+cDmCoIfVFX2pBBp5JHWPByQKvvA37Cq/n40sEaaOSW0kYwRUAPg4A0Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.236) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DWmmJp9DZzsJirVqwnDaGWW1CwYh74LsXTFYv6GpP9U=; b=LR1EPCoQP8sSWAVDwbF0BlzWZCet3R3+G5SVPaviLq8B9iBCmlZk57RA7Xn7vqUi6hnBHNOwO5ZGehuKj6D9eWEQni530O6gpblwfejT3TnLtKidmxHOeNvuigvJ9E6NK+muFpbJw2resrEbDwUWKSOn9hvPMCqMKKV/o34QS7ozJDomlLB7ETf2LvGZd6JRYEeH7O3tuimFdOqSJCOtNN/4vqDD3SJj5jzTABh3THGYw3kEZnndN7vq6n9fnmlhJlDNAC8oivZkxd1sJD/NMmB1Nlx4m5y2dzjHrPwY3LFEO3iDZitGZRlCh6RCo8KJnwX3ST4QuZvPP85NGBEGiw== Received: from MWHPR02CA0015.namprd02.prod.outlook.com (2603:10b6:300:4b::25) by DM5PR12MB1337.namprd12.prod.outlook.com (2603:10b6:3:6e::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.17; Mon, 30 May 2022 17:07:51 +0000 Received: from CO1NAM11FT039.eop-nam11.prod.protection.outlook.com (2603:10b6:300:4b:cafe::e2) by MWHPR02CA0015.outlook.office365.com (2603:10b6:300:4b::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:07:51 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.236) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.236 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.236; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.236) by CO1NAM11FT039.mail.protection.outlook.com (10.13.174.110) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:07:51 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by DRHQMAIL109.nvidia.com (10.27.9.19) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 30 May 2022 17:07:50 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 30 May 2022 10:07:50 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.22 via Frontend Transport; Mon, 30 May 2022 10:07:47 -0700 From: Avihai Horon To: , Cornelia Huck , "Alex Williamson" , Juan Quintela , "Dr . David Alan Gilbert" CC: Joao Martins , Yishai Hadas , Jason Gunthorpe , Mark Bloch , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH v2 02/11] vfio/migration: Skip pre-copy if dirty page tracking is not supported Date: Mon, 30 May 2022 20:07:30 +0300 Message-ID: <20220530170739.19072-3-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20220530170739.19072-1-avihaih@nvidia.com> References: <20220530170739.19072-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 97bca9f4-2cbc-4084-1c00-08da425eee0c X-MS-TrafficTypeDiagnostic: DM5PR12MB1337:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LSZtB1UMo80T2t1wKR/9u4W+9wKtMZsRxl1k4hPp4l7WSAUG+Tf3c6OualPqzawS9zslUv9uhMxHE4Q9hGR/z3As8zkx/20t8001aUPuu/GECLPFqn9Keqx6u8d8nb7URLtjhyLuCKIVfLzdT1dldqQF5G2WDEBxVC/nXAYhHt42a1sBluQsf9rS6Vkz9qOBGt+GUt/xHbCwPPETk9FiWZNuJIFrUxk9X4irUUu56QHcBSktWcGr/dhBSBXMmWZZQEQa8Hz+Qh4CIE3ExrjQfXxEGkG/CaYIo0SHmcC8Sgtwm1Ahkv93X2a0fUR8CGDBmQtZIBdJKUl+amVJn0NjeYN/b/wMZkK1gppNV9RbviCg7/biF7HMmEd4su7bEpX6l3liqYD/wbR67M/X+PcbfeG33gQvImf7G20XcVeECKnbMNJ2PnSblW5wqkL3TU1TE7JkEGDikhsdLrZWKX7SgigHGAcEIlQaYZG3Bx/bArgK6iWAp53U28+fs63q8AI3pGHju2/q3Sjy0/Rij07owX20wm2yJrFQJ1Sb41KodQwP0VVU4iDgHvUBbfpzD672mEG5sfNGNgDxJgSkjt+AWm5XRedBBlhBR+14+4iuPb4E9Y0McP4iWHOOoAoPRYXFp9TJq5ZyrtDq7UuUVjX/SoTLzkm0FgZvqKZx96qQhqAknE+OaO8sGqn3AQBiaOSvccIV/biZAHMVrWlwPwwob9CUkvBJnC2owUl5Db+5dU0= X-Forefront-Antispam-Report: CIP:12.22.5.236; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(316002)(356005)(47076005)(508600001)(36756003)(186003)(107886003)(81166007)(1076003)(2616005)(426003)(336012)(86362001)(110136005)(40460700003)(54906003)(36860700001)(8676002)(82310400005)(26005)(8936002)(83380400001)(5660300002)(4326008)(6666004)(7696005)(70586007)(2906002)(70206006)(14143004)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2022 17:07:51.3382 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 97bca9f4-2cbc-4084-1c00-08da425eee0c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.236]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT039.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1337 Received-SPF: softfail client-ip=2a01:111:f400:7e8d::620; envelope-from=avihaih@nvidia.com; helo=NAM04-BN8-obe.outbound.protection.outlook.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Currently, if IOMMU of a VFIO container doesn't support dirty page tracking, migration is blocked completely. This is because a DMA-able VFIO device can dirty RAM pages without updating QEMU about it, thus breaking the migration. However, this doesn't mean that migration can't be done at all. If migration pre-copy phase is skipped, the VFIO device doesn't have a chance to dirty RAM pages that have been migrated already, thus eliminating the problem previously mentioned. Hence, in such case allow migration but skip pre-copy phase. Signed-off-by: Avihai Horon --- hw/vfio/migration.c | 9 ++++++++- migration/migration.c | 5 +++++ migration/migration.h | 3 +++ 3 files changed, 16 insertions(+), 1 deletion(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 34f9f894ed..d8f9b086c2 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -863,10 +863,17 @@ int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) struct vfio_region_info *info = NULL; int ret = -ENOTSUP; - if (!vbasedev->enable_migration || !container->dirty_pages_supported) { + if (!vbasedev->enable_migration) { goto add_blocker; } + if (!container->dirty_pages_supported) { + warn_report_once( + "%s: IOMMU of the device's VFIO container doesn't support dirty page tracking, migration pre-copy phase will be skipped", + vbasedev->name); + migrate_get_current()->skip_precopy = true; + } + ret = vfio_get_dev_region_info(vbasedev, VFIO_REGION_TYPE_MIGRATION_DEPRECATED, VFIO_REGION_SUBTYPE_MIGRATION_DEPRECATED, diff --git a/migration/migration.c b/migration/migration.c index 31739b2af9..217f0e3e94 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -3636,6 +3636,11 @@ static MigIterateState migration_iteration_run(MigrationState *s) uint64_t pending_size, pend_pre, pend_compat, pend_post; bool in_postcopy = s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE; + if (s->skip_precopy) { + migration_completion(s); + return MIG_ITERATE_BREAK; + } + qemu_savevm_state_pending(s->to_dst_file, s->threshold_size, &pend_pre, &pend_compat, &pend_post); pending_size = pend_pre + pend_compat + pend_post; diff --git a/migration/migration.h b/migration/migration.h index 485d58b95f..0920a0950e 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -332,6 +332,9 @@ struct MigrationState { * This save hostname when out-going migration starts */ char *hostname; + + /* Whether to skip pre-copy phase of migration or not */ + bool skip_precopy; }; void migrate_set_state(int *state, int old_state, int new_state); From patchwork Mon May 30 17:07:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 12864859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9E018C433F5 for ; Mon, 30 May 2022 18:12:13 +0000 (UTC) Received: from localhost ([::1]:57612 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nvjsC-0007ih-Mj for qemu-devel@archiver.kernel.org; Mon, 30 May 2022 14:12:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54246) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvis4-0003TP-TW for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:00 -0400 Received: from mail-co1nam11on2060e.outbound.protection.outlook.com ([2a01:111:f400:7eab::60e]:26112 helo=NAM11-CO1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvis2-0008Ll-Hr for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:00 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mtmTED1a37dDf1Wo498cCsqKQ2ZzOPu5mvcvVa+MBPJIGfkFM3UgVe1UaCj5MN4mA+9ao74VGw177PMQVNqZudPzIhNcH7FSBbjTGmjcSTWMsYj668/+G5XgbOI8HwshaERtKyLJNsNSOLOMEKqEKkh+fpBP6ALYAhGkCmI2+gyIZr7jf2dI9ZNSqOH8X8Sd9jTVTxtbSwBjcFIDncwW0cD0+/Q5p4eIiBPIHyHs6mtqw5yWdLJvAq0nsG3UqaCJZ3CWMO0pgZW3yXlncD16+r8/LDXwMQj5soStX0DN4dotDl4rCie4X2FWqXD8ptnugK++eCzZTTbaBJhyMkhUSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=33dWet8F7c64nY1h8u8FSdJegji/Xrkz5DD1ILOqYGM=; b=D7xGCX1bUFpEt8rnaYK0FxwTIcWcwwedJRsRGGbn+pwg1JfhP8GQ6/SwY7ZxLthIZ4dD5NR1VIIABWjz/3EuhqcAjlxvvdvXI2zDDzifr1fzYeO3xMqvIeWgtu2bUw4wUQg1CX0o4+89guLIFlUyJ0jJUruzdzWTrVuwuILrJ+2OY9peDy+rdnrbcpZf1fZMnN+QGnCPsUKgZJmjfb0B1nKp+4PpUaFQYn0LBAuBR3WBfL9Tbtef+JCONNN8hnKjnUxvG1DzvyabGoppQApD1gMW6szYudjt95qoukLDtvaWTNHl6Bkn65GCBAlMiqSiqDqtX5XKJmLRX7kDxTjTIw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.236) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=33dWet8F7c64nY1h8u8FSdJegji/Xrkz5DD1ILOqYGM=; b=JTUfXZMet43l9BWhcRqDnQ46KGC/Bse92skmdQzSkZWCpV4xkcLdJgF30oBgFYqR9Lql8mSUXJStQ0DIprpLIWd4aXi8Ca+S5p3KKEJwu/3WOFWdkWnjb66uksakLvnS4or/0yVvdc//9jlDXKKZkb1DKWi5R4znLOh3M4Oqy0oLPWyeKrmcMlBBbM69dUEK/JTym2eCfGDigrgDimAaMoFk1EgXSYZTKaWdm499tCeBS4CBxooiwhHqDQU/sEwxdz/fT85XQKsFx8za6IJNjxQ212CaKNMUU+QcEu6gWY+00h0y6EtnaoozkPHDnfJELfmjpXyjwWmKVuBy194mvw== Received: from MW4PR04CA0215.namprd04.prod.outlook.com (2603:10b6:303:87::10) by MWHPR12MB1742.namprd12.prod.outlook.com (2603:10b6:300:112::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Mon, 30 May 2022 17:07:55 +0000 Received: from CO1NAM11FT026.eop-nam11.prod.protection.outlook.com (2603:10b6:303:87:cafe::bb) by MW4PR04CA0215.outlook.office365.com (2603:10b6:303:87::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:07:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.236) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.236 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.236; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.236) by CO1NAM11FT026.mail.protection.outlook.com (10.13.175.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:07:54 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by DRHQMAIL109.nvidia.com (10.27.9.19) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 30 May 2022 17:07:54 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 30 May 2022 10:07:53 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.22 via Frontend Transport; Mon, 30 May 2022 10:07:50 -0700 From: Avihai Horon To: , Cornelia Huck , "Alex Williamson" , Juan Quintela , "Dr . David Alan Gilbert" CC: Joao Martins , Yishai Hadas , Jason Gunthorpe , Mark Bloch , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH v2 03/11] migration/qemu-file: Add qemu_file_get_to_fd() Date: Mon, 30 May 2022 20:07:31 +0300 Message-ID: <20220530170739.19072-4-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20220530170739.19072-1-avihaih@nvidia.com> References: <20220530170739.19072-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 953c0b38-f026-42da-6957-08da425ef032 X-MS-TrafficTypeDiagnostic: MWHPR12MB1742:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MvUQii6WdxXcn8PwmNz9IFWor48bWhqaTkzqadQRKrq09brsx5Pj8yrhm9NhBulo7ZkFEdbKM65ddmCHeZIbfpuArwcbwepGXi3OEZ8XSQyOUbb9aWg57RPNMk2LDJgq9ppGFIt11wCtaE4VX/JG0kNHCMOS0xXNxgKDJ5r5V3SXDfxgrJMoiI/Bw87Wsxa6ZNNdGAOioIobVavmc5BDTo1LPqPqACGWBkMEYT970JSGFaGiduNtG/caKO2cbHisHNavz9DbKkkw3bjtwSdgr+u0s3smatqFIsJrXJ8kZ9Qig1Ju4pR77wWN557CL7DaidF9Lyul62DUg9CZ5sUjgy+foz1bVRxl3CexqaPzOClVvwBt5fG32gmA1I9ZyhGpQvXtwqYX29BHPRdC0pQj0nKHXXnArzwFTjopLopW/6qNLk75/N5pR07slQ2zyAvZRo0ALLOSnujUjY/Vn9IOTe61tclQ8aruNeG8VbPCtSQ3wp1KCQRCfF7dYGaP3R/GNnMcvQsjL/3qwdDjI8o5As9vTFOIAdHkpInc82Oc4+vyzjap432guOsj/UoPi4zlRkI48OWrT+B8cJK0tbvHe5eHaH1y8PRd10CIr81pxoGcIGcESKwiHtzViCuIF52ehyuuv0wY8oE8JG+XPo51WppLUkn5QTRRJqtDMTsDBuJlcZkOdc7ofU9joKOg/fFS7n6lQxp5QfsXCqtTjzzyNA== X-Forefront-Antispam-Report: CIP:12.22.5.236; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(36840700001)(40470700004)(46966006)(83380400001)(36860700001)(508600001)(86362001)(2906002)(40460700003)(81166007)(82310400005)(4326008)(8676002)(356005)(26005)(47076005)(6666004)(54906003)(70586007)(316002)(110136005)(70206006)(36756003)(5660300002)(2616005)(107886003)(7696005)(8936002)(1076003)(336012)(186003)(426003)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2022 17:07:54.9447 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 953c0b38-f026-42da-6957-08da425ef032 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.236]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT026.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1742 Received-SPF: softfail client-ip=2a01:111:f400:7eab::60e; envelope-from=avihaih@nvidia.com; helo=NAM11-CO1-obe.outbound.protection.outlook.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add new function qemu_file_get_to_fd() that allows reading data from QEMUFile and writing it straight into a given fd. This will be used later in VFIO migration code. Signed-off-by: Avihai Horon --- migration/qemu-file.c | 34 ++++++++++++++++++++++++++++++++++ migration/qemu-file.h | 1 + 2 files changed, 35 insertions(+) diff --git a/migration/qemu-file.c b/migration/qemu-file.c index 1479cddad9..cad3d32eb3 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -867,3 +867,37 @@ QIOChannel *qemu_file_get_ioc(QEMUFile *file) { return file->has_ioc ? QIO_CHANNEL(file->opaque) : NULL; } + +/* + * Read size bytes from QEMUFile f and write them to fd. + */ +int qemu_file_get_to_fd(QEMUFile *f, int fd, size_t size) +{ + while (size) { + size_t pending = f->buf_size - f->buf_index; + ssize_t rc; + + if (!pending) { + rc = qemu_fill_buffer(f); + if (rc < 0) { + return rc; + } + if (rc == 0) { + return -1; + } + continue; + } + + rc = write(fd, f->buf + f->buf_index, MIN(pending, size)); + if (rc < 0) { + return rc; + } + if (rc == 0) { + return -1; + } + f->buf_index += rc; + size -= rc; + } + + return 0; +} diff --git a/migration/qemu-file.h b/migration/qemu-file.h index 3f36d4dc8c..dd26037450 100644 --- a/migration/qemu-file.h +++ b/migration/qemu-file.h @@ -162,6 +162,7 @@ int qemu_file_shutdown(QEMUFile *f); QEMUFile *qemu_file_get_return_path(QEMUFile *f); void qemu_fflush(QEMUFile *f); void qemu_file_set_blocking(QEMUFile *f, bool block); +int qemu_file_get_to_fd(QEMUFile *f, int fd, size_t size); void ram_control_before_iterate(QEMUFile *f, uint64_t flags); void ram_control_after_iterate(QEMUFile *f, uint64_t flags); From patchwork Mon May 30 17:07:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 12864836 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 720FDC433EF for ; Mon, 30 May 2022 17:50:53 +0000 (UTC) Received: from localhost ([::1]:40428 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nvjXW-0000g0-N3 for qemu-devel@archiver.kernel.org; Mon, 30 May 2022 13:50:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54262) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvis7-0003gO-Uq for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:03 -0400 Received: from mail-dm6nam10on20630.outbound.protection.outlook.com ([2a01:111:f400:7e88::630]:18414 helo=NAM10-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvis5-0008M4-OI for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:03 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Qza9g1Jg51/dJcohJyhKUS1hG1Itssup65hsV/xWCkRlC+hBE/+vWVmzsyKU6uZSmURGRNsD1U1yk2QrbEOramLNkZrUpILWC+VDGJRf7kTWgmPMyViTKmbK7iRNapZT/rkt95QrLdyjC/bA7ndoRLy9wxVd+gQQckV9/0VRSRWf36Q11xHME0b5/ERAYqhE/mzDAPUFSDMxyw4T88tq7BbeH/FVtBEKYdiqngToXCW9BUb8P7ZQUQ5CueIdOGS3gZOcqDXki3IUAvjxFQYL59cR6yMGX2wJNIwbU9naxaWYX7XfT3gERInmgcE303RPu98bGi6BBT53PjkfsY1Pyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jypF+XZ5TJO5MGBaBe6ueZYxzrFZm4hXeUTs3lXLfRs=; b=ocGBuH+NpPc3is7hiDtEF4bDlhCK8WhQuOWbBEfXfbi39BBdFMQi3bVzVrfRkKYZ3qdY5X9FKwRLue0RGcW+B4bSuF7+BcZsMyxu8rnSdvc20BwPs98nBfNZAufikzE3/54CtPhPLUH+bAliQATcnAp0ecI30hngQNFKFwasfPMjAap5OUE+s/TmHTWfZzkVDR5ODZu6YGlmfrVuAXvL2lTkgAait3rEOj8i9tR5Tasi6jvyvOpU5VfowsrTJOP3vxNo597DwxPmV+K3xGsYOWbSbU0vCt8N2fGhFj8J7k7d4DVM4bkRyh1Ry7rgiOOEwNN5FS1fxCQu2Lt8Sfi1Iw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jypF+XZ5TJO5MGBaBe6ueZYxzrFZm4hXeUTs3lXLfRs=; b=pqd8qbAwOfEoKsNUW54KrAfJ8N7v3vZHwwT4M1J7wQLF5r1XfozNWGZ9CIHY1nIF9Xal/zefjwyhrfagb6Y69E8s/CWg8I9cbsNwlNASVDNm65UaAn4BFWEvMmCi4WX2Pq41APxilfkNi2gICtJZMtTD4xPwRhpykslOdEwiyvemTxKWtzbvGfXhIr3c6jKoXR8fYSEJqN7Vusc3FJ+WGQQETMyIsfd35Ji5oPaeZxwIHpLyzCSdmkqSVLP3DhU2QY3mk4jwrqWtdC/ZQ1Ye/ZFmAzHwJyPOAJVHvv4TVZOKXf9c27Qjk7jdnrqu6LQ7KnxGwzBIfeh8oWbdBFciUA== Received: from MWHPR08CA0044.namprd08.prod.outlook.com (2603:10b6:300:c0::18) by BN8PR12MB2994.namprd12.prod.outlook.com (2603:10b6:408:42::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.18; Mon, 30 May 2022 17:07:59 +0000 Received: from CO1NAM11FT030.eop-nam11.prod.protection.outlook.com (2603:10b6:300:c0:cafe::fc) by MWHPR08CA0044.outlook.office365.com (2603:10b6:300:c0::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:07:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.235) by CO1NAM11FT030.mail.protection.outlook.com (10.13.174.125) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:07:58 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 30 May 2022 17:07:58 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 30 May 2022 10:07:57 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.22 via Frontend Transport; Mon, 30 May 2022 10:07:54 -0700 From: Avihai Horon To: , Cornelia Huck , "Alex Williamson" , Juan Quintela , "Dr . David Alan Gilbert" CC: Joao Martins , Yishai Hadas , Jason Gunthorpe , Mark Bloch , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH v2 04/11] vfio/common: Change vfio_devices_all_running_and_saving() logic to equivalent one Date: Mon, 30 May 2022 20:07:32 +0300 Message-ID: <20220530170739.19072-5-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20220530170739.19072-1-avihaih@nvidia.com> References: <20220530170739.19072-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: fcdc6f24-c8d3-4398-af4d-08da425ef252 X-MS-TrafficTypeDiagnostic: BN8PR12MB2994:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5pURG42hkpy7fL+ILNVx76YfBKVArihXaK/UYbyUiNLhw6K3ORuiGir1ECymPTH7G/RcH4uNPWFzLT42JvK8duGy3LLVrR6bk1+zNF3nhhm6ve0nV1cE44pvABShF7k+5DCmXRbZkhvRewJ52eKH7oey8e6VQug4vjnF9GWhK0EH7J7phf4NcGjY5NT/5qMKuV2JnpvyaHKTEV4esDpeh+W9c04VIFFFkNIplPEfnAeU1gsrmSkERaONB0b9z4qTxMsDQlF8Q5JYmC1m4CPYKYduFVhIh4rI4Y2UASZfYKcGTC4FQ3Khjfm60bclwLBvUFgMkrea0AeX/QgxesolZ0A5odC8FB+upLRtSWb3uCL0+dpvXmO9SUjJPcHCtSTZ3clLiCzH3Yu8j5kAU05HnyJ2/dX8gZiYGTplgI24cDUgAgQuoDM6bYbxCipWycYHn76GWuDpEgxT/0ASlyBjwcdYxbWd2WMxZ+Tx3hhPd9sL9ZP8dnOcgpom/QOKSujN5bkIVx20xpRc1EVhSvXJwFQ2pJUcWXgh3hTeLuoKa+BSik7y3PqbtnQj9tqNNEg+z3ABmj+8IbzwSGAbQW3P4+Ts9bqZLGeDAsmlVIiUEpanpfzCWy7je87wF1uSnuK3jDvZikUsivthkpAUzBqMnGKJwy96jtpaV5Hg8D9wTmwpkGV3pmC78g4mRmJJQ649E7/ZKGABu79fEUx9VAhW/w== X-Forefront-Antispam-Report: CIP:12.22.5.235; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(46966006)(40470700004)(36840700001)(316002)(4326008)(7696005)(2616005)(8676002)(70206006)(70586007)(26005)(82310400005)(2906002)(86362001)(5660300002)(508600001)(8936002)(40460700003)(426003)(356005)(336012)(47076005)(36860700001)(36756003)(6666004)(186003)(1076003)(107886003)(54906003)(81166007)(110136005)(83380400001)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2022 17:07:58.5116 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: fcdc6f24-c8d3-4398-af4d-08da425ef252 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.235]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT030.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB2994 Received-SPF: softfail client-ip=2a01:111:f400:7e88::630; envelope-from=avihaih@nvidia.com; helo=NAM10-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" vfio_devices_all_running_and_saving() is used to check if migration is in pre-copy phase. This is done by checking if migration is in setup or active states and if all VFIO devices are in pre-copy state, i.e. _SAVING | _RUNNING. VFIO migration protocol v2 currently doesn't support pre-copy phase, so it doesn't have an equivalent VFIO pre-copy state like v1 has. As preparation for adding the v2 protocol and to make things easier, change vfio_devices_all_running_and_saving() logic to check if migration is active and if all VFIO devices are in running state. The new logic avoids using the VFIO pre-copy state and is equivalent to the previous one, indicating that migration is in pre-copy phase. No functional changes intended. Signed-off-by: Avihai Horon --- hw/vfio/common.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 29982c7af8..bbc6d375de 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -40,6 +40,7 @@ #include "trace.h" #include "qapi/error.h" #include "migration/migration.h" +#include "migration/misc.h" #include "sysemu/tpm.h" VFIOGroupList vfio_group_list = @@ -363,13 +364,16 @@ static bool vfio_devices_all_dirty_tracking(VFIOContainer *container) return true; } -static bool vfio_devices_all_running_and_saving(VFIOContainer *container) +/* + * Check if all VFIO devices are running and migration is active, which is + * essentially equivalent to the migration being in pre-copy phase. + */ +static bool vfio_devices_all_running_and_mig_active(VFIOContainer *container) { VFIOGroup *group; VFIODevice *vbasedev; - MigrationState *ms = migrate_get_current(); - if (!migration_is_setup_or_active(ms->state)) { + if (!migration_is_active(migrate_get_current())) { return false; } @@ -381,8 +385,7 @@ static bool vfio_devices_all_running_and_saving(VFIOContainer *container) return false; } - if ((migration->device_state & VFIO_DEVICE_STATE_V1_SAVING) && - (migration->device_state & VFIO_DEVICE_STATE_V1_RUNNING)) { + if (migration->device_state & VFIO_DEVICE_STATE_V1_RUNNING) { continue; } else { return false; @@ -461,7 +464,7 @@ static int vfio_dma_unmap(VFIOContainer *container, }; if (iotlb && container->dirty_pages_supported && - vfio_devices_all_running_and_saving(container)) { + vfio_devices_all_running_and_mig_active(container)) { return vfio_dma_unmap_bitmap(container, iova, size, iotlb); } From patchwork Mon May 30 17:07:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 12864867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 58895C433F5 for ; Mon, 30 May 2022 18:16:30 +0000 (UTC) Received: from localhost ([::1]:38440 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nvjwL-0005k0-6f for qemu-devel@archiver.kernel.org; Mon, 30 May 2022 14:16:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54278) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvisB-0003tI-V7 for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:07 -0400 Received: from mail-co1nam11on2060e.outbound.protection.outlook.com ([2a01:111:f400:7eab::60e]:5216 helo=NAM11-CO1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvisA-0008MJ-9u for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:07 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=M5C3ICtLYHIUykVC7v40z6avDB5IaThnwxJrIczOcmqOA3a/Xau3XBit+zXreoW92iJH1MktQeM7kZTs6NU7zHLi1b82gERzaIl6F2Y1j9+Fa+DRHUKHUkz3EX+en1V4LfvtSvbTgK51wktNVrQBOFGMG6jpo+t72AzfzWuaVGYZ0nbPuNRty6KFagt2s4KDIBLTgaJ6WRjADv5DCYlgH3YR9MdcQ9kXmUSxr4A4DkSrfU2Z+a+y3FXkioV2jYIOvIMvAnQMv1wAiLfiDLetVwXkpdlCaI3iqdi6LzCJS427wY4ko0sPRDsHNUPZjN0gFZPh7e9qU/Ssj8UD/jymig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DjlscHecdcMpLoybC6kEB/IIni+Nsq0JItrk38p+xG8=; b=YV9kUbdr1QQ/R0y+wJIeo9qD+IeAgzhIkJdOjAhYk0oe4kAwwUISExvQu11Oor5wNP0J2gOTytmJFhUelGKhyhy9dU+Dtp+TULw+Xk6PegTOuGeUCfc1gP8yXOKvP+nz9a+jz0hvzKuY+cbOAVPhMUNVw+U80SyD27o+Xp2M8rWyI42F1C9m4BMsdZsyD4mMhGrymvS7/52Vx4Bq2vPDCbVu+Ur1M/5UE1pWOg9rCXLoq7+xowW1yzJF0eLqWTwYZLrhyBwDARwGVJWsW0QGOvYcnVIbzr8eHqMYBvAJvBENOevbtaXA2s660eA1LjztAPOW4FbgZIfABa/VtIKiSA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.238) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DjlscHecdcMpLoybC6kEB/IIni+Nsq0JItrk38p+xG8=; b=p1KA81HzPakd1eUDtv9MYv3Hj0l964Xb2DX3SpQG7GP1akssFn0GA8Yc5STF31sFz7OMhJhPBlOcTQyP5IKjFxG/6M1MQ9XOh5zmiRpCpqDOfRyL1vJxixhQrSqzv9YmMO5nWdXuzhu/yDPjRKOC5JduELptamIgWS77av+yNIUMCjKe4ojbWqU2xGQlCB6aWYk88UWkvIpBMwqKElGlRhKa3g/YBcEldf1xScgYJ/oCh//qZkey+ZcN2rAwFj/uMZ3P4aFCCYpbuftrEFYtoFVW/HgBhtDEGx6u4MFgHsamTeW24ooUzPkrHnn/1MHkSG1Nzv1AP26FuKPh425CRQ== Received: from MW4PR04CA0314.namprd04.prod.outlook.com (2603:10b6:303:82::19) by BYAPR12MB4775.namprd12.prod.outlook.com (2603:10b6:a03:107::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Mon, 30 May 2022 17:08:02 +0000 Received: from CO1NAM11FT059.eop-nam11.prod.protection.outlook.com (2603:10b6:303:82:cafe::d6) by MW4PR04CA0314.outlook.office365.com (2603:10b6:303:82::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:08:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.238) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.238 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.238; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.238) by CO1NAM11FT059.mail.protection.outlook.com (10.13.174.160) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:08:01 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by DRHQMAIL105.nvidia.com (10.27.9.14) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 30 May 2022 17:08:01 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 30 May 2022 10:08:01 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.22 via Frontend Transport; Mon, 30 May 2022 10:07:58 -0700 From: Avihai Horon To: , Cornelia Huck , "Alex Williamson" , Juan Quintela , "Dr . David Alan Gilbert" CC: Joao Martins , Yishai Hadas , Jason Gunthorpe , Mark Bloch , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH v2 05/11] vfio/migration: Move migration v1 logic to vfio_migration_init() Date: Mon, 30 May 2022 20:07:33 +0300 Message-ID: <20220530170739.19072-6-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20220530170739.19072-1-avihaih@nvidia.com> References: <20220530170739.19072-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: bfb53061-0f4a-4b08-67c8-08da425ef462 X-MS-TrafficTypeDiagnostic: BYAPR12MB4775:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FskSo0/GV3UtBtFwRCcnBZLb7VsUrtSFfSjUy0TwzS9kARXRqAtWnsu+AAx52pOt/2KSXgqweJkGKp9iRCrgMKm4tN3xzfS2fLRpv+WpvfIEM/IQYeajNuOszHqYZAC3sQmkyX5cKQjLTe7YfxyTEqxosnHVANFB2lRXcWiJrKzBpzW3Ud5fvse20rTYQt7S239vEPSGGeyo+AqT3eiHRxw7XJokOSkns9WXp4Xi/laDYdNzgc8qlj2G7DFCT3BebbPWQLY1EIyDDep/nTONavIDLOpRIN53qM01zm+ct1yEAfRPzi6he0GVH+WWtxF/iMfbTc2md9xZBWPTP+BONClS4zJLt59IggPC+xy2HwAdpV9HorXkCdoLESxF64Yi9+di3n/xoHPqwUqI2BLgwZde85SK93eAUhk6JlbdU/uwoPVJidWtEoz8dzq2GCupiDrvtCAtWX44H+oYkPpXB9SJp7E/7C8FbI9zO+p0VMBNinA4/L0myyQPacdhbF0LHrS/taWPVf+MWtvau02X7qh8fUufOvl2D9OUXbnMkPcRpEOuAJ038oE8aiB+Tv3qIXfRR9BVzrPrCY5AOsVIoiHBW8jXanQDS01B5nVWI772k8mEJNXFQURw5Wmg05eHbynX8dSwvtfRbjNtknpdpKlToBFjfc7WIpDMeaBc86fdTzlrYdmwnW3q0eX7YOu9HGryxr+9eFfdZcvV2fOH0A== X-Forefront-Antispam-Report: CIP:12.22.5.238; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(36840700001)(46966006)(2616005)(26005)(2906002)(110136005)(54906003)(356005)(8676002)(316002)(36860700001)(7696005)(508600001)(47076005)(86362001)(6666004)(81166007)(336012)(186003)(40460700003)(82310400005)(8936002)(36756003)(426003)(83380400001)(107886003)(70206006)(70586007)(4326008)(1076003)(5660300002)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2022 17:08:01.9545 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bfb53061-0f4a-4b08-67c8-08da425ef462 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.238]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT059.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB4775 Received-SPF: softfail client-ip=2a01:111:f400:7eab::60e; envelope-from=avihaih@nvidia.com; helo=NAM11-CO1-obe.outbound.protection.outlook.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Move vfio_dev_get_region_info() logic from vfio_migration_probe() to vfio_migration_init(). This logic is specific to v1 protocol and moving it will make it easier to add the v2 protocol implementation later. No functional changes intended. Signed-off-by: Avihai Horon --- hw/vfio/migration.c | 30 +++++++++++++++--------------- hw/vfio/trace-events | 2 +- 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index d8f9b086c2..8a0deed0e4 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -789,14 +789,14 @@ static void vfio_migration_exit(VFIODevice *vbasedev) vbasedev->migration = NULL; } -static int vfio_migration_init(VFIODevice *vbasedev, - struct vfio_region_info *info) +static int vfio_migration_init(VFIODevice *vbasedev) { int ret; Object *obj; VFIOMigration *migration; char id[256] = ""; g_autofree char *path = NULL, *oid = NULL; + struct vfio_region_info *info = NULL; if (!vbasedev->ops->vfio_get_object) { return -EINVAL; @@ -807,6 +807,14 @@ static int vfio_migration_init(VFIODevice *vbasedev, return -EINVAL; } + ret = vfio_get_dev_region_info(vbasedev, + VFIO_REGION_TYPE_MIGRATION_DEPRECATED, + VFIO_REGION_SUBTYPE_MIGRATION_DEPRECATED, + &info); + if (ret) { + return ret; + } + vbasedev->migration = g_new0(VFIOMigration, 1); ret = vfio_region_setup(obj, vbasedev, &vbasedev->migration->region, @@ -824,6 +832,8 @@ static int vfio_migration_init(VFIODevice *vbasedev, goto err; } + g_free(info); + migration = vbasedev->migration; migration->vbasedev = vbasedev; @@ -846,6 +856,7 @@ static int vfio_migration_init(VFIODevice *vbasedev, return 0; err: + g_free(info); vfio_migration_exit(vbasedev); return ret; } @@ -860,7 +871,6 @@ int64_t vfio_mig_bytes_transferred(void) int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) { VFIOContainer *container = vbasedev->group->container; - struct vfio_region_info *info = NULL; int ret = -ENOTSUP; if (!vbasedev->enable_migration) { @@ -874,27 +884,17 @@ int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) migrate_get_current()->skip_precopy = true; } - ret = vfio_get_dev_region_info(vbasedev, - VFIO_REGION_TYPE_MIGRATION_DEPRECATED, - VFIO_REGION_SUBTYPE_MIGRATION_DEPRECATED, - &info); + ret = vfio_migration_init(vbasedev); if (ret) { goto add_blocker; } - ret = vfio_migration_init(vbasedev, info); - if (ret) { - goto add_blocker; - } - - trace_vfio_migration_probe(vbasedev->name, info->index); - g_free(info); + trace_vfio_migration_probe(vbasedev->name); return 0; add_blocker: error_setg(&vbasedev->migration_blocker, "VFIO device doesn't support migration"); - g_free(info); ret = migrate_add_blocker(vbasedev->migration_blocker, errp); if (ret < 0) { diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 582882db91..438402b619 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -148,7 +148,7 @@ vfio_display_edid_update(uint32_t prefx, uint32_t prefy) "%ux%u" vfio_display_edid_write_error(void) "" # migration.c -vfio_migration_probe(const char *name, uint32_t index) " (%s) Region %d" +vfio_migration_probe(const char *name) " (%s)" vfio_migration_set_state(const char *name, uint32_t state) " (%s) state %d" vfio_vmstate_change(const char *name, int running, const char *reason, uint32_t dev_state) " (%s) running %d reason %s device state %d" vfio_migration_state_notifier(const char *name, const char *state) " (%s) state %s" From patchwork Mon May 30 17:07:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 12864843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A22F5C433EF for ; Mon, 30 May 2022 17:58:46 +0000 (UTC) Received: from localhost ([::1]:56680 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nvjfB-0003Yr-NH for qemu-devel@archiver.kernel.org; Mon, 30 May 2022 13:58:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54300) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvisF-00049y-K8 for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:11 -0400 Received: from mail-dm6nam10on20611.outbound.protection.outlook.com ([2a01:111:f400:7e88::611]:46689 helo=NAM10-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvisD-0008MT-5P for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:11 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FSW6oANBiCDkEAh7KVmNV34FupFOe+8Kgu916NIGnMjeR1mZJvjdKPptL84uQC+5fWif83UyHAycAY9lJp1VkPkdPb9NO9hCUO3Xfs77c0gS/QNU3Ddel9IxC+B5M6MLCD5gEPugD60SOT9ObwHHTUWU145mZXNsYUw1Aa0d/s3BZ1Ul2MVmfqq1cAMifDj/8Uxks83H3xJCPyNj5frYt6T0ToCvSsc9K2YFJiVum7SdLUFo38YRkPc8SGpu/z1wXP/csSh6IIKTtHmmJri0q8NJH2T5G+knBX6DYnqqVUqvrFdWQGAopVe+ZzTnJKrtCoWmXeg5muwuXVrAUf5TZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TAcKiWMX79rHiJ0N+9JONfz6MYJ8RIJrv7lY5JEVzKA=; b=fH4R7SgwWx5ddFzh4GFmON/dZE1hBCpuoD7YS+7eNcrWfsPe5cHUPFKIk9hPQnACIoWzF/RDSyzj4nG6ufRt/ols5CINjoYOZGNtej2SQ+vwiUn8n6D0E/ay1DHD4HkJOuZDw+pCMRGXJADG9rhniSf4k5ch2GdqFuZ5s6OOJAyepYfF7sMe/S/Qit8sLIJucJNoPuazfqfFceG4iAwjr8qSLYRErU4DwAu72U661dDT8bH8B0w/ZwNvOgXl9+gPqDjwZiocPUeHfEcCDY1gt+dDvk6mLdGNH+Cm8xdnOuoUoXL+ytg+hgwFpaSdlAiY6vU/Yhk7Um+7wGko73P2DA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.236) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TAcKiWMX79rHiJ0N+9JONfz6MYJ8RIJrv7lY5JEVzKA=; b=cnS5mJF9GAPK59C32rAPRSIN0HHn6KlfL9vxg8VfTsztdajetkwi/LoZb8Jv7DrbD8Kfmzkmpk55eQaM2sF8snK2PO1jp04M9aSJtJlnCydhUA9denRS30m95d8jN0U668fFnL6SpLy1nxLJUsU4gYMopuQDRrVr4Z1RGw4JEPJYuxlwAXXJn8w+WiVWlHU4D6y0CQzDX+Z/zZ00iebAFOi6adiRIbhqhiqHjbcnfQrYcMBVkiai0kuYajMaDULBf1jkYPyB6MWOzhlkGoEKTtyOPuPevoXglrs5LpgQXIFQL85Rjmo7PBsgvbLzSG73CLBCGBGC38I8t77Fsrn7TQ== Received: from MWHPR10CA0069.namprd10.prod.outlook.com (2603:10b6:300:2c::31) by MWHPR12MB1360.namprd12.prod.outlook.com (2603:10b6:300:12::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Mon, 30 May 2022 17:08:05 +0000 Received: from CO1NAM11FT024.eop-nam11.prod.protection.outlook.com (2603:10b6:300:2c:cafe::e6) by MWHPR10CA0069.outlook.office365.com (2603:10b6:300:2c::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:08:05 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.236) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.236 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.236; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.236) by CO1NAM11FT024.mail.protection.outlook.com (10.13.174.162) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:08:05 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by DRHQMAIL109.nvidia.com (10.27.9.19) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 30 May 2022 17:08:05 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 30 May 2022 10:08:04 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.22 via Frontend Transport; Mon, 30 May 2022 10:08:01 -0700 From: Avihai Horon To: , Cornelia Huck , "Alex Williamson" , Juan Quintela , "Dr . David Alan Gilbert" CC: Joao Martins , Yishai Hadas , Jason Gunthorpe , Mark Bloch , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH v2 06/11] vfio/migration: Rename functions/structs related to v1 protocol Date: Mon, 30 May 2022 20:07:34 +0300 Message-ID: <20220530170739.19072-7-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20220530170739.19072-1-avihaih@nvidia.com> References: <20220530170739.19072-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 50a572a4-143a-4b83-e4aa-08da425ef69a X-MS-TrafficTypeDiagnostic: MWHPR12MB1360:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 14GiJsw5JtDs5YFpNx65qBCNPXJuOkUUYSV4RSTtgiukMS5KqxKI0LSlnY+ZaFwuY+t4H70Wr5Rfegs1lF5KZktJPu/p2NsWkIjeUDtSHTwSmEWiWnuuGaxW1Lc3nojqUME09ZfqYuYMO71ousFTbMdRjLN91c4kV26bT9wYkF3Q4pqo55S2jyEelJXYgyw46/kR7AwenYamhgOkUEsCWFEkEm6L1FyjmsdDBBSTnVx08VrHYNibnZyzQJk5vzDouqo6lUDuNZrQ8dvvyQJ6GnC1aLLdMAXQ5AA5AXZ/tLluibrpHoyVLIN9BYKzi4OdH8OXkDGmTVipH9HitXXNwKzlSaSFSffZ2MhVNjcfhqvbG3DWFhrcS2237A17Bm81E2P1ddbwvUAhPdDataCOOyCiuO5hbCCk1NnVObLe751/uwXjIMGTGZ2ZI0HsSa6vhJO/YDVA3omKMXSIqOGR1zkPbUNG2zYV8LO0FdwYz6opZsKOcgXgghlcsryRcxZ0oFfY1hBdGsp7cEGceaBON5bz/H0VbADUUdQZws50tPiYnQJur9EQFtxkOUn3x1gjGe3Ars/HeeLAuSF4yDIh4xHVrkkqdZzA1ymZzN7ftXe3aJDfJCuC0nqTYzCFUzx9V/zIxJBeeAxSAJ8AOWwP47BT1WJNW5LUH1SHaTGbMHMl+cMoP4FBN6vghlwXcRJz0Ohta3foe3nYBUQ46S83yA== X-Forefront-Antispam-Report: CIP:12.22.5.236; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(36840700001)(46966006)(40470700004)(40460700003)(186003)(86362001)(82310400005)(6666004)(81166007)(336012)(70206006)(107886003)(5660300002)(4326008)(70586007)(1076003)(426003)(36756003)(83380400001)(8936002)(30864003)(356005)(8676002)(2906002)(110136005)(54906003)(316002)(26005)(2616005)(508600001)(47076005)(36860700001)(7696005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2022 17:08:05.6801 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 50a572a4-143a-4b83-e4aa-08da425ef69a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.236]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT024.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1360 Received-SPF: softfail client-ip=2a01:111:f400:7e88::611; envelope-from=avihaih@nvidia.com; helo=NAM10-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" To avoid name collisions, rename functions and structs related to VFIO migration protocol v1. This will allow the two protocols to co-exist when v2 protocol is added, until v1 is removed. No functional changes intended. Signed-off-by: Avihai Horon --- hw/vfio/common.c | 6 +-- hw/vfio/migration.c | 82 +++++++++++++++++------------------ hw/vfio/trace-events | 2 +- include/hw/vfio/vfio-common.h | 2 +- 4 files changed, 46 insertions(+), 46 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index bbc6d375de..a3dd8221ed 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -355,8 +355,8 @@ static bool vfio_devices_all_dirty_tracking(VFIOContainer *container) return false; } - if ((vbasedev->pre_copy_dirty_page_tracking == ON_OFF_AUTO_OFF) - && (migration->device_state & VFIO_DEVICE_STATE_V1_RUNNING)) { + if ((vbasedev->pre_copy_dirty_page_tracking == ON_OFF_AUTO_OFF) && + (migration->device_state_v1 & VFIO_DEVICE_STATE_V1_RUNNING)) { return false; } } @@ -385,7 +385,7 @@ static bool vfio_devices_all_running_and_mig_active(VFIOContainer *container) return false; } - if (migration->device_state & VFIO_DEVICE_STATE_V1_RUNNING) { + if (migration->device_state_v1 & VFIO_DEVICE_STATE_V1_RUNNING) { continue; } else { return false; diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 8a0deed0e4..e40aa0ad80 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -107,8 +107,8 @@ static int vfio_mig_rw(VFIODevice *vbasedev, __u8 *buf, size_t count, * an error is returned. */ -static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t mask, - uint32_t value) +static int vfio_migration_v1_set_state(VFIODevice *vbasedev, uint32_t mask, + uint32_t value) { VFIOMigration *migration = vbasedev->migration; VFIORegion *region = &migration->region; @@ -145,7 +145,7 @@ static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t mask, return ret; } - migration->device_state = device_state; + migration->device_state_v1 = device_state; trace_vfio_migration_set_state(vbasedev->name, device_state); return 0; } @@ -260,8 +260,8 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size) return ret; } -static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, - uint64_t data_size) +static int vfio_v1_load_buffer(QEMUFile *f, VFIODevice *vbasedev, + uint64_t data_size) { VFIORegion *region = &vbasedev->migration->region; uint64_t data_offset = 0, size, report_size; @@ -288,7 +288,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, data_size = 0; } - trace_vfio_load_state_device_data(vbasedev->name, data_offset, size); + trace_vfio_v1_load_state_device_data(vbasedev->name, data_offset, size); while (size) { void *buf; @@ -394,7 +394,7 @@ static int vfio_load_device_config_state(QEMUFile *f, void *opaque) return qemu_file_get_error(f); } -static void vfio_migration_cleanup(VFIODevice *vbasedev) +static void vfio_migration_v1_cleanup(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; @@ -405,7 +405,7 @@ static void vfio_migration_cleanup(VFIODevice *vbasedev) /* ---------------------------------------------------------------------- */ -static int vfio_save_setup(QEMUFile *f, void *opaque) +static int vfio_v1_save_setup(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; @@ -431,8 +431,8 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) } } - ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_MASK, - VFIO_DEVICE_STATE_V1_SAVING); + ret = vfio_migration_v1_set_state(vbasedev, VFIO_DEVICE_STATE_MASK, + VFIO_DEVICE_STATE_V1_SAVING); if (ret) { error_report("%s: Failed to set state SAVING", vbasedev->name); return ret; @@ -448,11 +448,11 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) return 0; } -static void vfio_save_cleanup(void *opaque) +static void vfio_v1_save_cleanup(void *opaque) { VFIODevice *vbasedev = opaque; - vfio_migration_cleanup(vbasedev); + vfio_migration_v1_cleanup(vbasedev); trace_vfio_save_cleanup(vbasedev->name); } @@ -524,15 +524,15 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque) return 0; } -static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) +static int vfio_v1_save_complete_precopy(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; uint64_t data_size; int ret; - ret = vfio_migration_set_state(vbasedev, ~VFIO_DEVICE_STATE_V1_RUNNING, - VFIO_DEVICE_STATE_V1_SAVING); + ret = vfio_migration_v1_set_state(vbasedev, ~VFIO_DEVICE_STATE_V1_RUNNING, + VFIO_DEVICE_STATE_V1_SAVING); if (ret) { error_report("%s: Failed to set state STOP and SAVING", vbasedev->name); @@ -569,7 +569,8 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) return ret; } - ret = vfio_migration_set_state(vbasedev, ~VFIO_DEVICE_STATE_V1_SAVING, 0); + ret = vfio_migration_v1_set_state(vbasedev, ~VFIO_DEVICE_STATE_V1_SAVING, + 0); if (ret) { error_report("%s: Failed to set state STOPPED", vbasedev->name); return ret; @@ -592,7 +593,7 @@ static void vfio_save_state(QEMUFile *f, void *opaque) } } -static int vfio_load_setup(QEMUFile *f, void *opaque) +static int vfio_v1_load_setup(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; @@ -608,8 +609,8 @@ static int vfio_load_setup(QEMUFile *f, void *opaque) } } - ret = vfio_migration_set_state(vbasedev, ~VFIO_DEVICE_STATE_MASK, - VFIO_DEVICE_STATE_V1_RESUMING); + ret = vfio_migration_v1_set_state(vbasedev, ~VFIO_DEVICE_STATE_MASK, + VFIO_DEVICE_STATE_V1_RESUMING); if (ret) { error_report("%s: Failed to set state RESUMING", vbasedev->name); if (migration->region.mmaps) { @@ -619,11 +620,11 @@ static int vfio_load_setup(QEMUFile *f, void *opaque) return ret; } -static int vfio_load_cleanup(void *opaque) +static int vfio_v1_load_cleanup(void *opaque) { VFIODevice *vbasedev = opaque; - vfio_migration_cleanup(vbasedev); + vfio_migration_v1_cleanup(vbasedev); trace_vfio_load_cleanup(vbasedev->name); return 0; } @@ -661,7 +662,7 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) uint64_t data_size = qemu_get_be64(f); if (data_size) { - ret = vfio_load_buffer(f, vbasedev, data_size); + ret = vfio_v1_load_buffer(f, vbasedev, data_size); if (ret < 0) { return ret; } @@ -682,21 +683,21 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) return ret; } -static SaveVMHandlers savevm_vfio_handlers = { - .save_setup = vfio_save_setup, - .save_cleanup = vfio_save_cleanup, +static SaveVMHandlers savevm_vfio_v1_handlers = { + .save_setup = vfio_v1_save_setup, + .save_cleanup = vfio_v1_save_cleanup, .save_live_pending = vfio_save_pending, .save_live_iterate = vfio_save_iterate, - .save_live_complete_precopy = vfio_save_complete_precopy, + .save_live_complete_precopy = vfio_v1_save_complete_precopy, .save_state = vfio_save_state, - .load_setup = vfio_load_setup, - .load_cleanup = vfio_load_cleanup, + .load_setup = vfio_v1_load_setup, + .load_cleanup = vfio_v1_load_cleanup, .load_state = vfio_load_state, }; /* ---------------------------------------------------------------------- */ -static void vfio_vmstate_change(void *opaque, bool running, RunState state) +static void vfio_v1_vmstate_change(void *opaque, bool running, RunState state) { VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; @@ -736,21 +737,21 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) } } - ret = vfio_migration_set_state(vbasedev, mask, value); + ret = vfio_migration_v1_set_state(vbasedev, mask, value); if (ret) { /* * Migration should be aborted in this case, but vm_state_notify() * currently does not support reporting failures. */ error_report("%s: Failed to set device state 0x%x", vbasedev->name, - (migration->device_state & mask) | value); + (migration->device_state_v1 & mask) | value); if (migrate_get_current()->to_dst_file) { qemu_file_set_error(migrate_get_current()->to_dst_file, ret); } } vbasedev->migration->vm_running = running; trace_vfio_vmstate_change(vbasedev->name, running, RunState_str(state), - (migration->device_state & mask) | value); + (migration->device_state_v1 & mask) | value); } static void vfio_migration_state_notifier(Notifier *notifier, void *data) @@ -769,10 +770,10 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data) case MIGRATION_STATUS_CANCELLED: case MIGRATION_STATUS_FAILED: bytes_transferred = 0; - ret = vfio_migration_set_state(vbasedev, - ~(VFIO_DEVICE_STATE_V1_SAVING | - VFIO_DEVICE_STATE_V1_RESUMING), - VFIO_DEVICE_STATE_V1_RUNNING); + ret = vfio_migration_v1_set_state(vbasedev, + ~(VFIO_DEVICE_STATE_V1_SAVING | + VFIO_DEVICE_STATE_V1_RESUMING), + VFIO_DEVICE_STATE_V1_RUNNING); if (ret) { error_report("%s: Failed to set state RUNNING", vbasedev->name); } @@ -845,12 +846,11 @@ static int vfio_migration_init(VFIODevice *vbasedev) } strpadcpy(id, sizeof(id), path, '\0'); - register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, &savevm_vfio_handlers, - vbasedev); + register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, + &savevm_vfio_v1_handlers, vbasedev); - migration->vm_state = qdev_add_vm_change_state_handler(vbasedev->dev, - vfio_vmstate_change, - vbasedev); + migration->vm_state = qdev_add_vm_change_state_handler( + vbasedev->dev, vfio_v1_vmstate_change, vbasedev); migration->migration_state.notify = vfio_migration_state_notifier; add_migration_state_change_notifier(&migration->migration_state); return 0; diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 438402b619..ac8b04f52a 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -162,7 +162,7 @@ vfio_save_iterate(const char *name, int data_size) " (%s) data_size %d" vfio_save_complete_precopy(const char *name) " (%s)" vfio_load_device_config_state(const char *name) " (%s)" vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64 -vfio_load_state_device_data(const char *name, uint64_t data_offset, uint64_t data_size) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64 +vfio_v1_load_state_device_data(const char *name, uint64_t data_offset, uint64_t data_size) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64 vfio_load_cleanup(const char *name) " (%s)" vfio_get_dirty_bitmap(int fd, uint64_t iova, uint64_t size, uint64_t bitmap_size, uint64_t start) "container fd=%d, iova=0x%"PRIx64" size= 0x%"PRIx64" bitmap_size=0x%"PRIx64" start=0x%"PRIx64 vfio_iommu_map_dirty_notify(uint64_t iova_start, uint64_t iova_end) "iommu dirty @ 0x%"PRIx64" - 0x%"PRIx64 diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index e573f5a9f1..bbaf72ba00 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -62,7 +62,7 @@ typedef struct VFIOMigration { struct VFIODevice *vbasedev; VMChangeStateEntry *vm_state; VFIORegion region; - uint32_t device_state; + uint32_t device_state_v1; int vm_running; Notifier migration_state; uint64_t pending_bytes; From patchwork Mon May 30 17:07:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 12864870 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 94987C433F5 for ; Mon, 30 May 2022 18:21:09 +0000 (UTC) Received: from localhost ([::1]:47618 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nvk0m-0003u7-Vc for qemu-devel@archiver.kernel.org; Mon, 30 May 2022 14:21:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54316) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvisK-0004Nu-Bg for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:16 -0400 Received: from mail-sn1anam02on20614.outbound.protection.outlook.com ([2a01:111:f400:7ea9::614]:51678 helo=NAM02-SN1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvisG-0008Ms-8X for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:16 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=h496DP5xAtWyVhUpAxmVSHIqNFqvEe3ocHJ5gZ8yucw7I5Bqj6Ie2jdvwrvvOsOazPDXcSqb3lAid+MKuabarYc0GjJv9Yy+ySsrgzUiHV+Dbfzl+nNmzT2MCZ0r1j2sl9NI9c526Nkn2RY2W7ImVw+N/xvtjA+5UC0LRcVNOfK+/UHdgQanMavKjQeJXRLfoRA/QhDPtVBjUAX+Dy6xyryhgc+qQeB4GXEv8eS/PbeRsrpEz7yBhjUzJGocuMy1eqjzPcOsgxjCJzYiQzxQh8JJm6zhr+NfYXOfC3Y59HGZrZ2m0G0YUStBCb5nHL5ICa5BKi/AT2XFk+dSEDxF3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9/zOSCKw7zFSSB+1LnE/2WjnkVIfVsyyLiFlMm4RF1o=; b=CuvRtPVztVq/7l6MCzPTmCZiV5jsHV7KsWHw26YM7v8orwL9klTySwGNKKEcms8qZfFs0jMvyo1P9WtZ+FPRYAF/oqXby5kgzeN4XjJX0J7jekoz50/OWwjDp8K5m3pCho1lB2QKw6l9X83AmtXuo34My6dRK1zXdhmx8Hd6LsLFzDN3vx09YqigZ1fx7/Nsl6+XLmtpDs3bU1dh+f5oakOLBODKQkzBpBlDTjMcnc5uDC9hQTsEafx2ZoaH14GNUm1UedDf/mHIw62MrlJ9B+Wa1/SQv/qmk/9wUr2HmyXsSrKp6Jg13s/L7SOww7+GJVT9qdpExPhutXKXY/jaCQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9/zOSCKw7zFSSB+1LnE/2WjnkVIfVsyyLiFlMm4RF1o=; b=PGsteNYxN8zz4uu6ewkPkDq0kJjv4wDlZslRgmrH59r0IxF4nEXou8r1QMurIYJtqvLetpOzU/vfW2gISRjGCguUINMpvIcfLYtkJBtidZ4SaSs5joiZuPUh7YitWDEFtpA5h2dEbqdEjF8nLzzE/+aRfrn76109aeBKrrQaHIzZ8xqjw4AkumUiuvHt0ZJslqNJoywW4DoAd/+YkBvrutgS6oLJliZQ1aw9ryuWtGjWFICH9CatoRlvejLaY6MQMOKKIrq92LulJwytcv43CX2ETzRqOssVfwhEeFO/Ukvs1LyHDCMwObvRCJJQWFRtPLkRFo4KtfAdCViwgGNAEw== Received: from CO1PR15CA0099.namprd15.prod.outlook.com (2603:10b6:101:21::19) by DM6PR12MB4235.namprd12.prod.outlook.com (2603:10b6:5:220::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Mon, 30 May 2022 17:08:09 +0000 Received: from CO1NAM11FT035.eop-nam11.prod.protection.outlook.com (2603:10b6:101:21:cafe::92) by CO1PR15CA0099.outlook.office365.com (2603:10b6:101:21::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.19 via Frontend Transport; Mon, 30 May 2022 17:08:09 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.235) by CO1NAM11FT035.mail.protection.outlook.com (10.13.175.36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:08:08 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 30 May 2022 17:08:08 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 30 May 2022 10:08:08 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.22 via Frontend Transport; Mon, 30 May 2022 10:08:05 -0700 From: Avihai Horon To: , Cornelia Huck , "Alex Williamson" , Juan Quintela , "Dr . David Alan Gilbert" CC: Joao Martins , Yishai Hadas , Jason Gunthorpe , Mark Bloch , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH v2 07/11] vfio/migration: Implement VFIO migration protocol v2 Date: Mon, 30 May 2022 20:07:35 +0300 Message-ID: <20220530170739.19072-8-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20220530170739.19072-1-avihaih@nvidia.com> References: <20220530170739.19072-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7df47cd1-9995-4f66-4802-08da425ef884 X-MS-TrafficTypeDiagnostic: DM6PR12MB4235:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Yfp/bYY8hCIDni58U9PRYsQWhzQtMU0CZP6ThAYb6+x6uOjQ+cXZcl8uzFfTUaEpUSuDpbSA1v3VYAiZy9fJxkibYPCBELo7XaLB+RAcmRd6YUhbKHmgbL3+vtQBMj+LtLmAohQ2ttgQU5YC+ekyZsqCZ3ueky+L8JHbpX3Q9+0h+rI4X6wIMgPoPZT2orozACB0iUYPM49je9tCgsoMizl13yA8Z5asC/xiFUY3Z4M/gORnpzig0o37HLB01JC5Y/qzf8Q6vCmX9OzmmalxuG7JMIUsxerB1N+1LNk7bGmZQV67xHPt0wXlGfynqhEr46/WSaF35fEmbWK0rxQo8oQRXLjiF8UsVo2F6jw0FEDlzkEt7yRyJx5Z77SE0i7ajBpdW3eAVZ0CQymTB6OJ8zvMkq3FgIQfS5XuYH+kxhUfbw5MR+6qBGDxp675DoraAB8JpTZYf6ZAORORl0NAMFWd7xx/2jWiUZWTNVLiCphznpJ2RAriA0lXe2yCgrsYsAUvM8B2+dt4GbIuKQL6jHQjZaTOOck44YjWUl3nC0P8c07XxZRkzC7AewqJ6LyDMF8CMJhlsP2PDDBru4SPv/+vHCT67KYcNfbPLsvfMqxlDP9G9aLB8E5DPyKgq2k31EWGiWw3S2tLO+vQw8KYiSB7PNH+YixgbUho7h4th9D/t5nCnXuQNGnvX6DIpwI9uX0Q0VkXwUa9uV7h6sltyfmI0DkikpWt6JTl/WPI8frPg7vvQarYby3UztWQWZfn3KdCiJqpLGAiLJxS9IS6s0p25BflKWsPj6aywU3lt2U= X-Forefront-Antispam-Report: CIP:12.22.5.235; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(46966006)(40470700004)(36840700001)(70206006)(82310400005)(4326008)(8676002)(70586007)(86362001)(5660300002)(2616005)(1076003)(107886003)(30864003)(36860700001)(186003)(6666004)(40460700003)(508600001)(966005)(26005)(54906003)(8936002)(7696005)(110136005)(2906002)(316002)(81166007)(47076005)(83380400001)(426003)(336012)(36756003)(356005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2022 17:08:08.8718 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7df47cd1-9995-4f66-4802-08da425ef884 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.235]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT035.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4235 Received-SPF: softfail client-ip=2a01:111:f400:7ea9::614; envelope-from=avihaih@nvidia.com; helo=NAM02-SN1-obe.outbound.protection.outlook.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add implementation of VFIO migration protocol v2. The two protocols, v1 and v2, will co-exist and in next patch v1 protocol will be removed. There are several main differences between v1 and v2 protocols: - VFIO device state is now represented as a finite state machine instead of a bitmap. - Migration interface with kernel is now done using VFIO_DEVICE_FEATURE ioctl and normal read() and write() instead of the migration region. - VFIO migration protocol v2 currently doesn't support the pre-copy phase of migration. Detailed information about VFIO migration protocol v2 and difference compared to v1 can be found here [1]. [1] https://lore.kernel.org/all/20220224142024.147653-10-yishaih@nvidia.com/ Signed-off-by: Avihai Horon --- hw/vfio/common.c | 19 +- hw/vfio/migration.c | 365 ++++++++++++++++++++++++++++++---- hw/vfio/trace-events | 2 + include/hw/vfio/vfio-common.h | 5 + 4 files changed, 354 insertions(+), 37 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index a3dd8221ed..5541133ec9 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -355,10 +355,18 @@ static bool vfio_devices_all_dirty_tracking(VFIOContainer *container) return false; } - if ((vbasedev->pre_copy_dirty_page_tracking == ON_OFF_AUTO_OFF) && + if (!migration->v2 && + (vbasedev->pre_copy_dirty_page_tracking == ON_OFF_AUTO_OFF) && (migration->device_state_v1 & VFIO_DEVICE_STATE_V1_RUNNING)) { return false; } + + if (migration->v2 && + (vbasedev->pre_copy_dirty_page_tracking == ON_OFF_AUTO_OFF) && + (migration->device_state == VFIO_DEVICE_STATE_RUNNING || + migration->device_state == VFIO_DEVICE_STATE_RUNNING_P2P)) { + return false; + } } } return true; @@ -385,7 +393,14 @@ static bool vfio_devices_all_running_and_mig_active(VFIOContainer *container) return false; } - if (migration->device_state_v1 & VFIO_DEVICE_STATE_V1_RUNNING) { + if (!migration->v2 && + migration->device_state_v1 & VFIO_DEVICE_STATE_V1_RUNNING) { + continue; + } + + if (migration->v2 && + (migration->device_state == VFIO_DEVICE_STATE_RUNNING || + migration->device_state == VFIO_DEVICE_STATE_RUNNING_P2P)) { continue; } else { return false; diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index e40aa0ad80..de68eadb09 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -44,8 +44,83 @@ #define VFIO_MIG_FLAG_DEV_SETUP_STATE (0xffffffffef100003ULL) #define VFIO_MIG_FLAG_DEV_DATA_STATE (0xffffffffef100004ULL) +#define VFIO_MIG_DATA_BUFFER_SIZE (1024 * 1024) + static int64_t bytes_transferred; +static const char *mig_state_to_str(enum vfio_device_mig_state state) +{ + switch (state) { + case VFIO_DEVICE_STATE_ERROR: + return "ERROR"; + case VFIO_DEVICE_STATE_STOP: + return "STOP"; + case VFIO_DEVICE_STATE_RUNNING: + return "RUNNING"; + case VFIO_DEVICE_STATE_STOP_COPY: + return "STOP_COPY"; + case VFIO_DEVICE_STATE_RESUMING: + return "RESUMING"; + case VFIO_DEVICE_STATE_RUNNING_P2P: + return "RUNNING_P2P"; + default: + return "UNKNOWN STATE"; + } +} + +static int vfio_migration_set_state(VFIODevice *vbasedev, + enum vfio_device_mig_state new_state, + enum vfio_device_mig_state recover_state) +{ + VFIOMigration *migration = vbasedev->migration; + uint64_t buf[DIV_ROUND_UP(sizeof(struct vfio_device_feature) + + sizeof(struct vfio_device_feature_mig_state), + sizeof(uint64_t))] = {}; + struct vfio_device_feature *feature = (void *)buf; + struct vfio_device_feature_mig_state *mig_state = (void *)feature->data; + int ret; + + feature->argsz = sizeof(buf); + feature->flags = + VFIO_DEVICE_FEATURE_SET | VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE; + mig_state->device_state = new_state; + ret = ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature); + if (ret) { + /* Try to put the device in some good state */ + mig_state->device_state = recover_state; + if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) { + hw_error("%s: Device in error state, can't recover", + vbasedev->name); + } + + error_report("%s: Failed changing device state to %s", vbasedev->name, + mig_state_to_str(new_state)); + migration->device_state = recover_state; + + return -1; + } + + if (mig_state->data_fd != -1) { + if (migration->data_fd != -1) { + /* + * This can happen if the device is asynchronously reset and + * terminates a data transfer. + */ + error_report("%s: data_fd out of sync", vbasedev->name); + close(mig_state->data_fd); + + return -1; + } + + migration->data_fd = mig_state->data_fd; + } + migration->device_state = new_state; + + trace_vfio_migration_set_state(vbasedev->name, new_state); + + return 0; +} + static inline int vfio_mig_access(VFIODevice *vbasedev, void *val, int count, off_t off, bool iswrite) { @@ -260,6 +335,22 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size) return ret; } +static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, + uint64_t data_size) +{ + VFIOMigration *migration = vbasedev->migration; + int ret; + + ret = qemu_file_get_to_fd(f, migration->data_fd, data_size); + if (ret) { + return ret; + } + + trace_vfio_load_state_device_data(vbasedev->name, data_size); + + return 0; +} + static int vfio_v1_load_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t data_size) { @@ -394,6 +485,14 @@ static int vfio_load_device_config_state(QEMUFile *f, void *opaque) return qemu_file_get_error(f); } +static void vfio_migration_cleanup(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + + close(migration->data_fd); + migration->data_fd = -1; +} + static void vfio_migration_v1_cleanup(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; @@ -405,6 +504,18 @@ static void vfio_migration_v1_cleanup(VFIODevice *vbasedev) /* ---------------------------------------------------------------------- */ +static int vfio_save_setup(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + + trace_vfio_save_setup(vbasedev->name); + + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_SETUP_STATE); + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + return qemu_file_get_error(f); +} + static int vfio_v1_save_setup(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque; @@ -448,6 +559,14 @@ static int vfio_v1_save_setup(QEMUFile *f, void *opaque) return 0; } +static void vfio_save_cleanup(void *opaque) +{ + VFIODevice *vbasedev = opaque; + + vfio_migration_cleanup(vbasedev); + trace_vfio_save_cleanup(vbasedev->name); +} + static void vfio_v1_save_cleanup(void *opaque) { VFIODevice *vbasedev = opaque; @@ -524,6 +643,69 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque) return 0; } +/* Returns 1 if end-of-stream is reached, 0 if more data and -1 if error */ +static int vfio_save_block(QEMUFile *f, VFIOMigration *migration) +{ + ssize_t data_size; + + data_size = read(migration->data_fd, migration->data_buffer, + migration->data_buffer_size); + if (data_size < 0) { + return -1; + } + if (data_size == 0) { + return 1; + } + + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); + qemu_put_be64(f, data_size); + qemu_put_buffer_async(f, migration->data_buffer, data_size, false); + qemu_fflush(f); + bytes_transferred += data_size; + + trace_vfio_save_block(migration->vbasedev->name, data_size); + + return qemu_file_get_error(f); +} + +static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + enum vfio_device_mig_state recover_state; + int ret; + + /* We reach here with device state STOP or STOP_COPY only */ + recover_state = VFIO_DEVICE_STATE_STOP; + ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP_COPY, + recover_state); + if (ret) { + return ret; + } + + do { + ret = vfio_save_block(f, vbasedev->migration); + if (ret < 0) { + return ret; + } + } while (!ret); + + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + ret = qemu_file_get_error(f); + if (ret) { + return ret; + } + + ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP, + recover_state); + if (ret) { + return ret; + } + + trace_vfio_save_complete_precopy(vbasedev->name); + + return 0; +} + static int vfio_v1_save_complete_precopy(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque; @@ -593,6 +775,14 @@ static void vfio_save_state(QEMUFile *f, void *opaque) } } +static int vfio_load_setup(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + + return vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_RESUMING, + vbasedev->migration->device_state); +} + static int vfio_v1_load_setup(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque; @@ -620,6 +810,15 @@ static int vfio_v1_load_setup(QEMUFile *f, void *opaque) return ret; } +static int vfio_load_cleanup(void *opaque) +{ + VFIODevice *vbasedev = opaque; + + vfio_migration_cleanup(vbasedev); + trace_vfio_load_cleanup(vbasedev->name); + return 0; +} + static int vfio_v1_load_cleanup(void *opaque) { VFIODevice *vbasedev = opaque; @@ -662,7 +861,11 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) uint64_t data_size = qemu_get_be64(f); if (data_size) { - ret = vfio_v1_load_buffer(f, vbasedev, data_size); + if (vbasedev->migration->v2) { + ret = vfio_load_buffer(f, vbasedev, data_size); + } else { + ret = vfio_v1_load_buffer(f, vbasedev, data_size); + } if (ret < 0) { return ret; } @@ -683,6 +886,16 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) return ret; } +static SaveVMHandlers savevm_vfio_handlers = { + .save_setup = vfio_save_setup, + .save_cleanup = vfio_save_cleanup, + .save_live_complete_precopy = vfio_save_complete_precopy, + .save_state = vfio_save_state, + .load_setup = vfio_load_setup, + .load_cleanup = vfio_load_cleanup, + .load_state = vfio_load_state, +}; + static SaveVMHandlers savevm_vfio_v1_handlers = { .save_setup = vfio_v1_save_setup, .save_cleanup = vfio_v1_save_cleanup, @@ -697,6 +910,34 @@ static SaveVMHandlers savevm_vfio_v1_handlers = { /* ---------------------------------------------------------------------- */ +static void vfio_vmstate_change(void *opaque, bool running, RunState state) +{ + VFIODevice *vbasedev = opaque; + enum vfio_device_mig_state new_state; + int ret; + + if (running) { + new_state = VFIO_DEVICE_STATE_RUNNING; + } else { + new_state = VFIO_DEVICE_STATE_STOP; + } + + ret = vfio_migration_set_state(vbasedev, new_state, + VFIO_DEVICE_STATE_ERROR); + if (ret) { + /* + * Migration should be aborted in this case, but vm_state_notify() + * currently does not support reporting failures. + */ + if (migrate_get_current()->to_dst_file) { + qemu_file_set_error(migrate_get_current()->to_dst_file, ret); + } + } + + trace_vfio_vmstate_change(vbasedev->name, running, RunState_str(state), + new_state); +} + static void vfio_v1_vmstate_change(void *opaque, bool running, RunState state) { VFIODevice *vbasedev = opaque; @@ -770,12 +1011,17 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data) case MIGRATION_STATUS_CANCELLED: case MIGRATION_STATUS_FAILED: bytes_transferred = 0; - ret = vfio_migration_v1_set_state(vbasedev, - ~(VFIO_DEVICE_STATE_V1_SAVING | - VFIO_DEVICE_STATE_V1_RESUMING), - VFIO_DEVICE_STATE_V1_RUNNING); - if (ret) { - error_report("%s: Failed to set state RUNNING", vbasedev->name); + if (migration->v2) { + vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_RUNNING, + VFIO_DEVICE_STATE_ERROR); + } else { + ret = vfio_migration_v1_set_state(vbasedev, + ~(VFIO_DEVICE_STATE_V1_SAVING | + VFIO_DEVICE_STATE_V1_RESUMING), + VFIO_DEVICE_STATE_V1_RUNNING); + if (ret) { + error_report("%s: Failed to set state RUNNING", vbasedev->name); + } } } } @@ -784,12 +1030,35 @@ static void vfio_migration_exit(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; - vfio_region_exit(&migration->region); - vfio_region_finalize(&migration->region); + if (migration->v2) { + g_free(migration->data_buffer); + } else { + vfio_region_exit(&migration->region); + vfio_region_finalize(&migration->region); + } g_free(vbasedev->migration); vbasedev->migration = NULL; } +static int vfio_migration_query_flags(VFIODevice *vbasedev, uint64_t *mig_flags) +{ + uint64_t buf[DIV_ROUND_UP(sizeof(struct vfio_device_feature) + + sizeof(struct vfio_device_feature_migration), + sizeof(uint64_t))] = {}; + struct vfio_device_feature *feature = (void *)buf; + struct vfio_device_feature_migration *mig = (void *)feature->data; + + feature->argsz = sizeof(buf); + feature->flags = VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_MIGRATION; + if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) { + return -EOPNOTSUPP; + } + + *mig_flags = mig->flags; + + return 0; +} + static int vfio_migration_init(VFIODevice *vbasedev) { int ret; @@ -798,6 +1067,7 @@ static int vfio_migration_init(VFIODevice *vbasedev) char id[256] = ""; g_autofree char *path = NULL, *oid = NULL; struct vfio_region_info *info = NULL; + uint64_t mig_flags; if (!vbasedev->ops->vfio_get_object) { return -EINVAL; @@ -808,32 +1078,48 @@ static int vfio_migration_init(VFIODevice *vbasedev) return -EINVAL; } - ret = vfio_get_dev_region_info(vbasedev, - VFIO_REGION_TYPE_MIGRATION_DEPRECATED, - VFIO_REGION_SUBTYPE_MIGRATION_DEPRECATED, - &info); - if (ret) { - return ret; - } + ret = vfio_migration_query_flags(vbasedev, &mig_flags); + if (!ret) { + /* Migration v2 */ + /* Basic migration functionality must be supported */ + if (!(mig_flags & VFIO_MIGRATION_STOP_COPY)) { + return -EOPNOTSUPP; + } + vbasedev->migration = g_new0(VFIOMigration, 1); + vbasedev->migration->data_buffer_size = VFIO_MIG_DATA_BUFFER_SIZE; + vbasedev->migration->data_buffer = + g_malloc0(vbasedev->migration->data_buffer_size); + vbasedev->migration->data_fd = -1; + vbasedev->migration->v2 = true; + } else { + /* Migration v1 */ + ret = vfio_get_dev_region_info(vbasedev, + VFIO_REGION_TYPE_MIGRATION_DEPRECATED, + VFIO_REGION_SUBTYPE_MIGRATION_DEPRECATED, + &info); + if (ret) { + return ret; + } - vbasedev->migration = g_new0(VFIOMigration, 1); + vbasedev->migration = g_new0(VFIOMigration, 1); - ret = vfio_region_setup(obj, vbasedev, &vbasedev->migration->region, - info->index, "migration"); - if (ret) { - error_report("%s: Failed to setup VFIO migration region %d: %s", - vbasedev->name, info->index, strerror(-ret)); - goto err; - } + ret = vfio_region_setup(obj, vbasedev, &vbasedev->migration->region, + info->index, "migration"); + if (ret) { + error_report("%s: Failed to setup VFIO migration region %d: %s", + vbasedev->name, info->index, strerror(-ret)); + goto err; + } - if (!vbasedev->migration->region.size) { - error_report("%s: Invalid zero-sized VFIO migration region %d", - vbasedev->name, info->index); - ret = -EINVAL; - goto err; - } + if (!vbasedev->migration->region.size) { + error_report("%s: Invalid zero-sized VFIO migration region %d", + vbasedev->name, info->index); + ret = -EINVAL; + goto err; + } - g_free(info); + g_free(info); + } migration = vbasedev->migration; migration->vbasedev = vbasedev; @@ -846,11 +1132,20 @@ static int vfio_migration_init(VFIODevice *vbasedev) } strpadcpy(id, sizeof(id), path, '\0'); - register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, - &savevm_vfio_v1_handlers, vbasedev); + if (migration->v2) { + register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, + &savevm_vfio_handlers, vbasedev); + + migration->vm_state = qdev_add_vm_change_state_handler( + vbasedev->dev, vfio_vmstate_change, vbasedev); + } else { + register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, + &savevm_vfio_v1_handlers, vbasedev); + + migration->vm_state = qdev_add_vm_change_state_handler( + vbasedev->dev, vfio_v1_vmstate_change, vbasedev); + } - migration->vm_state = qdev_add_vm_change_state_handler( - vbasedev->dev, vfio_v1_vmstate_change, vbasedev); migration->migration_state.notify = vfio_migration_state_notifier; add_migration_state_change_notifier(&migration->migration_state); return 0; diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index ac8b04f52a..6e8c5958b9 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -163,6 +163,8 @@ vfio_save_complete_precopy(const char *name) " (%s)" vfio_load_device_config_state(const char *name) " (%s)" vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64 vfio_v1_load_state_device_data(const char *name, uint64_t data_offset, uint64_t data_size) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64 +vfio_load_state_device_data(const char *name, uint64_t data_size) " (%s) size 0x%"PRIx64 vfio_load_cleanup(const char *name) " (%s)" vfio_get_dirty_bitmap(int fd, uint64_t iova, uint64_t size, uint64_t bitmap_size, uint64_t start) "container fd=%d, iova=0x%"PRIx64" size= 0x%"PRIx64" bitmap_size=0x%"PRIx64" start=0x%"PRIx64 vfio_iommu_map_dirty_notify(uint64_t iova_start, uint64_t iova_end) "iommu dirty @ 0x%"PRIx64" - 0x%"PRIx64 +vfio_save_block(const char *name, int data_size) " (%s) data_size %d" diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index bbaf72ba00..2ec3346fea 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -66,6 +66,11 @@ typedef struct VFIOMigration { int vm_running; Notifier migration_state; uint64_t pending_bytes; + enum vfio_device_mig_state device_state; + int data_fd; + void *data_buffer; + size_t data_buffer_size; + bool v2; } VFIOMigration; typedef struct VFIOAddressSpace { From patchwork Mon May 30 17:07:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 12864840 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B9AE4C433EF for ; Mon, 30 May 2022 17:55:51 +0000 (UTC) Received: from localhost ([::1]:49404 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nvjcM-00070r-M0 for qemu-devel@archiver.kernel.org; Mon, 30 May 2022 13:55:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54334) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvisN-0004b3-JG for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:19 -0400 Received: from mail-dm6nam10on20604.outbound.protection.outlook.com ([2a01:111:f400:7e88::604]:32352 helo=NAM10-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvisK-0008N6-6o for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:19 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=T6jR/dtyQEL2NHiIdyMgk9IFWtEbAh/YsYngZuS8epNKYbYkfUuxElt7nhciRabcJCTOknLsae8RHidQhhfetntEpQmI1Yy/hnel+PMtLLLrqk0to+5SI8KmRn266pAJI3QUvQU/qpajVmZj0CgW/J3yB/mfXBEJYZbWIJAUyT27n4BnsXX2owPA/ZXAGwFZvrnS1ux6yUTPPXLIGUXX/XkkThi/obiYNguumIA2AccHLI6kyP1BUKcZaZoZ+0WPkdocO94hBhwrp70y/ky7VpD0nU12lCSEoeYOWh3qppK5VsOssAtZ7/k+0DtsH1VAwjcqp73p1d4GOrlaAqrHOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZJuF1217+IQ4i7VgbZUxRxtNbyHUTT+Gl6kqze0jgY8=; b=UYy2z+mLZRADTk/yQh7H2yNOQNRAR9LUbTGvx0jvo0JrOVLkJ3ubAXmo72v5+0CrzF3SEqGPsPYfl8UBIaeV0fHyUmK0HDQTu8LyCN3oDf8fbmokbCG7zYLUolgWYW/ipvgYsLV5Fekr82AM7ftu4E2Dxgv7LI0TRCvOjOjEZ7XZ8HJXV8u0+JWWwkifnwICdLkQl5+iBNHH8MzKjw++FX8LbHm9NaTUBsuWrghN8GPw0rGpS7/CsiRnuiJiTz0HVYmCeyFiZXz988LhGr41WN5a590vWbRc3yVxjFXIbQqVLf8YYiZSf1OKsBUBoTVmykZeQA7PCCDlN6gnMezKFg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.238) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZJuF1217+IQ4i7VgbZUxRxtNbyHUTT+Gl6kqze0jgY8=; b=FpS1cZcHpYGkyRRJlpG9ww3yU9whsW73IJtyvZrErmkxVn9TWxX9kMGaQ48rCU7X3VU4sSyQQhdFrZ3O6fEWRdISMpe6aCwXdAADKBjemhrsM/gbt6SBf7yaUqsz9yY71cow3KCxXKTBRZAp6IopaFoDI2CFFAAmBt7v+FAIibHBf+MqODl6/lXBS0rLwRzS1kmDSZOeT9Ho7GdKAC1RSVdLwQI2eg1h35OZU4Li30i/qCmqOqLJZ+P3WW69M0eJXWbdLY32EcU6Cp9qIwRqSao77kflgBC4WqNnlPNoAkvsdEQZLaIOoVwF6bOmX1pIn610JTjrCHYIzgZb/Cy7zw== Received: from CO1PR15CA0101.namprd15.prod.outlook.com (2603:10b6:101:21::21) by DM6PR12MB4337.namprd12.prod.outlook.com (2603:10b6:5:2a9::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Mon, 30 May 2022 17:08:12 +0000 Received: from CO1NAM11FT035.eop-nam11.prod.protection.outlook.com (2603:10b6:101:21:cafe::bc) by CO1PR15CA0101.outlook.office365.com (2603:10b6:101:21::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.19 via Frontend Transport; Mon, 30 May 2022 17:08:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.238) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.238 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.238; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.238) by CO1NAM11FT035.mail.protection.outlook.com (10.13.175.36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:08:12 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by DRHQMAIL105.nvidia.com (10.27.9.14) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 30 May 2022 17:08:11 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 30 May 2022 10:08:11 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.22 via Frontend Transport; Mon, 30 May 2022 10:08:08 -0700 From: Avihai Horon To: , Cornelia Huck , "Alex Williamson" , Juan Quintela , "Dr . David Alan Gilbert" CC: Joao Martins , Yishai Hadas , Jason Gunthorpe , Mark Bloch , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH v2 08/11] vfio/migration: Remove VFIO migration protocol v1 Date: Mon, 30 May 2022 20:07:36 +0300 Message-ID: <20220530170739.19072-9-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20220530170739.19072-1-avihaih@nvidia.com> References: <20220530170739.19072-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ae243338-c926-4820-ebd3-08da425efa97 X-MS-TrafficTypeDiagnostic: DM6PR12MB4337:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EOB423/lKCJqqNH6HeB+NBrJZAraqJ86FbpSxbERmPOh8JXFgDwhaxw4qRp0PiFCJlubHZlA+0iOq8zIIt8nNC6atVFEf4yscgxMTig6DSamYjGuk0CGjfqjhr0dNgrqjEvG0Sz78MTis3jCSEEvu+Mj4j4bvSCguPKvqWXHIKYvRivwxT0ZxM0im1KyeJSW3BEi5AZUHXlrhG9GzalaruujzFyp1FXegB6xf4K/LJXrZjWQZD0xWSgJMlOdVK+9iqazq6q0vFhj/l0AdYC+Co2p3OQj5B5JWfgHztd6fAC7yE7ICSFqnpAsfZjNa40I/fEp1xJbzrWwMqq2YfWGnNh/x0JOaKUrdbB/NP5lZsmsP8fI5feKPgDSeOT2KtfJZY02uGiiQmq9ubj41s/9hTC4xAETfgDgeWfskrq5d995y4JUek99xMkmzF1OaRvBHaq5bPGk290QLpptpnS5+1dPra1JUeX4DE1SyLI7nGsmBrgHJKLEL5+A79Y/fx89x65YeMvPi4xTph5rRGIZWu+NG6yr4yEJgpHZllsKIhDmFr24I+woxRkUB3qG/mwiB1MQqhwep37r5DydOC5bkK5zvhuX09ECjW1WQ3yAkJHF2KCk/nGuClcMNFmuK+ypXvaLVoCkipu4YUgTV+uT3glqf6DKsAVgiuNBL02hrlP7aKgNcWZ7yrj2UfEyLQhWLzBgSiz1XyIvUcYBEfl5ww== X-Forefront-Antispam-Report: CIP:12.22.5.238; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(46966006)(40470700004)(36840700001)(1076003)(356005)(47076005)(81166007)(7696005)(26005)(316002)(6666004)(70206006)(70586007)(54906003)(8676002)(30864003)(5660300002)(8936002)(40460700003)(82310400005)(336012)(186003)(426003)(36756003)(110136005)(508600001)(36860700001)(86362001)(107886003)(2616005)(2906002)(83380400001)(4326008)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2022 17:08:12.3559 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ae243338-c926-4820-ebd3-08da425efa97 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.238]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT035.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4337 Received-SPF: softfail client-ip=2a01:111:f400:7e88::604; envelope-from=avihaih@nvidia.com; helo=NAM10-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Now that v2 protocol implementation has been added, remove the deprecated v1 implementation. Signed-off-by: Avihai Horon --- hw/vfio/common.c | 19 +- hw/vfio/migration.c | 698 +--------------------------------- hw/vfio/trace-events | 5 - include/hw/vfio/vfio-common.h | 5 - 4 files changed, 24 insertions(+), 703 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 5541133ec9..00c6cb0ffe 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -355,14 +355,7 @@ static bool vfio_devices_all_dirty_tracking(VFIOContainer *container) return false; } - if (!migration->v2 && - (vbasedev->pre_copy_dirty_page_tracking == ON_OFF_AUTO_OFF) && - (migration->device_state_v1 & VFIO_DEVICE_STATE_V1_RUNNING)) { - return false; - } - - if (migration->v2 && - (vbasedev->pre_copy_dirty_page_tracking == ON_OFF_AUTO_OFF) && + if ((vbasedev->pre_copy_dirty_page_tracking == ON_OFF_AUTO_OFF) && (migration->device_state == VFIO_DEVICE_STATE_RUNNING || migration->device_state == VFIO_DEVICE_STATE_RUNNING_P2P)) { return false; @@ -393,14 +386,8 @@ static bool vfio_devices_all_running_and_mig_active(VFIOContainer *container) return false; } - if (!migration->v2 && - migration->device_state_v1 & VFIO_DEVICE_STATE_V1_RUNNING) { - continue; - } - - if (migration->v2 && - (migration->device_state == VFIO_DEVICE_STATE_RUNNING || - migration->device_state == VFIO_DEVICE_STATE_RUNNING_P2P)) { + if (migration->device_state == VFIO_DEVICE_STATE_RUNNING || + migration->device_state == VFIO_DEVICE_STATE_RUNNING_P2P) { continue; } else { return false; diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index de68eadb09..852759e6ca 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -121,220 +121,6 @@ static int vfio_migration_set_state(VFIODevice *vbasedev, return 0; } -static inline int vfio_mig_access(VFIODevice *vbasedev, void *val, int count, - off_t off, bool iswrite) -{ - int ret; - - ret = iswrite ? pwrite(vbasedev->fd, val, count, off) : - pread(vbasedev->fd, val, count, off); - if (ret < count) { - error_report("vfio_mig_%s %d byte %s: failed at offset 0x%" - HWADDR_PRIx", err: %s", iswrite ? "write" : "read", count, - vbasedev->name, off, strerror(errno)); - return (ret < 0) ? ret : -EINVAL; - } - return 0; -} - -static int vfio_mig_rw(VFIODevice *vbasedev, __u8 *buf, size_t count, - off_t off, bool iswrite) -{ - int ret, done = 0; - __u8 *tbuf = buf; - - while (count) { - int bytes = 0; - - if (count >= 8 && !(off % 8)) { - bytes = 8; - } else if (count >= 4 && !(off % 4)) { - bytes = 4; - } else if (count >= 2 && !(off % 2)) { - bytes = 2; - } else { - bytes = 1; - } - - ret = vfio_mig_access(vbasedev, tbuf, bytes, off, iswrite); - if (ret) { - return ret; - } - - count -= bytes; - done += bytes; - off += bytes; - tbuf += bytes; - } - return done; -} - -#define vfio_mig_read(f, v, c, o) vfio_mig_rw(f, (__u8 *)v, c, o, false) -#define vfio_mig_write(f, v, c, o) vfio_mig_rw(f, (__u8 *)v, c, o, true) - -#define VFIO_MIG_STRUCT_OFFSET(f) \ - offsetof(struct vfio_device_migration_info, f) -/* - * Change the device_state register for device @vbasedev. Bits set in @mask - * are preserved, bits set in @value are set, and bits not set in either @mask - * or @value are cleared in device_state. If the register cannot be accessed, - * the resulting state would be invalid, or the device enters an error state, - * an error is returned. - */ - -static int vfio_migration_v1_set_state(VFIODevice *vbasedev, uint32_t mask, - uint32_t value) -{ - VFIOMigration *migration = vbasedev->migration; - VFIORegion *region = &migration->region; - off_t dev_state_off = region->fd_offset + - VFIO_MIG_STRUCT_OFFSET(device_state); - uint32_t device_state; - int ret; - - ret = vfio_mig_read(vbasedev, &device_state, sizeof(device_state), - dev_state_off); - if (ret < 0) { - return ret; - } - - device_state = (device_state & mask) | value; - - if (!VFIO_DEVICE_STATE_VALID(device_state)) { - return -EINVAL; - } - - ret = vfio_mig_write(vbasedev, &device_state, sizeof(device_state), - dev_state_off); - if (ret < 0) { - int rret; - - rret = vfio_mig_read(vbasedev, &device_state, sizeof(device_state), - dev_state_off); - - if ((rret < 0) || (VFIO_DEVICE_STATE_IS_ERROR(device_state))) { - hw_error("%s: Device in error state 0x%x", vbasedev->name, - device_state); - return rret ? rret : -EIO; - } - return ret; - } - - migration->device_state_v1 = device_state; - trace_vfio_migration_set_state(vbasedev->name, device_state); - return 0; -} - -static void *get_data_section_size(VFIORegion *region, uint64_t data_offset, - uint64_t data_size, uint64_t *size) -{ - void *ptr = NULL; - uint64_t limit = 0; - int i; - - if (!region->mmaps) { - if (size) { - *size = MIN(data_size, region->size - data_offset); - } - return ptr; - } - - for (i = 0; i < region->nr_mmaps; i++) { - VFIOMmap *map = region->mmaps + i; - - if ((data_offset >= map->offset) && - (data_offset < map->offset + map->size)) { - - /* check if data_offset is within sparse mmap areas */ - ptr = map->mmap + data_offset - map->offset; - if (size) { - *size = MIN(data_size, map->offset + map->size - data_offset); - } - break; - } else if ((data_offset < map->offset) && - (!limit || limit > map->offset)) { - /* - * data_offset is not within sparse mmap areas, find size of - * non-mapped area. Check through all list since region->mmaps list - * is not sorted. - */ - limit = map->offset; - } - } - - if (!ptr && size) { - *size = limit ? MIN(data_size, limit - data_offset) : data_size; - } - return ptr; -} - -static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size) -{ - VFIOMigration *migration = vbasedev->migration; - VFIORegion *region = &migration->region; - uint64_t data_offset = 0, data_size = 0, sz; - int ret; - - ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset)); - if (ret < 0) { - return ret; - } - - ret = vfio_mig_read(vbasedev, &data_size, sizeof(data_size), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_size)); - if (ret < 0) { - return ret; - } - - trace_vfio_save_buffer(vbasedev->name, data_offset, data_size, - migration->pending_bytes); - - qemu_put_be64(f, data_size); - sz = data_size; - - while (sz) { - void *buf; - uint64_t sec_size; - bool buf_allocated = false; - - buf = get_data_section_size(region, data_offset, sz, &sec_size); - - if (!buf) { - buf = g_try_malloc(sec_size); - if (!buf) { - error_report("%s: Error allocating buffer ", __func__); - return -ENOMEM; - } - buf_allocated = true; - - ret = vfio_mig_read(vbasedev, buf, sec_size, - region->fd_offset + data_offset); - if (ret < 0) { - g_free(buf); - return ret; - } - } - - qemu_put_buffer(f, buf, sec_size); - - if (buf_allocated) { - g_free(buf); - } - sz -= sec_size; - data_offset += sec_size; - } - - ret = qemu_file_get_error(f); - - if (!ret && size) { - *size = data_size; - } - - bytes_transferred += data_size; - return ret; -} - static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t data_size) { @@ -351,96 +137,6 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, return 0; } -static int vfio_v1_load_buffer(QEMUFile *f, VFIODevice *vbasedev, - uint64_t data_size) -{ - VFIORegion *region = &vbasedev->migration->region; - uint64_t data_offset = 0, size, report_size; - int ret; - - do { - ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset)); - if (ret < 0) { - return ret; - } - - if (data_offset + data_size > region->size) { - /* - * If data_size is greater than the data section of migration region - * then iterate the write buffer operation. This case can occur if - * size of migration region at destination is smaller than size of - * migration region at source. - */ - report_size = size = region->size - data_offset; - data_size -= size; - } else { - report_size = size = data_size; - data_size = 0; - } - - trace_vfio_v1_load_state_device_data(vbasedev->name, data_offset, size); - - while (size) { - void *buf; - uint64_t sec_size; - bool buf_alloc = false; - - buf = get_data_section_size(region, data_offset, size, &sec_size); - - if (!buf) { - buf = g_try_malloc(sec_size); - if (!buf) { - error_report("%s: Error allocating buffer ", __func__); - return -ENOMEM; - } - buf_alloc = true; - } - - qemu_get_buffer(f, buf, sec_size); - - if (buf_alloc) { - ret = vfio_mig_write(vbasedev, buf, sec_size, - region->fd_offset + data_offset); - g_free(buf); - - if (ret < 0) { - return ret; - } - } - size -= sec_size; - data_offset += sec_size; - } - - ret = vfio_mig_write(vbasedev, &report_size, sizeof(report_size), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_size)); - if (ret < 0) { - return ret; - } - } while (data_size); - - return 0; -} - -static int vfio_update_pending(VFIODevice *vbasedev) -{ - VFIOMigration *migration = vbasedev->migration; - VFIORegion *region = &migration->region; - uint64_t pending_bytes = 0; - int ret; - - ret = vfio_mig_read(vbasedev, &pending_bytes, sizeof(pending_bytes), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(pending_bytes)); - if (ret < 0) { - migration->pending_bytes = 0; - return ret; - } - - migration->pending_bytes = pending_bytes; - trace_vfio_update_pending(vbasedev->name, pending_bytes); - return 0; -} - static int vfio_save_device_config_state(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque; @@ -493,15 +189,6 @@ static void vfio_migration_cleanup(VFIODevice *vbasedev) migration->data_fd = -1; } -static void vfio_migration_v1_cleanup(VFIODevice *vbasedev) -{ - VFIOMigration *migration = vbasedev->migration; - - if (migration->region.mmaps) { - vfio_region_unmap(&migration->region); - } -} - /* ---------------------------------------------------------------------- */ static int vfio_save_setup(QEMUFile *f, void *opaque) @@ -516,49 +203,6 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) return qemu_file_get_error(f); } -static int vfio_v1_save_setup(QEMUFile *f, void *opaque) -{ - VFIODevice *vbasedev = opaque; - VFIOMigration *migration = vbasedev->migration; - int ret; - - trace_vfio_save_setup(vbasedev->name); - - qemu_put_be64(f, VFIO_MIG_FLAG_DEV_SETUP_STATE); - - if (migration->region.mmaps) { - /* - * Calling vfio_region_mmap() from migration thread. Memory API called - * from this function require locking the iothread when called from - * outside the main loop thread. - */ - qemu_mutex_lock_iothread(); - ret = vfio_region_mmap(&migration->region); - qemu_mutex_unlock_iothread(); - if (ret) { - error_report("%s: Failed to mmap VFIO migration region: %s", - vbasedev->name, strerror(-ret)); - error_report("%s: Falling back to slow path", vbasedev->name); - } - } - - ret = vfio_migration_v1_set_state(vbasedev, VFIO_DEVICE_STATE_MASK, - VFIO_DEVICE_STATE_V1_SAVING); - if (ret) { - error_report("%s: Failed to set state SAVING", vbasedev->name); - return ret; - } - - qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); - - ret = qemu_file_get_error(f); - if (ret) { - return ret; - } - - return 0; -} - static void vfio_save_cleanup(void *opaque) { VFIODevice *vbasedev = opaque; @@ -567,82 +211,6 @@ static void vfio_save_cleanup(void *opaque) trace_vfio_save_cleanup(vbasedev->name); } -static void vfio_v1_save_cleanup(void *opaque) -{ - VFIODevice *vbasedev = opaque; - - vfio_migration_v1_cleanup(vbasedev); - trace_vfio_save_cleanup(vbasedev->name); -} - -static void vfio_save_pending(QEMUFile *f, void *opaque, - uint64_t threshold_size, - uint64_t *res_precopy_only, - uint64_t *res_compatible, - uint64_t *res_postcopy_only) -{ - VFIODevice *vbasedev = opaque; - VFIOMigration *migration = vbasedev->migration; - int ret; - - ret = vfio_update_pending(vbasedev); - if (ret) { - return; - } - - *res_precopy_only += migration->pending_bytes; - - trace_vfio_save_pending(vbasedev->name, *res_precopy_only, - *res_postcopy_only, *res_compatible); -} - -static int vfio_save_iterate(QEMUFile *f, void *opaque) -{ - VFIODevice *vbasedev = opaque; - VFIOMigration *migration = vbasedev->migration; - uint64_t data_size; - int ret; - - qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); - - if (migration->pending_bytes == 0) { - ret = vfio_update_pending(vbasedev); - if (ret) { - return ret; - } - - if (migration->pending_bytes == 0) { - qemu_put_be64(f, 0); - qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); - /* indicates data finished, goto complete phase */ - return 1; - } - } - - ret = vfio_save_buffer(f, vbasedev, &data_size); - if (ret) { - error_report("%s: vfio_save_buffer failed %s", vbasedev->name, - strerror(errno)); - return ret; - } - - qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); - - ret = qemu_file_get_error(f); - if (ret) { - return ret; - } - - /* - * Reset pending_bytes as .save_live_pending is not called during savevm or - * snapshot case, in such case vfio_update_pending() at the start of this - * function updates pending_bytes. - */ - migration->pending_bytes = 0; - trace_vfio_save_iterate(vbasedev->name, data_size); - return 0; -} - /* Returns 1 if end-of-stream is reached, 0 if more data and -1 if error */ static int vfio_save_block(QEMUFile *f, VFIOMigration *migration) { @@ -706,62 +274,6 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) return 0; } -static int vfio_v1_save_complete_precopy(QEMUFile *f, void *opaque) -{ - VFIODevice *vbasedev = opaque; - VFIOMigration *migration = vbasedev->migration; - uint64_t data_size; - int ret; - - ret = vfio_migration_v1_set_state(vbasedev, ~VFIO_DEVICE_STATE_V1_RUNNING, - VFIO_DEVICE_STATE_V1_SAVING); - if (ret) { - error_report("%s: Failed to set state STOP and SAVING", - vbasedev->name); - return ret; - } - - ret = vfio_update_pending(vbasedev); - if (ret) { - return ret; - } - - while (migration->pending_bytes > 0) { - qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); - ret = vfio_save_buffer(f, vbasedev, &data_size); - if (ret < 0) { - error_report("%s: Failed to save buffer", vbasedev->name); - return ret; - } - - if (data_size == 0) { - break; - } - - ret = vfio_update_pending(vbasedev); - if (ret) { - return ret; - } - } - - qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); - - ret = qemu_file_get_error(f); - if (ret) { - return ret; - } - - ret = vfio_migration_v1_set_state(vbasedev, ~VFIO_DEVICE_STATE_V1_SAVING, - 0); - if (ret) { - error_report("%s: Failed to set state STOPPED", vbasedev->name); - return ret; - } - - trace_vfio_save_complete_precopy(vbasedev->name); - return ret; -} - static void vfio_save_state(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque; @@ -783,33 +295,6 @@ static int vfio_load_setup(QEMUFile *f, void *opaque) vbasedev->migration->device_state); } -static int vfio_v1_load_setup(QEMUFile *f, void *opaque) -{ - VFIODevice *vbasedev = opaque; - VFIOMigration *migration = vbasedev->migration; - int ret = 0; - - if (migration->region.mmaps) { - ret = vfio_region_mmap(&migration->region); - if (ret) { - error_report("%s: Failed to mmap VFIO migration region %d: %s", - vbasedev->name, migration->region.nr, - strerror(-ret)); - error_report("%s: Falling back to slow path", vbasedev->name); - } - } - - ret = vfio_migration_v1_set_state(vbasedev, ~VFIO_DEVICE_STATE_MASK, - VFIO_DEVICE_STATE_V1_RESUMING); - if (ret) { - error_report("%s: Failed to set state RESUMING", vbasedev->name); - if (migration->region.mmaps) { - vfio_region_unmap(&migration->region); - } - } - return ret; -} - static int vfio_load_cleanup(void *opaque) { VFIODevice *vbasedev = opaque; @@ -819,15 +304,6 @@ static int vfio_load_cleanup(void *opaque) return 0; } -static int vfio_v1_load_cleanup(void *opaque) -{ - VFIODevice *vbasedev = opaque; - - vfio_migration_v1_cleanup(vbasedev); - trace_vfio_load_cleanup(vbasedev->name); - return 0; -} - static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) { VFIODevice *vbasedev = opaque; @@ -861,11 +337,7 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) uint64_t data_size = qemu_get_be64(f); if (data_size) { - if (vbasedev->migration->v2) { - ret = vfio_load_buffer(f, vbasedev, data_size); - } else { - ret = vfio_v1_load_buffer(f, vbasedev, data_size); - } + ret = vfio_load_buffer(f, vbasedev, data_size); if (ret < 0) { return ret; } @@ -896,18 +368,6 @@ static SaveVMHandlers savevm_vfio_handlers = { .load_state = vfio_load_state, }; -static SaveVMHandlers savevm_vfio_v1_handlers = { - .save_setup = vfio_v1_save_setup, - .save_cleanup = vfio_v1_save_cleanup, - .save_live_pending = vfio_save_pending, - .save_live_iterate = vfio_save_iterate, - .save_live_complete_precopy = vfio_v1_save_complete_precopy, - .save_state = vfio_save_state, - .load_setup = vfio_v1_load_setup, - .load_cleanup = vfio_v1_load_cleanup, - .load_state = vfio_load_state, -}; - /* ---------------------------------------------------------------------- */ static void vfio_vmstate_change(void *opaque, bool running, RunState state) @@ -938,70 +398,12 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) new_state); } -static void vfio_v1_vmstate_change(void *opaque, bool running, RunState state) -{ - VFIODevice *vbasedev = opaque; - VFIOMigration *migration = vbasedev->migration; - uint32_t value, mask; - int ret; - - if (vbasedev->migration->vm_running == running) { - return; - } - - if (running) { - /* - * Here device state can have one of _SAVING, _RESUMING or _STOP bit. - * Transition from _SAVING to _RUNNING can happen if there is migration - * failure, in that case clear _SAVING bit. - * Transition from _RESUMING to _RUNNING occurs during resuming - * phase, in that case clear _RESUMING bit. - * In both the above cases, set _RUNNING bit. - */ - mask = ~VFIO_DEVICE_STATE_MASK; - value = VFIO_DEVICE_STATE_V1_RUNNING; - } else { - /* - * Here device state could be either _RUNNING or _SAVING|_RUNNING. Reset - * _RUNNING bit - */ - mask = ~VFIO_DEVICE_STATE_V1_RUNNING; - - /* - * When VM state transition to stop for savevm command, device should - * start saving data. - */ - if (state == RUN_STATE_SAVE_VM) { - value = VFIO_DEVICE_STATE_V1_SAVING; - } else { - value = 0; - } - } - - ret = vfio_migration_v1_set_state(vbasedev, mask, value); - if (ret) { - /* - * Migration should be aborted in this case, but vm_state_notify() - * currently does not support reporting failures. - */ - error_report("%s: Failed to set device state 0x%x", vbasedev->name, - (migration->device_state_v1 & mask) | value); - if (migrate_get_current()->to_dst_file) { - qemu_file_set_error(migrate_get_current()->to_dst_file, ret); - } - } - vbasedev->migration->vm_running = running; - trace_vfio_vmstate_change(vbasedev->name, running, RunState_str(state), - (migration->device_state_v1 & mask) | value); -} - static void vfio_migration_state_notifier(Notifier *notifier, void *data) { MigrationState *s = data; VFIOMigration *migration = container_of(notifier, VFIOMigration, migration_state); VFIODevice *vbasedev = migration->vbasedev; - int ret; trace_vfio_migration_state_notifier(vbasedev->name, MigrationStatus_str(s->state)); @@ -1011,31 +413,14 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data) case MIGRATION_STATUS_CANCELLED: case MIGRATION_STATUS_FAILED: bytes_transferred = 0; - if (migration->v2) { - vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_RUNNING, - VFIO_DEVICE_STATE_ERROR); - } else { - ret = vfio_migration_v1_set_state(vbasedev, - ~(VFIO_DEVICE_STATE_V1_SAVING | - VFIO_DEVICE_STATE_V1_RESUMING), - VFIO_DEVICE_STATE_V1_RUNNING); - if (ret) { - error_report("%s: Failed to set state RUNNING", vbasedev->name); - } - } + vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_RUNNING, + VFIO_DEVICE_STATE_ERROR); } } static void vfio_migration_exit(VFIODevice *vbasedev) { - VFIOMigration *migration = vbasedev->migration; - - if (migration->v2) { - g_free(migration->data_buffer); - } else { - vfio_region_exit(&migration->region); - vfio_region_finalize(&migration->region); - } + g_free(vbasedev->migration->data_buffer); g_free(vbasedev->migration); vbasedev->migration = NULL; } @@ -1066,7 +451,6 @@ static int vfio_migration_init(VFIODevice *vbasedev) VFIOMigration *migration; char id[256] = ""; g_autofree char *path = NULL, *oid = NULL; - struct vfio_region_info *info = NULL; uint64_t mig_flags; if (!vbasedev->ops->vfio_get_object) { @@ -1079,48 +463,20 @@ static int vfio_migration_init(VFIODevice *vbasedev) } ret = vfio_migration_query_flags(vbasedev, &mig_flags); - if (!ret) { - /* Migration v2 */ - /* Basic migration functionality must be supported */ - if (!(mig_flags & VFIO_MIGRATION_STOP_COPY)) { - return -EOPNOTSUPP; - } - vbasedev->migration = g_new0(VFIOMigration, 1); - vbasedev->migration->data_buffer_size = VFIO_MIG_DATA_BUFFER_SIZE; - vbasedev->migration->data_buffer = - g_malloc0(vbasedev->migration->data_buffer_size); - vbasedev->migration->data_fd = -1; - vbasedev->migration->v2 = true; - } else { - /* Migration v1 */ - ret = vfio_get_dev_region_info(vbasedev, - VFIO_REGION_TYPE_MIGRATION_DEPRECATED, - VFIO_REGION_SUBTYPE_MIGRATION_DEPRECATED, - &info); - if (ret) { - return ret; - } - - vbasedev->migration = g_new0(VFIOMigration, 1); - - ret = vfio_region_setup(obj, vbasedev, &vbasedev->migration->region, - info->index, "migration"); - if (ret) { - error_report("%s: Failed to setup VFIO migration region %d: %s", - vbasedev->name, info->index, strerror(-ret)); - goto err; - } - - if (!vbasedev->migration->region.size) { - error_report("%s: Invalid zero-sized VFIO migration region %d", - vbasedev->name, info->index); - ret = -EINVAL; - goto err; - } + if (ret) { + return ret; + } - g_free(info); + /* Basic migration functionality must be supported */ + if (!(mig_flags & VFIO_MIGRATION_STOP_COPY)) { + return -EOPNOTSUPP; } + vbasedev->migration = g_new0(VFIOMigration, 1); + vbasedev->migration->data_buffer_size = VFIO_MIG_DATA_BUFFER_SIZE; + vbasedev->migration->data_buffer = + g_malloc0(vbasedev->migration->data_buffer_size); + vbasedev->migration->data_fd = -1; migration = vbasedev->migration; migration->vbasedev = vbasedev; @@ -1132,28 +488,16 @@ static int vfio_migration_init(VFIODevice *vbasedev) } strpadcpy(id, sizeof(id), path, '\0'); - if (migration->v2) { - register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, - &savevm_vfio_handlers, vbasedev); - - migration->vm_state = qdev_add_vm_change_state_handler( - vbasedev->dev, vfio_vmstate_change, vbasedev); - } else { - register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, - &savevm_vfio_v1_handlers, vbasedev); - - migration->vm_state = qdev_add_vm_change_state_handler( - vbasedev->dev, vfio_v1_vmstate_change, vbasedev); - } + register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, &savevm_vfio_handlers, + vbasedev); + migration->vm_state = qdev_add_vm_change_state_handler(vbasedev->dev, + vfio_vmstate_change, + vbasedev); migration->migration_state.notify = vfio_migration_state_notifier; add_migration_state_change_notifier(&migration->migration_state); - return 0; -err: - g_free(info); - vfio_migration_exit(vbasedev); - return ret; + return 0; } /* ---------------------------------------------------------------------- */ diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 6e8c5958b9..a24ea7d8b0 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -154,15 +154,10 @@ vfio_vmstate_change(const char *name, int running, const char *reason, uint32_t vfio_migration_state_notifier(const char *name, const char *state) " (%s) state %s" vfio_save_setup(const char *name) " (%s)" vfio_save_cleanup(const char *name) " (%s)" -vfio_save_buffer(const char *name, uint64_t data_offset, uint64_t data_size, uint64_t pending) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64" pending 0x%"PRIx64 -vfio_update_pending(const char *name, uint64_t pending) " (%s) pending 0x%"PRIx64 vfio_save_device_config_state(const char *name) " (%s)" -vfio_save_pending(const char *name, uint64_t precopy, uint64_t postcopy, uint64_t compatible) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" compatible 0x%"PRIx64 -vfio_save_iterate(const char *name, int data_size) " (%s) data_size %d" vfio_save_complete_precopy(const char *name) " (%s)" vfio_load_device_config_state(const char *name) " (%s)" vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64 -vfio_v1_load_state_device_data(const char *name, uint64_t data_offset, uint64_t data_size) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64 vfio_load_state_device_data(const char *name, uint64_t data_size) " (%s) size 0x%"PRIx64 vfio_load_cleanup(const char *name) " (%s)" vfio_get_dirty_bitmap(int fd, uint64_t iova, uint64_t size, uint64_t bitmap_size, uint64_t start) "container fd=%d, iova=0x%"PRIx64" size= 0x%"PRIx64" bitmap_size=0x%"PRIx64" start=0x%"PRIx64 diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 2ec3346fea..76d470178f 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -61,16 +61,11 @@ typedef struct VFIORegion { typedef struct VFIOMigration { struct VFIODevice *vbasedev; VMChangeStateEntry *vm_state; - VFIORegion region; - uint32_t device_state_v1; - int vm_running; Notifier migration_state; - uint64_t pending_bytes; enum vfio_device_mig_state device_state; int data_fd; void *data_buffer; size_t data_buffer_size; - bool v2; } VFIOMigration; typedef struct VFIOAddressSpace { From patchwork Mon May 30 17:07:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 12864844 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 150A3C433EF for ; Mon, 30 May 2022 17:59:03 +0000 (UTC) Received: from localhost ([::1]:58210 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nvjfS-0004bT-42 for qemu-devel@archiver.kernel.org; Mon, 30 May 2022 13:59:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54398) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvisS-0004vN-Pk for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:24 -0400 Received: from mail-mw2nam10on20603.outbound.protection.outlook.com ([2a01:111:f400:7e89::603]:31680 helo=NAM10-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvisQ-0008NQ-3z for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:24 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LU7zy5+yN3odRNhjGvGXXNCqZ7LPVwSSDG8c3K9uO+/w/q7FPaCbXBwnQk039he8I/Rga/FoXeT1OUefwcKqII7BJdtGaVn+2ptjMpkWw9ANGWw8amJVzBLi/lUUTpwvXhS04YzY3SBKrjXVxLdhRuFl5fDnlkgNG6t0tEjYrOLvNUbyvJfr3tK3LYXpyQwRpht1EKByOEOFnwxr607k84YMamQyJoQxIxzKZjVQqbgJ6ykEtNif7V577Vt+ykOdhMWT2c6vrv56fq83RWNgt8ZaRGQfH8fCxq2djZWx2H6J/IFHJbQYNGzxWFqA/6m5o4xTYXSXLmJOK+q2GKXu+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LnU8qOv6i2xNavvVVrbqS7u4puQ5WHvKn7v45SwSF4E=; b=UvvzX9f1IvE5cXLO3A40kXlDTDaxW8u3Rhoqof63RDL5fDEKwU3xuZfLbyFVQt2nncZAt8iMSHO/G0w8BCOm+3hUTPuPA17RkUhrwejYzlU+rqurtJJZWkfau+WFKHDuAJ7/u+xqR/FQspZuAnqJy0P4wtoAmH6HbxZKa1m//41Kwrq9ct+hs4cNwVV4Y6ElL+Y64C6OptVpubY3eWnEdOzNxCkzlJ9VcKaNkYOUQI1iVwpuhEdwi+z5Prv/bjLkNr70A61ut/c/Wb8FmdReV8QfHIpm4krMsq4LVRkCGfqeCGjb4scLBlC7+6zEQEWJbV0O8J3UJTgSiGpe3kCWew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.234) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LnU8qOv6i2xNavvVVrbqS7u4puQ5WHvKn7v45SwSF4E=; b=ORMG2osqX5n3hbvYLGmycn0FijNS02WIEGCxbpPF86EO5erj6vgnJETrQxhfrhVellaS85VHULAnUvUNL+XSNBcN7xeQrDPj8F1JQJHB9KGvQEz2kOb5YuNSzjGr7iW7dEML5RJFW4HGck81hujhKpH9hGquDi6Kg7O9AzWJaae+q5uOlEvOdTdGge52Xu58XpINBLzdSoB4bMJI+suHXoPvaaoTJr1cwtg+4fXUbc+qWzYZV38+ofIxfYk5RsiwPUNAmTDE+DE0RYJZ1P2ben+i1QnKEdOOr+cM8BYeLO4osdNmyYi5RftyQLe9S6mSSvyFHrO5fhs/mdfDbGS/YQ== Received: from BN9PR03CA0233.namprd03.prod.outlook.com (2603:10b6:408:f8::28) by MWHPR1201MB0239.namprd12.prod.outlook.com (2603:10b6:301:50::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Mon, 30 May 2022 17:08:16 +0000 Received: from BN8NAM11FT022.eop-nam11.prod.protection.outlook.com (2603:10b6:408:f8:cafe::3e) by BN9PR03CA0233.outlook.office365.com (2603:10b6:408:f8::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:08:16 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.234) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.234 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.234; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.234) by BN8NAM11FT022.mail.protection.outlook.com (10.13.176.112) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:08:16 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by DRHQMAIL101.nvidia.com (10.27.9.10) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 30 May 2022 17:08:15 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 30 May 2022 10:08:14 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.22 via Frontend Transport; Mon, 30 May 2022 10:08:11 -0700 From: Avihai Horon To: , Cornelia Huck , "Alex Williamson" , Juan Quintela , "Dr . David Alan Gilbert" CC: Joao Martins , Yishai Hadas , Jason Gunthorpe , Mark Bloch , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH v2 09/11] vfio/migration: Reset device if setting recover state fails Date: Mon, 30 May 2022 20:07:37 +0300 Message-ID: <20220530170739.19072-10-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20220530170739.19072-1-avihaih@nvidia.com> References: <20220530170739.19072-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b7ca6b28-4a36-4305-ffb0-08da425efcca X-MS-TrafficTypeDiagnostic: MWHPR1201MB0239:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vbremTIWLXZyELrVxs+ewf/bJ3Er3TeBaoBxXGgEGR9TbyJyEJ3+Ej4M8ASmXDaLo21kpdfEfdJiA38PSgEbbAq68U7g+j0rUGPmF/zqX0qWM0RSknPRSZyUnHgxLjEWzzeTuOg0GNcFGOD3oWbI1aS9pqR7IwRnHPqF7oOk1mHCGHzqVlwqMX19MSHtkS/kAl0Xlj4X+SZAVJmkxgMkEQq6shwPexevv17FcZixDNniFJZVqa+OxqmcpVhDWfH+1+WSbsIhgi1Sb5J7jGh6c7+NqSaXuG2LDDiLxQ6bAEVsGPJIIrxijyauxa8t5KLPotJoNgt7iZRyRBQwJ7b/FfwV19Bd5TtIFpXM7X4jRLROqkFmtBvoUun1XEaGrhIQMGbYdGsxbtymYRWcBDvKFBoTZalRaLyvlHhnCRojnnXRHMPiqwwAZuCZeRfiLzoyua2Mn39M8o5lhelBshmoSCCXX0XQsA4kDYHO/ZXIPB4isRI2mMEovrHVg0S8m/v3Fol/T0izVIlS3Xcjz2EjDClRLZuBqnW8l4QtChY2Mj3TDuLIY1iBl8PWnBE0CjYyDnLHDudIe6HKJSX715aN+bsmyHPCYe474BpsNO91lUgK8PPKA/tu9EwfOUrLT/HgxSVmVRGVOnEWwFKTI7NorP9qKOo8D9NgYq02AaGkisG8MwExdQhK8dAlfJvM0SnPjvoRCDHbsU/jvAyMa6NNjw== X-Forefront-Antispam-Report: CIP:12.22.5.234; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(46966006)(36840700001)(40470700004)(316002)(4326008)(54906003)(110136005)(8936002)(2906002)(8676002)(5660300002)(426003)(336012)(47076005)(7696005)(26005)(83380400001)(1076003)(6666004)(186003)(2616005)(107886003)(86362001)(36756003)(40460700003)(36860700001)(356005)(82310400005)(81166007)(70206006)(70586007)(508600001)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2022 17:08:16.0149 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b7ca6b28-4a36-4305-ffb0-08da425efcca X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.234]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT022.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1201MB0239 Received-SPF: softfail client-ip=2a01:111:f400:7e89::603; envelope-from=avihaih@nvidia.com; helo=NAM10-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" If vfio_migration_set_state() fails to set the device in the requested state it tries to put it in a recover state. If setting the device in the recover state fails as well, hw_error is triggered and the VM is aborted. To improve user experience and avoid VM data loss, reset the device with VFIO_RESET_DEVICE instead of aborting the VM. Signed-off-by: Avihai Horon --- hw/vfio/migration.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 852759e6ca..6c34502611 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -89,8 +89,16 @@ static int vfio_migration_set_state(VFIODevice *vbasedev, /* Try to put the device in some good state */ mig_state->device_state = recover_state; if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) { - hw_error("%s: Device in error state, can't recover", - vbasedev->name); + if (ioctl(vbasedev->fd, VFIO_DEVICE_RESET)) { + hw_error("%s: Device in error state, can't recover", + vbasedev->name); + } + + error_report( + "%s: Device was reset due to failure in changing device state to recover state %s", + vbasedev->name, mig_state_to_str(recover_state)); + + return -1; } error_report("%s: Failed changing device state to %s", vbasedev->name, From patchwork Mon May 30 17:07:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 12864846 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B30A0C433EF for ; Mon, 30 May 2022 18:04:01 +0000 (UTC) Received: from localhost ([::1]:38546 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nvjkG-0002EH-Pe for qemu-devel@archiver.kernel.org; Mon, 30 May 2022 14:04:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54402) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvisS-0004wI-Vl for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:25 -0400 Received: from mail-mw2nam12on20625.outbound.protection.outlook.com ([2a01:111:f400:fe5a::625]:48705 helo=NAM12-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvisQ-0008NW-QN for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:24 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nZsd2uUZIckUGe1kXxzsIR38hntlhHto05qtRPAp6duYpwk+J+n2GniC1uifXpQmArhkY4WmibVV4XhEY0/+SASgCk82FPcCgYKQNCq5gH7dlmli3a8tw3OC5JfQtpOJKbno8enW1KSyDXLVudV9b8mS7THtOkSmwpF+2ykZQwMG+uNfiK5IFNb+Ag0ln7Jksrw9STeM/s8lskRsvAfvUhN0GBdnk0nCVZJsRFFHjeEFK3GL4YxS2Ttlka9a+HeRTaFBxHO3vKy5p3zXhhNFxgU0uZXotTiFPbMxL1tx0+QPy9BNvC5DArGUrFGd15bjGo8G4VrHYgswZeNOnlXKxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=liYUGVmuX0kZmO+aQDDBrQ7rfJPkkF+ldMbOwennimQ=; b=CfrBcTi2qdBQgKU85kN1dNiEk/DPgzy24gvV/4TGyvOJnRB584dEqLGphL/CjyGB+mFibELDbm3O1keAlj1EpV7Uj9Nf2tNs3E05theLXJgVvdhV2fUEcRpDS70EE1IZEzLYfpi0rgsL3JgSIXeAJbBP2b/CJ1t1i2O7kY+/v9NO87EDQRKL7WwQPgvB194hrlMHvn+LCiPH9WWQ374zc7yQMr95fXrQZRMyT5dKvaGIHKyVwuEiDZrbeTcZivJ4pdxNng4PNhjOIxvdEVw4pp+3FzvbtUBuD33krKnuPMLSwrCGyO9bW1aGnuVzIUvMvMRn/ACOm5AY3atWNnfWTg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=liYUGVmuX0kZmO+aQDDBrQ7rfJPkkF+ldMbOwennimQ=; b=DRmACLktMqyd6b58g9loM1Zh04F/lWdPv65BvJHvY/PvCrCyi+RQwSP3L9Zcmx/Aywmw0bT+afPd5AI+7yz+Ile8LZaND/lhPzsVkUh60jfxA3ltUmqthBN0A6XmQTa7VY4cteocL2JMa3XEIyyECGBRQIvqeVIX/UVjhF9J5Y513msYdbkWJqTyU5oCy9Df73tB+Uh2e+t+CYR/qWy6SVIXbXxZ3yPsF38KdOcjqZlhB9o88KXMvsLdjucQmVh+v2PZvpyxpawqPs6KYwKbSujLphg4yzwekx1dK9RSJyhcifFa3/WSm2noR2lwAzKXSX9be+7AcjNYUmwg7CS1iQ== Received: from CO2PR04CA0178.namprd04.prod.outlook.com (2603:10b6:104:4::32) by IA1PR12MB6307.namprd12.prod.outlook.com (2603:10b6:208:3e5::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Mon, 30 May 2022 17:08:19 +0000 Received: from CO1NAM11FT048.eop-nam11.prod.protection.outlook.com (2603:10b6:104:4:cafe::b6) by CO2PR04CA0178.outlook.office365.com (2603:10b6:104:4::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:08:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.235) by CO1NAM11FT048.mail.protection.outlook.com (10.13.175.148) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:08:19 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 30 May 2022 17:08:18 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 30 May 2022 10:08:18 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.22 via Frontend Transport; Mon, 30 May 2022 10:08:15 -0700 From: Avihai Horon To: , Cornelia Huck , "Alex Williamson" , Juan Quintela , "Dr . David Alan Gilbert" CC: Joao Martins , Yishai Hadas , Jason Gunthorpe , Mark Bloch , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH v2 10/11] vfio: Alphabetize migration section of VFIO trace-events file Date: Mon, 30 May 2022 20:07:38 +0300 Message-ID: <20220530170739.19072-11-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20220530170739.19072-1-avihaih@nvidia.com> References: <20220530170739.19072-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 82af9299-0d8b-438a-017a-08da425efeae X-MS-TrafficTypeDiagnostic: IA1PR12MB6307:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: uVYozSL64rFJXRj4TB81gQhLBNsuoLgUXUuVxRXF+MAtnMkOn+kxICHsRditEjuEQkAW0v9T0E7Ar9enlBM91MTSvKwcgdPyrzCEzckEB1qE3sMZ5qpvsB+2WO2p0G3WSosbk5Bq7cNdV/G2998kuH7LRS95bp+pmdB+ZUUl7Zd5myhqQIi0qE8SBBUKewqTtRaipDPNX/3wbT+Jsteo2Fj/rD3Vu+XV26bJIpFSiqFocpPqMehToN8nGPBuGIGx8DzMQJ2MuApjoVEN9jplcADmZE2nmnu1PEDAFZd4YAaJFGQ7TiR3sKyVtP1G36+Omlsyqh+H0si+Cha1t63Fz95oj32FSHlhUpGz3FCmfWEbCGK9g6my9ZVGrPgwAV9XtYNpwxx65wCMFeyM6hgopFb638nHzDvG3TP+lykzE8G7xqWNwwO06ql1VIVpmKJQBWciwJ4HhqHD3GzIBZiAQy03e8ReAB7eqAoRxdRgcScFM1pcHCPknXLfvU/ndMiSDfNVwfFBqWyhW4luCjNEgLrSNti3sKYl57HeIv28sZV/DXsWNNOthdakRzrhRzyEKOPAKfC9BE9YxBA8ZP8e0sQ4aOpXjF77bl4tA1d+RYmpZSXlf5FLTYMwpywpwpNMsagb8WIEWW3CV51tOImUb9TrRwBDMFUerveHuVWaMlh5x1tBzW34MDyJoTqwDdZhgZuNy6u1KCEAnKv040/ZNw== X-Forefront-Antispam-Report: CIP:12.22.5.235; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(36840700001)(46966006)(54906003)(70206006)(8936002)(110136005)(70586007)(1076003)(8676002)(2616005)(316002)(4326008)(86362001)(107886003)(356005)(7696005)(6666004)(26005)(81166007)(36756003)(426003)(336012)(83380400001)(82310400005)(5660300002)(36860700001)(40460700003)(2906002)(186003)(47076005)(508600001)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2022 17:08:19.2272 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 82af9299-0d8b-438a-017a-08da425efeae X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.235]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT048.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6307 Received-SPF: softfail client-ip=2a01:111:f400:fe5a::625; envelope-from=avihaih@nvidia.com; helo=NAM12-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Sort the migration section of VFIO trace events file alphabetically and move two misplaced traces to common.c section. Signed-off-by: Avihai Horon --- hw/vfio/trace-events | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index a24ea7d8b0..d3cba59bfd 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -119,6 +119,8 @@ vfio_region_sparse_mmap_header(const char *name, int index, int nr_areas) "Devic vfio_region_sparse_mmap_entry(int i, unsigned long start, unsigned long end) "sparse entry %d [0x%lx - 0x%lx]" vfio_get_dev_region(const char *name, int index, uint32_t type, uint32_t subtype) "%s index %d, %08x/%0x8" vfio_dma_unmap_overflow_workaround(void) "" +vfio_get_dirty_bitmap(int fd, uint64_t iova, uint64_t size, uint64_t bitmap_size, uint64_t start) "container fd=%d, iova=0x%"PRIx64" size= 0x%"PRIx64" bitmap_size=0x%"PRIx64" start=0x%"PRIx64 +vfio_iommu_map_dirty_notify(uint64_t iova_start, uint64_t iova_end) "iommu dirty @ 0x%"PRIx64" - 0x%"PRIx64 # platform.c vfio_platform_base_device_init(char *name, int groupid) "%s belongs to group #%d" @@ -148,18 +150,16 @@ vfio_display_edid_update(uint32_t prefx, uint32_t prefy) "%ux%u" vfio_display_edid_write_error(void) "" # migration.c +vfio_load_cleanup(const char *name) " (%s)" +vfio_load_device_config_state(const char *name) " (%s)" +vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64 +vfio_load_state_device_data(const char *name, uint64_t data_size) " (%s) size 0x%"PRIx64 vfio_migration_probe(const char *name) " (%s)" vfio_migration_set_state(const char *name, uint32_t state) " (%s) state %d" -vfio_vmstate_change(const char *name, int running, const char *reason, uint32_t dev_state) " (%s) running %d reason %s device state %d" vfio_migration_state_notifier(const char *name, const char *state) " (%s) state %s" -vfio_save_setup(const char *name) " (%s)" +vfio_save_block(const char *name, int data_size) " (%s) data_size %d" vfio_save_cleanup(const char *name) " (%s)" -vfio_save_device_config_state(const char *name) " (%s)" vfio_save_complete_precopy(const char *name) " (%s)" -vfio_load_device_config_state(const char *name) " (%s)" -vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64 -vfio_load_state_device_data(const char *name, uint64_t data_size) " (%s) size 0x%"PRIx64 -vfio_load_cleanup(const char *name) " (%s)" -vfio_get_dirty_bitmap(int fd, uint64_t iova, uint64_t size, uint64_t bitmap_size, uint64_t start) "container fd=%d, iova=0x%"PRIx64" size= 0x%"PRIx64" bitmap_size=0x%"PRIx64" start=0x%"PRIx64 -vfio_iommu_map_dirty_notify(uint64_t iova_start, uint64_t iova_end) "iommu dirty @ 0x%"PRIx64" - 0x%"PRIx64 -vfio_save_block(const char *name, int data_size) " (%s) data_size %d" +vfio_save_device_config_state(const char *name) " (%s)" +vfio_save_setup(const char *name) " (%s)" +vfio_vmstate_change(const char *name, int running, const char *reason, uint32_t dev_state) " (%s) running %d reason %s device state %d" From patchwork Mon May 30 17:07:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 12864856 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 60FCFC433F5 for ; Mon, 30 May 2022 18:07:29 +0000 (UTC) Received: from localhost ([::1]:47082 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nvjnc-0000TZ-CG for qemu-devel@archiver.kernel.org; Mon, 30 May 2022 14:07:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54430) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvisX-0005BA-4j for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:29 -0400 Received: from mail-co1nam11on20630.outbound.protection.outlook.com ([2a01:111:f400:7eab::630]:31904 helo=NAM11-CO1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvisU-0008PY-UA for qemu-devel@nongnu.org; Mon, 30 May 2022 13:08:28 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dZYnjr76hgpYD6IFlWEtsdYjv2uDA25QwAxZ7U/fM0iixgAa2CykHFJVMX8n6/EFCEUK/oWUqzAuuLlGGIIIZzxVaw2sr65EAXj5qslpTxp4HPg+LhfQABNpcion9szslSVo5Bs3Lap44H/PCmz4nkyLqpzcCbYeOjI7R/IAnrNFQ8i0YOQTAfH4GaeiFUdjsyB5TnSQINtnkg8ZUKWU2rwUV6SnxBuuJozK4b33nF+o5fhAyqJyGa1y6RNDRLlRfMrz97PtnKTthMtqwAvaM3By12Y8W0yUVa7BV9M5SnYBhVxPnhfzxmlTCixTDu/A+FvbK4Smxn9aHd5oJ4VGBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PSPaPjjDN9b8FTrPUpEHjY11iJitw0hUj9yIciMkLkE=; b=Uh73WDhN6QdIsjAL3kuyDcxiB4YKD/jyRFcyigsMcdTlO7RTbBKGnFV4bqpRKiGaox6JLTuJ6ci7RQJzUhzoiNOd/cV8kt+/f8mU3SU+a+7mFQDy44T6yd6BtOFs7QGDGw9oGIs+yCzI1kqtYDbU2SB3fHT01uK26ksS330sgU4XOxnJ427PnmAgufHV39JY0b0KeV6Y7R9HxRJzBPimjZcdkj2LHTq7xwhKQuaZ6TE6SrD/frSmFP+bLJSOXSdDGkHXV46jDoNbaoO9JUQGTCzIQfmR9dYp+o95PzGnp3JpfksAWXEUUUbX7K+FpS3r2AdxP70wbJH1teLwvxhFoQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.238) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PSPaPjjDN9b8FTrPUpEHjY11iJitw0hUj9yIciMkLkE=; b=OncJ0ubDiPz1OwAQZ3YsQgk6xWIM+gPZDcnAV7FODnCmhALKdN1kc2ylb8v2rYMGt65Nlc4iCzDY4WfDHWJtaL4gCY9A+O3Dczt8hImAJJxcbs62zugDT8oEm0SqTCqxVTyUiHh3Vq5uAlMsoNexu5VkKtz7aEfDYLbe4oAemkCjS35n7aRFKkR422RRz78ao9KX+ObWdsWKUfPq9HaxSYjJmmHnrQGontYBE6Sr5cqrPCImnf+9yKHBmpppQ7oRhYvOK6gFrSqJ21OZUk8QdVt7oBJHAG82QbNhHURBluSG28MXVXn5e3QDefylWRQmrSbSy1s5EttnOklr/NTFPA== Received: from CO1PR15CA0084.namprd15.prod.outlook.com (2603:10b6:101:20::28) by BN6PR1201MB0004.namprd12.prod.outlook.com (2603:10b6:404:ac::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Mon, 30 May 2022 17:08:23 +0000 Received: from CO1NAM11FT046.eop-nam11.prod.protection.outlook.com (2603:10b6:101:20:cafe::67) by CO1PR15CA0084.outlook.office365.com (2603:10b6:101:20::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.19 via Frontend Transport; Mon, 30 May 2022 17:08:23 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.238) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.238 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.238; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.238) by CO1NAM11FT046.mail.protection.outlook.com (10.13.174.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5293.13 via Frontend Transport; Mon, 30 May 2022 17:08:22 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by DRHQMAIL105.nvidia.com (10.27.9.14) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 30 May 2022 17:08:22 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 30 May 2022 10:08:21 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.22 via Frontend Transport; Mon, 30 May 2022 10:08:18 -0700 From: Avihai Horon To: , Cornelia Huck , "Alex Williamson" , Juan Quintela , "Dr . David Alan Gilbert" CC: Joao Martins , Yishai Hadas , Jason Gunthorpe , Mark Bloch , Maor Gottlieb , Kirti Wankhede , Tarun Gupta , Avihai Horon Subject: [PATCH v2 11/11] docs/devel: Align vfio-migration docs to VFIO migration v2 Date: Mon, 30 May 2022 20:07:39 +0300 Message-ID: <20220530170739.19072-12-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20220530170739.19072-1-avihaih@nvidia.com> References: <20220530170739.19072-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 430a0034-c8de-4553-6081-08da425f00e5 X-MS-TrafficTypeDiagnostic: BN6PR1201MB0004:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: eiNSNyNT7JTsj5T07pRLZSk1W5QZXPBMKb0D9MbZkWiI22cV/dU1NeC8i003lROnF+5PGmhunOSmmfAHjE+N9gERj2AlqEdnQJugp14OogYTZUUwakEeQIOa0Q0Vsd6Z3uMnY2SFSKf/7JDKjmPSFiKNhxABKhzXoPLdDSMrOQOYIViJdgeHh8mSDxlyd9zuqkhWSt4IZgNTZL6OC71s50+gISAgg9ywt21aUxlMFuxPnsBjgfahvJtYR8YekNLmV2NNzCpEBpeqD231rmH6rKk+Kgji0wqLePjeFfeOkpwQPbMewTGfSG7ODAjkIuRfHb2/34FI8m9bFMz688Bg2xqbiTJPxFTh/0yl3YMqoLmN9Jd6FT5A55L4qTlNzlAvN7rxx0xX3b9EHJ2wYFPS44KCSY/NN1nDIXCM/p/U6aND1ilRwChtt3uc+PoGfofQdu3uy3yMyNw2kfAohnNSJ0EOPdeK37IvFF9iEjSlyBngrBUku7ZrPkk81aY4Q2ErXVfhv/LEnlzlGYLJ/bkrl+LfnA8akZMlr9CYDYNWG7VaFd4wireZ2dc72D1aHq7LHC7kbNk1k2lyJ6qfZR9pCq1j+xQmfkLAkxKPhNcbo0AyUhDhLuabN3aqxHmaZqGux9Ul3nU1HFZu64ALf/6nRc8eebtvuJHJdJyEX/GTpzE0Qd8ZLD5lCw3fdF6KmMTm2Cw02XjJt8SPFDGIttMhkg== X-Forefront-Antispam-Report: CIP:12.22.5.238; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(36840700001)(40470700004)(46966006)(26005)(86362001)(40460700003)(6666004)(508600001)(8676002)(356005)(7696005)(316002)(5660300002)(4326008)(110136005)(54906003)(36756003)(83380400001)(36860700001)(336012)(426003)(186003)(8936002)(2906002)(1076003)(2616005)(82310400005)(70586007)(70206006)(107886003)(81166007)(47076005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2022 17:08:22.9455 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 430a0034-c8de-4553-6081-08da425f00e5 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.238]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT046.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR1201MB0004 Received-SPF: softfail client-ip=2a01:111:f400:7eab::630; envelope-from=avihaih@nvidia.com; helo=NAM11-CO1-obe.outbound.protection.outlook.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Align the vfio-migration documentation to VFIO migration protocol v2. Signed-off-by: Avihai Horon --- docs/devel/vfio-migration.rst | 77 +++++++++++++++-------------------- 1 file changed, 33 insertions(+), 44 deletions(-) diff --git a/docs/devel/vfio-migration.rst b/docs/devel/vfio-migration.rst index 9ff6163c88..09744af5a6 100644 --- a/docs/devel/vfio-migration.rst +++ b/docs/devel/vfio-migration.rst @@ -7,46 +7,35 @@ the guest is running on source host and restoring this saved state on the destination host. This document details how saving and restoring of VFIO devices is done in QEMU. -Migration of VFIO devices consists of two phases: the optional pre-copy phase, -and the stop-and-copy phase. The pre-copy phase is iterative and allows to -accommodate VFIO devices that have a large amount of data that needs to be -transferred. The iterative pre-copy phase of migration allows for the guest to -continue whilst the VFIO device state is transferred to the destination, this -helps to reduce the total downtime of the VM. VFIO devices can choose to skip -the pre-copy phase of migration by returning pending_bytes as zero during the -pre-copy phase. +Migration of VFIO devices currently consists of a single stop-and-copy phase. +During the stop-and-copy phase the guest is stopped and the entire VFIO device +data is transferred to the destination. + +The pre-copy phase of migration is currently not supported for VFIO devices, +so VFIO device data is not transferred during pre-copy phase. A detailed description of the UAPI for VFIO device migration can be found in -the comment for the ``vfio_device_migration_info`` structure in the header -file linux-headers/linux/vfio.h. +the comment for the ``vfio_device_mig_state`` structure in the header file +linux-headers/linux/vfio.h. VFIO implements the device hooks for the iterative approach as follows: -* A ``save_setup`` function that sets up the migration region and sets _SAVING - flag in the VFIO device state. - -* A ``load_setup`` function that sets up the migration region on the - destination and sets _RESUMING flag in the VFIO device state. - -* A ``save_live_pending`` function that reads pending_bytes from the vendor - driver, which indicates the amount of data that the vendor driver has yet to - save for the VFIO device. +* A ``save_setup`` function that sets up migration on the source. -* A ``save_live_iterate`` function that reads the VFIO device's data from the - vendor driver through the migration region during iterative phase. +* A ``load_setup`` function that sets the VFIO device on the destination in + _RESUMING state. * A ``save_state`` function to save the device config space if it is present. -* A ``save_live_complete_precopy`` function that resets _RUNNING flag from the - VFIO device state and iteratively copies the remaining data for the VFIO - device until the vendor driver indicates that no data remains (pending bytes - is zero). +* A ``save_live_complete_precopy`` function that sets the VFIO device in + _STOP_COPY state and iteratively copies the data for the VFIO device until + the vendor driver indicates that no data remains. * A ``load_state`` function that loads the config section and the data - sections that are generated by the save functions above + sections that are generated by the save functions above. * ``cleanup`` functions for both save and load that perform any migration - related cleanup, including unmapping the migration region + related cleanup. The VFIO migration code uses a VM state change handler to change the VFIO @@ -71,13 +60,13 @@ tracking can identify dirtied pages, but any page pinned by the vendor driver can also be written by the device. There is currently no device or IOMMU support for dirty page tracking in hardware. -By default, dirty pages are tracked when the device is in pre-copy as well as -stop-and-copy phase. So, a page pinned by the vendor driver will be copied to -the destination in both phases. Copying dirty pages in pre-copy phase helps -QEMU to predict if it can achieve its downtime tolerances. If QEMU during -pre-copy phase keeps finding dirty pages continuously, then it understands -that even in stop-and-copy phase, it is likely to find dirty pages and can -predict the downtime accordingly. +By default, dirty pages are tracked during pre-copy as well as stop-and-copy +phase. So, a page pinned by the vendor driver will be copied to the destination +in both phases. Copying dirty pages in pre-copy phase helps QEMU to predict if +it can achieve its downtime tolerances. If QEMU during pre-copy phase keeps +finding dirty pages continuously, then it understands that even in stop-and-copy +phase, it is likely to find dirty pages and can predict the downtime +accordingly. QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking`` which disables querying the dirty bitmap during pre-copy phase. If it is set to @@ -111,23 +100,23 @@ Live migration save path | migrate_init spawns migration_thread Migration thread then calls each device's .save_setup() - (RUNNING, _SETUP, _RUNNING|_SAVING) + (RUNNING, _SETUP, _RUNNING) | - (RUNNING, _ACTIVE, _RUNNING|_SAVING) - If device is active, get pending_bytes by .save_live_pending() - If total pending_bytes >= threshold_size, call .save_live_iterate() - Data of VFIO device for pre-copy phase is copied + (RUNNING, _ACTIVE, _RUNNING) + Migration thread calls each .save_live_pending() handler + If total pending_bytes >= threshold_size, call each .save_live_iterate() handler + Data of this iteration for pre-copy phase is copied Iterate till total pending bytes converge and are less than threshold | On migration completion, vCPU stops and calls .save_live_complete_precopy for - each active device. The VFIO device is then transitioned into _SAVING state - (FINISH_MIGRATE, _DEVICE, _SAVING) + each active device. The VFIO device is then transitioned into _STOP_COPY state + (FINISH_MIGRATE, _DEVICE, _STOP_COPY) | For the VFIO device, iterate in .save_live_complete_precopy until pending data is 0 - (FINISH_MIGRATE, _DEVICE, _STOPPED) + (FINISH_MIGRATE, _DEVICE, _STOP) | - (FINISH_MIGRATE, _COMPLETED, _STOPPED) + (FINISH_MIGRATE, _COMPLETED, _STOP) Migraton thread schedules cleanup bottom half and exits Live migration resume path @@ -136,7 +125,7 @@ Live migration resume path :: Incoming migration calls .load_setup for each device - (RESTORE_VM, _ACTIVE, _STOPPED) + (RESTORE_VM, _ACTIVE, _STOP) | For each device, .load_state is called for that device section data (RESTORE_VM, _ACTIVE, _RESUMING)