From patchwork Mon May 1 14:01:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13227513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E00C0C77B7C for ; Mon, 1 May 2023 14:02:47 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ptU6Z-0004ZW-Tj; Mon, 01 May 2023 10:02:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU6X-0004KT-JG for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:13 -0400 Received: from mail-mw2nam04on20607.outbound.protection.outlook.com ([2a01:111:f400:7e8c::607] helo=NAM04-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU6R-0006c9-O9 for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:12 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DjvU3NxIp54FxIltLU7y8Zg+v+4lXGsWYwH3r997hIBdF1g2OPvFyKgWk37m8d9ZPpKi7+pL+phVrwvsO+3IAaETB0+imPU3MbI5r0O9rIrif/8PI4o/91j1Gc1pD8MX7OGNoAMzAv+plZim5Pou+uk05hAB5lcASSRW2fDgFbXnVeBX+wAXslmb6exN2BQ+SxLb9NcObXU0bq8guSVILx7XCHUhy9smoalXyf5ywDr5CZ1zJDy2CcxglyyzOrUSGvUuqr4hpLSArKX2yzfNHmF9e0CZn6sRTq8AEtvdPpvBmE9yE7MalVjsWrEaLHW125/K03wraznVhwIWUZWvaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pjYcg5/K060uERlZwG6z11hQsalO9jRhsYSQmnT5r4c=; b=cpNLgvLU0o/HjI8H3lGwAUJa6E2BKWH1xDL9GRmZMMzRdwjaLte1BNAH9VT0fRk7YKHDVmLXYiQ5cJ/kjon9qU5SH2d+nKFkTDaku0YFfsMwkq9AipGDa/4pUkt7ofYFQJf6KSFZZ3poBVJB+lDQFbyfqJbhu/6xGnnCuA8ukhoxf0yKzKj8qwxfWhHmb7pukYl1d2qhEH/4y1un0uJ8LF7v4mh6U/gtjS6HfX2/xUTnO5CwiJ79C1xEX1uCPHp0aBdQXFFIUCJ6zJQ1AwpNZ1JxamvhTjIlPiZBdiJlrud6i6f+q3QHEqxZfUYaAs4ONzX/SNmMv09nP2RUXbHKGg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pjYcg5/K060uERlZwG6z11hQsalO9jRhsYSQmnT5r4c=; b=qNvAzBTEIxgOQdXNGFSu0xbYnPbjzhLevKRzgLWMuT7Wc6jxnKyPBJnuSKZo0tLJ57KTp77glq67+eqeKy8lgflHFgJdpoIWRyYRVkKawsLR2Up3pXSXoRD0yuwrkCx9M8pUKvwZ7cu6G/cgMWe3RkZX+NMA9FLmEQbnaQn1X5B8k1fjBRWQPBQat7VxGDvOTYzUiyp28WxpQXbzm5AtsBC2eEMcRby2vvb7p+NGHDGQSqnLNYvAFb9iEIvfFHAEcUVoQ6/Q9wMlurJi9OzSMS3QsOkFW+B0guuaR4NXvs/2h+ycRPOl5UIsDa0rxjuHtMUbhmtIBihDmLO8r4I/sw== Received: from DM5PR07CA0080.namprd07.prod.outlook.com (2603:10b6:4:ad::45) by IA1PR12MB7637.namprd12.prod.outlook.com (2603:10b6:208:427::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.30; Mon, 1 May 2023 14:02:03 +0000 Received: from DM6NAM11FT051.eop-nam11.prod.protection.outlook.com (2603:10b6:4:ad:cafe::71) by DM5PR07CA0080.outlook.office365.com (2603:10b6:4:ad::45) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.30 via Frontend Transport; Mon, 1 May 2023 14:02:03 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT051.mail.protection.outlook.com (10.13.172.243) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.19 via Frontend Transport; Mon, 1 May 2023 14:02:02 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Mon, 1 May 2023 07:01:51 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Mon, 1 May 2023 07:01:51 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Mon, 1 May 2023 07:01:47 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Juan Quintela , Peter Xu , Leonardo Bras , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Paolo Bonzini , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 1/8] migration: Add precopy initial data capability Date: Mon, 1 May 2023 17:01:34 +0300 Message-ID: <20230501140141.11743-2-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230501140141.11743-1-avihaih@nvidia.com> References: <20230501140141.11743-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT051:EE_|IA1PR12MB7637:EE_ X-MS-Office365-Filtering-Correlation-Id: f419f778-f8c8-4ffb-b15c-08db4a4ca3e9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: utzL2imSjJNCsGlY928b1GAOUakPTzP5rz+Ex9k1jvUi9lXczz03QKV349A5ThGDM+X+k7nbogwszjYuYMQMOW6iw5sopP840hDLS+xos/HysGIhd/BEnl9yxGPNUi2B4ocJDV7/RN7BHeXJzucLqemNgHVWyI8Up00OZdmcsCtkSL0bartNEHcGtTUI9qnbxwGwtfEIWJ/f55rbPQco3bYeXrcNQ6k29XNBPV4tgY73grX9V71DlL9vM63+nSCkdiMBgSYENDgbq/cHQjwGU6rvIF3dTvTkGZ8FJPHzLh3p0EBDLMKIQ1K3x8HG0IMMdjAHEl//Cv5JThD2c1MMfBIZEQd30ntS71ZW2hDhlXoPlgB5tlmjGt+lQJhW/IlqcsQ3Jtney2nITaS3Qpqsc2GDY0ooEW8XaiRgif9zjoZGSYgwTqK4P3al3w7Pn+5mdKiIft6Ez5gQSERzuV/mZyxnF6YFp7vtK+BwXC5931UGXGkxNvis8YJSyvOlQOkHGhF4tfArSzPluyxCMOpEhfQSXIiax/wh0oazU/IFAzppi5/PVaFJjoDZttsHH0pG3Wet7swroDwDYmqid1lMd1S57g+RAF+ziv9/F1NWfUd/z7Sb2rm2i/VFbsfV6FKWZ4WcHWS/RihN6MSmweq18irbpjRnIvnUeRIcD/BxvJbRd0un8eAT25XXGrNaZ/I0FF0o44n3LtC1MmcxwNlxtQ== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(396003)(39860400002)(136003)(346002)(376002)(451199021)(40470700004)(46966006)(36840700001)(7416002)(5660300002)(356005)(7636003)(8676002)(40460700003)(82740400003)(8936002)(41300700001)(316002)(40480700001)(36756003)(82310400005)(86362001)(2906002)(36860700001)(186003)(2616005)(478600001)(83380400001)(6666004)(7696005)(426003)(336012)(47076005)(26005)(1076003)(54906003)(4326008)(6916009)(70586007)(70206006); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 May 2023 14:02:02.9622 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f419f778-f8c8-4ffb-b15c-08db4a4ca3e9 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT051.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB7637 Received-SPF: softfail client-ip=2a01:111:f400:7e8c::607; envelope-from=avihaih@nvidia.com; helo=NAM04-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Migration downtime estimation is calculated based on bandwidth and remaining migration data. This assumes that loading of migration data in the destination takes a negligible amount of time and that downtime depends only on network speed. While this may be true for RAM, it's not necessarily true for other migration users. For example, loading the data of a VFIO device in the destination might require from the device to allocate resources, prepare internal data structures and so on. These operations can take a significant amount of time which can increase migration downtime. This patch adds a new capability "precopy initial data" that allows the source to send initial precopy data and the destination to ACK that this data has been loaded. Migration will not attempt to stop the source VM and complete the migration until this ACK is received. This will allow migration users to send initial precopy data which can be used to reduce downtime (e.g., by pre-allocating resources), while making sure that the source will stop the VM and complete the migration only after this initial precopy data is sent and loaded in the destination so it will have full effect. This new capability relies on the return path capability to communicate from the destination back to the source. The actual implementation of the capability will be added in the following patches. Signed-off-by: Avihai Horon --- qapi/migration.json | 9 ++++++++- migration/options.h | 1 + migration/options.c | 20 ++++++++++++++++++++ 3 files changed, 29 insertions(+), 1 deletion(-) diff --git a/qapi/migration.json b/qapi/migration.json index 82000adce4..d496148386 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -478,6 +478,13 @@ # should not affect the correctness of postcopy migration. # (since 7.1) # +# @precopy-initial-data: If enabled, migration will not attempt to stop source +# VM and complete the migration until an ACK is received +# from the destination that initial precopy data has +# been loaded. This can improve downtime if there are +# migration users that support precopy initial data. +# (since 8.1) +# # Features: # @unstable: Members @x-colo and @x-ignore-shared are experimental. # @@ -492,7 +499,7 @@ 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate', { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] }, 'validate-uuid', 'background-snapshot', - 'zero-copy-send', 'postcopy-preempt'] } + 'zero-copy-send', 'postcopy-preempt', 'precopy-initial-data'] } ## # @MigrationCapabilityStatus: diff --git a/migration/options.h b/migration/options.h index 3c322867cd..d004b6321e 100644 --- a/migration/options.h +++ b/migration/options.h @@ -44,6 +44,7 @@ bool migrate_pause_before_switchover(void); bool migrate_postcopy_blocktime(void); bool migrate_postcopy_preempt(void); bool migrate_postcopy_ram(void); +bool migrate_precopy_initial_data(void); bool migrate_rdma_pin_all(void); bool migrate_release_ram(void); bool migrate_return_path(void); diff --git a/migration/options.c b/migration/options.c index 53b7fc5d5d..c4ef0c60c7 100644 --- a/migration/options.c +++ b/migration/options.c @@ -184,6 +184,8 @@ Property migration_properties[] = { DEFINE_PROP_MIG_CAP("x-zero-copy-send", MIGRATION_CAPABILITY_ZERO_COPY_SEND), #endif + DEFINE_PROP_MIG_CAP("x-precopy-initial-data", + MIGRATION_CAPABILITY_PRECOPY_INITIAL_DATA), DEFINE_PROP_END_OF_LIST(), }; @@ -286,6 +288,13 @@ bool migrate_postcopy_ram(void) return s->capabilities[MIGRATION_CAPABILITY_POSTCOPY_RAM]; } +bool migrate_precopy_initial_data(void) +{ + MigrationState *s = migrate_get_current(); + + return s->capabilities[MIGRATION_CAPABILITY_PRECOPY_INITIAL_DATA]; +} + bool migrate_rdma_pin_all(void) { MigrationState *s = migrate_get_current(); @@ -546,6 +555,17 @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp) } } + if (new_caps[MIGRATION_CAPABILITY_PRECOPY_INITIAL_DATA]) { + if (!new_caps[MIGRATION_CAPABILITY_RETURN_PATH]) { + error_setg(errp, "Precopy initial data requires return path"); + return false; + } + + /* Disable this capability until it's implemented */ + error_setg(errp, "Precopy initial data is not implemented yet"); + return false; + } + return true; } From patchwork Mon May 1 14:01:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13227519 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EABBDC77B61 for ; Mon, 1 May 2023 14:03:59 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ptU6d-0004gW-7Q; Mon, 01 May 2023 10:02:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU6Z-0004YY-J4 for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:15 -0400 Received: from mail-dm6nam12on20600.outbound.protection.outlook.com ([2a01:111:f400:fe59::600] helo=NAM12-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU6W-0006ce-Ta for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:15 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=K6XpmzUZTYWZ+0bS5lkU4Z4bdYFkkMWD0bqHDM89ZcrtVLXECB9SYEmgTlSFEMQ/d6F6StQulOApCLhFM/hDeqrnGREWtOKNWxoN2Q2j3WOEZfge1xAB6ev62YbkF20Wxs2SLFL6FtLqDDwy5RuNuhq+Y3F0IKccdl2mfPmIm5TA80IMhuTvejIYHIjukkec9X4H8jlQqp/t3ZmKTk1CJS6KA3/lFc9nej73U50wDOkmtzvKTyN9/yMJYnJEEvJbS8nDZ9XN9pxEO5qAMIH8Qsm+1NzQ+ZIN9KKP3wQfLpvBvbCnOuDHR13/0pGfaPpY40hnGjnCpuGaWqGqroLyFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3SF43iDpgEWDoLPcClIWjZNLYo6aeKbWHvGiaqhP6P8=; b=mIh11qUvy6PlEHjYJl5ms61PJ9QHQ25kYnsDYCTBQyRwrI4hW8I24sSnC9w5l4qJOIjz7pHkxT2rwUSQQYQbQX1PUEjoldrBf4NsMcNkTf5EinBQCdsr7DFfgHxBzdin2mwgvq2R78g7Yqn+o0lnGpyU6osebkQjF86IWzcdC+AhyGUzF0t6yaKc7O/BTIOztdaZbQfjqAi6HGmNi7cNZhhxa6nXs1oJuT989qeQbv3NDq5yjpz1f/KCpHURRaLhDrtWxVcnd778W64Xkuq0B2LfteTcqvG2r8HjnDctnqtCyi41dDDF0A6iJvK6O9xKMlp/ohBzlP+nGGfnqa55BQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3SF43iDpgEWDoLPcClIWjZNLYo6aeKbWHvGiaqhP6P8=; b=tXNAdlGyR+qeGtnyvM2p9+/9UK+bx7NcYj/Pi5msWUgNizLCBqlvomZXD/djpQd/BMLujqLt0m+f73JrfLp6NqVMQdPVMZrKYCnt0nLZdP5F2KjK51/m86l/EqoG+UlTPhirrUR+ADvDMgT6/dDOY17+vQ/vBksgtQ8Q4rLPZTQVWPvNJGpW84xVd2JAXMyNVLXp78SycFbimSNv7iT968VhuDHT9DbmdZJGKgphl+PTcBQu+V2xnQaqUAj6Y+Y4IjSsCpe6gfXjzE+QOft173oqEPDhvAeRxfkENDYAcc4aGeu1IBBtzSdBJ2vVev/EFzwPL2My74TyAQedcNUZ3A== Received: from DM5PR07CA0065.namprd07.prod.outlook.com (2603:10b6:4:ad::30) by CY5PR12MB6597.namprd12.prod.outlook.com (2603:10b6:930:43::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.30; Mon, 1 May 2023 14:02:06 +0000 Received: from DM6NAM11FT051.eop-nam11.prod.protection.outlook.com (2603:10b6:4:ad:cafe::77) by DM5PR07CA0065.outlook.office365.com (2603:10b6:4:ad::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.31 via Frontend Transport; Mon, 1 May 2023 14:02:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT051.mail.protection.outlook.com (10.13.172.243) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.19 via Frontend Transport; Mon, 1 May 2023 14:02:06 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Mon, 1 May 2023 07:01:56 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Mon, 1 May 2023 07:01:56 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Mon, 1 May 2023 07:01:52 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Juan Quintela , Peter Xu , Leonardo Bras , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Paolo Bonzini , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 2/8] migration: Add precopy initial data handshake Date: Mon, 1 May 2023 17:01:35 +0300 Message-ID: <20230501140141.11743-3-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230501140141.11743-1-avihaih@nvidia.com> References: <20230501140141.11743-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT051:EE_|CY5PR12MB6597:EE_ X-MS-Office365-Filtering-Correlation-Id: a26143c2-be10-4371-177e-08db4a4ca621 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: HLUPFQjTt2rfWhuIg7c394X4ztyIY9bO7gcnwtoGFUaaD55DIh98j1fLMROPF2/IH9BCmjJEWN6U7J0l9i2DfiN5DfBw1dufJkNPdILMit2Mbi2I6d/9YfoMyAIdc1jKqMUc0Yje3F69eYk2zollWCubZ/2ikucFECAnj4neqtgLzUSGaiazLvIeLQrDOFfSXLJMsCqZHMlSAbrOTsOT78eX4g9KqzHn8RT7nbO2lui+EYYcHMk7iWMy52AMJLSngAZbpkDJGCec/twSfq+JioiWS6+yg4I0jpgZuZM9QmU5W7flcZKcfwRbcd//jO1OtJxwU/VHqRAWMtHQbzXR4ymrQtiuM3DfA2ZzTidl+JbQn3OHmsEdEK+HQlPIp41JHS/HNuvU15HyGdrztzriVzbSeiHB3SV61NpEC0obA+Wv2gYe9VtvIGBctylVYEYetekIKJ3xZS86EWs3xDe6R3ZvtPKmxi9AhGlZ+nyzyG7pTVD9pz8TalBG/PILCBDMAspBz43T8P5hXPt9gwfKivAPj32X4MdtZ47EmulHkFhYZyLFRqDxIsIeLi+B1uKkUzU904JnOjsbZJ/3VQAoRbCiyeR7VYvYlQmBpplNTpgqf8vlnnzXT6bE6Ic6IMtvr7QdSBYg689Ns7cEY+rWM1FJ359IRhfEnQO15HV//WqNtLXero8+P/EcZaD2L9VoVLryW1sEwpJA7Ruvl0vXIw== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(39860400002)(396003)(376002)(346002)(136003)(451199021)(36840700001)(46966006)(40470700004)(2616005)(40460700003)(36860700001)(6666004)(7696005)(82310400005)(40480700001)(26005)(1076003)(186003)(36756003)(86362001)(5660300002)(83380400001)(7416002)(8676002)(8936002)(2906002)(478600001)(70206006)(70586007)(47076005)(356005)(7636003)(41300700001)(54906003)(6916009)(82740400003)(4326008)(316002)(336012)(426003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 May 2023 14:02:06.6651 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a26143c2-be10-4371-177e-08db4a4ca621 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT051.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6597 Received-SPF: softfail client-ip=2a01:111:f400:fe59::600; envelope-from=avihaih@nvidia.com; helo=NAM12-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add precopy initial data handshake between source and destination upon migration setup. The purpose of the handshake is to notify the destination that precopy initial data is used and which migration users (i.e., SaveStateEntry) are going to use it. The handshake is done in two levels. First, a general enable command is sent to notify the destination migration code that precopy initial data is used. Then, for each migration user in the source that supports precopy initial data, an enable command is sent to its counterpart in the destination: If both support it, precopy initial data will be used for them. If source doesn't support it, precopy initial data will not be used for them. If source supports it and destination doesn't, migration will be failed. To implement it, a new migration command MIG_CMD_INITIAL_DATA_ENABLE is added, as well as a new SaveVMHandlers handler initial_data_advise. Calling the handler advises the migration user that precopy initial data is used and its return value indicates whether precopy initial data is supported by it. Signed-off-by: Avihai Horon --- include/migration/register.h | 6 +++ migration/migration.h | 3 ++ migration/savevm.h | 1 + migration/migration.c | 4 ++ migration/savevm.c | 102 +++++++++++++++++++++++++++++++++++ migration/trace-events | 2 + 6 files changed, 118 insertions(+) diff --git a/include/migration/register.h b/include/migration/register.h index a8dfd8fefd..0a73f3883e 100644 --- a/include/migration/register.h +++ b/include/migration/register.h @@ -71,6 +71,12 @@ typedef struct SaveVMHandlers { int (*load_cleanup)(void *opaque); /* Called when postcopy migration wants to resume from failure */ int (*resume_prepare)(MigrationState *s, void *opaque); + + /* + * Advises that precopy initial data was requested to be enabled. Returns + * true if it's supported or false otherwise. Called both in src and dest. + */ + bool (*initial_data_advise)(void *opaque); } SaveVMHandlers; int register_savevm_live(const char *idstr, diff --git a/migration/migration.h b/migration/migration.h index 3a918514e7..4f615e9dbc 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -204,6 +204,9 @@ struct MigrationIncomingState { * contains valid information. */ QemuMutex page_request_mutex; + + /* Indicates whether precopy initial data was enabled and should be used */ + bool initial_data_enabled; }; MigrationIncomingState *migration_incoming_get_current(void); diff --git a/migration/savevm.h b/migration/savevm.h index fb636735f0..d47ab4ad18 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -58,6 +58,7 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint64_t *start_list, uint64_t *length_list); void qemu_savevm_send_colo_enable(QEMUFile *f); +void qemu_savevm_send_initial_data_enable(MigrationState *ms, QEMUFile *f); void qemu_savevm_live_state(QEMUFile *f); int qemu_save_device_state(QEMUFile *f); diff --git a/migration/migration.c b/migration/migration.c index abcadbb619..68cdf5b184 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2964,6 +2964,10 @@ static void *migration_thread(void *opaque) qemu_savevm_send_colo_enable(s->to_dst_file); } + if (migrate_precopy_initial_data()) { + qemu_savevm_send_initial_data_enable(s, s->to_dst_file); + } + qemu_savevm_state_setup(s->to_dst_file); qemu_savevm_wait_unplug(s, MIGRATION_STATUS_SETUP, diff --git a/migration/savevm.c b/migration/savevm.c index a9181b444b..2740defdf0 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -71,6 +71,13 @@ const unsigned int postcopy_ram_discard_version; +typedef struct { + uint8_t general_enable; + uint8_t reserved[7]; + uint8_t idstr[256]; + uint32_t instance_id; +} InitialDataInfo; + /* Subcommands for QEMU_VM_COMMAND */ enum qemu_vm_cmd { MIG_CMD_INVALID = 0, /* Must be 0 */ @@ -90,6 +97,8 @@ enum qemu_vm_cmd { MIG_CMD_ENABLE_COLO, /* Enable COLO */ MIG_CMD_POSTCOPY_RESUME, /* resume postcopy on dest */ MIG_CMD_RECV_BITMAP, /* Request for recved bitmap on dst */ + + MIG_CMD_INITIAL_DATA_ENABLE, /* Enable precopy initial data in dest */ MIG_CMD_MAX }; @@ -109,6 +118,8 @@ static struct mig_cmd_args { [MIG_CMD_POSTCOPY_RESUME] = { .len = 0, .name = "POSTCOPY_RESUME" }, [MIG_CMD_PACKAGED] = { .len = 4, .name = "PACKAGED" }, [MIG_CMD_RECV_BITMAP] = { .len = -1, .name = "RECV_BITMAP" }, + [MIG_CMD_INITIAL_DATA_ENABLE] = { .len = sizeof(InitialDataInfo), + .name = "INITIAL_DATA_ENABLE" }, [MIG_CMD_MAX] = { .len = -1, .name = "MAX" }, }; @@ -1036,6 +1047,40 @@ static void qemu_savevm_command_send(QEMUFile *f, qemu_fflush(f); } +void qemu_savevm_send_initial_data_enable(MigrationState *ms, QEMUFile *f) +{ + SaveStateEntry *se; + InitialDataInfo buf; + + /* Enable precopy initial data generally in the migration */ + memset(&buf, 0, sizeof(buf)); + buf.general_enable = 1; + qemu_savevm_command_send(f, MIG_CMD_INITIAL_DATA_ENABLE, sizeof(buf), + (uint8_t *)&buf); + trace_savevm_send_initial_data_enable(buf.general_enable, (char *)buf.idstr, + buf.instance_id); + + /* Enable precopy initial data for each migration user that supports it */ + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { + if (!se->ops || !se->ops->initial_data_advise) { + continue; + } + + if (!se->ops->initial_data_advise(se->opaque)) { + continue; + } + + memset(&buf, 0, sizeof(buf)); + memcpy(buf.idstr, se->idstr, sizeof(buf.idstr)); + buf.instance_id = se->instance_id; + + qemu_savevm_command_send(f, MIG_CMD_INITIAL_DATA_ENABLE, sizeof(buf), + (uint8_t *)&buf); + trace_savevm_send_initial_data_enable( + buf.general_enable, (char *)buf.idstr, buf.instance_id); + } +} + void qemu_savevm_send_colo_enable(QEMUFile *f) { trace_savevm_send_colo_enable(); @@ -2314,6 +2359,60 @@ static int loadvm_process_enable_colo(MigrationIncomingState *mis) return ret; } +static int loadvm_handle_initial_data_enable(MigrationIncomingState *mis) +{ + InitialDataInfo buf; + SaveStateEntry *se; + ssize_t read_size; + + read_size = qemu_get_buffer(mis->from_src_file, (void *)&buf, sizeof(buf)); + if (read_size != sizeof(buf)) { + error_report("%s: Could not get data buffer, read_size %ld, len %lu", + __func__, read_size, sizeof(buf)); + + return -EIO; + } + + /* Enable precopy initial data generally in the migration */ + if (buf.general_enable) { + mis->initial_data_enabled = true; + trace_loadvm_handle_initial_data_enable( + buf.general_enable, (char *)buf.idstr, buf.instance_id); + + return 0; + } + + /* Enable precopy initial data for a specific migration user */ + se = find_se((char *)buf.idstr, buf.instance_id); + if (!se) { + error_report("%s: Could not find SaveStateEntry, idstr '%s', " + "instance_id %" PRIu32, + __func__, buf.idstr, buf.instance_id); + + return -ENOENT; + } + + if (!se->ops || !se->ops->initial_data_advise) { + error_report("%s: '%s' doesn't have required " + "initial_data_advise op", + __func__, buf.idstr); + + return -EOPNOTSUPP; + } + + if (!se->ops->initial_data_advise(se->opaque)) { + error_report("%s: '%s' doesn't support precopy initial data", __func__, + buf.idstr); + + return -EOPNOTSUPP; + } + + trace_loadvm_handle_initial_data_enable(buf.general_enable, + (char *)buf.idstr, buf.instance_id); + + return 0; +} + /* * Process an incoming 'QEMU_VM_COMMAND' * 0 just a normal return @@ -2397,6 +2496,9 @@ static int loadvm_process_command(QEMUFile *f) case MIG_CMD_ENABLE_COLO: return loadvm_process_enable_colo(mis); + + case MIG_CMD_INITIAL_DATA_ENABLE: + return loadvm_handle_initial_data_enable(mis); } return 0; diff --git a/migration/trace-events b/migration/trace-events index 92161eeac5..21ae471126 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -23,6 +23,7 @@ loadvm_postcopy_ram_handle_discard_end(void) "" loadvm_postcopy_ram_handle_discard_header(const char *ramid, uint16_t len) "%s: %ud" loadvm_process_command(const char *s, uint16_t len) "com=%s len=%d" loadvm_process_command_ping(uint32_t val) "0x%x" +loadvm_handle_initial_data_enable(uint8_t general_enable, const char *idstr, int instance_id) "general_enable=%u, idstr=%s, instance_id=%u" postcopy_ram_listen_thread_exit(void) "" postcopy_ram_listen_thread_start(void) "" qemu_savevm_send_postcopy_advise(void) "" @@ -38,6 +39,7 @@ savevm_send_postcopy_run(void) "" savevm_send_postcopy_resume(void) "" savevm_send_colo_enable(void) "" savevm_send_recv_bitmap(char *name) "%s" +savevm_send_initial_data_enable(uint8_t general_enable, const char *idstr, int instance_id) "general_enable=%u, idstr=%s, instance_id=%u" savevm_state_setup(void) "" savevm_state_resume_prepare(void) "" savevm_state_header(void) "" From patchwork Mon May 1 14:01:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13227516 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DADDDC77B73 for ; Mon, 1 May 2023 14:03:23 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ptU6u-00052h-H7; Mon, 01 May 2023 10:02:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU6a-0004bp-Dx for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:16 -0400 Received: from mail-bn8nam12on20630.outbound.protection.outlook.com ([2a01:111:f400:fe5b::630] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU6Y-0006cu-12 for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:16 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=H2Bam4i0932SjmwBWYuohhHoWqGYl59oFay7o7xr1VMdCyDJPYOXanBItGEKQ1XgHZr27Dw2VYHzN0kPrJAqDeK4PQZ+sxbnIhZf0zrXezQnu51E7n7caI7+cJQENEDIA5B4okrJDXg1KqFK2OELgiJCWCu9zcio5F0Ktk0IgbEmNQbytgxfl5zwjw6fYmWz+cgssrQf54R76mTMcrbVeQcvyHgReHF3yln6Sx+ZkTbIWqRi5AjKejqqdcw/3gJYwNMqBLKC12LhOWTc/iAdsM0z7hWAq83CFZhivGP39aRKU328rqm58Mn5mrpw0/xZYgj2quNuZIFMWTVjrinlvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nkC3MkkLTfP/MUY8J0V0d2aqwjxvhrrgyVYHS0AZowM=; b=eoFAvmYVZu7FneWdl/JBr8WUUdPi6JY5qhNmhG+InUIYRD4llIJqY5PyrP3HAnXZOKGvlbC+pwidi1dTFKk4Wj41xAWiBtOWyFBWDJpUpYt34xbBiMlpnlip5NjOlGvCnT++yrB7lk0GUpiZPcedOs/9D7ARNjyTdnM4S3JF6VkHP4LeP2zQZSlobuelk0WwqelhUpu7b0mxJ0vd0ytzqINepi88ZzjBYnjqKnmKJWKETIYszGVz1i7VfXRGXMB4Y/Ut6eDhHY687iINvXs23wf6UNcFdGIrKQwsEuU+i/ScSzfz4LjwWVoqrIw+3ZzCeMVfA8Eqx8tVhX/2Rgm/kQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nkC3MkkLTfP/MUY8J0V0d2aqwjxvhrrgyVYHS0AZowM=; b=KJFU8QmKBkaBKrblGXJRix1r3U+hrzNPzbG0EuhmY1SovEZXqa2KbFu0A4ReyVBjsmaImT09YrvUtRywoGJjnKLkrSrcNpc96vYMhDO5+vZnoMOjwtWkly2k6FubrYc0JbB3yQBGlvUhfP2kjz3nXYaD0LKcGTYLBxO78tc7V3xg6h3LNxuIYFhmem73UI975nC/i+4AuJ45wauGAfNBipvegce3tz7CaLIAXA8pLUi7f49iinZCeRbEY1Ba6ai8EIy/d2ziWZw569BvZdx9vkxGB1IVBJ+2qPRltLj0fc5FyPeP7q3a+XoQm8WfEijY6CXFr4CFdI0CqeXyxK/RgQ== Received: from DM5PR07CA0066.namprd07.prod.outlook.com (2603:10b6:4:ad::31) by CH2PR12MB4072.namprd12.prod.outlook.com (2603:10b6:610:7e::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.30; Mon, 1 May 2023 14:02:10 +0000 Received: from DM6NAM11FT051.eop-nam11.prod.protection.outlook.com (2603:10b6:4:ad:cafe::ac) by DM5PR07CA0066.outlook.office365.com (2603:10b6:4:ad::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.30 via Frontend Transport; Mon, 1 May 2023 14:02:10 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT051.mail.protection.outlook.com (10.13.172.243) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.19 via Frontend Transport; Mon, 1 May 2023 14:02:09 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Mon, 1 May 2023 07:02:01 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Mon, 1 May 2023 07:02:01 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Mon, 1 May 2023 07:01:56 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Juan Quintela , Peter Xu , Leonardo Bras , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Paolo Bonzini , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 3/8] migration: Add precopy initial data loaded ACK functionality Date: Mon, 1 May 2023 17:01:36 +0300 Message-ID: <20230501140141.11743-4-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230501140141.11743-1-avihaih@nvidia.com> References: <20230501140141.11743-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT051:EE_|CH2PR12MB4072:EE_ X-MS-Office365-Filtering-Correlation-Id: 561df61f-d7ed-4432-f84b-08db4a4ca810 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /5CQD6PkrZe5pgSHd93AYxh0gzsu+BAT4wJ9fubAXoAaNG1Hdg9zLGQLDaWPNkqJ3wZj6sHVradsnPKmDAmgcpwGsxpvPJkMa4e7JlMKHvf00ADoHIU6nkdPQAV5TAtOW/AagovtVoZ4kmDypShNMFablJcMHB/TCjPyKUXsxwNpuRBWFjCMO4KT7KE/6ZeURJutEpkr68MYAXavdK60Rlb+4vc5MoHJZB2+c9GEjahUyILuJs7XLUOfMNqzdZbhoEbV4qrOm0zGSiMPdjRIjYwpyBNUR9MwYFJCbq8AmA2xtkfC2jBbImM4YqZd4BO0NQqzg3iARFuXIaNxm46d1773KXe/N7QdGH32XE7Q/blalPs9BCml+LDPglHCyoqiSaCRImaW1b+vm/Egnwo71T9vKKTn0hkIBgjCYJSYoAm775ed6O/MDK5GTa5Su76/f19DBJWgt+s7ZxT9n3rRzpro76WotrRD9YgHA84gBczb/9LseJHmI5ZzHHaH8K2Jyqqhxq08mtkFslYpHSbfx9p31qQ2WAdAwb2QpZ23WED17IB01JHaFuc6Ra2JJhMBJHeDa5ol9Ar751XEdo1TqcKVpf+s64ZduTfJannmpdxoCc3eA8gwbzlGFpDVpIyMBkO90lHAmmIUFb47cLm+DVrCYm6vVo4nonA/lhdlqY9T/qbJ7KPWnzPtGhncvdquwbTDqz6ongAo4MMF7d51UA== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(136003)(376002)(39860400002)(346002)(396003)(451199021)(36840700001)(40470700004)(46966006)(82310400005)(86362001)(36756003)(7636003)(82740400003)(356005)(7416002)(8676002)(5660300002)(4326008)(8936002)(316002)(6916009)(41300700001)(70206006)(2906002)(40480700001)(70586007)(83380400001)(426003)(336012)(40460700003)(2616005)(47076005)(36860700001)(6666004)(7696005)(54906003)(478600001)(26005)(186003)(1076003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 May 2023 14:02:09.9305 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 561df61f-d7ed-4432-f84b-08db4a4ca810 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT051.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4072 Received-SPF: softfail client-ip=2a01:111:f400:fe5b::630; envelope-from=avihaih@nvidia.com; helo=NAM12-BN8-obe.outbound.protection.outlook.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add the core functionality of precopy initial data, which allows the destination to ACK that initial data has been loaded and the source to wait for this ACK before completing the migration. A new return path command MIG_RP_MSG_INITIAL_DATA_LOADED_ACK is added. It is sent by the destination after precopy initial data is loaded to ACK to the source that precopy initial data has been loaded. In addition, two new SaveVMHandlers handlers are added: 1. is_initial_data_active which indicates whether precopy initial data is used for this migration user (i.e., SaveStateEntry). 2. initial_data_loaded which indicates whether precopy initial data has been loaded by this migration user. Signed-off-by: Avihai Horon --- include/migration/register.h | 7 ++++++ migration/migration.h | 12 +++++++++++ migration/migration.c | 41 ++++++++++++++++++++++++++++++++++-- migration/savevm.c | 39 ++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 5 files changed, 99 insertions(+), 2 deletions(-) diff --git a/include/migration/register.h b/include/migration/register.h index 0a73f3883e..297bbe9f73 100644 --- a/include/migration/register.h +++ b/include/migration/register.h @@ -77,6 +77,13 @@ typedef struct SaveVMHandlers { * true if it's supported or false otherwise. Called both in src and dest. */ bool (*initial_data_advise)(void *opaque); + /* + * Checks if precopy initial data is active. If it's inactive, + * initial_data_loaded check is skipped. + */ + bool (*is_initial_data_active)(void *opaque); + /* Checks if precopy initial data has been loaded in dest */ + bool (*initial_data_loaded)(void *opaque); } SaveVMHandlers; int register_savevm_live(const char *idstr, diff --git a/migration/migration.h b/migration/migration.h index 4f615e9dbc..d865c23d87 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -207,6 +207,11 @@ struct MigrationIncomingState { /* Indicates whether precopy initial data was enabled and should be used */ bool initial_data_enabled; + /* + * Indicates whether an ACK that precopy initial data was loaded has been + * sent to source. + */ + bool initial_data_loaded_ack_sent; }; MigrationIncomingState *migration_incoming_get_current(void); @@ -435,6 +440,12 @@ struct MigrationState { /* QEMU_VM_VMDESCRIPTION content filled for all non-iterable devices. */ JSONWriter *vmdesc; + + /* + * Indicates whether an ACK that precopy initial data was loaded in + * destination has been received. + */ + bool initial_data_loaded_acked; }; void migrate_set_state(int *state, int old_state, int new_state); @@ -475,6 +486,7 @@ int migrate_send_rp_message_req_pages(MigrationIncomingState *mis, void migrate_send_rp_recv_bitmap(MigrationIncomingState *mis, char *block_name); void migrate_send_rp_resume_ack(MigrationIncomingState *mis, uint32_t value); +int migrate_send_rp_initial_data_loaded_ack(MigrationIncomingState *mis); void dirty_bitmap_mig_before_vm_start(void); void dirty_bitmap_mig_cancel_outgoing(void); diff --git a/migration/migration.c b/migration/migration.c index 68cdf5b184..304cab2fa1 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -77,6 +77,11 @@ enum mig_rp_message_type { MIG_RP_MSG_RECV_BITMAP, /* send recved_bitmap back to source */ MIG_RP_MSG_RESUME_ACK, /* tell source that we are ready to resume */ + MIG_RP_MSG_INITIAL_DATA_LOADED_ACK, /* + * Tell source precopy initial data is + * loaded. + */ + MIG_RP_MSG_MAX }; @@ -756,6 +761,12 @@ bool migration_has_all_channels(void) return true; } +int migrate_send_rp_initial_data_loaded_ack(MigrationIncomingState *mis) +{ + return migrate_send_rp_message(mis, MIG_RP_MSG_INITIAL_DATA_LOADED_ACK, 0, + NULL); +} + /* * Send a 'SHUT' message on the return channel with the given value * to indicate that we've finished with the RP. Non-0 value indicates @@ -1401,6 +1412,8 @@ void migrate_init(MigrationState *s) s->vm_was_running = false; s->iteration_initial_bytes = 0; s->threshold_size = 0; + + s->initial_data_loaded_acked = false; } int migrate_add_blocker_internal(Error *reason, Error **errp) @@ -1717,6 +1730,9 @@ static struct rp_cmd_args { [MIG_RP_MSG_REQ_PAGES_ID] = { .len = -1, .name = "REQ_PAGES_ID" }, [MIG_RP_MSG_RECV_BITMAP] = { .len = -1, .name = "RECV_BITMAP" }, [MIG_RP_MSG_RESUME_ACK] = { .len = 4, .name = "RESUME_ACK" }, + [MIG_RP_MSG_INITIAL_DATA_LOADED_ACK] = { .len = 0, + .name = + "INITIAL_DATA_LOADED_ACK" }, [MIG_RP_MSG_MAX] = { .len = -1, .name = "MAX" }, }; @@ -1955,6 +1971,11 @@ retry: } break; + case MIG_RP_MSG_INITIAL_DATA_LOADED_ACK: + ms->initial_data_loaded_acked = true; + trace_source_return_path_thread_initial_data_loaded_ack(); + break; + default: break; } @@ -2704,6 +2725,20 @@ static void migration_update_counters(MigrationState *s, bandwidth, s->threshold_size); } +static bool initial_data_loaded_acked(MigrationState *s) +{ + if (!migrate_precopy_initial_data()) { + return true; + } + + /* No reason to wait for precopy initial data loaded ACK if VM is stopped */ + if (!runstate_is_running()) { + return true; + } + + return s->initial_data_loaded_acked; +} + /* Migration thread iteration status */ typedef enum { MIG_ITERATE_RESUME, /* Resume current iteration */ @@ -2719,6 +2754,7 @@ static MigIterateState migration_iteration_run(MigrationState *s) { uint64_t must_precopy, can_postcopy; bool in_postcopy = s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE; + bool initial_data_loaded = initial_data_loaded_acked(s); qemu_savevm_state_pending_estimate(&must_precopy, &can_postcopy); uint64_t pending_size = must_precopy + can_postcopy; @@ -2731,7 +2767,8 @@ static MigIterateState migration_iteration_run(MigrationState *s) trace_migrate_pending_exact(pending_size, must_precopy, can_postcopy); } - if (!pending_size || pending_size < s->threshold_size) { + if ((!pending_size || pending_size < s->threshold_size) && + initial_data_loaded) { trace_migration_thread_low_pending(pending_size); migration_completion(s); return MIG_ITERATE_BREAK; @@ -2739,7 +2776,7 @@ static MigIterateState migration_iteration_run(MigrationState *s) /* Still a significant amount to transfer */ if (!in_postcopy && must_precopy <= s->threshold_size && - qatomic_read(&s->start_postcopy)) { + initial_data_loaded && qatomic_read(&s->start_postcopy)) { if (postcopy_start(s)) { error_report("%s: postcopy failed to start", __func__); } diff --git a/migration/savevm.c b/migration/savevm.c index 2740defdf0..7a94deda3b 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -2504,6 +2504,39 @@ static int loadvm_process_command(QEMUFile *f) return 0; } +static int qemu_loadvm_initial_data_loaded_ack(MigrationIncomingState *mis) +{ + SaveStateEntry *se; + int ret; + + if (!mis->initial_data_enabled || mis->initial_data_loaded_ack_sent) { + return 0; + } + + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { + if (!se->ops || !se->ops->initial_data_loaded) { + continue; + } + + if (!se->ops->is_initial_data_active || + !se->ops->is_initial_data_active(se->opaque)) { + continue; + } + + if (!se->ops->initial_data_loaded(se->opaque)) { + return 0; + } + } + + ret = migrate_send_rp_initial_data_loaded_ack(mis); + if (!ret) { + mis->initial_data_loaded_ack_sent = true; + trace_loadvm_initial_data_loaded_acked(); + } + + return ret; +} + /* * Read a footer off the wire and check that it matches the expected section * @@ -2826,6 +2859,12 @@ retry: if (ret < 0) { goto out; } + + ret = qemu_loadvm_initial_data_loaded_ack(mis); + if (ret < 0) { + goto out; + } + break; case QEMU_VM_COMMAND: ret = loadvm_process_command(f); diff --git a/migration/trace-events b/migration/trace-events index 21ae471126..a0e1d3b2fd 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -24,6 +24,7 @@ loadvm_postcopy_ram_handle_discard_header(const char *ramid, uint16_t len) "%s: loadvm_process_command(const char *s, uint16_t len) "com=%s len=%d" loadvm_process_command_ping(uint32_t val) "0x%x" loadvm_handle_initial_data_enable(uint8_t general_enable, const char *idstr, int instance_id) "general_enable=%u, idstr=%s, instance_id=%u" +loadvm_initial_data_loaded_acked(void) "" postcopy_ram_listen_thread_exit(void) "" postcopy_ram_listen_thread_start(void) "" qemu_savevm_send_postcopy_advise(void) "" @@ -182,6 +183,7 @@ source_return_path_thread_loop_top(void) "" source_return_path_thread_pong(uint32_t val) "0x%x" source_return_path_thread_shut(uint32_t val) "0x%x" source_return_path_thread_resume_ack(uint32_t v) "%"PRIu32 +source_return_path_thread_initial_data_loaded_ack(void) "" migration_thread_low_pending(uint64_t pending) "%" PRIu64 migrate_transferred(uint64_t tranferred, uint64_t time_spent, uint64_t bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %" PRIu64 " max_size %" PRId64 process_incoming_migration_co_end(int ret, int ps) "ret=%d postcopy-state=%d" From patchwork Mon May 1 14:01:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13227514 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 49E8AC77B73 for ; Mon, 1 May 2023 14:03:21 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ptU6u-00052D-FF; Mon, 01 May 2023 10:02:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU6f-0004ke-1i for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:22 -0400 Received: from mail-dm6nam10on20606.outbound.protection.outlook.com ([2a01:111:f400:7e88::606] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU6d-0006dp-2t for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:20 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=T0UEIZLBqilG84roc74jwGpvPUR3NNEsSWjtwkp0Hzmha843N/qkKWKv2BB9Ot9bok2rPA5m9I02CLkTzrUGUExt2dJ+6YvDrgydUYLYbB5skLbHu9UPSAB2VTLg1am9GALcR6xAUfebaktfIPsY/VQDaq6Lhvbrpr2fMa2FliVQGwP/D0BPUBOngB4uulqpQgdN76JLqC5asIoMNgUN/ZaXOVOyhddAFD4gd/ZJeuWTmP9cN8WThm2pieJCu/ZLSfv1lz5FQse9+8BUd7MahqxMUiGPwhW1Ft5cLWvulimopp9n2ktTDKd1h/j0X8bChc6hFaEXY6jdXeFO0eMNPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=i1GiRRZFWNAqz0aOicqY70qhCawjtdnOioRW4j5drN4=; b=HrA8n6EktpstAl2HSlwL2loqdobl8JtG1eh0Ftovrd+eqjKRVbJzDNVMBXjPBmddfk3nsAZDDi4vaM/isBIxMjayqdm5Qbpqo8izTa5s6OJrcc/j8jAN+YjaB8o+IC38drD0WCCtNKM6EDvFrDFJ3u/88jo47Nn06GgmQqDMPcJ7xEt43eeMKbYqSiTqkKP94NnCAwyG7GjbsqMNstplT41ex83b6+M7xntUbTzUSxzzlUZ2edhhIUV9teIGJlGclIMIhpghZywoflrVMzsxrroo+v1veJRcNqZPclh1tJfbbExCJ+zfYgbepsAo17Lf+JEvo0PjcvFMPsRTm2q0sA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=i1GiRRZFWNAqz0aOicqY70qhCawjtdnOioRW4j5drN4=; b=r+XYXSanN7C3Sv3zTfmU9uwkvvpPZ5QFe9qUIidGej6T7AhaUKqZ6L93MwCV/gKFjN8pbPm7zacYXE5586Xy6XZ+Jz0VGJIq1HKJKRCdmNPrxAC0yVTgSzqr6g21ahSPxlcWHDL7XAmirxBE/+ZbYhC5XERkaYzJcDtqWLZHBBH82p2YKGa3c8XGO9cwxxBVmVvIUHkhp5AQdrydu/uxxocLniGzL47olfFCM2bxiJa9OARxtVMlyrBgqCzYFe1QjHDILQh/0E3PU4fBJrdharbFeRPvt/7LI2Sw4Ypi8QfpjuNhf/Ljvi6xj0358CngYcZ+Zba7hJ7wHOUOUVeXng== Received: from DM6PR08CA0060.namprd08.prod.outlook.com (2603:10b6:5:1e0::34) by SN7PR12MB7786.namprd12.prod.outlook.com (2603:10b6:806:349::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.30; Mon, 1 May 2023 14:02:14 +0000 Received: from DM6NAM11FT029.eop-nam11.prod.protection.outlook.com (2603:10b6:5:1e0:cafe::e3) by DM6PR08CA0060.outlook.office365.com (2603:10b6:5:1e0::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.31 via Frontend Transport; Mon, 1 May 2023 14:02:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT029.mail.protection.outlook.com (10.13.173.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.20 via Frontend Transport; Mon, 1 May 2023 14:02:14 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Mon, 1 May 2023 07:02:06 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Mon, 1 May 2023 07:02:06 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Mon, 1 May 2023 07:02:01 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Juan Quintela , Peter Xu , Leonardo Bras , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Paolo Bonzini , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 4/8] migration: Enable precopy initial data capability Date: Mon, 1 May 2023 17:01:37 +0300 Message-ID: <20230501140141.11743-5-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230501140141.11743-1-avihaih@nvidia.com> References: <20230501140141.11743-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT029:EE_|SN7PR12MB7786:EE_ X-MS-Office365-Filtering-Correlation-Id: 78b0daa7-d662-42ac-ba95-08db4a4caabe X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: OyaITLCJJLvBvdvvCPAI9JHYzqqAkPzhpKXU5jrSyAkmG/+jl9ZO+MFcjfEmA33HdylwbrE+zxgLiFhtbcjQbAxIANMtlmS+krKzNeuD2Z2rjYePUiJtheZfDDZtAzwjyiFXhRuSzaMQnR5csKRSxhCnIv48iXx+X/KJwJoUS0KwI3DjhnFw02yipnoJ+SrnoOrmg9A1vEsgJsdLNVKCmK0IN6JTnPd0oft8qLIgv+WT85IZ6RxKsXUmjETLy6f1X1AJtbSX/5FTW8ip0DGZJtaFoQFeJCj7VE5ihBTHYg2b1uqYPz0xpSQRWMku3NlUXTccFDL27tnTi/ksSX777vKGtGenIAeHgaMWUdVpabYaUKWa5x2VHGD6DNpRUHFIYK9VR7t9g7KfSLZkvOvmrclStyeTMDMuxJ6s3z8LfVll0UK5ZMdWMyPfovJXeYuTm0GrfgpcXqNG/94yGZjF9RIkQdUcBFjJ4Y/rFfN1ObdfXkg+TDRePVWpI/JtwilMWacx8NaCcoKKbnLQQO8at6QO5UtRt22jCaN6jeBINsf/HEQCeSDkrYUjDryRWY5YKZSDVsJcrcLIVIT+1A0sgklIlPRfLn/XQlDfzSHsmm7sqEKsHiaybHnvyUuUsDSk8skCPies/hqYhRxtqQR6savIyu98eWlTn9a0y3HoKrdFQJgoKA0NxO1L/eDkWWjAefDUASPPiRwWP1mAOfiUgw== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(346002)(39860400002)(396003)(136003)(376002)(451199021)(36840700001)(46966006)(40470700004)(40480700001)(40460700003)(316002)(1076003)(82310400005)(26005)(2616005)(41300700001)(336012)(426003)(47076005)(83380400001)(36860700001)(478600001)(186003)(70206006)(70586007)(54906003)(7696005)(6666004)(4326008)(6916009)(4744005)(82740400003)(7636003)(356005)(2906002)(36756003)(5660300002)(86362001)(7416002)(8936002)(8676002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 May 2023 14:02:14.4091 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 78b0daa7-d662-42ac-ba95-08db4a4caabe X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT029.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7786 Received-SPF: softfail client-ip=2a01:111:f400:7e88::606; envelope-from=avihaih@nvidia.com; helo=NAM10-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Now that precopy initial data logic has been implemented, enable the capability. Signed-off-by: Avihai Horon Reviewed-by: Juan Quintela --- migration/options.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/migration/options.c b/migration/options.c index c4ef0c60c7..77a570f866 100644 --- a/migration/options.c +++ b/migration/options.c @@ -560,10 +560,6 @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp) error_setg(errp, "Precopy initial data requires return path"); return false; } - - /* Disable this capability until it's implemented */ - error_setg(errp, "Precopy initial data is not implemented yet"); - return false; } return true; From patchwork Mon May 1 14:01:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13227517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F2638C77B73 for ; Mon, 1 May 2023 14:03:34 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ptU73-0005Su-VK; Mon, 01 May 2023 10:02:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU6k-0004wn-P1 for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:28 -0400 Received: from mail-dm6nam11on20606.outbound.protection.outlook.com ([2a01:111:f400:7eaa::606] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU6i-0006ff-Tb for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:26 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IkLx7hBw5i4178nxMMZNRarLZi4zbbBpP/aCp53vke9zKJh/v009IdUUpN8TFP6NZ0ieyZNpNL8VOrVDBhxqC/Pi3obvcdfiq+dTTRNWw/LUB6cwxd+P6dFlrp7Q3GEzv+ixj7n6BNZXyV4Wg4SGtvyjOrCp7vaTiN9D5nVUE8++bZ5D2frQOc8Dfeb6ErdFnpmPxpJLi6TkMrrrhlVnIejCpJP2X8NGAuIBNMjg3I5Z6YPtIgbeR1uH/YHTTsS36cS7RtXKF0u26uRubPraSHfv+G3D9V+wpRm+YJwA97cXAcXybPptFFQcEONIcHD4fNlKaZPKWCYPLx+GrNdCuA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+15kJovdY8T0+ENdJBkufdN6PfdXoc7VUs7mtrJnV3M=; b=PJuSydyl5aEwLRgilDeOyHKv23v7rahLR4IX9GyZW8iNnHiUjVshTG9U4TmhXQR+86FChA8w4Tk2M36fwf60CoN31ZXnjd9mdbqvTqX1hz5DCK44WqmxyK9KjtZnER0rrx3Z59a1v8OeXnMUEwHRbeYxqmQiiBoEoKYy4rQJiD7/NR++WemnCIjqv7+SpR+LXHV6fLnlf3S0qtiJfbyBPZ46kTQfVx7SM/XhJczX2gjuPwjluY+Jv/ZbF0Aq2syth+8YUOu51vvJla/KBxk8LiU83l405JngKYSuQ4yUMqtxVAKhCI9XChkeCOgTcD68w2S8eLFaVwOW0PQdy00QDg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+15kJovdY8T0+ENdJBkufdN6PfdXoc7VUs7mtrJnV3M=; b=p0Ad64S3tXPuZMuOqsIysBEHUWfNTs2FQeMrfp49NsEmN8yUP70TaB/B0ngB6N57DXGXvbcUO1A85IGxZ8thknGj3uulZQcpCl62GghPKf0iSkwysfflpwoFlBbKmjwrtQcKr+0UrMOZ39QJzMs1acXExyIGawX7GWWIVFrd/qxzRgFrPxZgJEeuFetZTsrI6I+glvn/iqyyIHtzphkgZxdaisZRbwUwHupxnPF7xtxhFQLKLWIJ3rfV7lEWL0CnMLOsso5HAYtlkFoX2ni40kR2R0ytyN6Sh90PSEHAA4VHDcPBfPeWLylQGL2p5Mdtd/t1eIDkX3TrI76JVZ0VLA== Received: from DM5PR07CA0089.namprd07.prod.outlook.com (2603:10b6:4:ae::18) by IA1PR12MB7519.namprd12.prod.outlook.com (2603:10b6:208:418::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.30; Mon, 1 May 2023 14:02:19 +0000 Received: from DM6NAM11FT103.eop-nam11.prod.protection.outlook.com (2603:10b6:4:ae:cafe::af) by DM5PR07CA0089.outlook.office365.com (2603:10b6:4:ae::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.31 via Frontend Transport; Mon, 1 May 2023 14:02:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT103.mail.protection.outlook.com (10.13.172.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.19 via Frontend Transport; Mon, 1 May 2023 14:02:19 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Mon, 1 May 2023 07:02:11 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Mon, 1 May 2023 07:02:11 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Mon, 1 May 2023 07:02:06 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Juan Quintela , Peter Xu , Leonardo Bras , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Paolo Bonzini , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 5/8] tests: Add migration precopy initial data capability test Date: Mon, 1 May 2023 17:01:38 +0300 Message-ID: <20230501140141.11743-6-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230501140141.11743-1-avihaih@nvidia.com> References: <20230501140141.11743-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT103:EE_|IA1PR12MB7519:EE_ X-MS-Office365-Filtering-Correlation-Id: 9a9ccc00-aec8-4357-624f-08db4a4cad95 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 60VlgvfklhUHp3kzd4ee0kyzbBl/6oPAyR2E/nxwCj539+jfIOgVt5rLQN1qkrYKNpR4xXcfn9pcJBYdZeXrdms1JX6VolA6TuPKfBCDF9poiah3o1uCxhLtD6zdU69U/yvIjmujXuqLasOFEXLU6++t+TsD/MA8obEzso24m4WClfeeNXNosVrW19IWCnRB6BRKrZb/xikq/y5f16vvijS6wYi1as1wnXw7CjNfe0gxEVcUQFJiEcHJ+grYuXKSf6zo6BOsSeNnVuB6l6tZIDJwNzcflFTAAoLZ7exWDEdh/S6QPsqvxa84fT7GvcxSS9voSlkm8E2YOrvrAq0IuWOmOckoHNbE0hgn51yD3Ob7BIddtntbA9FAQOYg4UabYWTe+WxKSHRA5csQgLTLOc0/YcSyrs+xbloPbARN1/mqdwIl+3IH6Uj4hz4Xg1NZ+k/Jq0iQ3R+xL99SEkZOpWdMq0N4Z8QNCOKTqHMi4+nqeWxFwZTbSRFfFt1uGnbn5Hu9VlGUr766FQ1f7Npyd9si3U371EKU3QqJm5G5Z3vNnFZOkipZpzqaoB1W5BLR3pUJLeCnxpy+cOOyROQWGq5h+Hd5qwy/XbI4AMjh9Ug/iPp9u789iuCyrrDfytojb3RhGO1swGeWueguU2lkPIsSSgRrg0nQT9B4IEuIUZjU14WjVzXMW7MSSOzRVb843pS+AivNkMHXx3DSwgUIQA== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(346002)(136003)(396003)(376002)(39850400004)(451199021)(36840700001)(46966006)(40470700004)(70206006)(70586007)(2906002)(426003)(86362001)(2616005)(82310400005)(7416002)(6916009)(5660300002)(8676002)(8936002)(40460700003)(36756003)(41300700001)(316002)(40480700001)(4326008)(6666004)(54906003)(478600001)(7696005)(186003)(26005)(1076003)(82740400003)(356005)(7636003)(47076005)(336012)(36860700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 May 2023 14:02:19.1713 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9a9ccc00-aec8-4357-624f-08db4a4cad95 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT103.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB7519 Received-SPF: softfail client-ip=2a01:111:f400:7eaa::606; envelope-from=avihaih@nvidia.com; helo=NAM11-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add migration precopy initial data capability test. The test runs without migration users that support this capability, but is still useful to make sure it didn't break anything. Signed-off-by: Avihai Horon Reviewed-by: Juan Quintela --- tests/qtest/migration-test.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 60dd53d3ec..71d30bd330 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -1533,6 +1533,25 @@ static void test_precopy_tcp_plain(void) test_precopy_common(&args); } +static void *test_migrate_initial_data_start(QTestState *from, QTestState *to) +{ + + migrate_set_capability(from, "return-path", true); + migrate_set_capability(from, "precopy-initial-data", true); + + return NULL; +} + +static void test_precopy_tcp_initial_data(void) +{ + MigrateCommon args = { + .listen_uri = "tcp:127.0.0.1:0", + .start_hook = test_migrate_initial_data_start, + }; + + test_precopy_common(&args); +} + #ifdef CONFIG_GNUTLS static void test_precopy_tcp_tls_psk_match(void) { @@ -2557,6 +2576,10 @@ int main(int argc, char **argv) #endif /* CONFIG_GNUTLS */ qtest_add_func("/migration/precopy/tcp/plain", test_precopy_tcp_plain); + + qtest_add_func("/migration/precopy/tcp/plain/precopy-initial-data", + test_precopy_tcp_initial_data); + #ifdef CONFIG_GNUTLS qtest_add_func("/migration/precopy/tcp/tls/psk/match", test_precopy_tcp_tls_psk_match); From patchwork Mon May 1 14:01:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13227518 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BF2D5C77B73 for ; Mon, 1 May 2023 14:03:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ptU6x-0005Ck-Ro; Mon, 01 May 2023 10:02:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU6t-0004zn-I6 for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:36 -0400 Received: from mail-dm6nam12on20626.outbound.protection.outlook.com ([2a01:111:f400:fe59::626] helo=NAM12-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU6r-0006gi-FE for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:34 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QT7YUeB5oQBo5oVFqNV1qZiCdkA5JjCR8/kxc4JHyoJ/L9lg6HmVsFEGE7PBvdbEgCk+55O9HfHXxfEo9sGfi2bAKlVb6yVHrCNBs3lB7pL2ptJ/2hh8IdhxaLRwFS2w0BTMWKRJM0ShQREb+v9VhERkWKDSWakf8G6NhBVmIn7MEHD2ng98KEREV2q1JlHicsjOA0qhigXJdcTmDCVTJTyZBgezmuRRzrNsQMf3qN59ZR4N6jl4Chr8DNnEjabuSfZAAfXu4gsw5iIb0e/skZSHGPq6c+T/k2Tb3Ql0VK6WBKID59cFMhrc4N3k7duy7PUfMtnv144j4QanPFRXQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+mlkwswwEYDXeDfwevrgq1hri9O7HNKvLwcLCvotfqE=; b=fqxAmJj4UncWfBlXeNTEqTORIWpnR0UBEPc/gj7fmncMEKLaNIF2kILCMuC2l+UCeeHRm1xlr0u7VmeKCPC27iiay8EIxCAFmbyqPOXj3FRQZvE/AVKTLVRcSRAK0Hw7ssihhWjvV39z6ktW86zU+LWx/X3hy1eylOg6oJC6uPI6mU4ty0XxGoDNaH+ktHN/djItqJ7lhXBxDFLREKwlnzJd5EGU2FeLaqRpwF2ZCEn6x/UikPP4U5geVVCNxvvS4TiC0QDebaVcqlU0e5wsPAFke+UomyqQM805+ehShSjZ4xZtOgRVRc0OxtyUfcLdIhFl36+cpjHc24PI6byfOQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+mlkwswwEYDXeDfwevrgq1hri9O7HNKvLwcLCvotfqE=; b=aQVDGcF6tVbpt+55uRFy106wwFu6kwRLboxCSQ00YhLfkyowXuUPAt4VSEGsEJAZmFRquCl9ZicLQjTi5rozgK/bCK3brf/Tt//12EwNBE+Hh9uAwQe9K9/JzIcqREamZ5EKRtLBn3CxqyslqXXc/ApmcfojLdHeN8ZRH+KgoKtfGKb4LjItrGOUSSA0zAKJ3Ycbi8Av3Cs46lBKZUyvz4+PgKIDuX5zcx1+HC9Eh3TGedo8wrQvBVGHpSaHsQG7/obLtpMHiyAGi+2AT6gT7h54MkbWGPLdcM6dKh9VNUYhImO9/JJ8KusSUFo/9pvlSXzdNlgGOaKaAv2fZmZgAQ== Received: from BN7PR06CA0067.namprd06.prod.outlook.com (2603:10b6:408:34::44) by SA1PR12MB8843.namprd12.prod.outlook.com (2603:10b6:806:379::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.30; Mon, 1 May 2023 14:02:29 +0000 Received: from BN8NAM11FT037.eop-nam11.prod.protection.outlook.com (2603:10b6:408:34:cafe::79) by BN7PR06CA0067.outlook.office365.com (2603:10b6:408:34::44) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.30 via Frontend Transport; Mon, 1 May 2023 14:02:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN8NAM11FT037.mail.protection.outlook.com (10.13.177.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.20 via Frontend Transport; Mon, 1 May 2023 14:02:28 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Mon, 1 May 2023 07:02:16 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Mon, 1 May 2023 07:02:16 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Mon, 1 May 2023 07:02:11 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Juan Quintela , Peter Xu , Leonardo Bras , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Paolo Bonzini , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 6/8] vfio/migration: Refactor vfio_save_block() to return saved data size Date: Mon, 1 May 2023 17:01:39 +0300 Message-ID: <20230501140141.11743-7-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230501140141.11743-1-avihaih@nvidia.com> References: <20230501140141.11743-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT037:EE_|SA1PR12MB8843:EE_ X-MS-Office365-Filtering-Correlation-Id: f42f8601-f87d-4a8b-c277-08db4a4cb368 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 9hH+djy1OZDEEZOV+8QOOcxUS8khNPkeZSRKj54yWS9IG7cDhzWLf7wGsuFUC6ahg3TG4Rh1bpS538mK2w92aqBT91iBi9HB42ZG3N39Z0frifoR56AMkVyp7dHo4PRu2s58mkg/Olnt3jTmqt2O7b53eTvi+Uqa5daFOmsD4vBDkwDJ4jlOW/4sOCK8PWQruhkdrXjLHZoxGxCosofLnWBCBB4s1H9yEHw+jc+I9oEaroo7Sg5VNkrl3t6dhDCU0pc3m8IcYhJ58oT7JNAHScRWBeJto31khtB/n8cUo5LYhSCVgCqca3yl36ywzgCAOrQuOpsHxDbi/aJyID81IaHwvGoBdXY1DoqDAKsXl4qd1kyPL6qJQmx+umotwxhNeOO87ivm0yZwUEiiUJpKfKucHi6kWHE7fsBtT0siMgV566w8qK7Ie/VNYHbIARMs+bKEd5P5JUj7dcX1uiQsU6WckCoSZXaN9WCcPh2A60jYiqd5+rddPQ3/elUnS1yb6OtoIOoD4F6IHOuFZOnoV6TEEZehkqM1UsQ5R9YzCr117xcsGMy9Y0B4XkIJ1zaw4vAd4D7oixchUgrCDh4GiYZQXhxtg0sKDKH3dnXBn7hhqr4Gw+ump0Su9hQ+WBas0WOdc5IB73O4/pvLHZmngjew5jy6tPk7DDfDBSPwY7/iqUGxxNLbIKO4gko4vVIDgqp9qG2l8oLJsEM5xaATxQ== X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(346002)(39860400002)(136003)(376002)(396003)(451199021)(40470700004)(36840700001)(46966006)(36756003)(40480700001)(478600001)(70206006)(70586007)(54906003)(6916009)(4326008)(336012)(316002)(40460700003)(426003)(26005)(2616005)(6666004)(186003)(1076003)(82310400005)(7696005)(2906002)(8936002)(41300700001)(5660300002)(8676002)(7416002)(7636003)(356005)(82740400003)(86362001)(83380400001)(47076005)(36860700001)(66574015); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 May 2023 14:02:28.8983 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f42f8601-f87d-4a8b-c277-08db4a4cb368 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT037.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB8843 Received-SPF: softfail client-ip=2a01:111:f400:fe59::626; envelope-from=avihaih@nvidia.com; helo=NAM12-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Refactor vfio_save_block() to return the size of saved data on success and -errno on error. This will be used in next patch to implement VFIO migration pre-copy support. Signed-off-by: Avihai Horon Reviewed-by: Cédric Le Goater Reviewed-by: Juan Quintela --- hw/vfio/migration.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 6b58dddb88..235978fd68 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -241,8 +241,8 @@ static int vfio_query_stop_copy_size(VFIODevice *vbasedev, return 0; } -/* Returns 1 if end-of-stream is reached, 0 if more data and -errno if error */ -static int vfio_save_block(QEMUFile *f, VFIOMigration *migration) +/* Returns the size of saved data on success and -errno on error */ +static ssize_t vfio_save_block(QEMUFile *f, VFIOMigration *migration) { ssize_t data_size; @@ -252,7 +252,7 @@ static int vfio_save_block(QEMUFile *f, VFIOMigration *migration) return -errno; } if (data_size == 0) { - return 1; + return 0; } qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); @@ -262,7 +262,7 @@ static int vfio_save_block(QEMUFile *f, VFIOMigration *migration) trace_vfio_save_block(migration->vbasedev->name, data_size); - return qemu_file_get_error(f); + return qemu_file_get_error(f) ?: data_size; } /* ---------------------------------------------------------------------- */ @@ -335,6 +335,7 @@ static void vfio_state_pending_exact(void *opaque, uint64_t *must_precopy, static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) { VFIODevice *vbasedev = opaque; + ssize_t data_size; int ret; /* We reach here with device state STOP only */ @@ -345,11 +346,11 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) } do { - ret = vfio_save_block(f, vbasedev->migration); - if (ret < 0) { - return ret; + data_size = vfio_save_block(f, vbasedev->migration); + if (data_size < 0) { + return data_size; } - } while (!ret); + } while (data_size); qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); ret = qemu_file_get_error(f); From patchwork Mon May 1 14:01:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13227523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A2649C77B61 for ; Mon, 1 May 2023 14:06:04 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ptU7W-0005rv-HH; Mon, 01 May 2023 10:03:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU73-0005Rj-Af for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:45 -0400 Received: from mail-bn8nam12on20603.outbound.protection.outlook.com ([2a01:111:f400:fe5b::603] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU6x-0006h9-5N for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:41 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dCmu7JdU3Wn3qRSIoVB6EZ1I+sXDhaLuiI42nz2PAW2lffe0QfaiF2VxcCyf84KWNpg/w4cDynC+hKtHNZLpV3bAJEBAeFfYaxIVTvkz1Kr0oU7MAGK985SEZ/DeYBDLvp21Lm4B7QwED1JDSxHBovzaLDi+dsNGqsnWp25IAqqZm43ryA2oa6EBufRycNFqsYXDRgkJyRMGeOWcpG7oSpBp8PDbCkH/5VSH6BbnaKOsKe8VrAvc242nzCMpTjxt6N0sdwiUZvb/s1bj24j+feISmkSqEFmWRsPie75PSVENhdvB8jyB6sE5RoFYgS6P68LF7N8PrN3PXwwg0ymePQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7pjyyMuaW5ZzDu6v3C5hJg81rrSrjc4nS7NizjC8b8U=; b=X8W2ShoW8HV5uwbBBk6AP6iM/AEpRY6J6mFKFUb55FPFvC/T1UKBsoXoYHOBqGIgsod2IT79QDU4b2NBulKUbI1evmE454/2dovh189NSfq3+Bwr0Ye6YzdzopSAbEFkgiS2Z9U7lAugfO7WvE49E4jYUaN5yydWDqCfHKKzw2GJqDnC87htU/ToozaSZuTVjVyPrXHT2Yc+2wZR+ZLvRkn4yPA0goXVeQkbKrpztDCB3sQdnNBP7UvN/OGwKYWHLgJlPQA0/hdWYY7cukel2qs+1h6ZfE6nlFT8B0gOZ0oPA1j3pBBNA4A4BgoMkpiOBToRXYcpyYw8grTh+8Jxjw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7pjyyMuaW5ZzDu6v3C5hJg81rrSrjc4nS7NizjC8b8U=; b=qbAC0kDaEo90vBdwacDi1HAoWtA7ScPz0R40dmlmhypWZJSP1YSyP2+EDjXOy6Oh/oro8Iv1RkAgrxaxloKX9yMpqo43CUP5BEgkMORVURZ5J0vNYHgZQSdrg+zZG3L+zJi3IVTIA2ujzsQKyWaOdkfPjGw1HgOD5i7+RXoW/w1V6UqK2kfKq3UoyFeRgu0Ituuh6wWh8ZUBDjboZzEKe4BBn8YYhJ/8pIPfqO+7Rrn0mp5JOU+NZwI2PYka6vasBEQNM6i+6VllUvDngjwbPKWsrGsRuUoQArtgDViL8FhjzomcpKmL9g2WmpkfaU5smuVjtWj1lSqZMWBk/7FIbg== Received: from DS0PR17CA0015.namprd17.prod.outlook.com (2603:10b6:8:191::23) by SJ0PR12MB5611.namprd12.prod.outlook.com (2603:10b6:a03:426::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.30; Mon, 1 May 2023 14:02:35 +0000 Received: from DM6NAM11FT056.eop-nam11.prod.protection.outlook.com (2603:10b6:8:191:cafe::c1) by DS0PR17CA0015.outlook.office365.com (2603:10b6:8:191::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.31 via Frontend Transport; Mon, 1 May 2023 14:02:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT056.mail.protection.outlook.com (10.13.173.99) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.19 via Frontend Transport; Mon, 1 May 2023 14:02:33 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Mon, 1 May 2023 07:02:21 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Mon, 1 May 2023 07:02:20 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Mon, 1 May 2023 07:02:16 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Juan Quintela , Peter Xu , Leonardo Bras , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Paolo Bonzini , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 7/8] vfio/migration: Add VFIO migration pre-copy support Date: Mon, 1 May 2023 17:01:40 +0300 Message-ID: <20230501140141.11743-8-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230501140141.11743-1-avihaih@nvidia.com> References: <20230501140141.11743-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT056:EE_|SJ0PR12MB5611:EE_ X-MS-Office365-Filtering-Correlation-Id: 8af6f638-4cc7-46fd-2f48-08db4a4cb608 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: r8j5KUgDB7cMJcNCz8rG36JcVk7RF5AHTrV3BVK5hF9GVLjPT0LDS6aDkW6onyN//lvtZagZ/htJ7dR37+0U4CWPywfLucfkRXC8rrDy+svlyEDJdTv4qUbEL2nDo68rJo3YY7kCJMephgktye63etKqPxNHpBUeiqfudiCjhNbr9e10XkMM2ENBI54Q3vRW6hq/2D03+B0gvjjQrLxCPXMVYbFCE/tQllrm7oTrqo4GGV20OgwgXVV/cjXHUFIC2cTbbFvea/uWpS+4243uo14IA1WXC6vxiiaEocjLhT1u+PKxskXrue9zMFhiya5aeYg2a3kOFLXutOHDBDZ40uxzP1Qb+Gqpzm1eUYeMVmACsaxHBaDp6xKVfXJw1Ha7AcLikNWJpv/WErKsQLV0BEO2mYRhUh2DX3vyS1PGtv8zzb4F9tsfNQX5dKIbBg0HS5E6reRyY4iQg6KxuzwwPvOl0HaOdlO6xTrpVOLeZMqi1toKxT0ceVYY62pDc8UcWBF6a2Ym13lFOSOuiF//i7DhehSdKYdx52W2FXZNN5nCsF6oUs4KOerVDz/tnkPvdMLgrZvMaw1gK7edZbj13zOWhWLqRlf52l1A39CFldD5dgOq2TW/MiUKsMpCKbi6MqP7EY9N0dgZqfcFiNBAUOnjooyyRLGfAQD0OqUX/IzbwQSC60G80wdygXetClgQxjt8smYpT/obWFje2WRb89XgP5Q1EhFN7Gl4jBxKN/vnlZwoLE+AZJhQtmgT+lfZ X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(396003)(39860400002)(136003)(346002)(376002)(451199021)(40470700004)(46966006)(36840700001)(7416002)(5660300002)(356005)(7636003)(8676002)(40460700003)(82740400003)(8936002)(41300700001)(316002)(40480700001)(36756003)(82310400005)(86362001)(2906002)(30864003)(36860700001)(186003)(2616005)(478600001)(83380400001)(6666004)(7696005)(426003)(336012)(47076005)(26005)(966005)(1076003)(54906003)(4326008)(6916009)(70586007)(70206006); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 May 2023 14:02:33.3454 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8af6f638-4cc7-46fd-2f48-08db4a4cb608 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT056.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5611 Received-SPF: softfail client-ip=2a01:111:f400:fe5b::603; envelope-from=avihaih@nvidia.com; helo=NAM12-BN8-obe.outbound.protection.outlook.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Pre-copy support allows the VFIO device data to be transferred while the VM is running. This helps to accommodate VFIO devices that have a large amount of data that needs to be transferred, and it can reduce migration downtime. Pre-copy support is optional in VFIO migration protocol v2. Implement pre-copy of VFIO migration protocol v2 and use it for devices that support it. Full description of it can be found here [1]. [1] https://lore.kernel.org/kvm/20221206083438.37807-3-yishaih@nvidia.com/ Signed-off-by: Avihai Horon --- docs/devel/vfio-migration.rst | 35 +++++--- include/hw/vfio/vfio-common.h | 3 + hw/vfio/common.c | 6 +- hw/vfio/migration.c | 155 ++++++++++++++++++++++++++++++++-- hw/vfio/trace-events | 4 +- 5 files changed, 181 insertions(+), 22 deletions(-) diff --git a/docs/devel/vfio-migration.rst b/docs/devel/vfio-migration.rst index 1b68ccf115..e896b2a673 100644 --- a/docs/devel/vfio-migration.rst +++ b/docs/devel/vfio-migration.rst @@ -7,12 +7,14 @@ the guest is running on source host and restoring this saved state on the destination host. This document details how saving and restoring of VFIO devices is done in QEMU. -Migration of VFIO devices currently consists of a single stop-and-copy phase. -During the stop-and-copy phase the guest is stopped and the entire VFIO device -data is transferred to the destination. - -The pre-copy phase of migration is currently not supported for VFIO devices. -Support for VFIO pre-copy will be added later on. +Migration of VFIO devices consists of two phases: the optional pre-copy phase, +and the stop-and-copy phase. The pre-copy phase is iterative and allows to +accommodate VFIO devices that have a large amount of data that needs to be +transferred. The iterative pre-copy phase of migration allows for the guest to +continue whilst the VFIO device state is transferred to the destination, this +helps to reduce the total downtime of the VM. VFIO devices opt-in to pre-copy +support by reporting the VFIO_MIGRATION_PRE_COPY flag in the +VFIO_DEVICE_FEATURE_MIGRATION ioctl. Note that currently VFIO migration is supported only for a single device. This is due to VFIO migration's lack of P2P support. However, P2P support is planned @@ -29,10 +31,20 @@ VFIO implements the device hooks for the iterative approach as follows: * A ``load_setup`` function that sets the VFIO device on the destination in _RESUMING state. +* A ``state_pending_estimate`` function that reports an estimate of the + remaining pre-copy data that the vendor driver has yet to save for the VFIO + device. + * A ``state_pending_exact`` function that reads pending_bytes from the vendor driver, which indicates the amount of data that the vendor driver has yet to save for the VFIO device. +* An ``is_active_iterate`` function that indicates ``save_live_iterate`` is + active only when the VFIO device is in pre-copy states. + +* A ``save_live_iterate`` function that reads the VFIO device's data from the + vendor driver during iterative pre-copy phase. + * A ``save_state`` function to save the device config space if it is present. * A ``save_live_complete_precopy`` function that sets the VFIO device in @@ -111,8 +123,10 @@ Flow of state changes during Live migration =========================================== Below is the flow of state change during live migration. -The values in the brackets represent the VM state, the migration state, and +The values in the parentheses represent the VM state, the migration state, and the VFIO device state, respectively. +The text in the square brackets represents the flow if the VFIO device supports +pre-copy. Live migration save path ------------------------ @@ -124,11 +138,12 @@ Live migration save path | migrate_init spawns migration_thread Migration thread then calls each device's .save_setup() - (RUNNING, _SETUP, _RUNNING) + (RUNNING, _SETUP, _RUNNING [_PRE_COPY]) | - (RUNNING, _ACTIVE, _RUNNING) - If device is active, get pending_bytes by .state_pending_exact() + (RUNNING, _ACTIVE, _RUNNING [_PRE_COPY]) + If device is active, get pending_bytes by .state_pending_{estimate,exact}() If total pending_bytes >= threshold_size, call .save_live_iterate() + [Data of VFIO device for pre-copy phase is copied] Iterate till total pending bytes converge and are less than threshold | On migration completion, vCPU stops and calls .save_live_complete_precopy for diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index eed244f25f..fa42955d4c 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -66,6 +66,9 @@ typedef struct VFIOMigration { int data_fd; void *data_buffer; size_t data_buffer_size; + uint64_t precopy_init_size; + uint64_t precopy_dirty_size; + uint64_t mig_flags; } VFIOMigration; typedef struct VFIOAddressSpace { diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 4d01ea3515..4f01f0148e 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -491,7 +491,8 @@ static bool vfio_devices_all_dirty_tracking(VFIOContainer *container) } if (vbasedev->pre_copy_dirty_page_tracking == ON_OFF_AUTO_OFF && - migration->device_state == VFIO_DEVICE_STATE_RUNNING) { + (migration->device_state == VFIO_DEVICE_STATE_RUNNING || + migration->device_state == VFIO_DEVICE_STATE_PRE_COPY)) { return false; } } @@ -536,7 +537,8 @@ static bool vfio_devices_all_running_and_mig_active(VFIOContainer *container) return false; } - if (migration->device_state == VFIO_DEVICE_STATE_RUNNING) { + if (migration->device_state == VFIO_DEVICE_STATE_RUNNING || + migration->device_state == VFIO_DEVICE_STATE_PRE_COPY) { continue; } else { return false; diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 235978fd68..980be1f614 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -68,6 +68,8 @@ static const char *mig_state_to_str(enum vfio_device_mig_state state) return "STOP_COPY"; case VFIO_DEVICE_STATE_RESUMING: return "RESUMING"; + case VFIO_DEVICE_STATE_PRE_COPY: + return "PRE_COPY"; default: return "UNKNOWN STATE"; } @@ -241,6 +243,22 @@ static int vfio_query_stop_copy_size(VFIODevice *vbasedev, return 0; } +static int vfio_query_precopy_size(VFIOMigration *migration) +{ + struct vfio_precopy_info precopy = { + .argsz = sizeof(precopy), + }; + + if (ioctl(migration->data_fd, VFIO_MIG_GET_PRECOPY_INFO, &precopy)) { + return -errno; + } + + migration->precopy_init_size = precopy.initial_bytes; + migration->precopy_dirty_size = precopy.dirty_bytes; + + return 0; +} + /* Returns the size of saved data on success and -errno on error */ static ssize_t vfio_save_block(QEMUFile *f, VFIOMigration *migration) { @@ -249,6 +267,11 @@ static ssize_t vfio_save_block(QEMUFile *f, VFIOMigration *migration) data_size = read(migration->data_fd, migration->data_buffer, migration->data_buffer_size); if (data_size < 0) { + /* Pre-copy emptied all the device state for now */ + if (errno == ENOMSG) { + return 0; + } + return -errno; } if (data_size == 0) { @@ -265,6 +288,31 @@ static ssize_t vfio_save_block(QEMUFile *f, VFIOMigration *migration) return qemu_file_get_error(f) ?: data_size; } +static void vfio_update_estimated_pending_data(VFIOMigration *migration, + uint64_t data_size) +{ + if (!data_size) { + /* + * Pre-copy emptied all the device state for now, update estimated sizes + * accordingly. + */ + migration->precopy_init_size = 0; + migration->precopy_dirty_size = 0; + + return; + } + + if (migration->precopy_init_size) { + uint64_t init_size = MIN(migration->precopy_init_size, data_size); + + migration->precopy_init_size -= init_size; + data_size -= init_size; + } + + migration->precopy_dirty_size -= MIN(migration->precopy_dirty_size, + data_size); +} + /* ---------------------------------------------------------------------- */ static int vfio_save_setup(QEMUFile *f, void *opaque) @@ -285,6 +333,31 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) return -ENOMEM; } + if (migration->mig_flags & VFIO_MIGRATION_PRE_COPY) { + int ret; + + migration->precopy_init_size = 0; + migration->precopy_dirty_size = 0; + + switch (migration->device_state) { + case VFIO_DEVICE_STATE_RUNNING: + ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_PRE_COPY, + VFIO_DEVICE_STATE_RUNNING); + if (ret) { + return ret; + } + + vfio_query_precopy_size(migration); + + break; + case VFIO_DEVICE_STATE_STOP: + /* vfio_save_complete_precopy() will go to STOP_COPY */ + break; + default: + return -EINVAL; + } + } + trace_vfio_save_setup(vbasedev->name, migration->data_buffer_size); qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); @@ -303,22 +376,36 @@ static void vfio_save_cleanup(void *opaque) trace_vfio_save_cleanup(vbasedev->name); } +static void vfio_state_pending_estimate(void *opaque, uint64_t *must_precopy, + uint64_t *can_postcopy) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + + if (migration->device_state != VFIO_DEVICE_STATE_PRE_COPY) { + return; + } + + *must_precopy += + migration->precopy_init_size + migration->precopy_dirty_size; + + trace_vfio_state_pending_estimate(vbasedev->name, *must_precopy, + *can_postcopy, + migration->precopy_init_size, + migration->precopy_dirty_size); +} + /* * Migration size of VFIO devices can be as little as a few KBs or as big as * many GBs. This value should be big enough to cover the worst case. */ #define VFIO_MIG_STOP_COPY_SIZE (100 * GiB) -/* - * Only exact function is implemented and not estimate function. The reason is - * that during pre-copy phase of migration the estimate function is called - * repeatedly while pending RAM size is over the threshold, thus migration - * can't converge and querying the VFIO device pending data size is useless. - */ static void vfio_state_pending_exact(void *opaque, uint64_t *must_precopy, uint64_t *can_postcopy) { VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; uint64_t stop_copy_size = VFIO_MIG_STOP_COPY_SIZE; /* @@ -328,8 +415,49 @@ static void vfio_state_pending_exact(void *opaque, uint64_t *must_precopy, vfio_query_stop_copy_size(vbasedev, &stop_copy_size); *must_precopy += stop_copy_size; + if (migration->device_state == VFIO_DEVICE_STATE_PRE_COPY) { + migration->precopy_init_size = 0; + migration->precopy_dirty_size = 0; + vfio_query_precopy_size(migration); + + *must_precopy += + migration->precopy_init_size + migration->precopy_dirty_size; + } + trace_vfio_state_pending_exact(vbasedev->name, *must_precopy, *can_postcopy, - stop_copy_size); + stop_copy_size, migration->precopy_init_size, + migration->precopy_dirty_size); +} + +static bool vfio_is_active_iterate(void *opaque) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + + return migration->device_state == VFIO_DEVICE_STATE_PRE_COPY; +} + +static int vfio_save_iterate(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + ssize_t data_size; + + data_size = vfio_save_block(f, migration); + if (data_size < 0) { + return data_size; + } + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + vfio_update_estimated_pending_data(migration, data_size); + + trace_vfio_save_iterate(vbasedev->name); + + /* + * A VFIO device's pre-copy dirty_bytes is not guaranteed to reach zero. + * Return 1 so following handlers will not be potentially blocked. + */ + return 1; } static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) @@ -338,7 +466,7 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) ssize_t data_size; int ret; - /* We reach here with device state STOP only */ + /* We reach here with device state STOP or STOP_COPY only */ ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP_COPY, VFIO_DEVICE_STATE_STOP); if (ret) { @@ -457,7 +585,10 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) static const SaveVMHandlers savevm_vfio_handlers = { .save_setup = vfio_save_setup, .save_cleanup = vfio_save_cleanup, + .state_pending_estimate = vfio_state_pending_estimate, .state_pending_exact = vfio_state_pending_exact, + .is_active_iterate = vfio_is_active_iterate, + .save_live_iterate = vfio_save_iterate, .save_live_complete_precopy = vfio_save_complete_precopy, .save_state = vfio_save_state, .load_setup = vfio_load_setup, @@ -470,13 +601,18 @@ static const SaveVMHandlers savevm_vfio_handlers = { static void vfio_vmstate_change(void *opaque, bool running, RunState state) { VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; enum vfio_device_mig_state new_state; int ret; if (running) { new_state = VFIO_DEVICE_STATE_RUNNING; } else { - new_state = VFIO_DEVICE_STATE_STOP; + new_state = + (migration->device_state == VFIO_DEVICE_STATE_PRE_COPY && + (state == RUN_STATE_FINISH_MIGRATE || state == RUN_STATE_PAUSED)) ? + VFIO_DEVICE_STATE_STOP_COPY : + VFIO_DEVICE_STATE_STOP; } /* @@ -603,6 +739,7 @@ static int vfio_migration_init(VFIODevice *vbasedev) migration->vbasedev = vbasedev; migration->device_state = VFIO_DEVICE_STATE_RUNNING; migration->data_fd = -1; + migration->mig_flags = mig_flags; vbasedev->dirty_pages_supported = vfio_dma_logging_supported(vbasedev); diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 646e42fd27..fd6893cb43 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -162,6 +162,8 @@ vfio_save_block(const char *name, int data_size) " (%s) data_size %d" vfio_save_cleanup(const char *name) " (%s)" vfio_save_complete_precopy(const char *name, int ret) " (%s) ret %d" vfio_save_device_config_state(const char *name) " (%s)" +vfio_save_iterate(const char *name) " (%s)" vfio_save_setup(const char *name, uint64_t data_buffer_size) " (%s) data buffer size 0x%"PRIx64 -vfio_state_pending_exact(const char *name, uint64_t precopy, uint64_t postcopy, uint64_t stopcopy_size) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" stopcopy size 0x%"PRIx64 +vfio_state_pending_estimate(const char *name, uint64_t precopy, uint64_t postcopy, uint64_t precopy_init_size, uint64_t precopy_dirty_size) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" precopy initial size 0x%"PRIx64" precopy dirty size 0x%"PRIx64 +vfio_state_pending_exact(const char *name, uint64_t precopy, uint64_t postcopy, uint64_t stopcopy_size, uint64_t precopy_init_size, uint64_t precopy_dirty_size) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" stopcopy size 0x%"PRIx64" precopy initial size 0x%"PRIx64" precopy dirty size 0x%"PRIx64 vfio_vmstate_change(const char *name, int running, const char *reason, const char *dev_state) " (%s) running %d reason %s device state %s" From patchwork Mon May 1 14:01:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Avihai Horon X-Patchwork-Id: 13227515 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ADEB8C77B7C for ; Mon, 1 May 2023 14:03:21 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ptU7a-0005zI-1Z; Mon, 01 May 2023 10:03:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU75-0005eM-LN for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:48 -0400 Received: from mail-mw2nam12on20626.outbound.protection.outlook.com ([2a01:111:f400:fe5a::626] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptU73-0006hI-B4 for qemu-devel@nongnu.org; Mon, 01 May 2023 10:02:47 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HGd8yY2g8D5gD8WU7nIHBZsbfPJAYiABlsiEIXEyjJbx9u2iFPME+dy6C6xnD6n0Xz1WeeWl8nHVnDK0xBvVDuvepKQ3Mv4mY28n1dCsfBRuSdYgdiE8OgtbvgL/WKIjQbFsGmGk0lpc3yz1BSujb1RzDY5fvbA/KEzhpreUFUz4KbuN93nKEMzUVUT0/PegaWnsO6T5hAvz2g7nBRVXhgvUV15M0516woV6Rrw6fQik/owJbmdUQbciZbC9pYf/zMMeF1bgVUsnbn38yKZ+fvyFVOv9TAnIlDyGf4X6eEaB+0Hlo/SFp4CfwhDNA8fEtKclA1tLLzHPJ1WIaq5Tlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mei7l4emdoKlPtLh2wmucZw9iYajU0TQamA9f7tTGt0=; b=fDAgophXO9L9uA37oEWk8uEjrmGozZWsuG4i1QoghkExFprZh2iRLyvwnOks5o76TuBkRweMz7WtpTSRwppHpZcJulSTHb3kWkDgJyYUnftDK6RMmbP4les0e3lPPYDpQqBjP9rLcWJocJMLZwnB2dWDlpJRTmxx3pCEXEjsODHMOWqWMrTp1MXVFQvabYr7CBjrNzcz5p+74k7ER8N4qzFf5Eb2qwdY9uyRafH87LiToPpQND4ThJ/akyPlsGnWvoZXCncuzjfQCaXKYzMIregGYSpoFlKKUMXDXrk0SEyxTu1f9Shk88GC/1OVB9fZn+5NsOlE6RKWw9Ik3PQJGw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mei7l4emdoKlPtLh2wmucZw9iYajU0TQamA9f7tTGt0=; b=cV0fRBdMGGH5IhWR8dOlFCbccvUJz3BeI4wecqrM8n0I22LHO1dr6jz3Nze7rtBohLCn1DRYBhgo+qDP5VKMbhU+FA2SP53/I6Xxn4uxQy3FaTpFt6HfWj6+oKNCdMyHUbixcojXUzJ3INr740ZJJnI1IuwFt8/+UPZ7A8nK6sDDDEqxUvoKn3Fu40H5IJ0G1FsmL6vEe22Jg3fw5TITWTkcnFInrbS09ypTm44O9/yRf8oVYXkBbwBWo9YqX/3j7R6Gy6UDGYe+d3yaXY7hcy6L6QU7D1+JI0f01Kt//ySj3Z9ERiu2gXmG634dmNOC+EuWZMRhku4hrdk2DYGlAQ== Received: from DM6PR02CA0052.namprd02.prod.outlook.com (2603:10b6:5:177::29) by PH7PR12MB6587.namprd12.prod.outlook.com (2603:10b6:510:211::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.28; Mon, 1 May 2023 14:02:38 +0000 Received: from DM6NAM11FT048.eop-nam11.prod.protection.outlook.com (2603:10b6:5:177:cafe::97) by DM6PR02CA0052.outlook.office365.com (2603:10b6:5:177::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.30 via Frontend Transport; Mon, 1 May 2023 14:02:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT048.mail.protection.outlook.com (10.13.173.114) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.19 via Frontend Transport; Mon, 1 May 2023 14:02:38 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Mon, 1 May 2023 07:02:26 -0700 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Mon, 1 May 2023 07:02:25 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Mon, 1 May 2023 07:02:21 -0700 From: Avihai Horon To: CC: Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_G?= =?utf-8?q?oater?= , Juan Quintela , Peter Xu , Leonardo Bras , Eric Blake , Markus Armbruster , Thomas Huth , Laurent Vivier , Paolo Bonzini , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 8/8] vfio/migration: Add support for precopy initial data capability Date: Mon, 1 May 2023 17:01:41 +0300 Message-ID: <20230501140141.11743-9-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230501140141.11743-1-avihaih@nvidia.com> References: <20230501140141.11743-1-avihaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT048:EE_|PH7PR12MB6587:EE_ X-MS-Office365-Filtering-Correlation-Id: 7ba094be-2c0b-4695-dcbe-08db4a4cb8fb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: I6Rd8mWWWykI+mjMJwbXr1pG0cNqqkrOwJPRXVqX/AZt4/jNMl2vb0NSZNMKOkpuEwMp3trEx/A8nsU7zqg6E9X/OMPgjamzfbRGtJaSHJMZDOX3ncaGFn+L+PPIgsRIJBBAtw54TI1WDCQTid3TxlZVoP097wYKExCUL42bzmLnM/uxpO4hQ3/vjPz/25NuyLVsJTrgjjJIhEv0iIvJ48RUVj2aruYzuazO/7IPLhelCxvqrWk8Ua7TYRU8U/WrzaLWqfaz8DpFOg5iWWNk1Li2IaAdUZ38uMv4anTlZ10H4L2YK3EZcTkPvLHVAzrwBTi/Vm5uA5rMQJ6PIqFXW4vpqkajymlb+jiCt0hp980OxdpoXJHClfh4NuHOho2XTdVyGt3oY90E0XCdgk8NxmFJU/0TpCHRboVg+7ChBNb7EJtV+8FsUxa9/e/qd7KEKcM6JVtinFSEzfqP3qOHOfQCgLTNYYbjf+095gcKcrWpHi46yENiIy8eYbX+vo8vnbx/1FdLh+b6N5p8JUsu1eUx5OEszFW9CQJIbuVOVSKjmglYABczl+ceY0zgqAESvscGNr3xLwUqIrajzK7An6Tw41IZZvQMOnz8pNrOWiftyJIg1mgwxijaQ+H9dO/fY6WYClQvFRoP9CCOmkpu7KG/c9JNzYUnl6u6znZJfZFY0abT6iZjsUsZEMe5fMk2MHUQagQ+b7aJ9vX+ANKqWw== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(396003)(39860400002)(136003)(346002)(376002)(451199021)(40470700004)(46966006)(36840700001)(7416002)(5660300002)(356005)(7636003)(8676002)(40460700003)(82740400003)(8936002)(41300700001)(316002)(40480700001)(36756003)(82310400005)(86362001)(2906002)(36860700001)(186003)(2616005)(478600001)(83380400001)(6666004)(7696005)(426003)(336012)(47076005)(26005)(1076003)(54906003)(4326008)(6916009)(70586007)(70206006); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 May 2023 14:02:38.3134 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7ba094be-2c0b-4695-dcbe-08db4a4cb8fb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT048.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6587 Received-SPF: softfail client-ip=2a01:111:f400:fe5a::626; envelope-from=avihaih@nvidia.com; helo=NAM12-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Loading of a VFIO device's data can take a substantial amount of time as the device may need to allocate resources, prepare internal data structures, etc. This can increase migration downtime, especially for VFIO devices with a lot of resources. To solve this, VFIO migration uAPI defines "initial bytes" as part of its precopy data stream. Initial bytes can be used in various ways to improve VFIO migration performance. For example, it can be used to transfer device metadata to pre-allocate resources in the destination. However, for this to work we need to make sure that all initial bytes are sent and loaded in the destination before the source VM is stopped. Use migration precopy initial data capability to make sure a VFIO device's initial bytes are sent and loaded in the destination before the source stops the VM and attempts to complete the migration. This can significantly reduce migration downtime. Signed-off-by: Avihai Horon --- include/hw/vfio/vfio-common.h | 3 +++ hw/vfio/migration.c | 48 ++++++++++++++++++++++++++++++++++- 2 files changed, 50 insertions(+), 1 deletion(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index fa42955d4c..dd3b052682 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -69,6 +69,9 @@ typedef struct VFIOMigration { uint64_t precopy_init_size; uint64_t precopy_dirty_size; uint64_t mig_flags; + bool initial_data_active; + bool initial_data_sent; + bool initial_data_loaded; } VFIOMigration; typedef struct VFIOAddressSpace { diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 980be1f614..23f4f1f8a5 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -45,6 +45,7 @@ #define VFIO_MIG_FLAG_DEV_CONFIG_STATE (0xffffffffef100002ULL) #define VFIO_MIG_FLAG_DEV_SETUP_STATE (0xffffffffef100003ULL) #define VFIO_MIG_FLAG_DEV_DATA_STATE (0xffffffffef100004ULL) +#define VFIO_MIG_FLAG_DEV_INIT_DATA_SENT (0xffffffffef100005ULL) /* * This is an arbitrary size based on migration of mlx5 devices, where typically @@ -372,6 +373,8 @@ static void vfio_save_cleanup(void *opaque) g_free(migration->data_buffer); migration->data_buffer = NULL; + migration->initial_data_sent = false; + migration->initial_data_active = false; vfio_migration_cleanup(vbasedev); trace_vfio_save_cleanup(vbasedev->name); } @@ -447,10 +450,17 @@ static int vfio_save_iterate(QEMUFile *f, void *opaque) if (data_size < 0) { return data_size; } - qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); vfio_update_estimated_pending_data(migration, data_size); + if (migration->initial_data_active && !migration->precopy_init_size && + !migration->initial_data_sent) { + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_INIT_DATA_SENT); + migration->initial_data_sent = true; + } else { + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + } + trace_vfio_save_iterate(vbasedev->name); /* @@ -568,6 +578,12 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) } break; } + case VFIO_MIG_FLAG_DEV_INIT_DATA_SENT: + { + vbasedev->migration->initial_data_loaded = true; + + return 0; + } default: error_report("%s: Unknown tag 0x%"PRIx64, vbasedev->name, data); return -EINVAL; @@ -582,6 +598,33 @@ static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) return ret; } +static bool vfio_initial_data_advise(void *opaque) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + + migration->initial_data_active = + migration->mig_flags & VFIO_MIGRATION_PRE_COPY; + + return migration->initial_data_active; +} + +static bool vfio_is_initial_data_active(void *opaque) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + + return migration->initial_data_active; +} + +static bool vfio_initial_data_loaded(void *opaque) +{ + VFIODevice *vbasedev = opaque; + VFIOMigration *migration = vbasedev->migration; + + return migration->initial_data_loaded; +} + static const SaveVMHandlers savevm_vfio_handlers = { .save_setup = vfio_save_setup, .save_cleanup = vfio_save_cleanup, @@ -594,6 +637,9 @@ static const SaveVMHandlers savevm_vfio_handlers = { .load_setup = vfio_load_setup, .load_cleanup = vfio_load_cleanup, .load_state = vfio_load_state, + .initial_data_advise = vfio_initial_data_advise, + .is_initial_data_active = vfio_is_initial_data_active, + .initial_data_loaded = vfio_initial_data_loaded, }; /* ---------------------------------------------------------------------- */