From patchwork Thu Mar 30 18:03:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194692 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3804DC761A6 for ; Thu, 30 Mar 2023 18:05:21 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwd4-0003By-HE; Thu, 30 Mar 2023 14:04:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwcl-000387-KL for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:03:50 -0400 Received: from smtp-out2.suse.de ([195.135.220.29]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwcj-0001ze-WF for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:03:47 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id CFFEF1F85D; Thu, 30 Mar 2023 18:03:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199423; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MGlBEeszxSprvniwR7x90pTvScfxw2Gu+VCaq9pUiME=; b=SDmXM5PvsmLZBbQX1bpMbp5RvUOkvjK8j36nISmd/7D4jsKIyXuPKVxnvPk2P7aCZxDRm3 6UAIqIzzrCbCFLfxdecy8sBZhUHKC5jTn3Ta4JErv491ltYFnlJrat1N/LCojFumV5MGEk aTdYrOVZ9IvTO3wlik9A52oKQ8VKTm0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199423; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MGlBEeszxSprvniwR7x90pTvScfxw2Gu+VCaq9pUiME=; b=dm7MtvoG789Ci816XCrqzMgxxvwCiD2Xj2cIFiLrgTfkN0rc+aWSm1V3WqTB0PiecT7rEd c0axjXgWf1F0lAAw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 86B501348E; Thu, 30 Mar 2023 18:03:41 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id yMJwE/3OJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:03:41 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Nikolay Borisov Subject: [RFC PATCH v1 01/26] migration: Add support for 'file:' uri for source migration Date: Thu, 30 Mar 2023 15:03:11 -0300 Message-Id: <20230330180336.2791-2-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=195.135.220.29; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Nikolay Borisov Implement support for a "file:" uri so that a migration can be initiated directly to a file from QEMU. Unlike other migration protocol backends, the 'file' protocol cannot honour non-blocking mode. POSIX file/block storage will always report ready to read/write, regardless of how slow the underlying storage will be at servicing the request. For outgoing migration this limitation is not a serious problem as the migration data transfer always happens in a dedicated thread. It may, however, result in delays in honouring a request to cancel the migration operation. Reviewed-by: Daniel P. Berrangé Signed-off-by: Nikolay Borisov Signed-off-by: Fabiano Rosas --- migration/file.c | 21 +++++++++++++++++++++ migration/file.h | 9 +++++++++ migration/meson.build | 1 + migration/migration.c | 3 +++ 4 files changed, 34 insertions(+) create mode 100644 migration/file.c create mode 100644 migration/file.h diff --git a/migration/file.c b/migration/file.c new file mode 100644 index 0000000000..36d6178c75 --- /dev/null +++ b/migration/file.c @@ -0,0 +1,21 @@ +#include "qemu/osdep.h" +#include "channel.h" +#include "io/channel-file.h" +#include "file.h" +#include "qemu/error-report.h" + + +void file_start_outgoing_migration(MigrationState *s, const char *fname, Error **errp) +{ + QIOChannelFile *ioc; + + ioc = qio_channel_file_new_path(fname, O_CREAT | O_TRUNC | O_WRONLY, 0660, errp); + if (!ioc) { + error_report("Error creating a channel"); + return; + } + + qio_channel_set_name(QIO_CHANNEL(ioc), "migration-file-outgoing"); + migration_channel_connect(s, QIO_CHANNEL(ioc), NULL, NULL); + object_unref(OBJECT(ioc)); +} diff --git a/migration/file.h b/migration/file.h new file mode 100644 index 0000000000..d476eb1157 --- /dev/null +++ b/migration/file.h @@ -0,0 +1,9 @@ +#ifndef QEMU_MIGRATION_FILE_H +#define QEMU_MIGRATION_FILE_H + +void file_start_outgoing_migration(MigrationState *s, + const char *filename, + Error **errp); + +#endif + diff --git a/migration/meson.build b/migration/meson.build index 0d1bb9f96e..6c02298c70 100644 --- a/migration/meson.build +++ b/migration/meson.build @@ -17,6 +17,7 @@ softmmu_ss.add(files( 'colo.c', 'exec.c', 'fd.c', + 'file.c', 'global_state.c', 'migration-hmp-cmds.c', 'migration.c', diff --git a/migration/migration.c b/migration/migration.c index ae2025d9d8..58ff0cb7c7 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -20,6 +20,7 @@ #include "migration/blocker.h" #include "exec.h" #include "fd.h" +#include "file.h" #include "socket.h" #include "sysemu/runstate.h" #include "sysemu/sysemu.h" @@ -2523,6 +2524,8 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, exec_start_outgoing_migration(s, p, &local_err); } else if (strstart(uri, "fd:", &p)) { fd_start_outgoing_migration(s, p, &local_err); + } else if (strstart(uri, "file:", &p)) { + file_start_outgoing_migration(s, p, &local_err); } else { if (!(has_resume && resume)) { yank_unregister_instance(MIGRATION_YANK_INSTANCE); From patchwork Thu Mar 30 18:03:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194688 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EE636C77B60 for ; Thu, 30 Mar 2023 18:04:50 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwcu-0003Bi-Vu; Thu, 30 Mar 2023 14:03:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwcn-00038E-9F for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:03:50 -0400 Received: from smtp-out1.suse.de ([2001:67c:2178:6::1c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwcl-00020A-Mo for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:03:49 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 9D63821A2A; Thu, 30 Mar 2023 18:03:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199426; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QRj+Ox85NP22NK88Buh78DEflEosiGH3v9OMA2dkbCk=; b=QobSR4/64jairexcpHT5uKqud6Rz5S37GVY+MUyDkSyNOgnETiNgWKbCnYdTWINNPu1DYA cHM9gkPflbO1tLJYcEcUPqRoYFFg9cPNpkOUkvxddZEr0ePcQQCcccuoY1sEI2AdZliS5l adJrNMHMjD+Ek/jGMMAPIc3ISJs6Mo0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199426; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QRj+Ox85NP22NK88Buh78DEflEosiGH3v9OMA2dkbCk=; b=D8QW0ABaqGiD7qyUg6iOcb+dy4iZVGetCyjiVlv68xq/RTLi++VaA/qc7gjt6W4VoMuuEw sNc4oRMnOyw+f3CQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 4EE331348E; Thu, 30 Mar 2023 18:03:44 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id SIrDBQDPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:03:44 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Nikolay Borisov Subject: [RFC PATCH v1 02/26] migration: Add support for 'file:' uri for incoming migration Date: Thu, 30 Mar 2023 15:03:12 -0300 Message-Id: <20230330180336.2791-3-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1c; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Nikolay Borisov This is a counterpart to the 'file:' uri support for source migration, now a file can also serve as the source of an incoming migration. Unlike other migration protocol backends, the 'file' protocol cannot honour non-blocking mode. POSIX file/block storage will always report ready to read/write, regardless of how slow the underlying storage will be at servicing the request. For incoming migration this limitation may result in the main event loop not being fully responsive while loading the VM state. This won't impact the VM since it is not running at this phase, however, it may impact management applications. Reviewed-by: Daniel P. Berrangé Signed-off-by: Nikolay Borisov Signed-off-by: Fabiano Rosas --- docs/devel/migration.rst | 2 ++ migration/file.c | 15 +++++++++++++++ migration/file.h | 1 + migration/migration.c | 2 ++ 4 files changed, 20 insertions(+) diff --git a/docs/devel/migration.rst b/docs/devel/migration.rst index 6f65c23b47..1080211f8e 100644 --- a/docs/devel/migration.rst +++ b/docs/devel/migration.rst @@ -39,6 +39,8 @@ over any transport. - exec migration: do the migration using the stdin/stdout through a process. - fd migration: do the migration using a file descriptor that is passed to QEMU. QEMU doesn't care how this file descriptor is opened. +- file migration: do the migration using a file that is passed by name + to QEMU. In addition, support is included for migration using RDMA, which transports the page data using ``RDMA``, where the hardware takes care of diff --git a/migration/file.c b/migration/file.c index 36d6178c75..ab4e12926c 100644 --- a/migration/file.c +++ b/migration/file.c @@ -19,3 +19,18 @@ void file_start_outgoing_migration(MigrationState *s, const char *fname, Error * migration_channel_connect(s, QIO_CHANNEL(ioc), NULL, NULL); object_unref(OBJECT(ioc)); } + +void file_start_incoming_migration(const char *fname, Error **errp) +{ + QIOChannelFile *ioc; + + ioc = qio_channel_file_new_path(fname, O_RDONLY, 0, errp); + if (!ioc) { + error_report("Error creating a channel"); + return; + } + + qio_channel_set_name(QIO_CHANNEL(ioc), "migration-file-incoming"); + migration_channel_process_incoming(QIO_CHANNEL(ioc)); + object_unref(OBJECT(ioc)); +} diff --git a/migration/file.h b/migration/file.h index d476eb1157..cdbd291322 100644 --- a/migration/file.h +++ b/migration/file.h @@ -5,5 +5,6 @@ void file_start_outgoing_migration(MigrationState *s, const char *filename, Error **errp); +void file_start_incoming_migration(const char *fname, Error **errp); #endif diff --git a/migration/migration.c b/migration/migration.c index 58ff0cb7c7..5408d87453 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -527,6 +527,8 @@ static void qemu_start_incoming_migration(const char *uri, Error **errp) exec_start_incoming_migration(p, errp); } else if (strstart(uri, "fd:", &p)) { fd_start_incoming_migration(p, errp); + } else if (strstart(uri, "file:", &p)) { + file_start_incoming_migration(p, errp); } else { error_setg(errp, "unknown migration protocol: %s", uri); } From patchwork Thu Mar 30 18:03:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E113EC761A6 for ; Thu, 30 Mar 2023 18:08:43 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwd5-0003DI-N9; Thu, 30 Mar 2023 14:04:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwcr-00038M-Ds for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:03:54 -0400 Received: from smtp-out2.suse.de ([2001:67c:2178:6::1d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwcp-00020p-Th for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:03:53 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 2430F1F86C; Thu, 30 Mar 2023 18:03:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199430; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sk83qEGaqZFWkA6OXTepi0U673ImA1hvV5Fb8G0U1QU=; b=xSJwDUXJ46/3mfXHNfkMxpUgwYYde3XfIvo5PvnHAWsRopeTO0JPCR4OMX/9mNiPKYoKAR PoxG8Uw4CRkPY77evHucOpo5ivBAV3qa3F250qYYHJ6oqvXDH5nK4shYzdRmv+QzcURiUh qV762rtuAmXpUyPKkjwoUvIXk9IT79s= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199430; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sk83qEGaqZFWkA6OXTepi0U673ImA1hvV5Fb8G0U1QU=; b=snNOiWBP8rg59tROOHWrYQ+5bEw86us+sY2Df0o6OrdlW00yxFcS7iF4MtfuLZoqkoxGFi YN6fne0VXgLie1Dg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 1CAB61348E; Thu, 30 Mar 2023 18:03:46 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id wAjfNALPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:03:46 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Nikolay Borisov , Thomas Huth , Laurent Vivier , Paolo Bonzini Subject: [RFC PATCH v1 03/26] tests/qtest: migration: Add migrate_incoming_qmp helper Date: Thu, 30 Mar 2023 15:03:13 -0300 Message-Id: <20230330180336.2791-4-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1d; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Nikolay Borisov file-based migration requires the target to initiate its migration after the source has finished writing out the data in the file. Currently there's no easy way to initiate 'migrate-incoming', allow this by introducing migrate_incoming_qmp helper, similarly to migrate_qmp. Signed-off-by: Nikolay Borisov Signed-off-by: Fabiano Rosas Reviewed-by: Daniel P. Berrangé --- tests/qtest/migration-helpers.c | 19 +++++++++++++++++++ tests/qtest/migration-helpers.h | 4 ++++ 2 files changed, 23 insertions(+) diff --git a/tests/qtest/migration-helpers.c b/tests/qtest/migration-helpers.c index f6f3c6680f..8161495c27 100644 --- a/tests/qtest/migration-helpers.c +++ b/tests/qtest/migration-helpers.c @@ -130,6 +130,25 @@ void migrate_qmp(QTestState *who, const char *uri, const char *fmt, ...) qobject_unref(rsp); } + +void migrate_incoming_qmp(QTestState *who, const char *uri, const char *fmt, ...) +{ + va_list ap; + QDict *args, *rsp; + + va_start(ap, fmt); + args = qdict_from_vjsonf_nofail(fmt, ap); + va_end(ap); + + g_assert(!qdict_haskey(args, "uri")); + qdict_put_str(args, "uri", uri); + + rsp = qtest_qmp(who, "{ 'execute': 'migrate-incoming', 'arguments': %p}", args); + + g_assert(qdict_haskey(rsp, "return")); + qobject_unref(rsp); +} + /* * Note: caller is responsible to free the returned object via * qobject_unref() after use diff --git a/tests/qtest/migration-helpers.h b/tests/qtest/migration-helpers.h index a188b62787..53ddeaebb7 100644 --- a/tests/qtest/migration-helpers.h +++ b/tests/qtest/migration-helpers.h @@ -31,6 +31,10 @@ QDict *qmp_command(QTestState *who, const char *command, ...); G_GNUC_PRINTF(3, 4) void migrate_qmp(QTestState *who, const char *uri, const char *fmt, ...); +G_GNUC_PRINTF(3, 4) +void migrate_incoming_qmp(QTestState *who, const char *uri, + const char *fmt, ...); + QDict *migrate_query(QTestState *who); QDict *migrate_query_not_failed(QTestState *who); From patchwork Thu Mar 30 18:03:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194690 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EB188C761A6 for ; Thu, 30 Mar 2023 18:04:50 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwd6-0003EH-DB; Thu, 30 Mar 2023 14:04:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwcu-0003Bh-Tx for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:03:56 -0400 Received: from smtp-out1.suse.de ([195.135.220.28]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwct-00021L-Ei for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:03:56 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 97176219E8; Thu, 30 Mar 2023 18:03:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199433; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q+ZBwUdJwUxaCGnBgDhTSIaWa5ev6qXpjX6yCZCHVwA=; b=apOxVUCnJLDpZRp7mXox/gizyVQUdYBODUJp9cr6TbAsIPBATclFTIpWVmZqKUoEwnEDcN 6cd9quglBhO3dOAQEEQKamFaaotQhvUHgVZeWC6qo0ppQ+uQhKQTFj8FvH4GmMNnTJtdDQ l3d8lygmLsG081tzPG5yDZcfN67wNe0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199433; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q+ZBwUdJwUxaCGnBgDhTSIaWa5ev6qXpjX6yCZCHVwA=; b=G2WOSJvQSxVjvEhA+GpZD5suiVllN4BRPoMUWjWVysfyNovqouhNvijTEXxBazfbL9numo FKYdI7sZgOYGdgAA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 952971348E; Thu, 30 Mar 2023 18:03:50 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id IIbAFgbPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:03:50 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Nikolay Borisov , Thomas Huth , Laurent Vivier , Paolo Bonzini Subject: [RFC PATCH v1 04/26] tests/qtest: migration-test: Add tests for file-based migration Date: Thu, 30 Mar 2023 15:03:14 -0300 Message-Id: <20230330180336.2791-5-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=195.135.220.28; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Nikolay Borisov Add basic tests for file-based migration. Signed-off-by: Nikolay Borisov Signed-off-by: Fabiano Rosas --- (farosas) fix segfault when connect_uri is not set --- tests/qtest/migration-test.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 3b615b0da9..13e5cdd5a4 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -748,6 +748,7 @@ static void test_migrate_end(QTestState *from, QTestState *to, bool test_dest) cleanup("migsocket"); cleanup("src_serial"); cleanup("dest_serial"); + cleanup("migfile"); } #ifdef CONFIG_GNUTLS @@ -1371,6 +1372,14 @@ static void test_precopy_common(MigrateCommon *args) * hanging forever if migration didn't converge */ wait_for_migration_complete(from); + /* + * For file based migration the target must begin its migration after + * the source has finished + */ + if (args->connect_uri && strstr(args->connect_uri, "file:")) { + migrate_incoming_qmp(to, args->connect_uri, "{}"); + } + if (!got_stop) { qtest_qmp_eventwait(from, "STOP"); } @@ -1524,6 +1533,17 @@ static void test_precopy_unix_xbzrle(void) test_precopy_common(&args); } +static void test_precopy_file_stream_ram(void) +{ + g_autofree char *uri = g_strdup_printf("file:%s/migfile", tmpfs); + MigrateCommon args = { + .connect_uri = uri, + .listen_uri = "defer", + }; + + test_precopy_common(&args); +} + static void test_precopy_tcp_plain(void) { MigrateCommon args = { @@ -2515,6 +2535,10 @@ int main(int argc, char **argv) qtest_add_func("/migration/bad_dest", test_baddest); qtest_add_func("/migration/precopy/unix/plain", test_precopy_unix_plain); qtest_add_func("/migration/precopy/unix/xbzrle", test_precopy_unix_xbzrle); + + qtest_add_func("/migration/precopy/file/stream-ram", + test_precopy_file_stream_ram); + #ifdef CONFIG_GNUTLS qtest_add_func("/migration/precopy/unix/tls/psk", test_precopy_unix_tls_psk); From patchwork Thu Mar 30 18:03:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194714 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 56F01C6FD1D for ; Thu, 30 Mar 2023 18:08:45 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwd5-0003DH-Lp; Thu, 30 Mar 2023 14:04:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwd4-0003CT-Am for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:06 -0400 Received: from smtp-out1.suse.de ([195.135.220.28]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwcx-00021x-1b for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:01 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id D4B12219E8; Thu, 30 Mar 2023 18:03:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199436; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aMxFafmyez7g9sXkEaSgJKRvOg914CYHM/M+GAElHpQ=; b=1Pnf9G9/hku7NMHwGoswjnbWpLz9J17VEbWd4Z+IWLCGVHPvbRwYYWE/B8m5VPqxNGgX6B sdDkUqNEpb9dCqSwrCUJbcitSbgICbaKcvF4M+biYnoz8vz16/NOOAkrmRF1Ec4fbdzmEa U5JvdiaCatmJV8WZlcIL172TlxVE0MU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199436; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aMxFafmyez7g9sXkEaSgJKRvOg914CYHM/M+GAElHpQ=; b=czxCkFBvWPzPBfYwSs661LS6GlGga6tLdh80I9fcfcAWN5JAu+lYU7NjbFh+4pzuneUXB6 iRr4du8558DWjXAw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 180961348E; Thu, 30 Mar 2023 18:03:53 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 2PLnMwnPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:03:53 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Nikolay Borisov , John Snow , Cleber Rosa Subject: [RFC PATCH v1 05/26] migration: Initial support of fixed-ram feature for analyze-migration.py Date: Thu, 30 Mar 2023 15:03:15 -0300 Message-Id: <20230330180336.2791-6-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=195.135.220.28; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Nikolay Borisov In order to allow analyze-migration.py script to work with migration streams that have the 'fixed-ram' capability, it's required to have access to the stream's configuration object. This commit enables this by making migration json writer part of MigrationState struct, allowing the configuration object be serialized to json. Signed-off-by: Nikolay Borisov Signed-off-by: Fabiano Rosas --- migration/migration.c | 1 + migration/savevm.c | 18 ++++++++++--- scripts/analyze-migration.py | 51 +++++++++++++++++++++++++++++++++--- 3 files changed, 62 insertions(+), 8 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 5408d87453..177fb0de0f 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2260,6 +2260,7 @@ void migrate_init(MigrationState *s) error_free(s->error); s->error = NULL; s->hostname = NULL; + s->vmdesc = NULL; migrate_set_state(&s->state, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP); diff --git a/migration/savevm.c b/migration/savevm.c index aa54a67fda..92102c1fe5 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1206,13 +1206,25 @@ void qemu_savevm_non_migratable_list(strList **reasons) void qemu_savevm_state_header(QEMUFile *f) { + MigrationState *s = migrate_get_current(); + + s->vmdesc = json_writer_new(false); + trace_savevm_state_header(); qemu_put_be32(f, QEMU_VM_FILE_MAGIC); qemu_put_be32(f, QEMU_VM_FILE_VERSION); - if (migrate_get_current()->send_configuration) { + if (s->send_configuration) { qemu_put_byte(f, QEMU_VM_CONFIGURATION); - vmstate_save_state(f, &vmstate_configuration, &savevm_state, 0); + /* + * This starts the main json object and is paired with the + * json_writer_end_object in + * qemu_savevm_state_complete_precopy_non_iterable + */ + json_writer_start_object(s->vmdesc, NULL); + json_writer_start_object(s->vmdesc, "configuration"); + vmstate_save_state(f, &vmstate_configuration, &savevm_state, s->vmdesc); + json_writer_end_object(s->vmdesc); } } @@ -1237,8 +1249,6 @@ void qemu_savevm_state_setup(QEMUFile *f) Error *local_err = NULL; int ret; - ms->vmdesc = json_writer_new(false); - json_writer_start_object(ms->vmdesc, NULL); json_writer_int64(ms->vmdesc, "page_size", qemu_target_page_size()); json_writer_start_array(ms->vmdesc, "devices"); diff --git a/scripts/analyze-migration.py b/scripts/analyze-migration.py index b82a1b0c58..05af9efd2f 100755 --- a/scripts/analyze-migration.py +++ b/scripts/analyze-migration.py @@ -23,7 +23,7 @@ import collections import struct import sys - +import math def mkdir_p(path): try: @@ -119,11 +119,16 @@ def __init__(self, file, version_id, ramargs, section_key): self.file = file self.section_key = section_key self.TARGET_PAGE_SIZE = ramargs['page_size'] + self.TARGET_PAGE_BITS = math.log2(self.TARGET_PAGE_SIZE) self.dump_memory = ramargs['dump_memory'] self.write_memory = ramargs['write_memory'] + self.fixed_ram = ramargs['fixed-ram'] self.sizeinfo = collections.OrderedDict() + self.bitmap_offset = collections.OrderedDict() + self.pages_offset = collections.OrderedDict() self.data = collections.OrderedDict() self.data['section sizes'] = self.sizeinfo + self.ram_read = False self.name = '' if self.write_memory: self.files = { } @@ -140,7 +145,13 @@ def __str__(self): def getDict(self): return self.data + def write_or_dump_fixed_ram(self): + pass + def read(self): + if self.fixed_ram and self.ram_read: + return + # Read all RAM sections while True: addr = self.file.read64() @@ -167,7 +178,26 @@ def read(self): f.truncate(0) f.truncate(len) self.files[self.name] = f + + if self.fixed_ram: + bitmap_len = self.file.read32() + # skip the pages_offset which we don't need + offset = self.file.tell() + 8 + self.bitmap_offset[self.name] = offset + offset = ((offset + bitmap_len + self.TARGET_PAGE_SIZE - 1) // + self.TARGET_PAGE_SIZE) * self.TARGET_PAGE_SIZE + self.pages_offset[self.name] = offset + self.file.file.seek(offset + len) + flags &= ~self.RAM_SAVE_FLAG_MEM_SIZE + if self.fixed_ram: + self.ram_read = True + # now we should rewind to the ram page offset of the first + # ram section + if self.fixed_ram: + if self.write_memory or self.dump_memory: + self.write_or_dump_fixed_ram() + return if flags & self.RAM_SAVE_FLAG_COMPRESS: if flags & self.RAM_SAVE_FLAG_CONTINUE: @@ -208,7 +238,7 @@ def read(self): # End of RAM section if flags & self.RAM_SAVE_FLAG_EOS: - break + return if flags != 0: raise Exception("Unknown RAM flags: %x" % flags) @@ -521,6 +551,7 @@ def read(self, desc_only = False, dump_memory = False, write_memory = False): ramargs['page_size'] = self.vmsd_desc['page_size'] ramargs['dump_memory'] = dump_memory ramargs['write_memory'] = write_memory + ramargs['fixed-ram'] = False self.section_classes[('ram',0)][1] = ramargs while True: @@ -528,8 +559,20 @@ def read(self, desc_only = False, dump_memory = False, write_memory = False): if section_type == self.QEMU_VM_EOF: break elif section_type == self.QEMU_VM_CONFIGURATION: - section = ConfigurationSection(file) - section.read() + config_desc = self.vmsd_desc.get('configuration') + if config_desc is not None: + config = VMSDSection(file, 1, config_desc, 'configuration') + config.read() + caps = config.data.get("configuration/capabilities") + if caps is not None: + caps = caps.data["capabilities"] + if type(caps) != list: + caps = [caps] + for i in caps: + # chomp out string length + cap = i.data[1:].decode("utf8") + if cap == "fixed-ram": + ramargs['fixed-ram'] = True elif section_type == self.QEMU_VM_SECTION_START or section_type == self.QEMU_VM_SECTION_FULL: section_id = file.read32() name = file.readstr() From patchwork Thu Mar 30 18:03:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194700 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9BEE7C761A6 for ; Thu, 30 Mar 2023 18:06:15 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwd7-0003Ef-6S; Thu, 30 Mar 2023 14:04:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwd5-0003DC-Er for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:07 -0400 Received: from smtp-out2.suse.de ([2001:67c:2178:6::1d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwd3-00022H-FK for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:07 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 9CE571F86C; Thu, 30 Mar 2023 18:03:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199439; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mJoiJagPKO3t1jkKJrbz4/vjoeEXgERDW7YnvkLu2iI=; b=btmmURmEsKQ1qXpv/sGq7pgqjjyYP2Jemw1WRB9P5ID60kWlGxP9HG+BIJQJa5GTGx0mS4 U3kUr41m9NcclequkcpP49NquXjBNiD4otEur/WEvBqzv+Hbjv4mafouNBu4KVbX/mNo6t QRO81nk/lNJ+2KiamVMh1lK4zWTEGks= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199439; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mJoiJagPKO3t1jkKJrbz4/vjoeEXgERDW7YnvkLu2iI=; b=eMoloApYy1a7Z8XJwxMaIGRldLZfy/BGKw11gPG8OYaeg9QgTtIKdn+YmW0ZEjwtoUOkop T/hIBSwmHiS77qAQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 53D101348E; Thu, 30 Mar 2023 18:03:57 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id kPL9Bg3PJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:03:57 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Nikolay Borisov Subject: [RFC PATCH v1 06/26] io: add and implement QIO_CHANNEL_FEATURE_SEEKABLE for channel file Date: Thu, 30 Mar 2023 15:03:16 -0300 Message-Id: <20230330180336.2791-7-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1d; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Nikolay Borisov Add a generic QIOChannel feature SEEKABLE which would be used by the qemu_file* apis. For the time being this will be only implemented for file channels. Signed-off-by: Nikolay Borisov Signed-off-by: Fabiano Rosas Reviewed-by: Daniel P. Berrangé --- include/io/channel.h | 1 + io/channel-file.c | 8 ++++++++ 2 files changed, 9 insertions(+) diff --git a/include/io/channel.h b/include/io/channel.h index 153fbd2904..29461dda41 100644 --- a/include/io/channel.h +++ b/include/io/channel.h @@ -44,6 +44,7 @@ enum QIOChannelFeature { QIO_CHANNEL_FEATURE_LISTEN, QIO_CHANNEL_FEATURE_WRITE_ZERO_COPY, QIO_CHANNEL_FEATURE_READ_MSG_PEEK, + QIO_CHANNEL_FEATURE_SEEKABLE, }; diff --git a/io/channel-file.c b/io/channel-file.c index d76663e6ae..a0268232da 100644 --- a/io/channel-file.c +++ b/io/channel-file.c @@ -35,6 +35,10 @@ qio_channel_file_new_fd(int fd) ioc->fd = fd; + if (lseek(fd, 0, SEEK_CUR) != (off_t)-1) { + qio_channel_set_feature(QIO_CHANNEL(ioc), QIO_CHANNEL_FEATURE_SEEKABLE); + } + trace_qio_channel_file_new_fd(ioc, fd); return ioc; @@ -59,6 +63,10 @@ qio_channel_file_new_path(const char *path, return NULL; } + if (lseek(ioc->fd, 0, SEEK_CUR) != (off_t)-1) { + qio_channel_set_feature(QIO_CHANNEL(ioc), QIO_CHANNEL_FEATURE_SEEKABLE); + } + trace_qio_channel_file_new_path(ioc, path, flags, mode, ioc->fd); return ioc; From patchwork Thu Mar 30 18:03:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194708 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DA409C761A6 for ; Thu, 30 Mar 2023 18:08:20 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwd8-0003FV-Ov; Thu, 30 Mar 2023 14:04:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwd5-0003DQ-R4 for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:07 -0400 Received: from smtp-out1.suse.de ([2001:67c:2178:6::1c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwd4-00022h-0b for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:07 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 6FA2321A31; Thu, 30 Mar 2023 18:04:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199442; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KeuGyQB6RtYTAbbkJrvUO7WzLSrtPBHu7V3PK6VPtuk=; b=ed/qonEzEpOS6Bd95rDSLvfGLYh84zO8WyK2CcBTC4xi/eXdx3S+XFsqAWipGqroA0GMNE 6oZU7ll0Cnjxnj2E60UOxl1YgMulF8G0gAX7s5breKy7J3Wsc0DjHmYc2OvsFbBcR56R1w v+ghmoAIu22XomV7+g+CHbD1XezcGMY= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199442; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KeuGyQB6RtYTAbbkJrvUO7WzLSrtPBHu7V3PK6VPtuk=; b=bVTcQijmFnZCEVfXEH3/5rGUqRaswfXK6CZeR/eth+mDwo8segrderUxBm0kitu2uXipxa eCJt3/hGJvxH3uBw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 1AEB21348E; Thu, 30 Mar 2023 18:03:59 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id wDuYNA/PJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:03:59 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Nikolay Borisov Subject: [RFC PATCH v1 07/26] io: Add generic pwritev/preadv interface Date: Thu, 30 Mar 2023 15:03:17 -0300 Message-Id: <20230330180336.2791-8-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1c; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Nikolay Borisov Introduce basic pwritev/preadv support in the generic channel layer. Specific implementation will follow for the file channel as this is required in order to support migration streams with fixed location of each ram page. Signed-off-by: Nikolay Borisov Signed-off-by: Fabiano Rosas --- include/io/channel.h | 82 ++++++++++++++++++++++++++++++++++++++++++++ io/channel.c | 58 +++++++++++++++++++++++++++++++ 2 files changed, 140 insertions(+) diff --git a/include/io/channel.h b/include/io/channel.h index 29461dda41..28bce7ef17 100644 --- a/include/io/channel.h +++ b/include/io/channel.h @@ -129,6 +129,16 @@ struct QIOChannelClass { Error **errp); /* Optional callbacks */ + ssize_t (*io_pwritev)(QIOChannel *ioc, + const struct iovec *iov, + size_t niov, + off_t offset, + Error **errp); + ssize_t (*io_preadv)(QIOChannel *ioc, + const struct iovec *iov, + size_t niov, + off_t offset, + Error **errp); int (*io_shutdown)(QIOChannel *ioc, QIOChannelShutdown how, Error **errp); @@ -511,6 +521,78 @@ int qio_channel_set_blocking(QIOChannel *ioc, int qio_channel_close(QIOChannel *ioc, Error **errp); +/** + * qio_channel_pwritev_full + * @ioc: the channel object + * @iov: the array of memory regions to write data from + * @niov: the length of the @iov array + * @offset: offset in the channel where writes should begin + * @errp: pointer to a NULL-initialized error object + * + * Not all implementations will support this facility, so may report + * an error. To avoid errors, the caller may check for the feature + * flag QIO_CHANNEL_FEATURE_SEEKABLE prior to calling this method. + * + * Behaves as qio_channel_writev_full, apart from not supporting + * sending of file handles as well as beginning the write at the + * passed @offset + * + */ +ssize_t qio_channel_pwritev_full(QIOChannel *ioc, const struct iovec *iov, + size_t niov, off_t offset, Error **errp); + +/** + * qio_channel_pwritev + * @ioc: the channel object + * @buf: the memory region to write data into + * @buflen: the number of bytes to @buf + * @offset: offset in the channel where writes should begin + * @errp: pointer to a NULL-initialized error object + * + * Not all implementations will support this facility, so may report + * an error. To avoid errors, the caller may check for the feature + * flag QIO_CHANNEL_FEATURE_SEEKABLE prior to calling this method. + * + */ +ssize_t qio_channel_pwritev(QIOChannel *ioc, char *buf, size_t buflen, + off_t offset, Error **errp); + +/** + * qio_channel_preadv_full + * @ioc: the channel object + * @iov: the array of memory regions to read data into + * @niov: the length of the @iov array + * @offset: offset in the channel where writes should begin + * @errp: pointer to a NULL-initialized error object + * + * Not all implementations will support this facility, so may report + * an error. To avoid errors, the caller may check for the feature + * flag QIO_CHANNEL_FEATURE_SEEKABLE prior to calling this method. + * + * Behaves as qio_channel_readv_full, apart from not supporting + * receiving of file handles as well as beginning the read at the + * passed @offset + * + */ +ssize_t qio_channel_preadv_full(QIOChannel *ioc, const struct iovec *iov, + size_t niov, off_t offset, Error **errp); + +/** + * qio_channel_preadv + * @ioc: the channel object + * @buf: the memory region to write data into + * @buflen: the number of bytes to @buf + * @offset: offset in the channel where writes should begin + * @errp: pointer to a NULL-initialized error object + * + * Not all implementations will support this facility, so may report + * an error. To avoid errors, the caller may check for the feature + * flag QIO_CHANNEL_FEATURE_SEEKABLE prior to calling this method. + * + */ +ssize_t qio_channel_preadv(QIOChannel *ioc, char *buf, size_t buflen, + off_t offset, Error **errp); + /** * qio_channel_shutdown: * @ioc: the channel object diff --git a/io/channel.c b/io/channel.c index a8c7f11649..312445b3aa 100644 --- a/io/channel.c +++ b/io/channel.c @@ -445,6 +445,64 @@ GSource *qio_channel_add_watch_source(QIOChannel *ioc, } +ssize_t qio_channel_pwritev_full(QIOChannel *ioc, const struct iovec *iov, + size_t niov, off_t offset, Error **errp) +{ + QIOChannelClass *klass = QIO_CHANNEL_GET_CLASS(ioc); + + if (!klass->io_pwritev) { + error_setg(errp, "Channel does not support pwritev"); + return -1; + } + + if (!qio_channel_has_feature(ioc, QIO_CHANNEL_FEATURE_SEEKABLE)) { + error_setg_errno(errp, EINVAL, "Requested channel is not seekable"); + return -1; + } + + return klass->io_pwritev(ioc, iov, niov, offset, errp); +} + +ssize_t qio_channel_pwritev(QIOChannel *ioc, char *buf, size_t buflen, + off_t offset, Error **errp) +{ + struct iovec iov = { + .iov_base = buf, + .iov_len = buflen + }; + + return qio_channel_pwritev_full(ioc, &iov, 1, offset, errp); +} + +ssize_t qio_channel_preadv_full(QIOChannel *ioc, const struct iovec *iov, + size_t niov, off_t offset, Error **errp) +{ + QIOChannelClass *klass = QIO_CHANNEL_GET_CLASS(ioc); + + if (!klass->io_preadv) { + error_setg(errp, "Channel does not support preadv"); + return -1; + } + + if (!qio_channel_has_feature(ioc, QIO_CHANNEL_FEATURE_SEEKABLE)) { + error_setg_errno(errp, EINVAL, "Requested channel is not seekable"); + return -1; + } + + return klass->io_preadv(ioc, iov, niov, offset, errp); +} + +ssize_t qio_channel_preadv(QIOChannel *ioc, char *buf, size_t buflen, + off_t offset, Error **errp) +{ + struct iovec iov = { + .iov_base = buf, + .iov_len = buflen + }; + + return qio_channel_preadv_full(ioc, &iov, 1, offset, errp); +} + int qio_channel_shutdown(QIOChannel *ioc, QIOChannelShutdown how, Error **errp) From patchwork Thu Mar 30 18:03:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194710 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BEEF6C77B60 for ; Thu, 30 Mar 2023 18:08:23 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwd8-0003F4-4W; Thu, 30 Mar 2023 14:04:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwd6-0003EV-OW for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:08 -0400 Received: from smtp-out1.suse.de ([2001:67c:2178:6::1c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwd4-000231-8M for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:08 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 2F77321A3A; Thu, 30 Mar 2023 18:04:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199445; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lZHIAYJbTcNFGV7RvsNwFPqk358QZF0Jidk56nhi3ZQ=; b=WT7vGwlet30JmgnJaB9RJfEK3Kf9i3o7VcbxEQrz7NCxSCM3JLWkWXffQElg7n2FhFr3W6 cX0WDj4EJG6EJ6KjAi0ejWfGFuCfiEHkk4BmKi0NGOTC1O/RD1x9aRBkGGrcj5c5OqPMTF NDUHD5ddzb6JpVtiO4CzRcAw84V1xKI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199445; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lZHIAYJbTcNFGV7RvsNwFPqk358QZF0Jidk56nhi3ZQ=; b=lCnh6kGMUOOCsITKqviQldPM6rHYlvybgIBDDxlm3Zppa7/7iyUOqMW36cLRe25v/ecbvm 594eA/4lshrORwDA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id DA1281348E; Thu, 30 Mar 2023 18:04:02 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id aGubJxLPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:02 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Nikolay Borisov Subject: [RFC PATCH v1 08/26] io: implement io_pwritev/preadv for QIOChannelFile Date: Thu, 30 Mar 2023 15:03:18 -0300 Message-Id: <20230330180336.2791-9-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1c; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Nikolay Borisov The upcoming 'fixed-ram' feature will require qemu to write data to (and restore from) specific offsets of the migration file. Add a minimal implementation of pwritev/preadv and expose them via the io_pwritev and io_preadv interfaces. Signed-off-by: Nikolay Borisov Signed-off-by: Fabiano Rosas Reviewed-by: Daniel P. Berrangé --- io/channel-file.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/io/channel-file.c b/io/channel-file.c index a0268232da..a3d2f0bcf9 100644 --- a/io/channel-file.c +++ b/io/channel-file.c @@ -145,6 +145,56 @@ static ssize_t qio_channel_file_writev(QIOChannel *ioc, return ret; } +static ssize_t qio_channel_file_preadv(QIOChannel *ioc, + const struct iovec *iov, + size_t niov, + off_t offset, + Error **errp) +{ + QIOChannelFile *fioc = QIO_CHANNEL_FILE(ioc); + ssize_t ret; + + retry: + ret = preadv(fioc->fd, iov, niov, offset); + if (ret < 0) { + if (errno == EAGAIN) { + return QIO_CHANNEL_ERR_BLOCK; + } + if (errno == EINTR) { + goto retry; + } + + error_setg_errno(errp, errno, "Unable to read from file"); + return -1; + } + + return ret; +} + +static ssize_t qio_channel_file_pwritev(QIOChannel *ioc, + const struct iovec *iov, + size_t niov, + off_t offset, + Error **errp) +{ + QIOChannelFile *fioc = QIO_CHANNEL_FILE(ioc); + ssize_t ret; + + retry: + ret = pwritev(fioc->fd, iov, niov, offset); + if (ret <= 0) { + if (errno == EAGAIN) { + return QIO_CHANNEL_ERR_BLOCK; + } + if (errno == EINTR) { + goto retry; + } + error_setg_errno(errp, errno, "Unable to write to file"); + return -1; + } + return ret; +} + static int qio_channel_file_set_blocking(QIOChannel *ioc, bool enabled, Error **errp) @@ -227,6 +277,8 @@ static void qio_channel_file_class_init(ObjectClass *klass, ioc_klass->io_writev = qio_channel_file_writev; ioc_klass->io_readv = qio_channel_file_readv; ioc_klass->io_set_blocking = qio_channel_file_set_blocking; + ioc_klass->io_pwritev = qio_channel_file_pwritev; + ioc_klass->io_preadv = qio_channel_file_preadv; ioc_klass->io_seek = qio_channel_file_seek; ioc_klass->io_close = qio_channel_file_close; ioc_klass->io_create_watch = qio_channel_file_create_watch; From patchwork Thu Mar 30 18:03:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194698 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 649FBC761A6 for ; Thu, 30 Mar 2023 18:05:52 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdC-0003Fi-1d; Thu, 30 Mar 2023 14:04:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwd8-0003FN-LN for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:10 -0400 Received: from smtp-out2.suse.de ([195.135.220.29]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwd7-00023c-0X for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:10 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id E6B191F86C; Thu, 30 Mar 2023 18:04:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199447; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w2iIpM0EFvr9p+63Rr5g9tPKF2Wyn2N9bky1lumiT4A=; b=DmIt0xElaKmXFCoSxwJIB3Yb0z2AjHFWMrN/876ENsGnj6K50IljkYzb77GUGZzbE+H5aA PHpoaSYrsWdUafQzX0VXPz0EOyLQujOra1p6W3w9VGiBM84rtcmf8P/z+cwsHx79VYe1lc 9OAj6zsupPogExeCzaVfb9HgfxfS/j4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199447; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w2iIpM0EFvr9p+63Rr5g9tPKF2Wyn2N9bky1lumiT4A=; b=EdVcAXmzsvJShHrCgq9geeo8Omm2w8XVR34uyNNZIaceOHRK/LjfEjqjH9KdTfOvx4oO8i UeXBSVho9cyU63Dw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id A28D81348E; Thu, 30 Mar 2023 18:04:05 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id IG1EGhXPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:05 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Nikolay Borisov Subject: [RFC PATCH v1 09/26] migration/qemu-file: add utility methods for working with seekable channels Date: Thu, 30 Mar 2023 15:03:19 -0300 Message-Id: <20230330180336.2791-10-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=195.135.220.29; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Nikolay Borisov Add utility methods that will be needed when implementing 'fixed-ram' migration capability. qemu_file_is_seekable qemu_put_buffer_at qemu_get_buffer_at qemu_set_offset qemu_get_offset Signed-off-by: Nikolay Borisov Signed-off-by: Fabiano Rosas --- fixed total_transferred accounting restructured to use qio_channel_file_preadv instead of the _full variant --- include/migration/qemu-file-types.h | 2 + migration/qemu-file.c | 80 +++++++++++++++++++++++++++++ migration/qemu-file.h | 4 ++ 3 files changed, 86 insertions(+) diff --git a/include/migration/qemu-file-types.h b/include/migration/qemu-file-types.h index 2867e3da84..eb0325ee86 100644 --- a/include/migration/qemu-file-types.h +++ b/include/migration/qemu-file-types.h @@ -50,6 +50,8 @@ unsigned int qemu_get_be16(QEMUFile *f); unsigned int qemu_get_be32(QEMUFile *f); uint64_t qemu_get_be64(QEMUFile *f); +bool qemu_file_is_seekable(QEMUFile *f); + static inline void qemu_put_be64s(QEMUFile *f, const uint64_t *pv) { qemu_put_be64(f, *pv); diff --git a/migration/qemu-file.c b/migration/qemu-file.c index 102ab3b439..a1f7dbb3d9 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -30,6 +30,7 @@ #include "qemu-file.h" #include "trace.h" #include "qapi/error.h" +#include "io/channel-file.h" #define IO_BUF_SIZE 32768 #define MAX_IOV_SIZE MIN_CONST(IOV_MAX, 64) @@ -281,6 +282,10 @@ static void qemu_iovec_release_ram(QEMUFile *f) memset(f->may_free, 0, sizeof(f->may_free)); } +bool qemu_file_is_seekable(QEMUFile *f) +{ + return qio_channel_has_feature(f->ioc, QIO_CHANNEL_FEATURE_SEEKABLE); +} /** * Flushes QEMUFile buffer @@ -559,6 +564,81 @@ void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, size_t size) } } +void qemu_put_buffer_at(QEMUFile *f, const uint8_t *buf, size_t buflen, off_t pos) +{ + Error *err = NULL; + + if (f->last_error) { + return; + } + + qemu_fflush(f); + qio_channel_pwritev(f->ioc, (char *)buf, buflen, pos, &err); + + if (err) { + qemu_file_set_error_obj(f, -EIO, err); + } else { + f->total_transferred += buflen; + } + + return; +} + + +size_t qemu_get_buffer_at(QEMUFile *f, const uint8_t *buf, size_t buflen, off_t pos) +{ + Error *err = NULL; + ssize_t ret; + + if (f->last_error) { + return 0; + } + + ret = qio_channel_preadv(f->ioc, (char *)buf, buflen, pos, &err); + if (ret == -1 || err) { + goto error; + } + + return (size_t)ret; + + error: + qemu_file_set_error_obj(f, -EIO, err); + return 0; +} + +void qemu_set_offset(QEMUFile *f, off_t off, int whence) +{ + Error *err = NULL; + off_t ret; + + qemu_fflush(f); + + if (!qemu_file_is_writable(f)) { + f->buf_index = 0; + f->buf_size = 0; + } + + ret = qio_channel_io_seek(f->ioc, off, whence, &err); + if (ret == (off_t)-1) { + qemu_file_set_error_obj(f, -EIO, err); + } +} + +off_t qemu_get_offset(QEMUFile *f) +{ + Error *err = NULL; + off_t ret; + + qemu_fflush(f); + + ret = qio_channel_io_seek(f->ioc, 0, SEEK_CUR, &err); + if (ret == (off_t)-1) { + qemu_file_set_error_obj(f, -EIO, err); + } + return ret; +} + + void qemu_put_byte(QEMUFile *f, int v) { if (f->last_error) { diff --git a/migration/qemu-file.h b/migration/qemu-file.h index 9d0155a2a1..350273b441 100644 --- a/migration/qemu-file.h +++ b/migration/qemu-file.h @@ -149,6 +149,10 @@ QEMUFile *qemu_file_get_return_path(QEMUFile *f); void qemu_fflush(QEMUFile *f); void qemu_file_set_blocking(QEMUFile *f, bool block); int qemu_file_get_to_fd(QEMUFile *f, int fd, size_t size); +void qemu_set_offset(QEMUFile *f, off_t off, int whence); +off_t qemu_get_offset(QEMUFile *f); +void qemu_put_buffer_at(QEMUFile *f, const uint8_t *buf, size_t buflen, off_t pos); +size_t qemu_get_buffer_at(QEMUFile *f, const uint8_t *buf, size_t buflen, off_t pos); void ram_control_before_iterate(QEMUFile *f, uint64_t flags); void ram_control_after_iterate(QEMUFile *f, uint64_t flags); From patchwork Thu Mar 30 18:03:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4DE4BC761A6 for ; Thu, 30 Mar 2023 18:05:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdI-0003HN-Ee; Thu, 30 Mar 2023 14:04:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdG-0003Gx-Tv for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:18 -0400 Received: from smtp-out1.suse.de ([195.135.220.28]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdB-00024D-6D for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:15 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 202BB21A31; Thu, 30 Mar 2023 18:04:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199452; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HE58zlgiBSuVmP1Wz3AAzAfLvpRSOuwpAWdDcUm/D4I=; b=UiWgXvsV69PQosqyB+Iz/ADomI2tABJStAndzv1EOkaOlCEqtEAgGHpiE1pqQP1wo4woWC /rfBhNzPw/tJJmU4ox4CbYi58bOpvRBlYcaZIVYtoAZL5tb4mA+c0gA33bqGAwSan8iGwo 6+nESElmt/y3K/bTCkmz6N3btdvhgEY= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199452; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HE58zlgiBSuVmP1Wz3AAzAfLvpRSOuwpAWdDcUm/D4I=; b=pljTN5Wc6DrPnT/8aFOFLZKxfPGRhkZtbbeQvZT6779VNHf9tan21uq0dxXo6QswYtaETK Tt9krkfIjzVnm7AQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 6CE871348E; Thu, 30 Mar 2023 18:04:08 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id aPASDRjPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:08 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Nikolay Borisov , Paolo Bonzini , Peter Xu , David Hildenbrand , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Eric Blake , Markus Armbruster Subject: [RFC PATCH v1 10/26] migration/ram: Introduce 'fixed-ram' migration stream capability Date: Thu, 30 Mar 2023 15:03:20 -0300 Message-Id: <20230330180336.2791-11-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=195.135.220.28; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Nikolay Borisov Implement 'fixed-ram' feature. The core of the feature is to ensure that each ram page of the migration stream has a specific offset in the resulting migration stream. The reason why we'd want such behavior are two fold: - When doing a 'fixed-ram' migration the resulting file will have a bounded size, since pages which are dirtied multiple times will always go to a fixed location in the file, rather than constantly being added to a sequential stream. This eliminates cases where a vm with, say, 1G of ram can result in a migration file that's 10s of GBs, provided that the workload constantly redirties memory. - It paves the way to implement DIO-enabled save/restore of the migration stream as the pages are ensured to be written at aligned offsets. The feature requires changing the stream format. First, a bitmap is introduced which tracks which pages have been written (i.e are dirtied) during migration and subsequently it's being written in the resulting file, again at a fixed location for every ramblock. Zero pages are ignored as they'd be zero in the destination migration as well. With the changed format data would look like the following: |name len|name|used_len|pc*|bitmap_size|pages_offset|bitmap|pages| * pc - refers to the page_size/mr->addr members, so newly added members begin from "bitmap_size". This layout is initialized during ram_save_setup so instead of having a sequential stream of pages that follow the ramblock headers the dirty pages for a ramblock follow its header. Since all pages have a fixed location RAM_SAVE_FLAG_EOS is no longer generated on every migration iteration but there is effectively a single RAM_SAVE_FLAG_EOS right at the end. Signed-off-by: Nikolay Borisov Signed-off-by: Fabiano Rosas --- docs/devel/migration.rst | 36 +++++++++++++++ include/exec/ramblock.h | 8 ++++ migration/migration.c | 51 +++++++++++++++++++++- migration/migration.h | 1 + migration/ram.c | 94 +++++++++++++++++++++++++++++++++------- migration/savevm.c | 1 + qapi/migration.json | 2 +- 7 files changed, 176 insertions(+), 17 deletions(-) diff --git a/docs/devel/migration.rst b/docs/devel/migration.rst index 1080211f8e..84112d7f3f 100644 --- a/docs/devel/migration.rst +++ b/docs/devel/migration.rst @@ -568,6 +568,42 @@ Others (especially either older devices or system devices which for some reason don't have a bus concept) make use of the ``instance id`` for otherwise identically named devices. +Fixed-ram format +---------------- + +When the ``fixed-ram`` capability is enabled, a slightly different +stream format is used for the RAM section. Instead of having a +sequential stream of pages that follow the RAMBlock headers, the dirty +pages for a RAMBlock follow its header. This ensures that each RAM +page has a fixed offset in the resulting migration stream. + + - RAMBlock 1 + + - ID string length + - ID string + - Used size + - Shadow bitmap size + - Pages offset in migration stream* + + - Shadow bitmap + - Sequence of pages for RAMBlock 1 (* offset points here) + + - RAMBlock 2 + + - ID string length + - ID string + - Used size + - Shadow bitmap size + - Pages offset in migration stream* + + - Shadow bitmap + - Sequence of pages for RAMBlock 2 (* offset points here) + +The ``fixed-ram`` capaility can be enabled in both source and +destination with: + + ``migrate_set_capability fixed-ram on`` + Return path ----------- diff --git a/include/exec/ramblock.h b/include/exec/ramblock.h index adc03df59c..4360c772c2 100644 --- a/include/exec/ramblock.h +++ b/include/exec/ramblock.h @@ -43,6 +43,14 @@ struct RAMBlock { size_t page_size; /* dirty bitmap used during migration */ unsigned long *bmap; + /* shadow dirty bitmap used when migrating to a file */ + unsigned long *shadow_bmap; + /* + * offset in the file pages belonging to this ramblock are saved, + * used only during migration to a file. + */ + off_t bitmap_offset; + uint64_t pages_offset; /* bitmap of already received pages in postcopy */ unsigned long *receivedmap; diff --git a/migration/migration.c b/migration/migration.c index 177fb0de0f..29630523e2 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -168,7 +168,8 @@ INITIALIZE_MIGRATE_CAPS_SET(check_caps_background_snapshot, MIGRATION_CAPABILITY_XBZRLE, MIGRATION_CAPABILITY_X_COLO, MIGRATION_CAPABILITY_VALIDATE_UUID, - MIGRATION_CAPABILITY_ZERO_COPY_SEND); + MIGRATION_CAPABILITY_ZERO_COPY_SEND, + MIGRATION_CAPABILITY_FIXED_RAM); /* When we add fault tolerance, we could have several migrations at once. For now we don't need to add @@ -1341,6 +1342,28 @@ static bool migrate_caps_check(bool *cap_list, } #endif + if (cap_list[MIGRATION_CAPABILITY_FIXED_RAM]) { + if (cap_list[MIGRATION_CAPABILITY_MULTIFD]) { + error_setg(errp, "Directly mapped memory incompatible with multifd"); + return false; + } + + if (cap_list[MIGRATION_CAPABILITY_XBZRLE]) { + error_setg(errp, "Directly mapped memory incompatible with xbzrle"); + return false; + } + + if (cap_list[MIGRATION_CAPABILITY_COMPRESS]) { + error_setg(errp, "Directly mapped memory incompatible with compression"); + return false; + } + + if (cap_list[MIGRATION_CAPABILITY_POSTCOPY_RAM]) { + error_setg(errp, "Directly mapped memory incompatible with postcopy ram"); + return false; + } + } + if (cap_list[MIGRATION_CAPABILITY_POSTCOPY_RAM]) { /* This check is reasonably expensive, so only when it's being * set the first time, also it's only the destination that needs @@ -2736,6 +2759,11 @@ MultiFDCompression migrate_multifd_compression(void) return s->parameters.multifd_compression; } +int migrate_fixed_ram(void) +{ + return migrate_get_current()->enabled_capabilities[MIGRATION_CAPABILITY_FIXED_RAM]; +} + int migrate_multifd_zlib_level(void) { MigrationState *s; @@ -4324,6 +4352,20 @@ fail: return NULL; } +static int migrate_check_fixed_ram(MigrationState *s, Error **errp) +{ + if (!s->enabled_capabilities[MIGRATION_CAPABILITY_FIXED_RAM]) { + return 0; + } + + if (!qemu_file_is_seekable(s->to_dst_file)) { + error_setg(errp, "Directly mapped memory requires a seekable transport"); + return -1; + } + + return 0; +} + void migrate_fd_connect(MigrationState *s, Error *error_in) { Error *local_err = NULL; @@ -4390,6 +4432,12 @@ void migrate_fd_connect(MigrationState *s, Error *error_in) } } + if (migrate_check_fixed_ram(s, &local_err) < 0) { + migrate_fd_cleanup(s); + migrate_fd_error(s, local_err); + return; + } + if (resume) { /* Wakeup the main migration thread to do the recovery */ migrate_set_state(&s->state, MIGRATION_STATUS_POSTCOPY_PAUSED, @@ -4519,6 +4567,7 @@ static Property migration_properties[] = { DEFINE_PROP_STRING("tls-authz", MigrationState, parameters.tls_authz), /* Migration capabilities */ + DEFINE_PROP_MIG_CAP("x-fixed-ram", MIGRATION_CAPABILITY_FIXED_RAM), DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE), DEFINE_PROP_MIG_CAP("x-rdma-pin-all", MIGRATION_CAPABILITY_RDMA_PIN_ALL), DEFINE_PROP_MIG_CAP("x-auto-converge", MIGRATION_CAPABILITY_AUTO_CONVERGE), diff --git a/migration/migration.h b/migration/migration.h index 2da2f8a164..8cf3caecfe 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -416,6 +416,7 @@ bool migrate_zero_blocks(void); bool migrate_dirty_bitmaps(void); bool migrate_ignore_shared(void); bool migrate_validate_uuid(void); +int migrate_fixed_ram(void); bool migrate_auto_converge(void); bool migrate_use_multifd(void); diff --git a/migration/ram.c b/migration/ram.c index 96e8a19a58..56f0f782c8 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1310,9 +1310,14 @@ static int save_zero_page_to_file(PageSearchStatus *pss, int len = 0; if (buffer_is_zero(p, TARGET_PAGE_SIZE)) { - len += save_page_header(pss, block, offset | RAM_SAVE_FLAG_ZERO); - qemu_put_byte(file, 0); - len += 1; + if (migrate_fixed_ram()) { + /* for zero pages we don't need to do anything */ + len = 1; + } else { + len += save_page_header(pss, block, offset | RAM_SAVE_FLAG_ZERO); + qemu_put_byte(file, 0); + len += 1; + } ram_release_page(block->idstr, offset); } return len; @@ -1394,14 +1399,20 @@ static int save_normal_page(PageSearchStatus *pss, RAMBlock *block, { QEMUFile *file = pss->pss_channel; - ram_transferred_add(save_page_header(pss, block, - offset | RAM_SAVE_FLAG_PAGE)); - if (async) { - qemu_put_buffer_async(file, buf, TARGET_PAGE_SIZE, - migrate_release_ram() && - migration_in_postcopy()); + if (migrate_fixed_ram()) { + qemu_put_buffer_at(file, buf, TARGET_PAGE_SIZE, + block->pages_offset + offset); + set_bit(offset >> TARGET_PAGE_BITS, block->shadow_bmap); } else { - qemu_put_buffer(file, buf, TARGET_PAGE_SIZE); + ram_transferred_add(save_page_header(pss, block, + offset | RAM_SAVE_FLAG_PAGE)); + if (async) { + qemu_put_buffer_async(file, buf, TARGET_PAGE_SIZE, + migrate_release_ram() && + migration_in_postcopy()); + } else { + qemu_put_buffer(file, buf, TARGET_PAGE_SIZE); + } } ram_transferred_add(TARGET_PAGE_SIZE); stat64_add(&ram_atomic_counters.normal, 1); @@ -2731,6 +2742,8 @@ static void ram_save_cleanup(void *opaque) block->clear_bmap = NULL; g_free(block->bmap); block->bmap = NULL; + g_free(block->shadow_bmap); + block->shadow_bmap = NULL; } xbzrle_cleanup(); @@ -3098,6 +3111,7 @@ static void ram_list_init_bitmaps(void) */ block->bmap = bitmap_new(pages); bitmap_set(block->bmap, 0, pages); + block->shadow_bmap = bitmap_new(block->used_length >> TARGET_PAGE_BITS); block->clear_bmap_shift = shift; block->clear_bmap = bitmap_new(clear_bmap_size(pages, shift)); } @@ -3287,6 +3301,33 @@ static int ram_save_setup(QEMUFile *f, void *opaque) if (migrate_ignore_shared()) { qemu_put_be64(f, block->mr->addr); } + + if (migrate_fixed_ram()) { + long num_pages = block->used_length >> TARGET_PAGE_BITS; + long bitmap_size = BITS_TO_LONGS(num_pages) * sizeof(unsigned long); + + /* Needed for external programs (think analyze-migration.py) */ + qemu_put_be32(f, bitmap_size); + + /* + * The bitmap starts after pages_offset, so add 8 to + * account for the pages_offset size. + */ + block->bitmap_offset = qemu_get_offset(f) + 8; + + /* + * Make pages_offset aligned to 1 MiB to account for + * migration file movement between filesystems with + * possibly different alignment restrictions when + * using O_DIRECT. + */ + block->pages_offset = ROUND_UP(block->bitmap_offset + + bitmap_size, 0x100000); + qemu_put_be64(f, block->pages_offset); + + /* Now prepare offset for next ramblock */ + qemu_set_offset(f, block->pages_offset + block->used_length, SEEK_SET); + } } } @@ -3306,6 +3347,18 @@ static int ram_save_setup(QEMUFile *f, void *opaque) return 0; } +static void ram_save_shadow_bmap(QEMUFile *f) +{ + RAMBlock *block; + + RAMBLOCK_FOREACH_MIGRATABLE(block) { + long num_pages = block->used_length >> TARGET_PAGE_BITS; + long bitmap_size = BITS_TO_LONGS(num_pages) * sizeof(unsigned long); + qemu_put_buffer_at(f, (uint8_t *)block->shadow_bmap, bitmap_size, + block->bitmap_offset); + } +} + /** * ram_save_iterate: iterative stage for migration * @@ -3413,9 +3466,15 @@ out: return ret; } - qemu_put_be64(f, RAM_SAVE_FLAG_EOS); - qemu_fflush(f); - ram_transferred_add(8); + /* + * For fixed ram we don't want to pollute the migration stream with + * EOS flags. + */ + if (!migrate_fixed_ram()) { + qemu_put_be64(f, RAM_SAVE_FLAG_EOS); + qemu_fflush(f); + ram_transferred_add(8); + } ret = qemu_file_get_error(f); } @@ -3461,6 +3520,9 @@ static int ram_save_complete(QEMUFile *f, void *opaque) pages = ram_find_and_save_block(rs); /* no more blocks to sent */ if (pages == 0) { + if (migrate_fixed_ram()) { + ram_save_shadow_bmap(f); + } break; } if (pages < 0) { @@ -3483,8 +3545,10 @@ static int ram_save_complete(QEMUFile *f, void *opaque) return ret; } - qemu_put_be64(f, RAM_SAVE_FLAG_EOS); - qemu_fflush(f); + if (!migrate_fixed_ram()) { + qemu_put_be64(f, RAM_SAVE_FLAG_EOS); + qemu_fflush(f); + } return 0; } diff --git a/migration/savevm.c b/migration/savevm.c index 92102c1fe5..1f1bc19224 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -241,6 +241,7 @@ static bool should_validate_capability(int capability) /* Validate only new capabilities to keep compatibility. */ switch (capability) { case MIGRATION_CAPABILITY_X_IGNORE_SHARED: + case MIGRATION_CAPABILITY_FIXED_RAM: return true; default: return false; diff --git a/qapi/migration.json b/qapi/migration.json index c84fa10e86..22eea58ce3 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -485,7 +485,7 @@ ## { 'enum': 'MigrationCapability', 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', - 'compress', 'events', 'postcopy-ram', + 'compress', 'events', 'postcopy-ram', 'fixed-ram', { 'name': 'x-colo', 'features': [ 'unstable' ] }, 'release-ram', 'block', 'return-path', 'pause-before-switchover', 'multifd', From patchwork Thu Mar 30 18:03:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4E857C761A6 for ; Thu, 30 Mar 2023 18:05:47 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdR-0003IC-65; Thu, 30 Mar 2023 14:04:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdI-0003HE-7u for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:20 -0400 Received: from smtp-out1.suse.de ([2001:67c:2178:6::1c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdG-00024U-C7 for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:19 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id DA36221A2E; Thu, 30 Mar 2023 18:04:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199454; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kVyI+EulfkwfpRcLylzEL5w/yl20jYRSdQzOG7bfwY4=; b=yXuMhPQZ4Z1Eu6ODsgB4D+ffQhYaYenXy8ke8pX7QkqQI5T57kXHlH8nttwV6tJM7AU+Ho L3wkWauCpn8cfILPRk5TjLQJxNwPvHIybe/TJkT9Mawvog3Hq1qBO3iZIICN3tiAyLLFKt 0KUALOBtMYr8Fq9oJKjkUI0b2YPC1AQ= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199454; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kVyI+EulfkwfpRcLylzEL5w/yl20jYRSdQzOG7bfwY4=; b=2Zf7kF94laUc6FReMIdKxuYUnlJ/K2qZSKJXScwpJmy1XYqIgzp26+t6e+Hf19BwK000eR 2waTCCnyzyXfjXCQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 922811348E; Thu, 30 Mar 2023 18:04:12 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id GOImFhzPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:12 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Nikolay Borisov Subject: [RFC PATCH v1 11/26] migration: Refactor precopy ram loading code Date: Thu, 30 Mar 2023 15:03:21 -0300 Message-Id: <20230330180336.2791-12-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1c; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Nikolay Borisov To facilitate the implementation of the 'fixed-ram' migration restore, factor out the code responsible for parsing the ramblocks headers. This also makes ram_load_precopy easier to comprehend. Signed-off-by: Nikolay Borisov Signed-off-by: Fabiano Rosas --- migration/ram.c | 142 +++++++++++++++++++++++++++--------------------- 1 file changed, 80 insertions(+), 62 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 56f0f782c8..5c085d6154 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -4319,6 +4319,83 @@ void colo_flush_ram_cache(void) trace_colo_flush_ram_cache_end(); } +static int parse_ramblock(QEMUFile *f, RAMBlock *block, ram_addr_t length) +{ + int ret = 0; + /* ADVISE is earlier, it shows the source has the postcopy capability on */ + bool postcopy_advised = migration_incoming_postcopy_advised(); + + assert(block); + + if (!qemu_ram_is_migratable(block)) { + error_report("block %s should not be migrated !", block->idstr); + ret = -EINVAL; + } + + if (length != block->used_length) { + Error *local_err = NULL; + + ret = qemu_ram_resize(block, length, &local_err); + if (local_err) { + error_report_err(local_err); + } + } + /* For postcopy we need to check hugepage sizes match */ + if (postcopy_advised && migrate_postcopy_ram() && + block->page_size != qemu_host_page_size) { + uint64_t remote_page_size = qemu_get_be64(f); + if (remote_page_size != block->page_size) { + error_report("Mismatched RAM page size %s " + "(local) %zd != %" PRId64, block->idstr, + block->page_size, remote_page_size); + ret = -EINVAL; + } + } + if (migrate_ignore_shared()) { + hwaddr addr = qemu_get_be64(f); + if (ramblock_is_ignored(block) && + block->mr->addr != addr) { + error_report("Mismatched GPAs for block %s " + "%" PRId64 "!= %" PRId64, block->idstr, + (uint64_t)addr, + (uint64_t)block->mr->addr); + ret = -EINVAL; + } + } + ram_control_load_hook(f, RAM_CONTROL_BLOCK_REG, block->idstr); + + return ret; +} + +static int parse_ramblocks(QEMUFile *f, ram_addr_t total_ram_bytes) +{ + int ret = 0; + + /* Synchronize RAM block list */ + while (!ret && total_ram_bytes) { + char id[256]; + RAMBlock *block; + ram_addr_t length; + int len = qemu_get_byte(f); + + qemu_get_buffer(f, (uint8_t *)id, len); + id[len] = 0; + length = qemu_get_be64(f); + + block = qemu_ram_block_by_name(id); + if (block) { + ret = parse_ramblock(f, block, length); + } else { + error_report("Unknown ramblock \"%s\", cannot accept " + "migration", id); + ret = -EINVAL; + } + total_ram_bytes -= length; + } + + return ret; +} + /** * ram_load_precopy: load pages in precopy case * @@ -4333,14 +4410,13 @@ static int ram_load_precopy(QEMUFile *f) { MigrationIncomingState *mis = migration_incoming_get_current(); int flags = 0, ret = 0, invalid_flags = 0, len = 0, i = 0; - /* ADVISE is earlier, it shows the source has the postcopy capability on */ - bool postcopy_advised = migration_incoming_postcopy_advised(); + if (!migrate_use_compression()) { invalid_flags |= RAM_SAVE_FLAG_COMPRESS_PAGE; } while (!ret && !(flags & RAM_SAVE_FLAG_EOS)) { - ram_addr_t addr, total_ram_bytes; + ram_addr_t addr; void *host = NULL, *host_bak = NULL; uint8_t ch; @@ -4411,65 +4487,7 @@ static int ram_load_precopy(QEMUFile *f) switch (flags & ~RAM_SAVE_FLAG_CONTINUE) { case RAM_SAVE_FLAG_MEM_SIZE: - /* Synchronize RAM block list */ - total_ram_bytes = addr; - while (!ret && total_ram_bytes) { - RAMBlock *block; - char id[256]; - ram_addr_t length; - - len = qemu_get_byte(f); - qemu_get_buffer(f, (uint8_t *)id, len); - id[len] = 0; - length = qemu_get_be64(f); - - block = qemu_ram_block_by_name(id); - if (block && !qemu_ram_is_migratable(block)) { - error_report("block %s should not be migrated !", id); - ret = -EINVAL; - } else if (block) { - if (length != block->used_length) { - Error *local_err = NULL; - - ret = qemu_ram_resize(block, length, - &local_err); - if (local_err) { - error_report_err(local_err); - } - } - /* For postcopy we need to check hugepage sizes match */ - if (postcopy_advised && migrate_postcopy_ram() && - block->page_size != qemu_host_page_size) { - uint64_t remote_page_size = qemu_get_be64(f); - if (remote_page_size != block->page_size) { - error_report("Mismatched RAM page size %s " - "(local) %zd != %" PRId64, - id, block->page_size, - remote_page_size); - ret = -EINVAL; - } - } - if (migrate_ignore_shared()) { - hwaddr addr = qemu_get_be64(f); - if (ramblock_is_ignored(block) && - block->mr->addr != addr) { - error_report("Mismatched GPAs for block %s " - "%" PRId64 "!= %" PRId64, - id, (uint64_t)addr, - (uint64_t)block->mr->addr); - ret = -EINVAL; - } - } - ram_control_load_hook(f, RAM_CONTROL_BLOCK_REG, - block->idstr); - } else { - error_report("Unknown ramblock \"%s\", cannot " - "accept migration", id); - ret = -EINVAL; - } - - total_ram_bytes -= length; - } + ret = parse_ramblocks(f, addr); break; case RAM_SAVE_FLAG_ZERO: From patchwork Thu Mar 30 18:03:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0E4CEC761A6 for ; Thu, 30 Mar 2023 18:05:37 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdS-0003J5-JN; Thu, 30 Mar 2023 14:04:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdJ-0003Hi-D9 for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:21 -0400 Received: from smtp-out2.suse.de ([2001:67c:2178:6::1d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdG-00024o-Pb for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:21 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id A42051F86C; Thu, 30 Mar 2023 18:04:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199457; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3/GTzm3pKNJ7Y2cbA6lCzYeLB9tTPes8/nCkr00r/LM=; b=N6nMrBlmLrOL6hSxYgBHqIRB0uxVDWXyYNCT1GgdNN+Z5aiVQ9ip97adtWRda+Rp7KkJeO gVAaMt0G582alKxLwX3G1bfpbyUDkIgS7alAXZG4tzZVcLk2W+VS5mhwjuHMP/uKyEmuo+ Dqy7exoDesMnQMckayTaRbh6AOK9oPA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199457; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3/GTzm3pKNJ7Y2cbA6lCzYeLB9tTPes8/nCkr00r/LM=; b=e1EAh7Fafg/aVgt+08YAzUWtQskXUbmzZ1gs6AasnCuivFRkMhPvtqqMQePGe+ZBdr7VWd W2xJaHwZkoWD9RDg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 5A85E1348E; Thu, 30 Mar 2023 18:04:15 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id eBKaCB/PJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:15 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Nikolay Borisov Subject: [RFC PATCH v1 12/26] migration: Add support for 'fixed-ram' migration restore Date: Thu, 30 Mar 2023 15:03:22 -0300 Message-Id: <20230330180336.2791-13-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1d; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Nikolay Borisov Add the necessary code to parse the format changes for the 'fixed-ram' capability. One of the more notable changes in behavior is that in the 'fixed-ram' case ram pages are restored in one go rather than constantly looping through the migration stream. Also due to idiosyncrasies of the format I have added the 'ram_migrated' since it was easier to simply return directly from ->load_state rather than introducing more conditionals around the code to prevent ->load_state being called multiple times (from qemu_loadvm_section_start_full/qemu_loadvm_section_part_end i.e. from multiple QEMU_VM_SECTION_(PART|END) flags). Signed-off-by: Nikolay Borisov Signed-off-by: Fabiano Rosas --- migration/migration.h | 2 + migration/ram.c | 105 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 105 insertions(+), 2 deletions(-) diff --git a/migration/migration.h b/migration/migration.h index 8cf3caecfe..84be34587f 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -96,6 +96,8 @@ struct MigrationIncomingState { bool have_listen_thread; QemuThread listen_thread; + bool ram_migrated; + /* For the kernel to send us notifications */ int userfault_fd; /* To notify the fault_thread to wake, e.g., when need to quit */ diff --git a/migration/ram.c b/migration/ram.c index 5c085d6154..1666ce6d5f 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -4396,6 +4396,100 @@ static int parse_ramblocks(QEMUFile *f, ram_addr_t total_ram_bytes) return ret; } +static void read_ramblock_fixed_ram(QEMUFile *f, RAMBlock *block, + long num_pages, unsigned long *bitmap) +{ + unsigned long set_bit_idx, clear_bit_idx; + unsigned long len; + ram_addr_t offset; + void *host; + size_t read, completed, read_len; + + for (set_bit_idx = find_first_bit(bitmap, num_pages); + set_bit_idx < num_pages; + set_bit_idx = find_next_bit(bitmap, num_pages, clear_bit_idx + 1)) { + + clear_bit_idx = find_next_zero_bit(bitmap, num_pages, set_bit_idx + 1); + + len = TARGET_PAGE_SIZE * (clear_bit_idx - set_bit_idx); + offset = set_bit_idx << TARGET_PAGE_BITS; + + for (read = 0, completed = 0; completed < len; offset += read) { + host = host_from_ram_block_offset(block, offset); + read_len = MIN(len, TARGET_PAGE_SIZE); + + read = qemu_get_buffer_at(f, host, read_len, + block->pages_offset + offset); + completed += read; + } + } +} + +static int parse_ramblocks_fixed_ram(QEMUFile *f) +{ + int ret = 0; + + while (!ret) { + char id[256]; + RAMBlock *block; + ram_addr_t length; + long num_pages, bitmap_size; + int len = qemu_get_byte(f); + g_autofree unsigned long *dirty_bitmap = NULL; + + qemu_get_buffer(f, (uint8_t *)id, len); + id[len] = 0; + length = qemu_get_be64(f); + + block = qemu_ram_block_by_name(id); + if (block) { + ret = parse_ramblock(f, block, length); + if (ret < 0) { + return ret; + } + } else { + error_report("Unknown ramblock \"%s\", cannot accept " + "migration", id); + ret = -EINVAL; + continue; + } + + /* 1. read the bitmap size */ + num_pages = length >> TARGET_PAGE_BITS; + bitmap_size = qemu_get_be32(f); + + assert(bitmap_size == BITS_TO_LONGS(num_pages) * sizeof(unsigned long)); + + block->pages_offset = qemu_get_be64(f); + + /* 2. read the actual bitmap */ + dirty_bitmap = g_malloc0(bitmap_size); + if (qemu_get_buffer(f, (uint8_t *)dirty_bitmap, bitmap_size) != bitmap_size) { + error_report("Error parsing dirty bitmap"); + return -EINVAL; + } + + read_ramblock_fixed_ram(f, block, num_pages, dirty_bitmap); + + /* Skip pages array */ + qemu_set_offset(f, block->pages_offset + length, SEEK_SET); + + /* Check if this is the last ramblock */ + if (qemu_get_be64(f) == RAM_SAVE_FLAG_EOS) { + ret = 1; + } else { + /* + * If not, adjust the internal file index to account for the + * previous 64 bit read + */ + qemu_file_skip(f, -8); + ret = 0; + } + } + + return ret; +} + /** * ram_load_precopy: load pages in precopy case * @@ -4415,7 +4509,7 @@ static int ram_load_precopy(QEMUFile *f) invalid_flags |= RAM_SAVE_FLAG_COMPRESS_PAGE; } - while (!ret && !(flags & RAM_SAVE_FLAG_EOS)) { + while (!ret && !(flags & RAM_SAVE_FLAG_EOS) && !mis->ram_migrated) { ram_addr_t addr; void *host = NULL, *host_bak = NULL; uint8_t ch; @@ -4487,7 +4581,14 @@ static int ram_load_precopy(QEMUFile *f) switch (flags & ~RAM_SAVE_FLAG_CONTINUE) { case RAM_SAVE_FLAG_MEM_SIZE: - ret = parse_ramblocks(f, addr); + if (migrate_fixed_ram()) { + ret = parse_ramblocks_fixed_ram(f); + if (ret == 1) { + mis->ram_migrated = true; + } + } else { + ret = parse_ramblocks(f, addr); + } break; case RAM_SAVE_FLAG_ZERO: From patchwork Thu Mar 30 18:03:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194706 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E2097C77B60 for ; Thu, 30 Mar 2023 18:08:08 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdY-0003Lm-9L; Thu, 30 Mar 2023 14:04:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdR-0003IT-9E for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:29 -0400 Received: from smtp-out1.suse.de ([2001:67c:2178:6::1c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdK-00025S-Sr for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:26 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 215FC21A2E; Thu, 30 Mar 2023 18:04:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199461; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mC4J7MkyhShPeuviGNSRXVLzIyJv7LGs1dCrXwd+IWo=; b=iW/tGpqXPmQvC8iRaAi6AFS/ApeMCva9iwzG7nVZmrsQENmS5SyLXowA4SB4Jz740Upah1 KdKsg3fh5YR0BrxRiwdq38L8tjiGAA4F9fG6GxIJwwDpoTh5e+3wXdZc0bcKCckHrfGgSx eeoav8wxVXbVf1Kx8NUa8UjIKiJmY1w= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199461; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mC4J7MkyhShPeuviGNSRXVLzIyJv7LGs1dCrXwd+IWo=; b=7z17LRMxvuaTPsVlomDUotrZb7YGZHgJXPPUGz1BULHssESn3k/NPbhJzc0G2AvE/50ko1 COYoIRRrR8XnGJAA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 22C8E1348E; Thu, 30 Mar 2023 18:04:17 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id IEVzNiHPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:17 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Nikolay Borisov , Thomas Huth , Laurent Vivier , Paolo Bonzini Subject: [RFC PATCH v1 13/26] tests/qtest: migration-test: Add tests for fixed-ram file-based migration Date: Thu, 30 Mar 2023 15:03:23 -0300 Message-Id: <20230330180336.2791-14-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1c; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Nikolay Borisov Add basic tests for 'fixed-ram' migration. Signed-off-by: Nikolay Borisov Signed-off-by: Fabiano Rosas --- tests/qtest/migration-test.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 13e5cdd5a4..84b4c761ad 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -1544,6 +1544,26 @@ static void test_precopy_file_stream_ram(void) test_precopy_common(&args); } +static void *migrate_fixed_ram_start(QTestState *from, QTestState *to) +{ + migrate_set_capability(from, "fixed-ram", true); + migrate_set_capability(to, "fixed-ram", true); + + return NULL; +} + +static void test_precopy_file_fixed_ram(void) +{ + g_autofree char *uri = g_strdup_printf("file:%s/migfile", tmpfs); + MigrateCommon args = { + .connect_uri = uri, + .listen_uri = "defer", + .start_hook = migrate_fixed_ram_start, + }; + + test_precopy_common(&args); +} + static void test_precopy_tcp_plain(void) { MigrateCommon args = { @@ -2538,6 +2558,8 @@ int main(int argc, char **argv) qtest_add_func("/migration/precopy/file/stream-ram", test_precopy_file_stream_ram); + qtest_add_func("/migration/precopy/file/fixed-ram", + test_precopy_file_fixed_ram); #ifdef CONFIG_GNUTLS qtest_add_func("/migration/precopy/unix/tls/psk", From patchwork Thu Mar 30 18:03:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194712 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BF69DC761A6 for ; Thu, 30 Mar 2023 18:08:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdY-0003MI-To; Thu, 30 Mar 2023 14:04:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdS-0003JC-L7 for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:30 -0400 Received: from smtp-out2.suse.de ([2001:67c:2178:6::1d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdR-000262-14 for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:30 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id AA8641F86C; Thu, 30 Mar 2023 18:04:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199463; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KK/TQbf2mnwP6VHEO79Vdk2BgwveWe5Obtn01e1sCM4=; b=UQdjHBs/99rN/nMpP9kbvjNNhG/+2Rp1Gu5P+vKDVpYncm2TVnNF11T+GUIkgsomPZzELr pEx23bWrPpb2UVEENK1x+yLsJ6hBhgdHH2BqE95AnUL4OTucQV5buI9mnTxZyhiH4rXOOD vU1I7rMkxqXAa0OIf3aA6EgjUm9UfIw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199463; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KK/TQbf2mnwP6VHEO79Vdk2BgwveWe5Obtn01e1sCM4=; b=jm2KF97cUfEcM1gE39C+zuuAOMzfd5zCTbME9IFgNXosVQOVo25jx50TdHJuK8NHiIDRYJ YZy3FT/Zx5le2bDw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 984D51348E; Thu, 30 Mar 2023 18:04:21 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id QI6cFyXPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:21 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela Subject: [RFC PATCH v1 14/26] migration: Add completion tracepoint Date: Thu, 30 Mar 2023 15:03:24 -0300 Message-Id: <20230330180336.2791-15-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1d; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add a completion tracepoint that provides basic stats for debug. Displays throughput (MB/s and pages/s) and total time (ms). Usage: $QEMU ... -trace migration_status Output: migration_status 1506 MB/s, 436725 pages/s, 8698 ms Signed-off-by: Fabiano Rosas --- migration/migration.c | 6 +++--- migration/migration.h | 4 +++- migration/savevm.c | 4 ++++ migration/trace-events | 1 + 4 files changed, 11 insertions(+), 4 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 29630523e2..17b26c1808 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -3811,7 +3811,7 @@ static uint64_t migration_total_bytes(MigrationState *s) ram_counters.multifd_bytes; } -static void migration_calculate_complete(MigrationState *s) +void migration_calculate_complete(MigrationState *s) { uint64_t bytes = migration_total_bytes(s); int64_t end_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); @@ -3843,8 +3843,7 @@ static void update_iteration_initial_status(MigrationState *s) s->iteration_initial_pages = ram_get_total_transferred_pages(); } -static void migration_update_counters(MigrationState *s, - int64_t current_time) +void migration_update_counters(MigrationState *s, int64_t current_time) { uint64_t transferred, transferred_pages, time_spent; uint64_t current_bytes; /* bytes transferred since the beginning */ @@ -3941,6 +3940,7 @@ static void migration_iteration_finish(MigrationState *s) case MIGRATION_STATUS_COMPLETED: migration_calculate_complete(s); runstate_set(RUN_STATE_POSTMIGRATE); + trace_migration_status((int)s->mbps / 8, (int)s->pages_per_second, s->total_time); break; case MIGRATION_STATUS_COLO: if (!migrate_colo_enabled()) { diff --git a/migration/migration.h b/migration/migration.h index 84be34587f..01c8201cfa 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -387,7 +387,9 @@ struct MigrationState { }; void migrate_set_state(int *state, int old_state, int new_state); - +void migration_calculate_complete(MigrationState *s); +void migration_update_counters(MigrationState *s, + int64_t current_time); void migration_fd_process_incoming(QEMUFile *f, Error **errp); void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp); void migration_incoming_process(void); diff --git a/migration/savevm.c b/migration/savevm.c index 1f1bc19224..b369d11b19 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1638,6 +1638,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) qemu_mutex_lock_iothread(); while (qemu_file_get_error(f) == 0) { + migration_update_counters(ms, qemu_clock_get_ms(QEMU_CLOCK_REALTIME)); if (qemu_savevm_state_iterate(f, false) > 0) { break; } @@ -1660,6 +1661,9 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) } migrate_set_state(&ms->state, MIGRATION_STATUS_SETUP, status); + migration_calculate_complete(ms); + trace_migration_status((int)ms->mbps / 8, (int)ms->pages_per_second, ms->total_time); + /* f is outer parameter, it should not stay in global migration state after * this function finished */ ms->to_dst_file = NULL; diff --git a/migration/trace-events b/migration/trace-events index 92161eeac5..23e4dad1ec 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -165,6 +165,7 @@ migration_return_path_end_after(int rp_error) "%d" migration_thread_after_loop(void) "" migration_thread_file_err(void) "" migration_thread_setup_complete(void) "" +migration_status(int mpbs, int pages_per_second, int64_t total_time) "%d MB/s, %d pages/s, %ld ms" open_return_path_on_source(void) "" open_return_path_on_source_continue(void) "" postcopy_start(void) "" From patchwork Thu Mar 30 18:03:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194709 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A06DCC6FD1D for ; Thu, 30 Mar 2023 18:08:22 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdY-0003Jz-6S; Thu, 30 Mar 2023 14:04:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdS-0003J9-HO for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:30 -0400 Received: from smtp-out2.suse.de ([2001:67c:2178:6::1d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdR-00026O-1i for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:30 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 42A4D1F8AA; Thu, 30 Mar 2023 18:04:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199466; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u9uAvRgIhFG02HKMriEQRRbGKMKeihMxDlDshlT9D8E=; b=Tll7eXHRt6kN0tiniG0XME1K99M9E9UFaaPL/iDPnf5kJ6mXE1EnFfGE/FjHJ9iiF8CqF0 iW3c/iIpY603rt23+HZLk8s6BdHQ91SEYKbEB5INdp4TbsVUZjQdOWUO6q3rgJ930CCaWT i2LuUUQn4nnKzAl7t6qENzfBML0eZqU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199466; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u9uAvRgIhFG02HKMriEQRRbGKMKeihMxDlDshlT9D8E=; b=JowCqT4vq1E9c78DKupvRnFAstmbU2zd33dsXbUdzbGV0Fv3bxZkk/EaG5w2WpuH1tNOG3 O6Pv0kBcFEF3AnCw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 288401348E; Thu, 30 Mar 2023 18:04:23 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id EIvDNyfPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:23 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela Subject: [RFC PATCH v1 15/26] migration/multifd: Remove direct "socket" references Date: Thu, 30 Mar 2023 15:03:25 -0300 Message-Id: <20230330180336.2791-16-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1d; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org We're about to enable support for other transports in multifd, so remove direct references to sockets. Signed-off-by: Fabiano Rosas --- migration/multifd.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index cbc0dfe39b..e613d85e24 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -512,6 +512,11 @@ static void multifd_send_terminate_threads(Error *err) } } +static int multifd_send_channel_destroy(QIOChannel *send) +{ + return socket_send_channel_destroy(send); +} + void multifd_save_cleanup(void) { int i; @@ -534,7 +539,7 @@ void multifd_save_cleanup(void) if (p->registered_yank) { migration_ioc_unregister_yank(p->c); } - socket_send_channel_destroy(p->c); + multifd_send_channel_destroy(p->c); p->c = NULL; qemu_mutex_destroy(&p->mutex); qemu_sem_destroy(&p->sem); @@ -889,20 +894,25 @@ static void multifd_new_send_channel_cleanup(MultiFDSendParams *p, static void multifd_new_send_channel_async(QIOTask *task, gpointer opaque) { MultiFDSendParams *p = opaque; - QIOChannel *sioc = QIO_CHANNEL(qio_task_get_source(task)); + QIOChannel *ioc = QIO_CHANNEL(qio_task_get_source(task)); Error *local_err = NULL; trace_multifd_new_send_channel_async(p->id); if (!qio_task_propagate_error(task, &local_err)) { - p->c = QIO_CHANNEL(sioc); + p->c = QIO_CHANNEL(ioc); qio_channel_set_delay(p->c, false); p->running = true; - if (multifd_channel_connect(p, sioc, local_err)) { + if (multifd_channel_connect(p, ioc, local_err)) { return; } } - multifd_new_send_channel_cleanup(p, sioc, local_err); + multifd_new_send_channel_cleanup(p, ioc, local_err); +} + +static void multifd_new_send_channel_create(gpointer opaque) +{ + socket_send_channel_create(multifd_new_send_channel_async, opaque); } int multifd_save_setup(Error **errp) @@ -951,7 +961,7 @@ int multifd_save_setup(Error **errp) p->write_flags = 0; } - socket_send_channel_create(multifd_new_send_channel_async, p); + multifd_new_send_channel_create(p); } for (i = 0; i < thread_count; i++) { From patchwork Thu Mar 30 18:03:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194702 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 838A3C6FD1D for ; Thu, 30 Mar 2023 18:06:23 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdd-0003Nd-U4; Thu, 30 Mar 2023 14:04:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdU-0003K0-4p for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:36 -0400 Received: from smtp-out1.suse.de ([2001:67c:2178:6::1c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdR-00026h-Te for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:31 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id CD4D521A3A; Thu, 30 Mar 2023 18:04:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199468; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OMQmZ9odr+w1MVMVyhbi8097U68hhrYNCCmZsyNq6zY=; b=lUPYDUU4FWBaBQCZ4wTp5TNPmwLnsHy/f4aQnmHq3O2E9PLRR7ae54YfcN2pGXQIOm2GWv ffqco5CgN54cJb7h7LK8yKLNNRcEYDb8Io47C304c8YwK/B99C6GoTaTEo2fEH3WkTKVOD ApHAgEAo6/5goOqRf7CbfewLE26GHWs= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199468; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OMQmZ9odr+w1MVMVyhbi8097U68hhrYNCCmZsyNq6zY=; b=plL4ELN0P6n8ygbqpL0CzJ77hIAIngA5LxJ+zHQp4M9VLMPNf8Gjw0XKbRcsW1HK5kT04Z 9WhUf8kYuGzjQjCA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id BA5211348E; Thu, 30 Mar 2023 18:04:26 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id aPafHyrPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:26 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela Subject: [RFC PATCH v1 16/26] migration/multifd: Allow multifd without packets Date: Thu, 30 Mar 2023 15:03:26 -0300 Message-Id: <20230330180336.2791-17-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1c; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org For the upcoming support to the new 'fixed-ram' migration stream format, we cannot use multifd packets because each write into the ramblock section in the migration file is expected to contain only the guest pages. They are written at their respective offsets relative to the ramblock section header. There is no space for the packet information and the expected gains from the new approach come partly from being able to write the pages sequentially without extraneous data in between. The new format also doesn't need the packets and all necessary information can be taken from the standard migration headers with some (future) changes to multifd code. Use the presence of the fixed-ram capability to decide whether to send packets. For now this has no effect as fixed-ram cannot yet be enabled with multifd. Signed-off-by: Fabiano Rosas --- migration/migration.c | 5 ++ migration/migration.h | 2 +- migration/multifd.c | 119 ++++++++++++++++++++++++++---------------- 3 files changed, 80 insertions(+), 46 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 17b26c1808..c647fbffa6 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2764,6 +2764,11 @@ int migrate_fixed_ram(void) return migrate_get_current()->enabled_capabilities[MIGRATION_CAPABILITY_FIXED_RAM]; } +bool migrate_multifd_use_packets(void) +{ + return !migrate_fixed_ram(); +} + int migrate_multifd_zlib_level(void) { MigrationState *s; diff --git a/migration/migration.h b/migration/migration.h index 01c8201cfa..d7a014ce57 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -421,7 +421,7 @@ bool migrate_dirty_bitmaps(void); bool migrate_ignore_shared(void); bool migrate_validate_uuid(void); int migrate_fixed_ram(void); - +bool migrate_multifd_use_packets(void); bool migrate_auto_converge(void); bool migrate_use_multifd(void); bool migrate_pause_before_switchover(void); diff --git a/migration/multifd.c b/migration/multifd.c index e613d85e24..9f6b2787ed 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -659,18 +659,22 @@ static void *multifd_send_thread(void *opaque) Error *local_err = NULL; int ret = 0; bool use_zero_copy_send = migrate_use_zero_copy_send(); + bool use_packets = migrate_multifd_use_packets(); thread = MigrationThreadAdd(p->name, qemu_get_thread_id()); trace_multifd_send_thread_start(p->id); rcu_register_thread(); - if (multifd_send_initial_packet(p, &local_err) < 0) { - ret = -1; - goto out; + if (use_packets) { + if (multifd_send_initial_packet(p, &local_err) < 0) { + ret = -1; + goto out; + } + + /* initial packet */ + p->num_packets = 1; } - /* initial packet */ - p->num_packets = 1; while (true) { qemu_sem_wait(&p->sem); @@ -681,11 +685,10 @@ static void *multifd_send_thread(void *opaque) qemu_mutex_lock(&p->mutex); if (p->pending_job) { - uint64_t packet_num = p->packet_num; uint32_t flags; p->normal_num = 0; - if (use_zero_copy_send) { + if (!use_packets || use_zero_copy_send) { p->iovs_num = 0; } else { p->iovs_num = 1; @@ -703,16 +706,20 @@ static void *multifd_send_thread(void *opaque) break; } } - multifd_send_fill_packet(p); + + if (use_packets) { + multifd_send_fill_packet(p); + p->num_packets++; + } + flags = p->flags; p->flags = 0; - p->num_packets++; p->total_normal_pages += p->normal_num; p->pages->num = 0; p->pages->block = NULL; qemu_mutex_unlock(&p->mutex); - trace_multifd_send(p->id, packet_num, p->normal_num, flags, + trace_multifd_send(p->id, p->packet_num, p->normal_num, flags, p->next_packet_size); if (use_zero_copy_send) { @@ -722,7 +729,7 @@ static void *multifd_send_thread(void *opaque) if (ret != 0) { break; } - } else { + } else if (use_packets) { /* Send header using the same writev call */ p->iov[0].iov_len = p->packet_len; p->iov[0].iov_base = p->packet; @@ -919,6 +926,7 @@ int multifd_save_setup(Error **errp) { int thread_count; uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size(); + bool use_packets = migrate_multifd_use_packets(); uint8_t i; if (!migrate_use_multifd()) { @@ -943,14 +951,20 @@ int multifd_save_setup(Error **errp) p->pending_job = 0; p->id = i; p->pages = multifd_pages_init(page_count); - p->packet_len = sizeof(MultiFDPacket_t) - + sizeof(uint64_t) * page_count; - p->packet = g_malloc0(p->packet_len); - p->packet->magic = cpu_to_be32(MULTIFD_MAGIC); - p->packet->version = cpu_to_be32(MULTIFD_VERSION); + + if (use_packets) { + p->packet_len = sizeof(MultiFDPacket_t) + + sizeof(uint64_t) * page_count; + p->packet = g_malloc0(p->packet_len); + p->packet->magic = cpu_to_be32(MULTIFD_MAGIC); + p->packet->version = cpu_to_be32(MULTIFD_VERSION); + + /* We need one extra place for the packet header */ + p->iov = g_new0(struct iovec, page_count + 1); + } else { + p->iov = g_new0(struct iovec, page_count); + } p->name = g_strdup_printf("multifdsend_%d", i); - /* We need one extra place for the packet header */ - p->iov = g_new0(struct iovec, page_count + 1); p->normal = g_new0(ram_addr_t, page_count); p->page_size = qemu_target_page_size(); p->page_count = page_count; @@ -1082,7 +1096,7 @@ void multifd_recv_sync_main(void) { int i; - if (!migrate_use_multifd()) { + if (!migrate_use_multifd() || !migrate_multifd_use_packets()) { return; } for (i = 0; i < migrate_multifd_channels(); i++) { @@ -1109,6 +1123,7 @@ static void *multifd_recv_thread(void *opaque) { MultiFDRecvParams *p = opaque; Error *local_err = NULL; + bool use_packets = migrate_multifd_use_packets(); int ret; trace_multifd_recv_thread_start(p->id); @@ -1121,17 +1136,20 @@ static void *multifd_recv_thread(void *opaque) break; } - ret = qio_channel_read_all_eof(p->c, (void *)p->packet, - p->packet_len, &local_err); - if (ret == 0 || ret == -1) { /* 0: EOF -1: Error */ - break; - } + if (use_packets) { + ret = qio_channel_read_all_eof(p->c, (void *)p->packet, + p->packet_len, &local_err); + if (ret == 0 || ret == -1) { /* 0: EOF -1: Error */ + break; + } - qemu_mutex_lock(&p->mutex); - ret = multifd_recv_unfill_packet(p, &local_err); - if (ret) { - qemu_mutex_unlock(&p->mutex); - break; + qemu_mutex_lock(&p->mutex); + ret = multifd_recv_unfill_packet(p, &local_err); + if (ret) { + qemu_mutex_unlock(&p->mutex); + break; + } + p->num_packets++; } flags = p->flags; @@ -1139,7 +1157,7 @@ static void *multifd_recv_thread(void *opaque) p->flags &= ~MULTIFD_FLAG_SYNC; trace_multifd_recv(p->id, p->packet_num, p->normal_num, flags, p->next_packet_size); - p->num_packets++; + p->total_normal_pages += p->normal_num; qemu_mutex_unlock(&p->mutex); @@ -1174,6 +1192,7 @@ int multifd_load_setup(Error **errp) { int thread_count; uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size(); + bool use_packets = migrate_multifd_use_packets(); uint8_t i; /* @@ -1198,9 +1217,12 @@ int multifd_load_setup(Error **errp) qemu_sem_init(&p->sem_sync, 0); p->quit = false; p->id = i; - p->packet_len = sizeof(MultiFDPacket_t) - + sizeof(uint64_t) * page_count; - p->packet = g_malloc0(p->packet_len); + + if (use_packets) { + p->packet_len = sizeof(MultiFDPacket_t) + + sizeof(uint64_t) * page_count; + p->packet = g_malloc0(p->packet_len); + } p->name = g_strdup_printf("multifdrecv_%d", i); p->iov = g_new0(struct iovec, page_count); p->normal = g_new0(ram_addr_t, page_count); @@ -1246,18 +1268,26 @@ void multifd_recv_new_channel(QIOChannel *ioc, Error **errp) { MultiFDRecvParams *p; Error *local_err = NULL; - int id; + bool use_packets = migrate_multifd_use_packets(); + int id, num_packets = 0; - id = multifd_recv_initial_packet(ioc, &local_err); - if (id < 0) { - multifd_recv_terminate_threads(local_err); - error_propagate_prepend(errp, local_err, - "failed to receive packet" - " via multifd channel %d: ", - qatomic_read(&multifd_recv_state->count)); - return; + if (use_packets) { + id = multifd_recv_initial_packet(ioc, &local_err); + if (id < 0) { + multifd_recv_terminate_threads(local_err); + error_propagate_prepend(errp, local_err, + "failed to receive packet" + " via multifd channel %d: ", + qatomic_read(&multifd_recv_state->count)); + return; + } + trace_multifd_recv_new_channel(id); + + /* initial packet */ + num_packets = 1; + } else { + id = 0; } - trace_multifd_recv_new_channel(id); p = &multifd_recv_state->params[id]; if (p->c != NULL) { @@ -1268,9 +1298,8 @@ void multifd_recv_new_channel(QIOChannel *ioc, Error **errp) return; } p->c = ioc; + p->num_packets = num_packets; object_ref(OBJECT(ioc)); - /* initial packet */ - p->num_packets = 1; p->running = true; qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p, From patchwork Thu Mar 30 18:03:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F2E0DC761A6 for ; Thu, 30 Mar 2023 18:08:26 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdd-0003NY-Si; Thu, 30 Mar 2023 14:04:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdY-0003Ls-Ak for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:36 -0400 Received: from smtp-out2.suse.de ([195.135.220.29]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdU-00027V-MI for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:34 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 6077D1F86C; Thu, 30 Mar 2023 18:04:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199471; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rGH+5ko9m4lHRTeAUdNka2KpY3vjmqBzojbedAqahic=; b=pMrGB0hj3AoDXSm1CT2NEgBiX984RFZ8mA5ZERa6wR2iGuKka1TbX6dH4ChuIRx7ohldAv nuLocOVdu6oqO2jCmlSgzCjQul+HyU/K4z+EPUpEFxAes+l3VigASY3NNeE3lLZwGlw+la 8d5JcanU+KMWkme0fIiiRMjvLwGKxVs= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199471; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rGH+5ko9m4lHRTeAUdNka2KpY3vjmqBzojbedAqahic=; b=5zcc3WXsJDAwHmC2fK/vl6+H3yzAFimi6tN13N2b7+xGi7EikniyC+NVfiN9yzFUQRRB2b mn8ecDN/2W8UEYBQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 51D521348E; Thu, 30 Mar 2023 18:04:29 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id eCx4Bi3PJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:29 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela Subject: [RFC PATCH v1 17/26] migration/multifd: Add outgoing QIOChannelFile support Date: Thu, 30 Mar 2023 15:03:27 -0300 Message-Id: <20230330180336.2791-18-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=195.135.220.29; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Allow multifd to open file-backed channels. This will be used when enabling the fixed-ram migration stream format which expects a seekable transport. The QIOChannel read and write methods will use the preadv/pwritev versions which don't update the file offset at each call so we can reuse the fd without re-opening for every channel. Note that this is just setup code and multifd cannot yet make use of the file channels. Signed-off-by: Fabiano Rosas --- include/io/channel-file.h | 1 + migration/file.c | 63 +++++++++++++++++++++++++++++++++++++-- migration/file.h | 6 +++- migration/migration.c | 11 ++++++- migration/migration.h | 1 + migration/multifd.c | 14 +++++++-- 6 files changed, 89 insertions(+), 7 deletions(-) diff --git a/include/io/channel-file.h b/include/io/channel-file.h index 50e8eb1138..85b6c34a72 100644 --- a/include/io/channel-file.h +++ b/include/io/channel-file.h @@ -22,6 +22,7 @@ #define QIO_CHANNEL_FILE_H #include "io/channel.h" +#include "io/task.h" #include "qom/object.h" #define TYPE_QIO_CHANNEL_FILE "qio-channel-file" diff --git a/migration/file.c b/migration/file.c index ab4e12926c..f674cd1bdb 100644 --- a/migration/file.c +++ b/migration/file.c @@ -1,20 +1,77 @@ #include "qemu/osdep.h" -#include "channel.h" #include "io/channel-file.h" #include "file.h" #include "qemu/error-report.h" +static struct FileOutgoingArgs { + char *fname; + int flags; + int mode; +} outgoing_args; + +static void qio_channel_file_connect_worker(QIOTask *task, gpointer opaque) +{ + /* noop */ +} + +static void file_migration_cancel(Error *errp) +{ + MigrationState *s; + + s = migrate_get_current(); + + migrate_set_state(&s->state, MIGRATION_STATUS_SETUP, + MIGRATION_STATUS_FAILED); + migration_cancel(errp); +} + +int file_send_channel_destroy(QIOChannel *ioc) +{ + if (ioc) { + qio_channel_close(ioc, NULL); + object_unref(OBJECT(ioc)); + } + g_free(outgoing_args.fname); + outgoing_args.fname = NULL; + + return 0; +} + +void file_send_channel_create(QIOTaskFunc f, void *data) +{ + QIOChannelFile *ioc; + QIOTask *task; + Error *errp = NULL; + + ioc = qio_channel_file_new_path(outgoing_args.fname, + outgoing_args.flags, + outgoing_args.mode, &errp); + if (!ioc) { + file_migration_cancel(errp); + return; + } + + task = qio_task_new(OBJECT(ioc), f, (gpointer)data, NULL); + qio_task_run_in_thread(task, qio_channel_file_connect_worker, + (gpointer)data, NULL, NULL); +} void file_start_outgoing_migration(MigrationState *s, const char *fname, Error **errp) { QIOChannelFile *ioc; + int flags = O_CREAT | O_TRUNC | O_WRONLY; + mode_t mode = 0660; - ioc = qio_channel_file_new_path(fname, O_CREAT | O_TRUNC | O_WRONLY, 0660, errp); + ioc = qio_channel_file_new_path(fname, flags, mode, errp); if (!ioc) { - error_report("Error creating a channel"); + error_report("Error creating migration outgoing channel"); return; } + outgoing_args.fname = g_strdup(fname); + outgoing_args.flags = flags; + outgoing_args.mode = mode; + qio_channel_set_name(QIO_CHANNEL(ioc), "migration-file-outgoing"); migration_channel_connect(s, QIO_CHANNEL(ioc), NULL, NULL); object_unref(OBJECT(ioc)); diff --git a/migration/file.h b/migration/file.h index cdbd291322..5e27ca6afd 100644 --- a/migration/file.h +++ b/migration/file.h @@ -1,10 +1,14 @@ #ifndef QEMU_MIGRATION_FILE_H #define QEMU_MIGRATION_FILE_H +#include "io/task.h" +#include "channel.h" + void file_start_outgoing_migration(MigrationState *s, const char *filename, Error **errp); void file_start_incoming_migration(const char *fname, Error **errp); +void file_send_channel_create(QIOTaskFunc f, void *data); +int file_send_channel_destroy(QIOChannel *ioc); #endif - diff --git a/migration/migration.c b/migration/migration.c index c647fbffa6..6594c2f404 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -194,7 +194,7 @@ static bool migration_needs_multiple_sockets(void) static bool uri_supports_multi_channels(const char *uri) { return strstart(uri, "tcp:", NULL) || strstart(uri, "unix:", NULL) || - strstart(uri, "vsock:", NULL); + strstart(uri, "vsock:", NULL) || strstart(uri, "file:", NULL); } static bool @@ -2740,6 +2740,15 @@ bool migrate_pause_before_switchover(void) MIGRATION_CAPABILITY_PAUSE_BEFORE_SWITCHOVER]; } +bool migrate_to_file(void) +{ + MigrationState *s; + + s = migrate_get_current(); + + return qemu_file_is_seekable(s->to_dst_file); +} + int migrate_multifd_channels(void) { MigrationState *s; diff --git a/migration/migration.h b/migration/migration.h index d7a014ce57..8459201958 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -425,6 +425,7 @@ bool migrate_multifd_use_packets(void); bool migrate_auto_converge(void); bool migrate_use_multifd(void); bool migrate_pause_before_switchover(void); +bool migrate_to_file(void); int migrate_multifd_channels(void); MultiFDCompression migrate_multifd_compression(void); int migrate_multifd_zlib_level(void); diff --git a/migration/multifd.c b/migration/multifd.c index 9f6b2787ed..50bd9b32eb 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -17,6 +17,7 @@ #include "exec/ramblock.h" #include "qemu/error-report.h" #include "qapi/error.h" +#include "file.h" #include "ram.h" #include "migration.h" #include "socket.h" @@ -27,6 +28,7 @@ #include "threadinfo.h" #include "qemu/yank.h" +#include "io/channel-file.h" #include "io/channel-socket.h" #include "yank_functions.h" @@ -514,7 +516,11 @@ static void multifd_send_terminate_threads(Error *err) static int multifd_send_channel_destroy(QIOChannel *send) { - return socket_send_channel_destroy(send); + if (migrate_to_file()) { + return file_send_channel_destroy(send); + } else { + return socket_send_channel_destroy(send); + } } void multifd_save_cleanup(void) @@ -919,7 +925,11 @@ static void multifd_new_send_channel_async(QIOTask *task, gpointer opaque) static void multifd_new_send_channel_create(gpointer opaque) { - socket_send_channel_create(multifd_new_send_channel_async, opaque); + if (migrate_to_file()) { + file_send_channel_create(multifd_new_send_channel_async, opaque); + } else { + socket_send_channel_create(multifd_new_send_channel_async, opaque); + } } int multifd_save_setup(Error **errp) From patchwork Thu Mar 30 18:03:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 51324C6FD1D for ; Thu, 30 Mar 2023 18:05:26 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdj-0003dj-L9; Thu, 30 Mar 2023 14:04:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdZ-0003MU-R3 for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:40 -0400 Received: from smtp-out2.suse.de ([2001:67c:2178:6::1d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdY-00027t-3Q for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:37 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id E5F421F8AA; Thu, 30 Mar 2023 18:04:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199473; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rIg2V0ZNpP25ldqpYVh8t7yPV7FWwSjZcjKTJKCqfrM=; b=aCspemCoKhukl3hqj1PhnpwnQVHGZ4yGW92MwrbFesPPAXUxbZ9TsxYaM6At+FPHBthsA2 ugsgKlVENfBJSpUMFWTBaIK6xT3xyu2b+ZcalBt8SYmJkU25lKZ4t18MopTF0aU6MOfKoY AJV64uUSsEogVMwlz8dM6V39Db+7CB4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199473; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rIg2V0ZNpP25ldqpYVh8t7yPV7FWwSjZcjKTJKCqfrM=; b=FgNIrXZTAKLUzdtFE7EXS+eahv0h9VxrH15RP6X7m2xJb/hoO25w7hLwVIro/RMHjNAIDd XYH2Nbt4lnNI42Dg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id D67ED1348E; Thu, 30 Mar 2023 18:04:31 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 6L/HJi/PJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:31 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela Subject: [RFC PATCH v1 18/26] migration/multifd: Add incoming QIOChannelFile support Date: Thu, 30 Mar 2023 15:03:28 -0300 Message-Id: <20230330180336.2791-19-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1d; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org On the receiving side we don't need to differentiate between main channel and threads, so whichever channel is defined first gets to be the main one. And since there are no packets, use the atomic channel count to index into the params array. Signed-off-by: Fabiano Rosas --- migration/file.c | 38 +++++++++++++++++++++++++++++++++----- migration/migration.c | 2 ++ migration/multifd.c | 7 ++++++- migration/multifd.h | 1 + 4 files changed, 42 insertions(+), 6 deletions(-) diff --git a/migration/file.c b/migration/file.c index f674cd1bdb..6f40894488 100644 --- a/migration/file.c +++ b/migration/file.c @@ -2,6 +2,7 @@ #include "io/channel-file.h" #include "file.h" #include "qemu/error-report.h" +#include "migration.h" static struct FileOutgoingArgs { char *fname; @@ -77,17 +78,44 @@ void file_start_outgoing_migration(MigrationState *s, const char *fname, Error * object_unref(OBJECT(ioc)); } +static void file_process_migration_incoming(QIOTask *task, gpointer opaque) +{ + QIOChannelFile *ioc = opaque; + + migration_channel_process_incoming(QIO_CHANNEL(ioc)); + object_unref(OBJECT(ioc)); +} + void file_start_incoming_migration(const char *fname, Error **errp) { QIOChannelFile *ioc; + QIOTask *task; + int channels = 1; + int i = 0, fd; ioc = qio_channel_file_new_path(fname, O_RDONLY, 0, errp); if (!ioc) { - error_report("Error creating a channel"); + goto out; + } + + if (migrate_use_multifd()) { + channels += migrate_multifd_channels(); + } + + fd = ioc->fd; + + do { + qio_channel_set_name(QIO_CHANNEL(ioc), "migration-file-incoming"); + task = qio_task_new(OBJECT(ioc), file_process_migration_incoming, + (gpointer)ioc, NULL); + + qio_task_run_in_thread(task, qio_channel_file_connect_worker, + (gpointer)ioc, NULL, NULL); + } while (++i < channels && (ioc = qio_channel_file_new_fd(fd))); + +out: + if (!ioc) { + error_report("Error creating migration incoming channel"); return; } - - qio_channel_set_name(QIO_CHANNEL(ioc), "migration-file-incoming"); - migration_channel_process_incoming(QIO_CHANNEL(ioc)); - object_unref(OBJECT(ioc)); } diff --git a/migration/migration.c b/migration/migration.c index 6594c2f404..258709aee1 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -794,6 +794,8 @@ void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp) } default_channel = (channel_magic == cpu_to_be32(QEMU_VM_FILE_MAGIC)); + } else if (migrate_use_multifd() && migrate_fixed_ram()) { + default_channel = multifd_recv_first_channel(); } else { default_channel = !mis->from_src_file; } diff --git a/migration/multifd.c b/migration/multifd.c index 50bd9b32eb..1332b426ce 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -1254,6 +1254,11 @@ int multifd_load_setup(Error **errp) return 0; } +bool multifd_recv_first_channel(void) +{ + return !multifd_recv_state; +} + bool multifd_recv_all_channels_created(void) { int thread_count = migrate_multifd_channels(); @@ -1296,7 +1301,7 @@ void multifd_recv_new_channel(QIOChannel *ioc, Error **errp) /* initial packet */ num_packets = 1; } else { - id = 0; + id = qatomic_read(&multifd_recv_state->count); } p = &multifd_recv_state->params[id]; diff --git a/migration/multifd.h b/migration/multifd.h index 7cfc265148..354150ff55 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -18,6 +18,7 @@ void multifd_save_cleanup(void); int multifd_load_setup(Error **errp); void multifd_load_cleanup(void); void multifd_load_shutdown(void); +bool multifd_recv_first_channel(void); bool multifd_recv_all_channels_created(void); void multifd_recv_new_channel(QIOChannel *ioc, Error **errp); void multifd_recv_sync_main(void); From patchwork Thu Mar 30 18:03:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 44CBFC761A6 for ; Thu, 30 Mar 2023 18:06:50 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwde-0003T1-JR; Thu, 30 Mar 2023 14:04:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdb-0003Mc-BY for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:40 -0400 Received: from smtp-out1.suse.de ([2001:67c:2178:6::1c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdZ-00029d-H6 for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:39 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 75B5821A28; Thu, 30 Mar 2023 18:04:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199476; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jTovoAIppISs4ZnHeN2RjtChtrIcKJohhC+tNPFXop4=; b=dS/yjbDdD62SFluxVmwBsuOt2tEiteF7943aepAkQLDb1hN+Qw7tRzuORHgAGiilSsNvuk th2xjFCffutZ8T00qFeFrqyaWnrCuML+qpb2ESv5Td//AQWVRdnT8GoHplvlY/nTHDhBLc +tFSmFtTdotqymPBBA65vH1ctp00Lwc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199476; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jTovoAIppISs4ZnHeN2RjtChtrIcKJohhC+tNPFXop4=; b=Wx4YVa3ZjwYMFuxyHcddbwqcdIk9ceu4S7TA85osKgA0Gs9LRb3bgVlMpSgIYOHN2S6R54 cyc21xMRleX2H+BQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 64C401348E; Thu, 30 Mar 2023 18:04:34 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id MMQZCzLPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:34 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela Subject: [RFC PATCH v1 19/26] migration/multifd: Add pages to the receiving side Date: Thu, 30 Mar 2023 15:03:29 -0300 Message-Id: <20230330180336.2791-20-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1c; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Currently multifd does not need to have knowledge of pages on the receiving side because all the information needed is within the packets that come in the stream. We're about to add support to fixed-ram migration, which cannot use packets because it expects the ramblock section in the migration file to contain only the guest pages data. Add a pointer to MultiFDPages in the multifd_recv_state and use the pages similarly to what we already do on the sending side. The pages are used to transfer data between the ram migration code in the main migration thread and the multifd receiving threads. Signed-off-by: Fabiano Rosas --- migration/multifd.c | 98 +++++++++++++++++++++++++++++++++++++++++++++ migration/multifd.h | 12 ++++++ 2 files changed, 110 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index 1332b426ce..20ef665218 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -1004,6 +1004,8 @@ int multifd_save_setup(Error **errp) struct { MultiFDRecvParams *params; + /* array of pages to receive */ + MultiFDPages_t *pages; /* number of created threads */ int count; /* syncs main thread and channels */ @@ -1014,6 +1016,66 @@ struct { MultiFDMethods *ops; } *multifd_recv_state; +static int multifd_recv_pages(QEMUFile *f) +{ + int i; + static int next_recv_channel; + MultiFDRecvParams *p = NULL; + MultiFDPages_t *pages = multifd_recv_state->pages; + + /* + * next_channel can remain from a previous migration that was + * using more channels, so ensure it doesn't overflow if the + * limit is lower now. + */ + next_recv_channel %= migrate_multifd_channels(); + for (i = next_recv_channel;; i = (i + 1) % migrate_multifd_channels()) { + p = &multifd_recv_state->params[i]; + + qemu_mutex_lock(&p->mutex); + if (p->quit) { + error_report("%s: channel %d has already quit!", __func__, i); + qemu_mutex_unlock(&p->mutex); + return -1; + } + if (!p->pending_job) { + p->pending_job++; + next_recv_channel = (i + 1) % migrate_multifd_channels(); + break; + } + qemu_mutex_unlock(&p->mutex); + } + + multifd_recv_state->pages = p->pages; + p->pages = pages; + qemu_mutex_unlock(&p->mutex); + qemu_sem_post(&p->sem); + + return 1; +} + +int multifd_recv_queue_page(QEMUFile *f, RAMBlock *block, ram_addr_t offset) +{ + MultiFDPages_t *pages = multifd_recv_state->pages; + + if (!pages->block) { + pages->block = block; + } + + pages->offset[pages->num] = offset; + pages->num++; + + if (pages->num < pages->allocated) { + return 1; + } + + if (multifd_recv_pages(f) < 0) { + return -1; + } + + return 1; +} + static void multifd_recv_terminate_threads(Error *err) { int i; @@ -1035,6 +1097,7 @@ static void multifd_recv_terminate_threads(Error *err) qemu_mutex_lock(&p->mutex); p->quit = true; + qemu_sem_post(&p->sem); /* * We could arrive here for two reasons: * - normal quit, i.e. everything went fine, just finished @@ -1083,9 +1146,12 @@ void multifd_load_cleanup(void) object_unref(OBJECT(p->c)); p->c = NULL; qemu_mutex_destroy(&p->mutex); + qemu_sem_destroy(&p->sem); qemu_sem_destroy(&p->sem_sync); g_free(p->name); p->name = NULL; + multifd_pages_clear(p->pages); + p->pages = NULL; p->packet_len = 0; g_free(p->packet); p->packet = NULL; @@ -1098,6 +1164,8 @@ void multifd_load_cleanup(void) qemu_sem_destroy(&multifd_recv_state->sem_sync); g_free(multifd_recv_state->params); multifd_recv_state->params = NULL; + multifd_pages_clear(multifd_recv_state->pages); + multifd_recv_state->pages = NULL; g_free(multifd_recv_state); multifd_recv_state = NULL; } @@ -1160,6 +1228,25 @@ static void *multifd_recv_thread(void *opaque) break; } p->num_packets++; + } else { + /* + * No packets, so we need to wait for the vmstate code to + * queue pages. + */ + qemu_sem_wait(&p->sem); + qemu_mutex_lock(&p->mutex); + if (!p->pending_job) { + qemu_mutex_unlock(&p->mutex); + break; + } + + for (int i = 0; i < p->pages->num; i++) { + p->normal[p->normal_num] = p->pages->offset[i]; + p->normal_num++; + } + + p->pages->num = 0; + p->host = p->pages->block->host; } flags = p->flags; @@ -1182,6 +1269,13 @@ static void *multifd_recv_thread(void *opaque) qemu_sem_post(&multifd_recv_state->sem_sync); qemu_sem_wait(&p->sem_sync); } + + if (!use_packets) { + qemu_mutex_lock(&p->mutex); + p->pending_job--; + p->pages->block = NULL; + qemu_mutex_unlock(&p->mutex); + } } if (local_err) { @@ -1216,6 +1310,7 @@ int multifd_load_setup(Error **errp) thread_count = migrate_multifd_channels(); multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state)); multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count); + multifd_recv_state->pages = multifd_pages_init(page_count); qatomic_set(&multifd_recv_state->count, 0); qemu_sem_init(&multifd_recv_state->sem_sync, 0); multifd_recv_state->ops = multifd_ops[migrate_multifd_compression()]; @@ -1224,9 +1319,12 @@ int multifd_load_setup(Error **errp) MultiFDRecvParams *p = &multifd_recv_state->params[i]; qemu_mutex_init(&p->mutex); + qemu_sem_init(&p->sem, 0); qemu_sem_init(&p->sem_sync, 0); p->quit = false; + p->pending_job = 0; p->id = i; + p->pages = multifd_pages_init(page_count); if (use_packets) { p->packet_len = sizeof(MultiFDPacket_t) diff --git a/migration/multifd.h b/migration/multifd.h index 354150ff55..2f008217c3 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -24,6 +24,7 @@ void multifd_recv_new_channel(QIOChannel *ioc, Error **errp); void multifd_recv_sync_main(void); int multifd_send_sync_main(QEMUFile *f); int multifd_queue_page(QEMUFile *f, RAMBlock *block, ram_addr_t offset); +int multifd_recv_queue_page(QEMUFile *f, RAMBlock *block, ram_addr_t offset); /* Multifd Compression flags */ #define MULTIFD_FLAG_SYNC (1 << 0) @@ -153,7 +154,11 @@ typedef struct { uint32_t page_size; /* number of pages in a full packet */ uint32_t page_count; + /* multifd flags for receiving ram */ + int read_flags; + /* sem where to wait for more work */ + QemuSemaphore sem; /* syncs main thread and channels */ QemuSemaphore sem_sync; @@ -167,6 +172,13 @@ typedef struct { uint32_t flags; /* global number of generated multifd packets */ uint64_t packet_num; + int pending_job; + /* array of pages to sent. + * The owner of 'pages' depends of 'pending_job' value: + * pending_job == 0 -> migration_thread can use it. + * pending_job != 0 -> multifd_channel can use it. + */ + MultiFDPages_t *pages; /* thread local variables. No locking required */ From patchwork Thu Mar 30 18:03:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 86CCBC77B60 for ; Thu, 30 Mar 2023 18:05:16 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdj-0003dx-Jn; Thu, 30 Mar 2023 14:04:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwde-0003UN-PH for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:42 -0400 Received: from smtp-out2.suse.de ([195.135.220.29]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdc-0002A6-QS for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:42 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 080141F86C; Thu, 30 Mar 2023 18:04:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199479; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jxYLf4ThcdyP8XrOyqsCtaDbjIGCxRI1LOVpHdLT+OI=; b=k/Bmc7SpIbLY2PW2jpPwEb7YNkiHLj3Y1fIls5mg+MHiR7PGkKCww70YkDQRWsD2OBRDS2 o3jYgk22zh93JIuUyZQ0CHltqvMeIxrELDnd/iEWZXEF8py+XfOBGNmYDR8O6TcWu9otDG ykep2nyVt+ezUyGqxUmIl21EbCz8XHc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199479; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jxYLf4ThcdyP8XrOyqsCtaDbjIGCxRI1LOVpHdLT+OI=; b=WV2yG0Cq3mkKOr+PnHzrN8KCWfZaiX1re6D3OiV9nDwVMn++tDeM61vkYcWsxPOignuMY8 501JdlLz0fHpNxAg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id E940C1348E; Thu, 30 Mar 2023 18:04:36 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id +N1sKzTPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:36 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela Subject: [RFC PATCH v1 20/26] io: Add a pwritev/preadv version that takes a discontiguous iovec Date: Thu, 30 Mar 2023 15:03:30 -0300 Message-Id: <20230330180336.2791-21-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=195.135.220.29; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org For the upcoming support to fixed-ram migration with multifd, we need to be able to accept an iovec array with non-contiguous data. Add a pwritev and preadv version that splits the array into contiguous segments before writing. With that we can have the ram code continue to add pages in any order and the multifd code continue to send large arrays for reading and writing. Signed-off-by: Fabiano Rosas --- Since iovs can be non contiguous, we'd need a separate array on the side to carry an extra file offset for each of them, so I'm relying on the fact that iovs are all within a same host page and passing in an encoded offset that takes the host page into account. --- include/io/channel.h | 50 +++++++++++++++++++++++++++ io/channel.c | 82 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 132 insertions(+) diff --git a/include/io/channel.h b/include/io/channel.h index 28bce7ef17..7c4d2432cf 100644 --- a/include/io/channel.h +++ b/include/io/channel.h @@ -33,8 +33,10 @@ OBJECT_DECLARE_TYPE(QIOChannel, QIOChannelClass, #define QIO_CHANNEL_ERR_BLOCK -2 #define QIO_CHANNEL_WRITE_FLAG_ZERO_COPY 0x1 +#define QIO_CHANNEL_WRITE_FLAG_WITH_OFFSET 0x2 #define QIO_CHANNEL_READ_FLAG_MSG_PEEK 0x1 +#define QIO_CHANNEL_READ_FLAG_WITH_OFFSET 0x2 typedef enum QIOChannelFeature QIOChannelFeature; @@ -541,6 +543,30 @@ int qio_channel_close(QIOChannel *ioc, ssize_t qio_channel_pwritev_full(QIOChannel *ioc, const struct iovec *iov, size_t niov, off_t offset, Error **errp); +/** + * qio_channel_write_full_all: + * @ioc: the channel object + * @iov: the array of memory regions to write data from + * @niov: the length of the @iov array + * @offset: the iovec offset in the file where to write the data + * @fds: an array of file handles to send + * @nfds: number of file handles in @fds + * @flags: write flags (QIO_CHANNEL_WRITE_FLAG_*) + * @errp: pointer to a NULL-initialized error object + * + * + * Selects between a writev or pwritev channel writer function. + * + * If QIO_CHANNEL_WRITE_FLAG_OFFSET is passed in flags, pwritev is + * used and @offset is expected to be a meaningful value, @fds and + * @nfds are ignored; otherwise uses writev and @offset is ignored. + * + * Returns: 0 if all bytes were written, or -1 on error + */ +int qio_channel_write_full_all(QIOChannel *ioc, const struct iovec *iov, + size_t niov, off_t offset, int *fds, size_t nfds, + int flags, Error **errp); + /** * qio_channel_pwritev * @ioc: the channel object @@ -577,6 +603,30 @@ ssize_t qio_channel_pwritev(QIOChannel *ioc, char *buf, size_t buflen, ssize_t qio_channel_preadv_full(QIOChannel *ioc, const struct iovec *iov, size_t niov, off_t offset, Error **errp); +/** + * qio_channel_read_full_all: + * @ioc: the channel object + * @iov: the array of memory regions to read data to + * @niov: the length of the @iov array + * @offset: the iovec offset in the file from where to read the data + * @fds: an array of file handles to send + * @nfds: number of file handles in @fds + * @flags: read flags (QIO_CHANNEL_READ_FLAG_*) + * @errp: pointer to a NULL-initialized error object + * + * + * Selects between a readv or preadv channel reader function. + * + * If QIO_CHANNEL_READ_FLAG_OFFSET is passed in flags, preadv is + * used and @offset is expected to be a meaningful value, @fds and + * @nfds are ignored; otherwise uses readv and @offset is ignored. + * + * Returns: 0 if all bytes were read, or -1 on error + */ +int qio_channel_read_full_all(QIOChannel *ioc, const struct iovec *iov, + size_t niov, off_t offset, + int flags, Error **errp); + /** * qio_channel_preadv * @ioc: the channel object diff --git a/io/channel.c b/io/channel.c index 312445b3aa..64b26040c2 100644 --- a/io/channel.c +++ b/io/channel.c @@ -463,6 +463,76 @@ ssize_t qio_channel_pwritev_full(QIOChannel *ioc, const struct iovec *iov, return klass->io_pwritev(ioc, iov, niov, offset, errp); } +static int qio_channel_preadv_pwritev_contiguous(QIOChannel *ioc, + const struct iovec *iov, + size_t niov, off_t offset, + bool is_write, Error **errp) +{ + ssize_t ret; + int i, slice_idx, slice_num; + uint64_t base, next, file_offset; + size_t len; + + slice_idx = 0; + slice_num = 1; + + /* + * If the iov array doesn't have contiguous elements, we need to + * split it in slices because we only have one (file) 'offset' for + * the whole iov. Do this here so callers don't need to break the + * iov array themselves. + */ + for (i = 0; i < niov; i++, slice_num++) { + base = (uint64_t) iov[i].iov_base; + + if (i != niov - 1) { + len = iov[i].iov_len; + next = (uint64_t) iov[i + 1].iov_base; + + if (base + len == next) { + continue; + } + } + + /* + * Use the offset of the first element of the segment that + * we're sending. + */ + file_offset = offset + (uint64_t) iov[slice_idx].iov_base; + + if (is_write) { + ret = qio_channel_pwritev_full(ioc, &iov[slice_idx], slice_num, + file_offset, errp); + } else { + ret = qio_channel_preadv_full(ioc, &iov[slice_idx], slice_num, + file_offset, errp); + } + + if (ret < 0) { + break; + } + + slice_idx += slice_num; + slice_num = 0; + } + + return (ret < 0) ? -1 : 0; +} + +int qio_channel_write_full_all(QIOChannel *ioc, + const struct iovec *iov, + size_t niov, off_t offset, + int *fds, size_t nfds, + int flags, Error **errp) +{ + if (flags & QIO_CHANNEL_WRITE_FLAG_WITH_OFFSET) { + return qio_channel_preadv_pwritev_contiguous(ioc, iov, niov, + offset, true, errp); + } + + return qio_channel_writev_full_all(ioc, iov, niov, NULL, 0, flags, errp); +} + ssize_t qio_channel_pwritev(QIOChannel *ioc, char *buf, size_t buflen, off_t offset, Error **errp) { @@ -492,6 +562,18 @@ ssize_t qio_channel_preadv_full(QIOChannel *ioc, const struct iovec *iov, return klass->io_preadv(ioc, iov, niov, offset, errp); } +int qio_channel_read_full_all(QIOChannel *ioc, const struct iovec *iov, + size_t niov, off_t offset, + int flags, Error **errp) +{ + if (flags & QIO_CHANNEL_READ_FLAG_WITH_OFFSET) { + return qio_channel_preadv_pwritev_contiguous(ioc, iov, niov, + offset, false, errp); + } + + return qio_channel_readv_full_all(ioc, iov, niov, NULL, NULL, errp); +} + ssize_t qio_channel_preadv(QIOChannel *ioc, char *buf, size_t buflen, off_t offset, Error **errp) { From patchwork Thu Mar 30 18:03:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7EB43C761A6 for ; Thu, 30 Mar 2023 18:06:21 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdl-0003gl-Hs; Thu, 30 Mar 2023 14:04:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdg-0003cR-Af for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:44 -0400 Received: from smtp-out2.suse.de ([2001:67c:2178:6::1d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwde-0002AT-NP for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:44 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 8EB9B1FD65; Thu, 30 Mar 2023 18:04:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199481; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bAPOD6ButsyUsf/nrocgicIUg/3F17VOW9SNc1mL0Vs=; b=dvGYwh66lPe6fnAQ4BHODVQL0HKsv1xbZysgCDNby5VdwE5Rb2rpTbsC6hR467q7Ao8u8M oPLK9BVK3+SECWSp+bcamjgHFaIzlz9rY4ZlgXeNaeh9veD2wWIiwomUaR/LUQH8TbdzR4 9KrInDdasEUSAxe8x9R5lgnQrZL4Oes= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199481; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bAPOD6ButsyUsf/nrocgicIUg/3F17VOW9SNc1mL0Vs=; b=92ONn2bZiBhhXZAfzIhmW1dLNnTqFyEAcshcuvmP9OH2owvittVFO9dbF9Ce/8e0R5EaPw L5265ob+ctVS5GBQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 7B48C1348E; Thu, 30 Mar 2023 18:04:39 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id ECGbEDfPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:39 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela Subject: [RFC PATCH v1 21/26] migration/ram: Add a wrapper for fixed-ram shadow bitmap Date: Thu, 30 Mar 2023 15:03:31 -0300 Message-Id: <20230330180336.2791-22-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1d; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org We'll need to set the shadow_bmap bits from outside ram.c soon and TARGET_PAGE_BITS is poisoned, so add a wrapper to it. Signed-off-by: Fabiano Rosas --- migration/ram.c | 9 +++++++++ migration/ram.h | 1 + 2 files changed, 10 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 1666ce6d5f..e9b28c16da 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3359,6 +3359,15 @@ static void ram_save_shadow_bmap(QEMUFile *f) } } +void ramblock_set_shadow_bmap(RAMBlock *block, ram_addr_t offset, bool set) +{ + if (set) { + set_bit(offset >> TARGET_PAGE_BITS, block->shadow_bmap); + } else { + clear_bit(offset >> TARGET_PAGE_BITS, block->shadow_bmap); + } +} + /** * ram_save_iterate: iterative stage for migration * diff --git a/migration/ram.h b/migration/ram.h index 81cbb0947c..8d8258fee1 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -98,6 +98,7 @@ int ram_dirty_bitmap_reload(MigrationState *s, RAMBlock *rb); bool ramblock_page_is_discarded(RAMBlock *rb, ram_addr_t start); void postcopy_preempt_shutdown_file(MigrationState *s); void *postcopy_preempt_thread(void *opaque); +void ramblock_set_shadow_bmap(RAMBlock *block, ram_addr_t offset, bool set); /* ram cache */ int colo_init_ram_cache(void); From patchwork Thu Mar 30 18:03:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B7752C77B6D for ; Thu, 30 Mar 2023 18:06:35 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdl-0003gd-Ek; Thu, 30 Mar 2023 14:04:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdi-0003e1-Vl for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:47 -0400 Received: from smtp-out1.suse.de ([2001:67c:2178:6::1c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdh-0002B0-7A for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:46 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 217B121A28; Thu, 30 Mar 2023 18:04:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199484; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=D40bqUGAizR7bdyPNJQVZOGajsk2QhjheNyi32cf6tk=; b=y6BJh6QrpmHA3ex9lWHnPW87xEqd9SwkP0cbWkFBvGLSz2VATp9csasjo4S3Kz8lD8/rwX Y4KJ/Y+KO9Wi7XeQrI8k++rgc87AvuwMF3tDjzhQW7nzF7KGFl+croIGoAbxVE7VlPsXme wYkBw6JHqi67NkpD7kof6kyQUiiYAjg= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199484; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=D40bqUGAizR7bdyPNJQVZOGajsk2QhjheNyi32cf6tk=; b=R3xaLC0QeqktNwiBZVNXkXAs9tXX8FkilugPc025hhsaby86BnQkcUH29YggOVjoTenLzt 0qGdOGAa/iziAbBg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 0F88E1348E; Thu, 30 Mar 2023 18:04:41 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 2ILcMTnPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:41 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela Subject: [RFC PATCH v1 22/26] migration/multifd: Support outgoing fixed-ram stream format Date: Thu, 30 Mar 2023 15:03:32 -0300 Message-Id: <20230330180336.2791-23-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1c; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The new fixed-ram stream format uses a file transport and puts ram pages in the migration file at their respective offsets and can be done in parallel by using the pwritev system call which takes iovecs and an offset. Add support to enabling the new format along with multifd to make use of the threading and page handling already in place. This requires multifd to stop sending headers and leaving the stream format to the fixed-ram code. When it comes time to write the data, we need to call a version of qio_channel_write that can take an offset. Usage on HMP is: (qemu) stop (qemu) migrate_set_capability multifd on (qemu) migrate_set_capability fixed-ram on (qemu) migrate_set_parameter max-bandwidth 0 (qemu) migrate_set_parameter multifd-channels 8 (qemu) migrate file:migfile Signed-off-by: Fabiano Rosas --- migration/migration.c | 5 ----- migration/multifd.c | 51 +++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 49 insertions(+), 7 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 258709aee1..77d24a5114 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1345,11 +1345,6 @@ static bool migrate_caps_check(bool *cap_list, #endif if (cap_list[MIGRATION_CAPABILITY_FIXED_RAM]) { - if (cap_list[MIGRATION_CAPABILITY_MULTIFD]) { - error_setg(errp, "Directly mapped memory incompatible with multifd"); - return false; - } - if (cap_list[MIGRATION_CAPABILITY_XBZRLE]) { error_setg(errp, "Directly mapped memory incompatible with xbzrle"); return false; diff --git a/migration/multifd.c b/migration/multifd.c index 20ef665218..cc70b20ff7 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -256,6 +256,19 @@ static void multifd_pages_clear(MultiFDPages_t *pages) g_free(pages); } +static void multifd_set_file_bitmap(MultiFDSendParams *p, bool set) +{ + MultiFDPages_t *pages = p->pages; + + if (!pages->block) { + return; + } + + for (int i = 0; i < p->normal_num; i++) { + ramblock_set_shadow_bmap(pages->block, pages->offset[i], set); + } +} + static void multifd_send_fill_packet(MultiFDSendParams *p) { MultiFDPacket_t *packet = p->packet; @@ -608,6 +621,17 @@ int multifd_send_sync_main(QEMUFile *f) } } + if (!migrate_multifd_use_packets()) { + for (i = 0; i < migrate_multifd_channels(); i++) { + MultiFDSendParams *p = &multifd_send_state->params[i]; + + qemu_sem_post(&p->sem); + continue; + } + + return 0; + } + /* * When using zero-copy, it's necessary to flush the pages before any of * the pages can be sent again, so we'll make sure the new version of the @@ -692,6 +716,8 @@ static void *multifd_send_thread(void *opaque) if (p->pending_job) { uint32_t flags; + uint64_t write_base; + p->normal_num = 0; if (!use_packets || use_zero_copy_send) { @@ -716,6 +742,16 @@ static void *multifd_send_thread(void *opaque) if (use_packets) { multifd_send_fill_packet(p); p->num_packets++; + write_base = 0; + } else { + multifd_set_file_bitmap(p, true); + + /* + * If we subtract the host page now, we don't need to + * pass it into qio_channel_write_full_all() below. + */ + write_base = p->pages->block->pages_offset - + (uint64_t)p->pages->block->host; } flags = p->flags; @@ -741,8 +777,9 @@ static void *multifd_send_thread(void *opaque) p->iov[0].iov_base = p->packet; } - ret = qio_channel_writev_full_all(p->c, p->iov, p->iovs_num, NULL, - 0, p->write_flags, &local_err); + ret = qio_channel_write_full_all(p->c, p->iov, p->iovs_num, + write_base, NULL, 0, + p->write_flags, &local_err); if (ret != 0) { break; } @@ -758,6 +795,13 @@ static void *multifd_send_thread(void *opaque) } else if (p->quit) { qemu_mutex_unlock(&p->mutex); break; + } else if (!use_packets) { + /* + * When migrating to a file there's not need for a SYNC + * packet, the channels are ready right away. + */ + qemu_sem_post(&multifd_send_state->channels_ready); + qemu_mutex_unlock(&p->mutex); } else { qemu_mutex_unlock(&p->mutex); /* sometimes there are spurious wakeups */ @@ -767,6 +811,7 @@ static void *multifd_send_thread(void *opaque) out: if (local_err) { trace_multifd_send_error(p->id); + multifd_set_file_bitmap(p, false); multifd_send_terminate_threads(local_err); error_free(local_err); } @@ -981,6 +1026,8 @@ int multifd_save_setup(Error **errp) if (migrate_use_zero_copy_send()) { p->write_flags = QIO_CHANNEL_WRITE_FLAG_ZERO_COPY; + } else if (!use_packets) { + p->write_flags |= QIO_CHANNEL_WRITE_FLAG_WITH_OFFSET; } else { p->write_flags = 0; } From patchwork Thu Mar 30 18:03:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 58B61C6FD1D for ; Thu, 30 Mar 2023 18:06:08 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdn-0003rQ-88; Thu, 30 Mar 2023 14:04:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdl-0003hJ-JJ for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:49 -0400 Received: from smtp-out1.suse.de ([195.135.220.28]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdj-0002BM-SO for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:49 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id AC43C21A2A; Thu, 30 Mar 2023 18:04:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199486; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xbnEYXpXSxKKvda2xSwA3+xm74lnwDfudsd/3P4vJn0=; b=FunnyHzyPcjgmVesHyQ7lQjgGS8xCZRLGYCCGTui31QgWUsJwR3tBl+OV9dSgMyf6r14XF cCgNllvpeE9bU73n2YEJ94ahR4Zm0fH0Sl+YeYuYaTeKRheB/Tlc7qPglPJ5rdzhFjkPyj vVzeX7U8qeRHO1vk/TYs95l/sDABGKE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199486; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xbnEYXpXSxKKvda2xSwA3+xm74lnwDfudsd/3P4vJn0=; b=t2p4OOeBp3MWeycDT+Dp8R4F0w1eUGSFMCHt4U4VO3I8xOeeoihVti1zH23quNJFKLkISL Vlkxce/Cpu8yH6DQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 961B71348E; Thu, 30 Mar 2023 18:04:44 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 0MoRFzzPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:44 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela Subject: [RFC PATCH v1 23/26] migration/multifd: Support incoming fixed-ram stream format Date: Thu, 30 Mar 2023 15:03:33 -0300 Message-Id: <20230330180336.2791-24-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=195.135.220.28; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org For the incoming fixed-ram migration we need to read the ramblock headers, get the pages bitmap and send the host address of each non-zero page to the multifd channel thread for writing. To read from the migration file we need a preadv function that can read into the iovs in segments of contiguous pages because (as in the writing case) the file offset applies to the entire iovec. Usage on HMP is: (qemu) migrate_set_capability multifd on (qemu) migrate_set_capability fixed-ram on (qemu) migrate_set_parameter max-bandwidth 0 (qemu) migrate_set_parameter multifd-channels 8 (qemu) migrate_incoming file:migfile (qemu) info status (qemu) c Signed-off-by: Fabiano Rosas --- migration/multifd.c | 26 ++++++++++++++++++++++++-- migration/ram.c | 9 +++++++-- 2 files changed, 31 insertions(+), 4 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index cc70b20ff7..36b5aedb16 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -141,6 +141,7 @@ static void nocomp_recv_cleanup(MultiFDRecvParams *p) static int nocomp_recv_pages(MultiFDRecvParams *p, Error **errp) { uint32_t flags = p->flags & MULTIFD_FLAG_COMPRESSION_MASK; + uint64_t read_base = 0; if (flags != MULTIFD_FLAG_NOCOMP) { error_setg(errp, "multifd %u: flags received %x flags expected %x", @@ -151,7 +152,13 @@ static int nocomp_recv_pages(MultiFDRecvParams *p, Error **errp) p->iov[i].iov_base = p->host + p->normal[i]; p->iov[i].iov_len = p->page_size; } - return qio_channel_readv_all(p->c, p->iov, p->normal_num, errp); + + if (migrate_fixed_ram()) { + read_base = p->pages->block->pages_offset - (uint64_t) p->host; + } + + return qio_channel_read_full_all(p->c, p->iov, p->normal_num, read_base, + p->read_flags, errp); } static MultiFDMethods multifd_nocomp_ops = { @@ -1221,9 +1228,21 @@ void multifd_recv_sync_main(void) { int i; - if (!migrate_use_multifd() || !migrate_multifd_use_packets()) { + if (!migrate_use_multifd()) { return; } + + if (!migrate_multifd_use_packets()) { + for (i = 0; i < migrate_multifd_channels(); i++) { + MultiFDRecvParams *p = &multifd_recv_state->params[i]; + + qemu_sem_post(&p->sem); + continue; + } + + return; + } + for (i = 0; i < migrate_multifd_channels(); i++) { MultiFDRecvParams *p = &multifd_recv_state->params[i]; @@ -1256,6 +1275,7 @@ static void *multifd_recv_thread(void *opaque) while (true) { uint32_t flags; + p->normal_num = 0; if (p->quit) { break; @@ -1377,6 +1397,8 @@ int multifd_load_setup(Error **errp) p->packet_len = sizeof(MultiFDPacket_t) + sizeof(uint64_t) * page_count; p->packet = g_malloc0(p->packet_len); + } else { + p->read_flags |= QIO_CHANNEL_READ_FLAG_WITH_OFFSET; } p->name = g_strdup_printf("multifdrecv_%d", i); p->iov = g_new0(struct iovec, page_count); diff --git a/migration/ram.c b/migration/ram.c index e9b28c16da..180e8e0d94 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -4427,8 +4427,13 @@ static void read_ramblock_fixed_ram(QEMUFile *f, RAMBlock *block, host = host_from_ram_block_offset(block, offset); read_len = MIN(len, TARGET_PAGE_SIZE); - read = qemu_get_buffer_at(f, host, read_len, - block->pages_offset + offset); + if (migrate_use_multifd()) { + multifd_recv_queue_page(f, block, offset); + read = read_len; + } else { + read = qemu_get_buffer_at(f, host, read_len, + block->pages_offset + offset); + } completed += read; } } From patchwork Thu Mar 30 18:03:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 33337C77B60 for ; Thu, 30 Mar 2023 18:05:37 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdz-0004AY-7F; Thu, 30 Mar 2023 14:05:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdr-0003vP-9D for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:59 -0400 Received: from smtp-out1.suse.de ([195.135.220.28]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdn-0002Bs-1F for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:52 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id E6C9F21A28; Thu, 30 Mar 2023 18:04:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199489; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LBctC7xfTNOsGsTjxQoB9Jw5X8LHmjUFXnq6SNzwf04=; b=rLM9/mNqXqK25iAcZYhUFKgYusksxGSZ30pblYjz8ncWBXyC6WU0AHV/S1bK5WuTCzIfk9 eAQLv2khIaM+VR9p+d9KZbiwrrgyaIm/UyU9hCzxBgd/m6VKBf1lmpigW8mSWmJDQVoYY9 wqh/LAxsRPfaISoQ217xb+X4NzzplNI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199489; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LBctC7xfTNOsGsTjxQoB9Jw5X8LHmjUFXnq6SNzwf04=; b=ZlUUCcZ3p5yt/LYKJ3/c8Elw0HavRczLL10aFh+rKNFjnpDq/B9KHX7VGTH0TPw8BvLUIY zKLkSR37kY4bDkBA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 2A7E41348E; Thu, 30 Mar 2023 18:04:46 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id CAhdOD7PJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:46 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Thomas Huth , Laurent Vivier , Paolo Bonzini Subject: [RFC PATCH v1 24/26] tests/qtest: Add a multifd + fixed-ram migration test Date: Thu, 30 Mar 2023 15:03:34 -0300 Message-Id: <20230330180336.2791-25-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=195.135.220.28; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Signed-off-by: Fabiano Rosas --- tests/qtest/migration-test.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 84b4c761ad..2e0911996d 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -1564,6 +1564,31 @@ static void test_precopy_file_fixed_ram(void) test_precopy_common(&args); } +static void *migrate_multifd_fixed_ram_start(QTestState *from, QTestState *to) +{ + migrate_fixed_ram_start(from, to); + + migrate_set_parameter_int(from, "multifd-channels", 4); + migrate_set_parameter_int(to, "multifd-channels", 4); + + migrate_set_capability(from, "multifd", true); + migrate_set_capability(to, "multifd", true); + + return NULL; +} + +static void test_multifd_file_fixed_ram(void) +{ + g_autofree char *uri = g_strdup_printf("file:%s/migfile", tmpfs); + MigrateCommon args = { + .connect_uri = uri, + .listen_uri = "defer", + .start_hook = migrate_multifd_fixed_ram_start, + }; + + test_precopy_common(&args); +} + static void test_precopy_tcp_plain(void) { MigrateCommon args = { @@ -2560,6 +2585,8 @@ int main(int argc, char **argv) test_precopy_file_stream_ram); qtest_add_func("/migration/precopy/file/fixed-ram", test_precopy_file_fixed_ram); + qtest_add_func("/migration/multifd/file/fixed-ram", + test_multifd_file_fixed_ram); #ifdef CONFIG_GNUTLS qtest_add_func("/migration/precopy/unix/tls/psk", From patchwork Thu Mar 30 18:03:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194704 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0C4F8C6FD1D for ; Thu, 30 Mar 2023 18:06:36 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phwdx-00045I-GZ; Thu, 30 Mar 2023 14:05:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdv-0003wW-Hu for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:59 -0400 Received: from smtp-out2.suse.de ([2001:67c:2178:6::1d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdr-0002CS-1x for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:04:56 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id EC8941F85D; Thu, 30 Mar 2023 18:04:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199492; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qtULdrZbNmgeG+8GbFopzbhmF1MwlpcwBBoHGMhsoDI=; b=alQn+VKXxNLFtMa3bWIiZTHrvnJnBWuT7EmgJ870A8adEcA/3ZXRB9l4wvIprMjd0ej8rB +44niIwUomaz2JBUflKiUH9UA07WFPzDsZFkfN2TmNaSv7xx89v32nqo2zSzTS9NQD6wjs 2vr98Y8gRQcHxx9jLXHQJeN8lCTosAc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199492; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qtULdrZbNmgeG+8GbFopzbhmF1MwlpcwBBoHGMhsoDI=; b=g/GzmAJagWpu4gYvj/WTG7Bp8xm2LPLvxWzzhtJ1IcCsEqjp3+0X/z7zDdhdt+L0wYqFoz 3hNQG/O9TDlqmdAw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 6CD981348E; Thu, 30 Mar 2023 18:04:50 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id gCn0DELPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:50 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela , Eric Blake , Markus Armbruster Subject: [RFC PATCH v1 25/26] migration: Add direct-io parameter Date: Thu, 30 Mar 2023 15:03:35 -0300 Message-Id: <20230330180336.2791-26-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1d; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add the direct-io migration parameter that tells the migration code to use O_DIRECT when opening the migration stream file whenever possible. This is currently only used for the secondary channels of fixed-ram migration, which can guarantee that writes are page aligned. However the parameter could be made to affect other types of file-based migrations in the future. Signed-off-by: Fabiano Rosas --- include/qemu/osdep.h | 2 ++ migration/file.c | 13 +++++++++++-- migration/migration-hmp-cmds.c | 9 +++++++++ migration/migration.c | 32 ++++++++++++++++++++++++++++++++ migration/migration.h | 1 + qapi/migration.json | 17 ++++++++++++++--- util/osdep.c | 9 +++++++++ 7 files changed, 78 insertions(+), 5 deletions(-) diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h index 9eff0be95b..19c1d5f999 100644 --- a/include/qemu/osdep.h +++ b/include/qemu/osdep.h @@ -570,6 +570,8 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive); bool qemu_has_ofd_lock(void); #endif +bool qemu_has_direct_io(void); + #if defined(__HAIKU__) && defined(__i386__) #define FMT_pid "%ld" #elif defined(WIN64) diff --git a/migration/file.c b/migration/file.c index 6f40894488..1a40da097d 100644 --- a/migration/file.c +++ b/migration/file.c @@ -43,9 +43,18 @@ void file_send_channel_create(QIOTaskFunc f, void *data) QIOChannelFile *ioc; QIOTask *task; Error *errp = NULL; + int flags = outgoing_args.flags; - ioc = qio_channel_file_new_path(outgoing_args.fname, - outgoing_args.flags, + if (migrate_use_direct_io() && qemu_has_direct_io()) { + /* + * Enable O_DIRECT for the secondary channels. These are used + * for sending ram pages and writes should be guaranteed to be + * aligned to at least page size. + */ + flags |= O_DIRECT; + } + + ioc = qio_channel_file_new_path(outgoing_args.fname, flags, outgoing_args.mode, &errp); if (!ioc) { file_migration_cancel(errp); diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index 72519ea99f..c9a8d84714 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -344,6 +344,11 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) } } } + if (params->has_direct_io) { + monitor_printf(mon, "%s: %s\n", + MigrationParameter_str(MIGRATION_PARAMETER_DIRECT_IO), + params->direct_io ? "on" : "off"); + } } qapi_free_MigrationParameters(params); @@ -600,6 +605,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) error_setg(&err, "The block-bitmap-mapping parameter can only be set " "through QMP"); break; + case MIGRATION_PARAMETER_DIRECT_IO: + p->has_direct_io = true; + visit_type_bool(v, param, &p->direct_io, &err); + break; default: assert(0); } diff --git a/migration/migration.c b/migration/migration.c index 77d24a5114..65798171e4 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1022,6 +1022,11 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) s->parameters.block_bitmap_mapping); } + if (s->parameters.has_direct_io) { + params->has_direct_io = true; + params->direct_io = s->parameters.direct_io; + } + return params; } @@ -1782,6 +1787,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params, dest->has_block_bitmap_mapping = true; dest->block_bitmap_mapping = params->block_bitmap_mapping; } + + if (params->has_direct_io) { + dest->direct_io = params->direct_io; + } } static void migrate_params_apply(MigrateSetParameters *params, Error **errp) @@ -1904,6 +1913,10 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp) QAPI_CLONE(BitmapMigrationNodeAliasList, params->block_bitmap_mapping); } + + if (params->has_direct_io) { + s->parameters.direct_io = params->direct_io; + } } void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp) @@ -2885,6 +2898,24 @@ bool migrate_postcopy_preempt(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_POSTCOPY_PREEMPT]; } +bool migrate_use_direct_io(void) +{ + MigrationState *s; + + /* For now O_DIRECT is only supported with fixed-ram */ + if (!migrate_fixed_ram()) { + return false; + } + + s = migrate_get_current(); + + if (s->parameters.has_direct_io) { + return s->parameters.direct_io; + } + + return false; +} + /* migration thread support */ /* * Something bad happened to the RP stream, mark an error @@ -4666,6 +4697,7 @@ static void migration_instance_init(Object *obj) params->has_announce_max = true; params->has_announce_rounds = true; params->has_announce_step = true; + params->has_direct_io = qemu_has_direct_io(); qemu_sem_init(&ms->postcopy_pause_sem, 0); qemu_sem_init(&ms->postcopy_pause_rp_sem, 0); diff --git a/migration/migration.h b/migration/migration.h index 8459201958..e0c9c78570 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -422,6 +422,7 @@ bool migrate_ignore_shared(void); bool migrate_validate_uuid(void); int migrate_fixed_ram(void); bool migrate_multifd_use_packets(void); +bool migrate_use_direct_io(void); bool migrate_auto_converge(void); bool migrate_use_multifd(void); bool migrate_pause_before_switchover(void); diff --git a/qapi/migration.json b/qapi/migration.json index 22eea58ce3..2190d98ded 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -776,6 +776,9 @@ # block device name if there is one, and to their node name # otherwise. (Since 5.2) # +# @direct-io: Open migration files with O_DIRECT when possible. Not +# all migration transports support this. (since 8.1) +# # Features: # @unstable: Member @x-checkpoint-delay is experimental. # @@ -796,7 +799,7 @@ 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', 'multifd-zlib-level' ,'multifd-zstd-level', - 'block-bitmap-mapping' ] } + 'block-bitmap-mapping', 'direct-io' ] } ## # @MigrateSetParameters: @@ -941,6 +944,9 @@ # block device name if there is one, and to their node name # otherwise. (Since 5.2) # +# @direct-io: Open migration files with O_DIRECT when possible. Not +# all migration transports support this. (since 8.1) +# # Features: # @unstable: Member @x-checkpoint-delay is experimental. # @@ -976,7 +982,8 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*direct-io': 'bool' } } ## # @migrate-set-parameters: @@ -1141,6 +1148,9 @@ # block device name if there is one, and to their node name # otherwise. (Since 5.2) # +# @direct-io: Open migration files with O_DIRECT when possible. Not +# all migration transports support this. (since 8.1) +# # Features: # @unstable: Member @x-checkpoint-delay is experimental. # @@ -1174,7 +1184,8 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*direct-io': 'bool' } } ## # @query-migrate-parameters: diff --git a/util/osdep.c b/util/osdep.c index e996c4744a..d0227a60ab 100644 --- a/util/osdep.c +++ b/util/osdep.c @@ -277,6 +277,15 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive) } #endif +bool qemu_has_direct_io(void) +{ +#ifdef O_DIRECT + return true; +#else + return false; +#endif +} + static int qemu_open_cloexec(const char *name, int flags, mode_t mode) { int ret; From patchwork Thu Mar 30 18:03:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13194707 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CA3A2C6FD1D for ; Thu, 30 Mar 2023 18:08:08 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1phweE-0004mH-Jh; Thu, 30 Mar 2023 14:05:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1phwdx-00046R-M3 for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:05:01 -0400 Received: from smtp-out1.suse.de ([2001:67c:2178:6::1c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1phwdv-0002Cs-9k for qemu-devel@nongnu.org; Thu, 30 Mar 2023 14:05:01 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 7D73F21A28; Thu, 30 Mar 2023 18:04:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1680199495; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YBOn1F0mFy3EHuJgQ6+dEFU+zvI08FJIqP8WSJO233c=; b=JHx7iQ3kwt4vaVjtMh723zyw14+XN7UJttIuJN06F7zqynrcbDnXbihHca8b7kPwFGRDcz L7yKmK4tJTXTLiBnc3pK5Fi1xrok7s/4WskVBK0aNyE1rJhYGhv9uoo1wIZrHDjThSNwHy yajTG6cGRl/4Tzum7wsi7LHaTrhGFjA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1680199495; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YBOn1F0mFy3EHuJgQ6+dEFU+zvI08FJIqP8WSJO233c=; b=CbMTRIRGQa7rslgowzPZc7GWH/0qaHVIE5CvJbVOdcg+o5hYbXE0d4HWVBjmhj+aUrOPzs tedWINadFerVFMBQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 6D45C1348E; Thu, 30 Mar 2023 18:04:53 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id aGEVDUXPJWQ5GwAAMHmgww (envelope-from ); Thu, 30 Mar 2023 18:04:53 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Claudio Fontana , jfehlig@suse.com, dfaggioli@suse.com, dgilbert@redhat.com, =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Juan Quintela Subject: [RFC PATCH v1 26/26] tests/migration/guestperf: Add file, fixed-ram and direct-io support Date: Thu, 30 Mar 2023 15:03:36 -0300 Message-Id: <20230330180336.2791-27-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230330180336.2791-1-farosas@suse.de> References: <20230330180336.2791-1-farosas@suse.de> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:67c:2178:6::1c; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add support to the new migration features: - 'file' transport; - 'fixed-ram' stream format capability; - 'direct-io' parameter; Usage: $ ./guestperf.py --binary --initrd \ --transport file --dst-file migfile --multifd --fixed-ram \ --multifd-channels 4 --output fixed-ram.json --verbose Signed-off-by: Fabiano Rosas --- tests/migration/guestperf/engine.py | 38 +++++++++++++++++++++++++-- tests/migration/guestperf/scenario.py | 14 ++++++++-- tests/migration/guestperf/shell.py | 18 +++++++++++-- 3 files changed, 64 insertions(+), 6 deletions(-) diff --git a/tests/migration/guestperf/engine.py b/tests/migration/guestperf/engine.py index e69d16a62c..a465336184 100644 --- a/tests/migration/guestperf/engine.py +++ b/tests/migration/guestperf/engine.py @@ -35,10 +35,11 @@ class Engine(object): def __init__(self, binary, dst_host, kernel, initrd, transport="tcp", - sleep=15, verbose=False, debug=False): + sleep=15, verbose=False, debug=False, dst_file="/tmp/migfile"): self._binary = binary # Path to QEMU binary self._dst_host = dst_host # Hostname of target host + self._dst_file = dst_file # Path to file (for file transport) self._kernel = kernel # Path to kernel image self._initrd = initrd # Path to stress initrd self._transport = transport # 'unix' or 'tcp' or 'rdma' @@ -203,6 +204,23 @@ def _migrate(self, hardware, scenario, src, dst, connect_uri): resp = dst.command("migrate-set-parameters", multifd_channels=scenario._multifd_channels) + if scenario._fixed_ram: + resp = src.command("migrate-set-capabilities", + capabilities = [ + { "capability": "fixed-ram", + "state": True } + ]) + resp = dst.command("migrate-set-capabilities", + capabilities = [ + { "capability": "fixed-ram", + "state": True } + ]) + + if scenario._direct_io: + resp = src.command("migrate-set-parameters", + direct_io=scenario._direct_io) + + resp = src.command("migrate", uri=connect_uri) post_copy = False @@ -233,6 +251,11 @@ def _migrate(self, hardware, scenario, src, dst, connect_uri): progress_history.append(progress) if progress._status == "completed": + if connect_uri[0:5] == "file:": + if self._verbose: + print("Migrating incoming") + dst.command("migrate-incoming", uri=connect_uri) + if self._verbose: print("Sleeping %d seconds for final guest workload run" % self._sleep) sleep_secs = self._sleep @@ -357,7 +380,11 @@ def _get_dst_args(self, hardware, uri): if self._dst_host != "localhost": tunnelled = True argv = self._get_common_args(hardware, tunnelled) - return argv + ["-incoming", uri] + + incoming = ["-incoming", uri] + if uri[0:5] == "file:": + incoming = ["-incoming", "defer"] + return argv + incoming @staticmethod def _get_common_wrapper(cpu_bind, mem_bind): @@ -417,6 +444,10 @@ def run(self, hardware, scenario, result_dir=os.getcwd()): os.remove(monaddr) except: pass + elif self._transport == "file": + if self._dst_host != "localhost": + raise Exception("Use unix migration transport for non-local host") + uri = "file:%s" % self._dst_file if self._dst_host != "localhost": dstmonaddr = ("localhost", 9001) @@ -453,6 +484,9 @@ def run(self, hardware, scenario, result_dir=os.getcwd()): if self._dst_host == "localhost" and os.path.exists(dstmonaddr): os.remove(dstmonaddr) + if uri[0:5] == "file:" and os.path.exists(uri[5:]): + os.remove(uri[5:]) + if self._verbose: print("Finished migration") diff --git a/tests/migration/guestperf/scenario.py b/tests/migration/guestperf/scenario.py index de70d9b2f5..29b6af41ac 100644 --- a/tests/migration/guestperf/scenario.py +++ b/tests/migration/guestperf/scenario.py @@ -30,7 +30,8 @@ def __init__(self, name, auto_converge=False, auto_converge_step=10, compression_mt=False, compression_mt_threads=1, compression_xbzrle=False, compression_xbzrle_cache=10, - multifd=False, multifd_channels=2): + multifd=False, multifd_channels=2, + fixed_ram=False, direct_io=False): self._name = name @@ -60,6 +61,11 @@ def __init__(self, name, self._multifd = multifd self._multifd_channels = multifd_channels + self._fixed_ram = fixed_ram + + self._direct_io = direct_io + + def serialize(self): return { "name": self._name, @@ -79,6 +85,8 @@ def serialize(self): "compression_xbzrle_cache": self._compression_xbzrle_cache, "multifd": self._multifd, "multifd_channels": self._multifd_channels, + "fixed_ram": self._fixed_ram, + "direct_io": self._direct_io, } @classmethod @@ -100,4 +108,6 @@ def deserialize(cls, data): data["compression_xbzrle"], data["compression_xbzrle_cache"], data["multifd"], - data["multifd_channels"]) + data["multifd_channels"], + data["fixed_ram"], + data["direct_io"]) diff --git a/tests/migration/guestperf/shell.py b/tests/migration/guestperf/shell.py index 8a809e3dda..0cb402adce 100644 --- a/tests/migration/guestperf/shell.py +++ b/tests/migration/guestperf/shell.py @@ -48,6 +48,7 @@ def __init__(self): parser.add_argument("--kernel", dest="kernel", default="/boot/vmlinuz-%s" % platform.release()) parser.add_argument("--initrd", dest="initrd", default="tests/migration/initrd-stress.img") parser.add_argument("--transport", dest="transport", default="unix") + parser.add_argument("--dst-file", dest="dst_file") # Hardware args @@ -71,7 +72,8 @@ def get_engine(self, args): transport=args.transport, sleep=args.sleep, debug=args.debug, - verbose=args.verbose) + verbose=args.verbose, + dst_file=args.dst_file) def get_hardware(self, args): def split_map(value): @@ -127,6 +129,13 @@ def __init__(self): parser.add_argument("--multifd-channels", dest="multifd_channels", default=2, type=int) + parser.add_argument("--fixed-ram", dest="fixed_ram", default=False, + action="store_true") + + parser.add_argument("--direct-io", dest="direct_io", default=False, + action="store_true") + + def get_scenario(self, args): return Scenario(name="perfreport", downtime=args.downtime, @@ -150,7 +159,12 @@ def get_scenario(self, args): compression_xbzrle_cache=args.compression_xbzrle_cache, multifd=args.multifd, - multifd_channels=args.multifd_channels) + multifd_channels=args.multifd_channels, + + fixed_ram=args.fixed_ram, + + direct_io=args.direct_io) + def run(self, argv): args = self._parser.parse_args(argv)