From patchwork Fri Dec 6 00:58:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13896182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 83203E77170 for ; Fri, 6 Dec 2024 01:01:01 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJMgE-0003a5-K9; Thu, 05 Dec 2024 19:58:50 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJMgD-0003ZD-6A for qemu-devel@nongnu.org; Thu, 05 Dec 2024 19:58:49 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJMg9-0005iM-1i for qemu-devel@nongnu.org; Thu, 05 Dec 2024 19:58:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733446722; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=es3XBObQHiGtgZBVTVbSw3WDegXk2q8ebNuAwrDJ0N0=; b=Dn711r+rrs9wHwbEzkNVDvYWJIybIpGIETjizzb9Sb1znKomRCaxlheHQ710RDE793W3Ye B+4DAW1HBqvA35h0/Iz6FAKGWMaowLabYfYcrWIBubzqE6ejAWa+XhFTlhQqYiKObpor3+ KflVG+01NyOhdolNu4OUcDNJZqGf0/Q= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-511-Uv6HYEu0NyCVRxUGyinsqw-1; Thu, 05 Dec 2024 19:58:41 -0500 X-MC-Unique: Uv6HYEu0NyCVRxUGyinsqw-1 X-Mimecast-MFC-AGG-ID: Uv6HYEu0NyCVRxUGyinsqw Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-6d893f0027fso42036656d6.0 for ; Thu, 05 Dec 2024 16:58:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733446720; x=1734051520; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=es3XBObQHiGtgZBVTVbSw3WDegXk2q8ebNuAwrDJ0N0=; b=ixnn3rs78YZxjKPeN0ZQW7OS+mfqMdatmQxYe9ASwS8gW4g5K7DVh+lqtd9JnjlHJQ qijqaC8/cxbKvMuvlK68cTtEcZsVCFn2isR4qPZxGecGwntLMPgw9JyTfoISEqE0h+dp XUgbgoC5CNYP2WGFHPw+kWXluLBv7vT4qXt7roqvfRzmbn29ogHPAHfu5KlCp2weucaJ aAbldfFdIsWPrpaDW6Tu04pvCC+EE+5XxioI3Oq1USBLkFh2lH8I0ddwBaBLxXH/LKce 0jcoch0ya7q+A4hayv685NSi27NcMrYSEWgwE8mLz64ms+OeJ9faJxj8/80WI+PMQL0g mBdQ== X-Gm-Message-State: AOJu0YyXLJ0W/tRMv5csmqtmgjD9K8wfEkWeWspisgpe9iFHuAcbbln5 0yfbSMQVTkHwfPaTeX0P7g6tBaebJja3qzj5IyDk2cTrnLWkMhPcwAd4gIui96OxMcgYDIlCbwF 1o/7MSECR5RVeCGHMmMRf31TmMpKnyr3yQGPaZjWECTAb2+X6oNGY/jzwPy5Z22f0D+9mnZ2Bx4 VnCJ2E+zodtGaRPvWVimiFw3qW4d8Z8rRjTA== X-Gm-Gg: ASbGnctdT4w+unLpAU7p1Pu3LaeJigmT+JiLXbhOg8yPQX9z9xBwEnaXzu+Qd6GU9oa qi+KDE+eRgPzmxlT2ffRMUNkluwKke/+AWsShZlPUU/92Ta3gSqJuFRAGpTEQByDO2Wu4mHwY8r fzoYKhsKKx+K67Mz42zSWQCKNBtFrTl+/knYCE+of+7yfE+M3zEVxgMiqKW1r75uthdJNg7V0sT y1Ayc4W0GaX8y/pekMW0vQgCdfH09fB7LHS5n/s01SSeE4JuMcSPJXZ3jmkoHB4Qqxqc2EMmBU7 QHCKEdY1fDxB5hxo0WGSF7Lpdg== X-Received: by 2002:ad4:5c46:0:b0:6d8:7ed4:336c with SMTP id 6a1803df08f44-6d8e70a2743mr24885956d6.9.1733446720024; Thu, 05 Dec 2024 16:58:40 -0800 (PST) X-Google-Smtp-Source: AGHT+IEUWRkX/DMHW6ZCiAHXl9KlYYZWdaC3iIDx0eVNrUEzC4vIQRmJBnzXGJUDRyid7bDrenVBIQ== X-Received: by 2002:ad4:5c46:0:b0:6d8:7ed4:336c with SMTP id 6a1803df08f44-6d8e70a2743mr24885556d6.9.1733446719594; Thu, 05 Dec 2024 16:58:39 -0800 (PST) Received: from x1n.redhat.com (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6d8dac016cbsm12635226d6.117.2024.12.05.16.58.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Dec 2024 16:58:38 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Cc: "Maciej S . Szmigiero" , peterx@redhat.com, =?utf-8?q?C=C3=A9dric_Le_Goater?= , Avihai Horon , Alex Williamson , Fabiano Rosas , Prasad Pandit Subject: [PATCH v2 1/7] migration/multifd: Further remove the SYNC on complete Date: Thu, 5 Dec 2024 19:58:28 -0500 Message-ID: <20241206005834.1050905-2-peterx@redhat.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241206005834.1050905-1-peterx@redhat.com> References: <20241206005834.1050905-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.996, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Commit 637280aeb2 ("migration/multifd: Avoid the final FLUSH in complete()") removed the FLUSH operation on complete() which should avoid one global sync on destination side, because it's not needed. However that commit overlooked multifd_ram_flush_and_sync() part of things, as that's always used together with the FLUSH message on the main channel. For very old binaries (multifd_flush_after_each_section==true), that's still needed because each EOS received on destination will enforce all-channel sync once. For new binaries (multifd_flush_after_each_section==false), the flush and sync shouldn't be needed just like the FLUSH, with the same reasoning. Currently, on new binaries we'll still send SYNC messages on multifd channels, even if FLUSH is omitted at the end. It'll make all recv threads hang at SYNC message. Multifd is still all working fine because luckily recv side cleanup code (mostly multifd_recv_sync_main()) is smart enough to make sure even if recv threads are stuck at SYNC it'll get kicked out. And since this is the completion phase of migration, nothing else will be sent after the SYNCs. This may be needed for VFIO in the future because VFIO can have data to push after ram_save_complete(), hence we don't want the recv thread to be stuck in SYNC message. Remove this limitation will make src even slightly faster too to avoid some more code. Stable branches do not need this patch, as no real bug I can think of that will go wrong there.. so not attaching Fixes to be clear on the backport not needed. Signed-off-by: Peter Xu Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/ram.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 05ff9eb328..7284c34bd8 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3283,9 +3283,16 @@ static int ram_save_complete(QEMUFile *f, void *opaque) } } - ret = multifd_ram_flush_and_sync(); - if (ret < 0) { - return ret; + if (migrate_multifd() && + migrate_multifd_flush_after_each_section()) { + /* + * Only the old dest QEMU will need this sync, because each EOS + * will require one SYNC message on each channel. + */ + ret = multifd_ram_flush_and_sync(); + if (ret < 0) { + return ret; + } } if (migrate_mapped_ram()) { From patchwork Fri Dec 6 00:58:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13896178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4CACCE77170 for ; Fri, 6 Dec 2024 00:59:39 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJMgF-0003aS-O0; Thu, 05 Dec 2024 19:58:51 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJMgD-0003ZA-5D for qemu-devel@nongnu.org; Thu, 05 Dec 2024 19:58:49 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJMg9-0005ki-T9 for qemu-devel@nongnu.org; Thu, 05 Dec 2024 19:58:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733446725; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=of03+bL8atAVQ8Sx7MKVcupnwu+B9QuMvu36EUykEjA=; b=NHOV1QF4BJsqGRRu1Z5+5vs11Lzxjjg8jSVL361T3IrIsldtisNt9xeWkpTfiD36XxzXbT k+hnnIlKjG968G4TdRac5EZY3ZLm/j9E8e7uC0OZpv6KVmLzQhHusnkTYA2JwY7LFEFLc8 B100Hca/+6PWoc/QN9bH8OleU5cbeWQ= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-411-gVYBKsjnOAO6leXwWcEFDQ-1; Thu, 05 Dec 2024 19:58:43 -0500 X-MC-Unique: gVYBKsjnOAO6leXwWcEFDQ-1 X-Mimecast-MFC-AGG-ID: gVYBKsjnOAO6leXwWcEFDQ Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-6d88cde9cedso28575866d6.2 for ; Thu, 05 Dec 2024 16:58:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733446722; x=1734051522; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=of03+bL8atAVQ8Sx7MKVcupnwu+B9QuMvu36EUykEjA=; b=AIYWUoIHyEvMwCcVc5D/uSWcaJhgW4oicaWBkQnchzFDB91BvgOerB4HXbRVYrqR2L mKYJfKxNDzXX6TNF8KxLBAMzfGNU0EDLutNWlHTXGC18kisFvY4VBbw6+xsXZMe2umGb x6qnsoltrudSlI5zpDlIePsjye/PQK8z8LjxFoZqTBQW+yOEjZz/YMwOMnmBicf6EEQh Ox/JGuiRwpCx5SB3Yzv0BifExmZl2mxeIjNFuTAtZFCw7OEId406o/Tg3ZC+qE5Zcgf0 TxiuocmMbXStxvGz1dyAG3Yaar0BY0pZq7Ejl0g71zu+1R+aTbEvGudIYoB6Zb6keQUE 3vsg== X-Gm-Message-State: AOJu0YxDoT9ZeDibxCCvi2pnwmTdfdqQc3hHYotb62FfdG8zvGmQw6oR TpoqzMTFyDxUYWx0agTmQ/0IttywsD6bOZXx7Na0XWifhuLHkAN67j9rbEy+gJ/wcdd2kKWrO9H hbF4DTdStAtz7B6iS7WC2MTb7DIs111bmr6CcQ/oP3+3Gr9RyCCzaeWynvbymr+8M/7RMG2n+Po 9gIWaxiE4v0y5LkQLch11N/tfwuSTX+GimmQ== X-Gm-Gg: ASbGncvRCTWaufHCucyN+GJXfqzKOdD2DssNITbofsR2liIR6TJtY3Xs4QI6/PWcs0X h2qYeitCm6nLPegHO3dX0mqz0dmfj4gXHEJSZcNvWFs/sgt+YKwSER2/7t5lVN5WM0ryK89Acjr oo2vsEQmrSISHNjCyAkF7Y9DgDchaAmaoWOPOtY3/OpYypaWO4WqeJ0OcQyPS1bKouVNjaEvIwy XaX1GZZ7fdIL1rStBSvxdQfSpRC1tBWFuLkgCeMLcBIhlvUc2DpqoKk9+vPTre/0mzltBpd6fyU sApipkAXcNYyOBdkTSC/sAnM+A== X-Received: by 2002:a05:6214:226b:b0:6d8:80e8:d567 with SMTP id 6a1803df08f44-6d8e7171871mr19171456d6.18.1733446722285; Thu, 05 Dec 2024 16:58:42 -0800 (PST) X-Google-Smtp-Source: AGHT+IGHqSvKx0Fdgv1BuBiyzutcmeiZvriVu0GpdIByj+uT5tm34qvjtlNTSxZPTwVTAIRnWWSCGw== X-Received: by 2002:a05:6214:226b:b0:6d8:80e8:d567 with SMTP id 6a1803df08f44-6d8e7171871mr19171086d6.18.1733446721867; Thu, 05 Dec 2024 16:58:41 -0800 (PST) Received: from x1n.redhat.com (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6d8dac016cbsm12635226d6.117.2024.12.05.16.58.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Dec 2024 16:58:40 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Cc: "Maciej S . Szmigiero" , peterx@redhat.com, =?utf-8?q?C=C3=A9dric_Le_Goater?= , Avihai Horon , Alex Williamson , Fabiano Rosas , Prasad Pandit Subject: [PATCH v2 2/7] migration/multifd: Allow to sync with sender threads only Date: Thu, 5 Dec 2024 19:58:29 -0500 Message-ID: <20241206005834.1050905-3-peterx@redhat.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241206005834.1050905-1-peterx@redhat.com> References: <20241206005834.1050905-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.996, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Teach multifd_send_sync_main() to sync with threads only. We already have such requests, which is when mapped-ram is enabled with multifd. In that case, no SYNC messages will be pushed to the stream when multifd syncs the sender threads because there's no destination threads waiting for that. The whole point of the sync is to make sure all threads flushed their jobs. So fundamentally we have a request to do the sync in different ways: - Either to sync the threads only, - Or to sync the threads but also with the destination side. Mapped-ram did it already because of the use_packet check in the sync handler of the sender thread. It works. However it may stop working when e.g. VFIO may start to reuse multifd channels to push device states. In that case VFIO has similar request on "thread-only sync" however we can't check a flag because such sync request can still come from RAM which needs the on-wire notifications. Paving way for that by allowing the multifd_send_sync_main() to specify what kind of sync the caller needs. We can use it for mapped-ram already. No functional change intended. Signed-off-by: Peter Xu --- migration/multifd.h | 19 ++++++++++++++++--- migration/multifd-nocomp.c | 7 ++++++- migration/multifd.c | 15 +++++++++------ 3 files changed, 31 insertions(+), 10 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index 50d58c0c9c..bd337631ec 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -19,6 +19,18 @@ typedef struct MultiFDRecvData MultiFDRecvData; typedef struct MultiFDSendData MultiFDSendData; +typedef enum { + /* No sync request */ + MULTIFD_SYNC_NONE = 0, + /* Sync locally on the sender threads without pushing messages */ + MULTIFD_SYNC_LOCAL, + /* + * Sync not only on the sender threads, but also push "SYNC" message to + * the wire (which is for a remote sync). + */ + MULTIFD_SYNC_ALL, +} MultiFDSyncReq; + bool multifd_send_setup(void); void multifd_send_shutdown(void); void multifd_send_channel_created(void); @@ -28,7 +40,7 @@ void multifd_recv_shutdown(void); bool multifd_recv_all_channels_created(void); void multifd_recv_new_channel(QIOChannel *ioc, Error **errp); void multifd_recv_sync_main(void); -int multifd_send_sync_main(void); +int multifd_send_sync_main(MultiFDSyncReq req); bool multifd_queue_page(RAMBlock *block, ram_addr_t offset); bool multifd_recv(void); MultiFDRecvData *multifd_get_recv_data(void); @@ -143,7 +155,7 @@ typedef struct { /* multifd flags for each packet */ uint32_t flags; /* - * The sender thread has work to do if either of below boolean is set. + * The sender thread has work to do if either of below field is set. * * @pending_job: a job is pending * @pending_sync: a sync request is pending @@ -152,7 +164,8 @@ typedef struct { * cleared by the multifd sender threads. */ bool pending_job; - bool pending_sync; + MultiFDSyncReq pending_sync; + MultiFDSendData *data; /* thread local variables. No locking required */ diff --git a/migration/multifd-nocomp.c b/migration/multifd-nocomp.c index 55191152f9..219f9e58ef 100644 --- a/migration/multifd-nocomp.c +++ b/migration/multifd-nocomp.c @@ -345,6 +345,8 @@ retry: int multifd_ram_flush_and_sync(void) { + MultiFDSyncReq req; + if (!migrate_multifd()) { return 0; } @@ -356,7 +358,10 @@ int multifd_ram_flush_and_sync(void) } } - return multifd_send_sync_main(); + /* File migrations only need to sync with threads */ + req = migrate_mapped_ram() ? MULTIFD_SYNC_LOCAL : MULTIFD_SYNC_ALL; + + return multifd_send_sync_main(req); } bool multifd_send_prepare_common(MultiFDSendParams *p) diff --git a/migration/multifd.c b/migration/multifd.c index 498e71fd10..2248bd2d46 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -523,7 +523,7 @@ static int multifd_zero_copy_flush(QIOChannel *c) return ret; } -int multifd_send_sync_main(void) +int multifd_send_sync_main(MultiFDSyncReq req) { int i; bool flush_zero_copy; @@ -543,8 +543,8 @@ int multifd_send_sync_main(void) * We should be the only user so far, so not possible to be set by * others concurrently. */ - assert(qatomic_read(&p->pending_sync) == false); - qatomic_set(&p->pending_sync, true); + assert(qatomic_read(&p->pending_sync) == MULTIFD_SYNC_NONE); + qatomic_set(&p->pending_sync, req); qemu_sem_post(&p->sem); } for (i = 0; i < migrate_multifd_channels(); i++) { @@ -635,14 +635,17 @@ static void *multifd_send_thread(void *opaque) */ qatomic_store_release(&p->pending_job, false); } else { + MultiFDSyncReq req = qatomic_read(&p->pending_sync); + /* * If not a normal job, must be a sync request. Note that * pending_sync is a standalone flag (unlike pending_job), so * it doesn't require explicit memory barriers. */ - assert(qatomic_read(&p->pending_sync)); + assert(req != MULTIFD_SYNC_NONE); - if (use_packets) { + /* Only push the SYNC message if it involves a remote sync */ + if (req == MULTIFD_SYNC_ALL) { p->flags = MULTIFD_FLAG_SYNC; multifd_send_fill_packet(p); ret = qio_channel_write_all(p->c, (void *)p->packet, @@ -654,7 +657,7 @@ static void *multifd_send_thread(void *opaque) stat64_add(&mig_stats.multifd_bytes, p->packet_len); } - qatomic_set(&p->pending_sync, false); + qatomic_set(&p->pending_sync, MULTIFD_SYNC_NONE); qemu_sem_post(&p->sem_sync); } } From patchwork Fri Dec 6 00:58:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13896181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 301EEE7716E for ; Fri, 6 Dec 2024 01:00:19 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJMgE-0003a6-Nw; Thu, 05 Dec 2024 19:58:50 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJMgD-0003ZC-5h for qemu-devel@nongnu.org; Thu, 05 Dec 2024 19:58:49 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJMgB-0005ks-1K for qemu-devel@nongnu.org; Thu, 05 Dec 2024 19:58:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733446726; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eNEQZ+vPm4wSXR/zObKahxluS5nzwSvUVA2E+LmCCII=; b=ObLgKXlBCcauAmbFid098fLBTPLc98NoOFTnUVb7HgpmtJJM1dKeofkS26TbocgjtTX1pd tx/dZ3gedGWqlcVI0wRVHIdOWajL4GYUFnWZOt7/tPDW/n7VOynFICIc9AtY33qbnPdPbH U1flHyF5ffFkgtUBZbtpTrCgNH6Kspo= Received: from mail-vk1-f200.google.com (mail-vk1-f200.google.com [209.85.221.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-83-RkZwR9mPMTOSVnF-Mnb7Uw-1; Thu, 05 Dec 2024 19:58:45 -0500 X-MC-Unique: RkZwR9mPMTOSVnF-Mnb7Uw-1 X-Mimecast-MFC-AGG-ID: RkZwR9mPMTOSVnF-Mnb7Uw Received: by mail-vk1-f200.google.com with SMTP id 71dfb90a1353d-515cf20f2c7so376676e0c.0 for ; Thu, 05 Dec 2024 16:58:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733446724; x=1734051524; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eNEQZ+vPm4wSXR/zObKahxluS5nzwSvUVA2E+LmCCII=; b=kzSemO+NoDlPzSJlVHz8oQcvNTm6DdKYOhLByR+LlpDUEKQBDj8DmzEGDPZTxjiNNh 4AtsEn107FpF+mGY3qEf63MmdWlUbsWLxGyEZfAX7PfKH2b8y5XaU5aL0IQPmtbVo7Fb RfvRvnU2253dftvN3VTBVPQvVyIXyc2xaPy1Pg/aEYjY5n2HIWz+1lm05+lIk9d1FaRP lwNB18XoLa9OOA6Jy+qg4gJYlE9+qEiuhwoMygaFXffXO3SmbjSdsBdWJFjKW/p+/EeL itPz6zPDzDb0mqRwSyMoCNMdJX8LDl3Son3XkxkNyqGzW76fE7LPUJ+mZiaRj+o9MCl/ ro4g== X-Gm-Message-State: AOJu0Yzoud3N/ad/3Uq70Tan/V4R0FHm4UttfG7zAit9SCmlWrUCMH1Z W+xfvqjQvL4guJhzDiymt+cAcwrxFgiFkazH/w+uLqV7jUx3aO1qsfXdrDSGsYxEOZS2Bd025jD mKpLqB+RU0DWMAuuGEn0zNEi6VfPMktBO4ig91KoyvbQV2BV5ybBp4rSt9xlGKlLqPi0aGx0Pro g7ivVMvixOG5L93SOQH3Af02ZH7cT1tFw34g== X-Gm-Gg: ASbGncu6Ull6mN2opODZg7i4r3p5B0yauOlziSM1LXpnhqjDXlA38qt1oACINrBnZXU uqDC8uf30Am/y+32BB1qD6ruPqU9VhbQUbWIy0r16v59yPl4KoX4VksUXmeLXAVC8BNkm5+Zqo0 vbPrjyGbOJlbW3B20cGg6lPUAo7GcRwDrpDFHp1twJY9HQ5lmvO2V3yJVYoKouIOdGU1dFFl4OQ GpUheaUv/aoDl1G1m/Ulmt/wD4WFn2daeenY32alCRSi06rhKLuwRTVFTE2Rpo3qOTYNkMmSNCE 5ewep+9JdSwAkEtWQrB8/2B2UA== X-Received: by 2002:a05:6122:320b:b0:50d:83e1:94fb with SMTP id 71dfb90a1353d-515fcb17e77mr2258133e0c.12.1733446724507; Thu, 05 Dec 2024 16:58:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IHVeQ36fQkMVtmBstad6PzSsgH1UitLG9zUgtKUvTEJ/gJIZmBcg3Str9mpOaZcVRjl7pN6YA== X-Received: by 2002:a05:6122:320b:b0:50d:83e1:94fb with SMTP id 71dfb90a1353d-515fcb17e77mr2258098e0c.12.1733446724038; Thu, 05 Dec 2024 16:58:44 -0800 (PST) Received: from x1n.redhat.com (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6d8dac016cbsm12635226d6.117.2024.12.05.16.58.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Dec 2024 16:58:43 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Cc: "Maciej S . Szmigiero" , peterx@redhat.com, =?utf-8?q?C=C3=A9dric_Le_Goater?= , Avihai Horon , Alex Williamson , Fabiano Rosas , Prasad Pandit Subject: [PATCH v2 3/7] migration/ram: Move RAM_SAVE_FLAG* into ram.h Date: Thu, 5 Dec 2024 19:58:30 -0500 Message-ID: <20241206005834.1050905-4-peterx@redhat.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241206005834.1050905-1-peterx@redhat.com> References: <20241206005834.1050905-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.996, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Firstly, we're going to use the multifd flag soon in multifd code, so ram.c isn't gonna work. Secondly, we have a separate RDMA flag dangling around, which is definitely not obvious. There's one comment that helps, but not too much. We should just put it altogether, so nothing will get overlooked. Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/ram.h | 25 +++++++++++++++++++++++++ migration/rdma.h | 7 ------- migration/ram.c | 21 --------------------- 3 files changed, 25 insertions(+), 28 deletions(-) diff --git a/migration/ram.h b/migration/ram.h index 0d1981f888..cfdcccd266 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -33,6 +33,31 @@ #include "exec/cpu-common.h" #include "io/channel.h" +/* + * RAM_SAVE_FLAG_ZERO used to be named RAM_SAVE_FLAG_COMPRESS, it + * worked for pages that were filled with the same char. We switched + * it to only search for the zero value. And to avoid confusion with + * RAM_SAVE_FLAG_COMPRESS_PAGE just rename it. + * + * RAM_SAVE_FLAG_FULL was obsoleted in 2009. + * + * RAM_SAVE_FLAG_COMPRESS_PAGE (0x100) was removed in QEMU 9.1. + */ +#define RAM_SAVE_FLAG_FULL 0x01 +#define RAM_SAVE_FLAG_ZERO 0x02 +#define RAM_SAVE_FLAG_MEM_SIZE 0x04 +#define RAM_SAVE_FLAG_PAGE 0x08 +#define RAM_SAVE_FLAG_EOS 0x10 +#define RAM_SAVE_FLAG_CONTINUE 0x20 +#define RAM_SAVE_FLAG_XBZRLE 0x40 +/* + * ONLY USED IN RDMA: Whenever this is found in the data stream, the flags + * will be passed to rdma functions in the incoming-migration side. + */ +#define RAM_SAVE_FLAG_HOOK 0x80 +#define RAM_SAVE_FLAG_MULTIFD_FLUSH 0x200 +/* We can't use any flag that is bigger than 0x200 */ + extern XBZRLECacheStats xbzrle_counters; /* Should be holding either ram_list.mutex, or the RCU lock. */ diff --git a/migration/rdma.h b/migration/rdma.h index a8d27f33b8..f55f28bbed 100644 --- a/migration/rdma.h +++ b/migration/rdma.h @@ -33,13 +33,6 @@ void rdma_start_incoming_migration(InetSocketAddress *host_port, Error **errp); #define RAM_CONTROL_ROUND 1 #define RAM_CONTROL_FINISH 3 -/* - * Whenever this is found in the data stream, the flags - * will be passed to rdma functions in the incoming-migration - * side. - */ -#define RAM_SAVE_FLAG_HOOK 0x80 - #define RAM_SAVE_CONTROL_NOT_SUPP -1000 #define RAM_SAVE_CONTROL_DELAYED -2000 diff --git a/migration/ram.c b/migration/ram.c index 7284c34bd8..44010ff325 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -71,27 +71,6 @@ /***********************************************************/ /* ram save/restore */ -/* - * RAM_SAVE_FLAG_ZERO used to be named RAM_SAVE_FLAG_COMPRESS, it - * worked for pages that were filled with the same char. We switched - * it to only search for the zero value. And to avoid confusion with - * RAM_SAVE_FLAG_COMPRESS_PAGE just rename it. - * - * RAM_SAVE_FLAG_FULL was obsoleted in 2009. - * - * RAM_SAVE_FLAG_COMPRESS_PAGE (0x100) was removed in QEMU 9.1. - */ -#define RAM_SAVE_FLAG_FULL 0x01 -#define RAM_SAVE_FLAG_ZERO 0x02 -#define RAM_SAVE_FLAG_MEM_SIZE 0x04 -#define RAM_SAVE_FLAG_PAGE 0x08 -#define RAM_SAVE_FLAG_EOS 0x10 -#define RAM_SAVE_FLAG_CONTINUE 0x20 -#define RAM_SAVE_FLAG_XBZRLE 0x40 -/* 0x80 is reserved in rdma.h for RAM_SAVE_FLAG_HOOK */ -#define RAM_SAVE_FLAG_MULTIFD_FLUSH 0x200 -/* We can't use any flag that is bigger than 0x200 */ - /* * mapped-ram migration supports O_DIRECT, so we need to make sure the * userspace buffer, the IO operation size and the file offset are From patchwork Fri Dec 6 00:58:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13896177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 01A95E7716E for ; Fri, 6 Dec 2024 00:59:38 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJMgJ-0003bN-0f; Thu, 05 Dec 2024 19:58:55 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJMgG-0003ah-9O for qemu-devel@nongnu.org; Thu, 05 Dec 2024 19:58:52 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJMgE-0005lQ-Ow for qemu-devel@nongnu.org; Thu, 05 Dec 2024 19:58:52 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733446729; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Jq9ieJQ9//qq2auSKoavGM6aHp5t8bjDLhHqkdKpCKw=; b=CDjghp9fRAHrEKbPQdu37wARtijK88aRV1yFHpge5Aaf5XKhs1kGYijhrGMvy/aSEIXwJ7 JMV9snlAmnzbXHihvLoEwI74Nf7Jm18MGja606flne5Gl6mmvRrFokExK2dgA7clCfK9to +3p69dnmC64nzso7KdTQj6+0s4VM8+w= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-96-FLqV5ex0N4SCPKB4n1bYbA-1; Thu, 05 Dec 2024 19:58:47 -0500 X-MC-Unique: FLqV5ex0N4SCPKB4n1bYbA-1 X-Mimecast-MFC-AGG-ID: FLqV5ex0N4SCPKB4n1bYbA Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-6d8860ab00dso27077736d6.0 for ; Thu, 05 Dec 2024 16:58:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733446726; x=1734051526; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Jq9ieJQ9//qq2auSKoavGM6aHp5t8bjDLhHqkdKpCKw=; b=nnSqCxGXjmxtczW796EPrz6TPdZVV1OuQB9Z21tPjryj6XocYMU1vZAGKqG1vh7a3e hib+wnoHVbc7C1CCm3El5B8HB994smDSG9PVo4QpBylYpF1h1piQ9zH+pnpIAasmvtCt WDVCDb/rcsls+BcJeXvlDv2cpyo5SYmzK0GYaHGssMVT4MJFTtZ60pVWStsRL3T2uMPK 3N/tQDLJWMn26T5144xxCtfGbprn/YbynFmcvAR8AQ+MQuZPjAGBe25gqgzTd1Zb/WrM 2lFRkhD5P3ttMS6aEOgV7Blpz4m6tokiqqJq0zCuIF2joKzI5WYHnejS4WYkLxpxx0RR D1yg== X-Gm-Message-State: AOJu0YwsatRlIYfq+T9xUhFEWK9EqDqQuZkPSl/VlHO0AynB+w094nmi dENtjL3X7hXXJQPz/i38Yqp71od6xu1cWEFYLZ7u4nKexhArOXeQVH7lo9jv4FI/kttwkW4zBjG axA/CrLJcAoAzCVfJGJipnsfpDQpCJ/I+l2veDBOThTeaAqIZ0MqvhryAitcXGQKl2ro9mnc4Ov e1RmSR/jlWvj8OBMxhEkhRh0cNCx7SeF7QMA== X-Gm-Gg: ASbGncugPWdKy8yPulj/vxxC+AXzTyX21tojODe0xZqvMNHuJL28t5/Pg9qzf9rSsL6 eljDAEgZ6MmvD8AK0YRiDaWpwgFbGjOi0+V+YapIILtCJ/dEh3rmf1TYusZwVqhKxJjiY2yQPaY cXR9iy6qZzvdneSUEEusTz5zJZ9MtWb+Ng1niUpYmm/d85kK9AUbWrHZE6ioL5yziaVBJIsZk3S wtOxu/6VqQ9V1OMW4qikezMQrsB1UbYHw/70c5LRIuPeHcRGS4iOyimskL3FCMfzRIyN/aCr1ij TOopT45r+aN3tkjGFq3K6sRYzA== X-Received: by 2002:a05:6214:400b:b0:6d8:8b9d:1502 with SMTP id 6a1803df08f44-6d8e71ebf0dmr17251036d6.30.1733446726556; Thu, 05 Dec 2024 16:58:46 -0800 (PST) X-Google-Smtp-Source: AGHT+IHXPWxDLHx/ej3m1LYVm/1cVuhXlhPgz4QiSfuXcagOoJw1NDxUyU37WNo3iFCTdLuErwJyhQ== X-Received: by 2002:a05:6214:400b:b0:6d8:8b9d:1502 with SMTP id 6a1803df08f44-6d8e71ebf0dmr17250716d6.30.1733446726181; Thu, 05 Dec 2024 16:58:46 -0800 (PST) Received: from x1n.redhat.com (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6d8dac016cbsm12635226d6.117.2024.12.05.16.58.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Dec 2024 16:58:45 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Cc: "Maciej S . Szmigiero" , peterx@redhat.com, =?utf-8?q?C=C3=A9dric_Le_Goater?= , Avihai Horon , Alex Williamson , Fabiano Rosas , Prasad Pandit Subject: [PATCH v2 4/7] migration/multifd: Unify RAM_SAVE_FLAG_MULTIFD_FLUSH messages Date: Thu, 5 Dec 2024 19:58:31 -0500 Message-ID: <20241206005834.1050905-5-peterx@redhat.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241206005834.1050905-1-peterx@redhat.com> References: <20241206005834.1050905-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.996, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org RAM_SAVE_FLAG_MULTIFD_FLUSH message should always be correlated to a sync request on src. Unify such message into one place, and conditionally send the message only if necessary. Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.h | 2 +- migration/multifd-nocomp.c | 27 +++++++++++++++++++++++++-- migration/ram.c | 18 ++++-------------- 3 files changed, 30 insertions(+), 17 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index bd337631ec..c9ae57ea02 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -350,7 +350,7 @@ static inline uint32_t multifd_ram_page_count(void) void multifd_ram_save_setup(void); void multifd_ram_save_cleanup(void); -int multifd_ram_flush_and_sync(void); +int multifd_ram_flush_and_sync(QEMUFile *f); size_t multifd_ram_payload_size(void); void multifd_ram_fill_packet(MultiFDSendParams *p); int multifd_ram_unfill_packet(MultiFDRecvParams *p, Error **errp); diff --git a/migration/multifd-nocomp.c b/migration/multifd-nocomp.c index 219f9e58ef..58372db0f4 100644 --- a/migration/multifd-nocomp.c +++ b/migration/multifd-nocomp.c @@ -20,6 +20,7 @@ #include "qemu/cutils.h" #include "qemu/error-report.h" #include "trace.h" +#include "qemu-file.h" static MultiFDSendData *multifd_ram_send; @@ -343,9 +344,10 @@ retry: return true; } -int multifd_ram_flush_and_sync(void) +int multifd_ram_flush_and_sync(QEMUFile *f) { MultiFDSyncReq req; + int ret; if (!migrate_multifd()) { return 0; @@ -361,7 +363,28 @@ int multifd_ram_flush_and_sync(void) /* File migrations only need to sync with threads */ req = migrate_mapped_ram() ? MULTIFD_SYNC_LOCAL : MULTIFD_SYNC_ALL; - return multifd_send_sync_main(req); + ret = multifd_send_sync_main(req); + if (ret) { + return ret; + } + + /* If we don't need to sync with remote at all, nothing else to do */ + if (req == MULTIFD_SYNC_LOCAL) { + return 0; + } + + /* + * Old QEMUs don't understand RAM_SAVE_FLAG_MULTIFD_FLUSH, it relies + * on RAM_SAVE_FLAG_EOS instead. + */ + if (migrate_multifd_flush_after_each_section()) { + return 0; + } + + qemu_put_be64(f, RAM_SAVE_FLAG_MULTIFD_FLUSH); + qemu_fflush(f); + + return 0; } bool multifd_send_prepare_common(MultiFDSendParams *p) diff --git a/migration/ram.c b/migration/ram.c index 44010ff325..90811aabd4 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1306,15 +1306,10 @@ static int find_dirty_block(RAMState *rs, PageSearchStatus *pss) (!migrate_multifd_flush_after_each_section() || migrate_mapped_ram())) { QEMUFile *f = rs->pss[RAM_CHANNEL_PRECOPY].pss_channel; - int ret = multifd_ram_flush_and_sync(); + int ret = multifd_ram_flush_and_sync(f); if (ret < 0) { return ret; } - - if (!migrate_mapped_ram()) { - qemu_put_be64(f, RAM_SAVE_FLAG_MULTIFD_FLUSH); - qemu_fflush(f); - } } /* Hit the end of the list */ @@ -3044,18 +3039,13 @@ static int ram_save_setup(QEMUFile *f, void *opaque, Error **errp) } bql_unlock(); - ret = multifd_ram_flush_and_sync(); + ret = multifd_ram_flush_and_sync(f); bql_lock(); if (ret < 0) { error_setg(errp, "%s: multifd synchronization failed", __func__); return ret; } - if (migrate_multifd() && !migrate_multifd_flush_after_each_section() - && !migrate_mapped_ram()) { - qemu_put_be64(f, RAM_SAVE_FLAG_MULTIFD_FLUSH); - } - qemu_put_be64(f, RAM_SAVE_FLAG_EOS); ret = qemu_fflush(f); if (ret < 0) { @@ -3190,7 +3180,7 @@ out: if (ret >= 0 && migration_is_running()) { if (migrate_multifd() && migrate_multifd_flush_after_each_section() && !migrate_mapped_ram()) { - ret = multifd_ram_flush_and_sync(); + ret = multifd_ram_flush_and_sync(f); if (ret < 0) { return ret; } @@ -3268,7 +3258,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque) * Only the old dest QEMU will need this sync, because each EOS * will require one SYNC message on each channel. */ - ret = multifd_ram_flush_and_sync(); + ret = multifd_ram_flush_and_sync(f); if (ret < 0) { return ret; } From patchwork Fri Dec 6 00:58:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13896180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C4ABEE77170 for ; Fri, 6 Dec 2024 01:00:12 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJMgI-0003bF-Cd; Thu, 05 Dec 2024 19:58:54 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJMgH-0003b5-JQ for qemu-devel@nongnu.org; Thu, 05 Dec 2024 19:58:53 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJMgG-0005lm-8l for qemu-devel@nongnu.org; Thu, 05 Dec 2024 19:58:53 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733446731; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/u1/cF9PD85EfXyRSPB3h8kEgVgxDKJ3kxfHiutEnP8=; b=MdlNGzvjZJ7HbHE6DJihTVBDLyMqWp86iyeHxTt4u6rrAiY/iXtDjkzI8TFNU7R5B+5/uX vkFdb7z9q3gRxGuJXwC3w73ax1+m959i3bGV91C90RK4mpcWJsq1UM+yh2RrX5/LRKMEf0 83CcB2QgiJlM0L6sKZjchH1I+TZn540= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-500-XaftsRzGMCqkM14pD0zE5g-1; Thu, 05 Dec 2024 19:58:49 -0500 X-MC-Unique: XaftsRzGMCqkM14pD0zE5g-1 X-Mimecast-MFC-AGG-ID: XaftsRzGMCqkM14pD0zE5g Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-6d8eb5ea994so573116d6.1 for ; Thu, 05 Dec 2024 16:58:49 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733446728; x=1734051528; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/u1/cF9PD85EfXyRSPB3h8kEgVgxDKJ3kxfHiutEnP8=; b=PqQ02f9uYSalALG7Rrla1BOTo+2l+mbgcpRQhlM09m8YUIkVlP6mKFKNyxshJl4W3F Q8SLut3/MJQUHJoRzdX7Wn+h20Eo2IN8iKSwamI2cDj1hXqLbtRaP0PBZaysyAhSHEf3 DStWNymYvNzpRahUQQyQpQxToT67IGutKlbQvOJEGO0MsVZSsL2be7OrI2I5WzHxlF4x jmHGaKIxXGptNncOf27zdRmmxVDHGtMurBlkpGaoyF09uTOjG0MbsuDFI1F9FNaTLkI7 TWz/JmLCazkD8I8Tvxho8zR3Pk30StT3FqFGph7zJ25qCC7T6ndcg5D6uNgRZEoLPcyC q2mA== X-Gm-Message-State: AOJu0YyoqZZvHFuuaZDSMtFAs3DIRcpFKTcYHaCBh6Dnw+hTCQXAzeqa qqfUrWw5+t4gXfAi7VoCIrllMwofTl6cRtATTl3udN6z2+SGx7acyRgm2GgxKSiQjkgHN8nJGRy GA/nGpo3Rt1uk20b9c6qQSfNGbFACIj7xLgeMadGCVQoSfiAyOzAf3TX0Rq4HXVOt4Ilm734h0O pvaB8PUw3WcVqDj8Ux+fMLXd3U5M64mDvjbA== X-Gm-Gg: ASbGncuxrXpNddNrDzuuX6SaD+IDNi3u8gkWlrXSgTn4T1yQmnNwQAmAYhL2d2ZSoEe 33V4Av1cmnG6kNKCu8tgRTp8zyaECoRx5DnE/weHIPlycXdu7aWTJM4scOuJBbe1Tr3NgbLtDfv L3zhqZICDbJqbopEEUzF3G1kCQs/ObjT4x8f2k80cr3u6YJKDWL8lw4HM/EzmQGJwvlabQUASZp IV0fe0aFMFbXw2WW8e7khhlzp5J4Gc4c9TUSRtJZN/uWlQ7ex7sZ1rF1UqtcxrjQVq906y3tgm3 BiaNecbGj6W62pWfCyWYpMTp/Q== X-Received: by 2002:a05:6214:2684:b0:6d8:9d81:2107 with SMTP id 6a1803df08f44-6d8e7132dcfmr14903966d6.20.1733446728565; Thu, 05 Dec 2024 16:58:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IEldja1CzrluAZz2bZtPWJ4lXVZKhQwebwnK0Z8A1dFoCgwYTCE/VVZSxC0eIrsP49jtVxyNQ== X-Received: by 2002:a05:6214:2684:b0:6d8:9d81:2107 with SMTP id 6a1803df08f44-6d8e7132dcfmr14903706d6.20.1733446728142; Thu, 05 Dec 2024 16:58:48 -0800 (PST) Received: from x1n.redhat.com (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6d8dac016cbsm12635226d6.117.2024.12.05.16.58.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Dec 2024 16:58:47 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Cc: "Maciej S . Szmigiero" , peterx@redhat.com, =?utf-8?q?C=C3=A9dric_Le_Goater?= , Avihai Horon , Alex Williamson , Fabiano Rosas , Prasad Pandit Subject: [PATCH v2 5/7] migration/multifd: Remove sync processing on postcopy Date: Thu, 5 Dec 2024 19:58:32 -0500 Message-ID: <20241206005834.1050905-6-peterx@redhat.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241206005834.1050905-1-peterx@redhat.com> References: <20241206005834.1050905-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.996, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Multifd never worked with postcopy, at least yet so far. Remove the sync processing there, because it's confusing, and they should never appear. Now if RAM_SAVE_FLAG_MULTIFD_FLUSH is observed, we fail hard instead of trying to invoke multifd code. Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/ram.c | 8 -------- 1 file changed, 8 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 90811aabd4..154ff5abd4 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3772,15 +3772,7 @@ int ram_load_postcopy(QEMUFile *f, int channel) TARGET_PAGE_SIZE); } break; - case RAM_SAVE_FLAG_MULTIFD_FLUSH: - multifd_recv_sync_main(); - break; case RAM_SAVE_FLAG_EOS: - /* normal exit */ - if (migrate_multifd() && - migrate_multifd_flush_after_each_section()) { - multifd_recv_sync_main(); - } break; default: error_report("Unknown combination of migration flags: 0x%x" From patchwork Fri Dec 6 00:58:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13896176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 057E5E77171 for ; Fri, 6 Dec 2024 00:59:38 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJMgK-0003cD-Pj; Thu, 05 Dec 2024 19:58:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJMgI-0003bG-M1 for qemu-devel@nongnu.org; Thu, 05 Dec 2024 19:58:54 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJMgG-0005lw-TJ for qemu-devel@nongnu.org; Thu, 05 Dec 2024 19:58:54 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733446732; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=drjbKA8KuZzlVuJlQXBYiI10JQger84ZUXuJUp7t2qg=; b=Wqei99SUNCGVFa9KdDqNsWCPlLvOa+ZPyuBhzLQ8ZSQnsY5fwbtzi8GFrUzeFikwjSOSut Grgpg0/ZeaqgIdKR+sHCGGm96gpX6vDbO3kbzxxd8vrYcCCew0a/VNdlmofv4n+vMpdS8o QhgdxSdiFhwN2e9AWyfijXJ507dVGkA= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-180-izdX7eyUPyeu5W1XxmY0_Q-1; Thu, 05 Dec 2024 19:58:51 -0500 X-MC-Unique: izdX7eyUPyeu5W1XxmY0_Q-1 X-Mimecast-MFC-AGG-ID: izdX7eyUPyeu5W1XxmY0_Q Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-6d87c55ca85so40361286d6.1 for ; Thu, 05 Dec 2024 16:58:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733446730; x=1734051530; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=drjbKA8KuZzlVuJlQXBYiI10JQger84ZUXuJUp7t2qg=; b=bnp2k23FslkSetNSBcwllOb6ViCALGtcb/l/1DfbqrKeXsioU/s7rU74YysXuhSJse fiGPjd3tUWt4/XMnSG1Dc5cZ7qcS5oirGRfMHGAh10yLwu4rkgEWVK8kBRwx7pybCe4y u+C9BUTXWDrK2oFmkT9fhmxfNnvt3kjw2tz3mrGTzk/ceZwbfiTuChRoZP1DhGeZQzet MQq1oZUU8XMx30RjmJeKEKM8yr8w1tZBM2jV4IXTjKIU/GKZgpxZGypDtAlY9aZHj3lA AhMapzA4HZ4LlPPcY/+qe2EyF0LOYfaLXSGu19FOiIJdVZ3oPKfjCMeZITt2hdP2TX56 buMw== X-Gm-Message-State: AOJu0Yw2JAe5ncSpkVqWkDsSNagoqBbbLMk493UmCUNHutrcnVO0T366 9/6znFmyPdmyUiddoCPFSA8VOeCZNwWTxz7oy9xN5iv0413NHwWOZQrTIfrJxzuMOvh4ulEhABV QEK7IfDFdpr2Ynj95XVyFQ53nkmuJXpYr7EMbHHDKGc+Hp225ZegCoY4d6rQAoBCqjDxZO5iDZH WHex45AM79HirE3Nme+zQX5RM6h/jnsWHs9w== X-Gm-Gg: ASbGncv14r6sBAVIVus9SChLo0ssP9rMR6IcnIBzmPktOy4TQvgACsRpnBlg1USzGGE 4GiOtPtx9+4cYdrH4zVWE5MqIJvy6iYi3YgyBUY/BHs1BDRa9GpbT8q4LE9XMvOwVsiUkpuJl1/ D/tuNYPVhlwEtRlzFVv8A2+8m1otniE6JGAGbQFpE+ihNImdTHnj1HzH9O2HRa7GqXXPt+CdDQf 2QVkZKOetI6v4UE6NpeR2rDD+Y2cupJjrIQ05xIvRFUmACiGJlbAj86sxK/naESBZku7GGOTxqL fErH+m3ebn3kfdKuKFxNFwJQNw== X-Received: by 2002:ad4:5b84:0:b0:6d4:1f6d:695f with SMTP id 6a1803df08f44-6d8e70a4561mr17971796d6.2.1733446730702; Thu, 05 Dec 2024 16:58:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IHEncKgYAyVwJxQJ2sxQIizv5+JUBBLheVgwUBG7yK1do59UnqfWleTbHlzw2eCjiKn29CnQQ== X-Received: by 2002:ad4:5b84:0:b0:6d4:1f6d:695f with SMTP id 6a1803df08f44-6d8e70a4561mr17971506d6.2.1733446730295; Thu, 05 Dec 2024 16:58:50 -0800 (PST) Received: from x1n.redhat.com (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6d8dac016cbsm12635226d6.117.2024.12.05.16.58.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Dec 2024 16:58:49 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Cc: "Maciej S . Szmigiero" , peterx@redhat.com, =?utf-8?q?C=C3=A9dric_Le_Goater?= , Avihai Horon , Alex Williamson , Fabiano Rosas , Prasad Pandit Subject: [PATCH v2 6/7] migration/multifd: Cleanup src flushes on condition check Date: Thu, 5 Dec 2024 19:58:33 -0500 Message-ID: <20241206005834.1050905-7-peterx@redhat.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241206005834.1050905-1-peterx@redhat.com> References: <20241206005834.1050905-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.996, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The src flush condition check is over complicated, and it's getting more out of control if postcopy will be involved. In general, we have two modes to do the sync: legacy or modern ways. Legacy uses per-section flush, modern uses per-round flush. Mapped-ram always uses the modern, which is per-round. Introduce two helpers, which can greatly simplify the code, and hopefully make it readable again. Signed-off-by: Peter Xu --- migration/multifd.h | 2 ++ migration/multifd-nocomp.c | 42 ++++++++++++++++++++++++++++++++++++++ migration/ram.c | 10 +++------ 3 files changed, 47 insertions(+), 7 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index c9ae57ea02..582040922f 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -351,6 +351,8 @@ static inline uint32_t multifd_ram_page_count(void) void multifd_ram_save_setup(void); void multifd_ram_save_cleanup(void); int multifd_ram_flush_and_sync(QEMUFile *f); +bool multifd_ram_sync_per_round(void); +bool multifd_ram_sync_per_section(void); size_t multifd_ram_payload_size(void); void multifd_ram_fill_packet(MultiFDSendParams *p); int multifd_ram_unfill_packet(MultiFDRecvParams *p, Error **errp); diff --git a/migration/multifd-nocomp.c b/migration/multifd-nocomp.c index 58372db0f4..c1f686c0ce 100644 --- a/migration/multifd-nocomp.c +++ b/migration/multifd-nocomp.c @@ -344,6 +344,48 @@ retry: return true; } +/* + * We have two modes for multifd flushes: + * + * - Per-section mode: this is the legacy way to flush, it requires one + * MULTIFD_FLAG_SYNC message for each RAM_SAVE_FLAG_EOS. + * + * - Per-round mode: this is the modern way to flush, it requires one + * MULTIFD_FLAG_SYNC message only for each round of RAM scan. Normally + * it's paired with a new RAM_SAVE_FLAG_MULTIFD_FLUSH message in network + * based migrations. + * + * One thing to mention is mapped-ram always use the modern way to sync. + */ + +/* Do we need a per-section multifd flush (legacy way)? */ +bool multifd_ram_sync_per_section(void) +{ + if (!migrate_multifd()) { + return false; + } + + if (migrate_mapped_ram()) { + return false; + } + + return migrate_multifd_flush_after_each_section(); +} + +/* Do we need a per-round multifd flush (modern way)? */ +bool multifd_ram_sync_per_round(void) +{ + if (!migrate_multifd()) { + return false; + } + + if (migrate_mapped_ram()) { + return true; + } + + return !migrate_multifd_flush_after_each_section(); +} + int multifd_ram_flush_and_sync(QEMUFile *f) { MultiFDSyncReq req; diff --git a/migration/ram.c b/migration/ram.c index 154ff5abd4..5d4bdefe69 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1302,9 +1302,7 @@ static int find_dirty_block(RAMState *rs, PageSearchStatus *pss) pss->page = 0; pss->block = QLIST_NEXT_RCU(pss->block, next); if (!pss->block) { - if (migrate_multifd() && - (!migrate_multifd_flush_after_each_section() || - migrate_mapped_ram())) { + if (multifd_ram_sync_per_round()) { QEMUFile *f = rs->pss[RAM_CHANNEL_PRECOPY].pss_channel; int ret = multifd_ram_flush_and_sync(f); if (ret < 0) { @@ -3178,8 +3176,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) out: if (ret >= 0 && migration_is_running()) { - if (migrate_multifd() && migrate_multifd_flush_after_each_section() && - !migrate_mapped_ram()) { + if (multifd_ram_sync_per_section()) { ret = multifd_ram_flush_and_sync(f); if (ret < 0) { return ret; @@ -3252,8 +3249,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque) } } - if (migrate_multifd() && - migrate_multifd_flush_after_each_section()) { + if (multifd_ram_sync_per_section()) { /* * Only the old dest QEMU will need this sync, because each EOS * will require one SYNC message on each channel. From patchwork Fri Dec 6 00:58:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13896179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C4AE8E77171 for ; Fri, 6 Dec 2024 01:00:12 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJMgN-0003cR-RS; Thu, 05 Dec 2024 19:58:59 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJMgM-0003cG-0c for qemu-devel@nongnu.org; Thu, 05 Dec 2024 19:58:58 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJMgK-0005md-JJ for qemu-devel@nongnu.org; Thu, 05 Dec 2024 19:58:57 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733446734; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rlpFg1t4p+x/wsjA8Y1ehZs0v0oTzhln8SN7DYaHVqQ=; b=bX0J4/XWx8zaeASCxCzz0aAkb7K4G0WRkzzRVVZQ8Hi52RJ2dbw28uQFFOZkVtdZ1oehZe biTEGe3+hiDkpfvsdbzifGxtBdiGXQySXtE6aLS1yUavOcHiPB6vQKlPUzN2ZPit1SnfAN rBmVhmxPTq3viMVBk+0hnuKjpN2MH7g= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-318-frkpDalANPSdz5MD5Y2AkA-1; Thu, 05 Dec 2024 19:58:53 -0500 X-MC-Unique: frkpDalANPSdz5MD5Y2AkA-1 X-Mimecast-MFC-AGG-ID: frkpDalANPSdz5MD5Y2AkA Received: by mail-qv1-f69.google.com with SMTP id 6a1803df08f44-6d8860ab00dso27078726d6.0 for ; Thu, 05 Dec 2024 16:58:53 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733446732; x=1734051532; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rlpFg1t4p+x/wsjA8Y1ehZs0v0oTzhln8SN7DYaHVqQ=; b=gSKkHhJi7cDbbhnRmWHMflARIADtBPBGo+0iXsmTFtXI43AvXmMRbGJCeW2c5p+Ukg 7S4R5XnHXEBUqjMgL8GT50JDPc7pXi0HkNz4syHSGgabPVLClk6XN3teXDgUVJYa/uwj 48vHlGUurXJXDuFmXlkiPv2HO0mboqVcrhHvbgDFx0C7NhWFYNTl1Q+LObwGfCZb8X1N Xc6mRN979Jt+pnfMJB/baqhnBglP2QEsYuNhsXiRb6z8K5xwH5Hi6YnhjhgmXt/Gjrc0 R+aDpIVbDW72bQ10zRoqnjynuqOwkhAp/9LVCh/6ntXwhKofrl9NTrpHNl4Dn/UeMO/t ySFA== X-Gm-Message-State: AOJu0YxgQBNgO0GXA2l5K0W6pifDhoPukAEurlYeeehFG3dy6YBINZAq X0W8A0eAx0FabrXk1QIWZ9+oVubaahwpSs6trvkjaHZ7YY7e0TuWfuDr5z3IDQxvcPJjTRhigfh qJnKYqmVxR9Rs2wWm0gqgTbW5UC3f/LO46JUNZu4anwaV82KDqFJxPqK1myzQvmiIzmZ9nJOTqT C8TeTqquSgaWXXcBlIpwubcZlBlMnCXMowqA== X-Gm-Gg: ASbGncsghqfdP4Y6mmj5+vt0O436ZW400vd3fvRfziBlZNNE2bdtnFea7mXaSlUzitw U936/bJQ14HFw7WshP3EDxJV5yDMOK4eQd49K3x/i9qMqQgO8r5172UGqDoeitg3MDW+OQ4Q7pP AU6NyuFTYTVAVMTR4+Hu9w5yM7S6ZXz1g+Vq2fvM6cpIN/PVC3illcPpQEZEGDXg5+uODJkQZ1Q usBdp1TAp1g6IUxU1mSOI9mAlg3UipDfs9brEWj56h1tvLQGBKlfpfsPI7JnA+WXCE4TkeyoN2h Rc6C8Dsq/J0TpoxhmtJDxXRyYg== X-Received: by 2002:ad4:5745:0:b0:6d8:8a8f:75b0 with SMTP id 6a1803df08f44-6d8e7128d28mr20532646d6.14.1733446732512; Thu, 05 Dec 2024 16:58:52 -0800 (PST) X-Google-Smtp-Source: AGHT+IEzrpn8ChXzP6fq6lz8dwmTdEkw/KsgeF2Fo83weEfj60YxFPRjaAk05xlnPU2UU1EVdJnDgg== X-Received: by 2002:ad4:5745:0:b0:6d8:8a8f:75b0 with SMTP id 6a1803df08f44-6d8e7128d28mr20532266d6.14.1733446732154; Thu, 05 Dec 2024 16:58:52 -0800 (PST) Received: from x1n.redhat.com (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6d8dac016cbsm12635226d6.117.2024.12.05.16.58.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Dec 2024 16:58:51 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Cc: "Maciej S . Szmigiero" , peterx@redhat.com, =?utf-8?q?C=C3=A9dric_Le_Goater?= , Avihai Horon , Alex Williamson , Fabiano Rosas , Prasad Pandit Subject: [PATCH v2 7/7] migration/multifd: Document the reason to sync for save_setup() Date: Thu, 5 Dec 2024 19:58:34 -0500 Message-ID: <20241206005834.1050905-8-peterx@redhat.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241206005834.1050905-1-peterx@redhat.com> References: <20241206005834.1050905-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.996, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org It's not straightforward to see why src QEMU needs to sync multifd during setup() phase. After all, there's no page queued at that point. For old QEMUs, there's a solid reason: EOS requires it to work. While it's clueless on the new QEMUs which do not take EOS message as sync requests. One will figure that out only when this is conditionally removed. In fact, the author did try it out. Logically we could still avoid doing this on new machine types, however that needs a separate compat field and that can be an overkill in some trivial overhead in setup() phase. Let's instead document it completely, to avoid someone else tries this again and do the debug one more time, or anyone confused on why this ever existed. Signed-off-by: Peter Xu --- migration/ram.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 5d4bdefe69..ddee703585 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3036,6 +3036,33 @@ static int ram_save_setup(QEMUFile *f, void *opaque, Error **errp) migration_ops->ram_save_target_page = ram_save_target_page_legacy; } + /* + * This operation is unfortunate.. + * + * For legacy QEMUs using per-section sync + * ======================================= + * + * This must exist because the EOS below requires the SYNC messages + * per-channel to work. + * + * For modern QEMUs using per-round sync + * ===================================== + * + * Logically this sync is not needed (because we know there's nothing + * in the multifd queue yet!). However as a side effect, this makes + * sure the dest side won't receive any data before it properly reaches + * ram_load_precopy(). + * + * Without this sync, src QEMU can send data too soon so that dest may + * not have been ready to receive it (e.g., rb->receivedmap may be + * uninitialized, for example). + * + * Logically "wait for recv setup ready" shouldn't need to involve src + * QEMU at all, however to be compatible with old QEMUs, let's stick + * with this. Fortunately the overhead is low to sync during setup + * because the VM is running, so at least it's not accounted as part of + * downtime. + */ bql_unlock(); ret = multifd_ram_flush_and_sync(f); bql_lock();