From patchwork Fri Dec 6 22:47:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13897910 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 730ADE77173 for ; Fri, 6 Dec 2024 22:49:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJh7N-0004Ls-5D; Fri, 06 Dec 2024 17:48:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJh7H-0004Km-OY for qemu-devel@nongnu.org; Fri, 06 Dec 2024 17:48:08 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJh7F-0002PI-Uc for qemu-devel@nongnu.org; Fri, 06 Dec 2024 17:48:07 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733525284; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=st2vOIgs8A0o+X0EEzA+Hhgn5ZdqD6a20MPWzcMAv0s=; b=gVW/iST5Al2MDjZ3lqh390j06ER32yBZ9WHIIuK4ZMGDjsqJeT5AYAyNBGkGvDRaTLDz+W 85HikFVp4knFn0sx/PSujwGwtZjmF9l31tz86v1SgKGa9UjIywXqkDwMLT0Kz6GeQCJlD2 +r5ywrMhZeyVjqfr2ah6EUp8YXwv4cg= Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-49-9ZyUuV1jMtysXOIlThy7cA-1; Fri, 06 Dec 2024 17:48:01 -0500 X-MC-Unique: 9ZyUuV1jMtysXOIlThy7cA-1 X-Mimecast-MFC-AGG-ID: 9ZyUuV1jMtysXOIlThy7cA Received: by mail-io1-f69.google.com with SMTP id ca18e2360f4ac-841aa90930cso237554739f.1 for ; Fri, 06 Dec 2024 14:48:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525280; x=1734130080; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=st2vOIgs8A0o+X0EEzA+Hhgn5ZdqD6a20MPWzcMAv0s=; b=TJCC0PKlvCJN/AMNrVBRUtnDxcRdsD2UNE17qzQQrUCrxRY16E2AVAcRkv+UtO2zYW EB821xcN3rTL2+jKU1nWtfKWiTmCWt14HXHRZNE/mjUnk+X11QJ/1+7xKSgu0YOpWabI fgN9gVFQt1BdHMBrFLGdwUbi7Jpwj9DBsso7/z/50Gh3UoihWGoZ3JEOBGte3k+1hjtG WdHR/Pk0mExVlGSQKYXW0Qrcv6ifSn629eO8ljo2KoXQZWPdpsFlAQ/uLwEFNXN9y/Fz KjH5Ly3DEsKvTTRURZVgrAOwY7tDJPq0Bt7VW+gVRAXN/0x4Xvcxw1mx66GT6k0DeIGx iOlQ== X-Gm-Message-State: AOJu0YwxTYUgs1lhZb68LK5qVCrpZR3me3AFIggEBAa5Sq1UItCsD++N OynCOYmLxRCWDZLAWI8f4yHY6oM9Vy/xXzVY2cc8yZy22S8ho6qqMAK8tw7GnxMioPIS6zP0P9i n/XOyNWNB0W+sAODvwrA2cOsegGoouZ/bxscyKq52xEOryxpoxUnFvxsdSLthj7hx0Y7A8Q3nmg F5ymdMfUpCJjvZyiF/N0BoeGB8yzoGZ0+ghw== X-Gm-Gg: ASbGncu4F2P0xgbF1yPgEocLLSCBAjfswfY+U2L+vn3MYytshoHTj1Fxl0XBwNM4xXj ShgL5PsgUocP/3Sf2MnSn3n0aq9afqdboGE0dJRA3ok2WSgXQIJt0JDtQc6RgBCDRuvr/JYB9WA 1Ta9G6ltXhqOtkrEU9bHoMPSu6X4xN46K6ioIOHKu8lXz9CKOu3ZsX9MOOTPRV4ziU7WBGRL1I6 RoQVEuC7vGJ/RW2/uqGeNuIWj3r8izsVWYdnH5A5yQCdweFyoRMkrbSg9X8OppcIsYt8Xt3OzfB 7MdstYt6dL78eZTd85kYjJrldQ== X-Received: by 2002:a05:6602:6b8c:b0:835:4931:b110 with SMTP id ca18e2360f4ac-8447e21faa8mr730657839f.5.1733525280392; Fri, 06 Dec 2024 14:48:00 -0800 (PST) X-Google-Smtp-Source: AGHT+IFt71zdrodR2Kwojdh8cwWyWrQ7qPOAo2hkK6TCT0xhiBm2HPMgH/ncrLLvL+Sma5LIEVZ9xQ== X-Received: by 2002:a05:6602:6b8c:b0:835:4931:b110 with SMTP id ca18e2360f4ac-8447e21faa8mr730654839f.5.1733525279941; Fri, 06 Dec 2024 14:47:59 -0800 (PST) Received: from x1n.redhat.com (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-84473a5bd28sm125601439f.38.2024.12.06.14.47.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Dec 2024 14:47:59 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Cc: Prasad Pandit , "Maciej S . Szmigiero" , =?utf-8?q?C=C3=A9dric_?= =?utf-8?q?Le_Goater?= , Alex Williamson , peterx@redhat.com, Avihai Horon , Fabiano Rosas Subject: [PATCH v3 1/7] migration/multifd: Further remove the SYNC on complete Date: Fri, 6 Dec 2024 17:47:49 -0500 Message-ID: <20241206224755.1108686-2-peterx@redhat.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241206224755.1108686-1-peterx@redhat.com> References: <20241206224755.1108686-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.996, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Commit 637280aeb2 ("migration/multifd: Avoid the final FLUSH in complete()") stopped sending the RAM_SAVE_FLAG_MULTIFD_FLUSH flag at ram_save_complete(), because the sync on the destination side is not needed due to the last iteration of find_dirty_block() having already done it. However, that commit overlooked that multifd_ram_flush_and_sync() on the source side is also not needed at ram_save_complete(), for the same reason. Moreover, removing the RAM_SAVE_FLAG_MULTIFD_FLUSH but keeping the multifd_ram_flush_and_sync() means that currently the recv threads will hang when receiving the MULTIFD_FLAG_SYNC message, waiting for the destination sync which only happens when RAM_SAVE_FLAG_MULTIFD_FLUSH is received. Luckily, multifd is still all working fine because recv side cleanup code (mostly multifd_recv_sync_main()) is smart enough to make sure even if recv threads are stuck at SYNC it'll get kicked out. And since this is the completion phase of migration, nothing else will be sent after the SYNCs. This needs to be fixed because in the future VFIO will have data to push after ram_save_complete() and we don't want the recv thread to be stuck in the MULTIFD_FLAG_SYNC message. Remove the unnecessary (and buggy) invocation of multifd_ram_flush_and_sync(). For very old binaries (multifd_flush_after_each_section==true), the flush_and_sync is still needed because each EOS received on destination will enforce all-channel sync once. Stable branches do not need this patch, as no real bug I can think of that will go wrong there.. so not attaching Fixes to be clear on the backport not needed. Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/ram.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 05ff9eb328..7284c34bd8 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3283,9 +3283,16 @@ static int ram_save_complete(QEMUFile *f, void *opaque) } } - ret = multifd_ram_flush_and_sync(); - if (ret < 0) { - return ret; + if (migrate_multifd() && + migrate_multifd_flush_after_each_section()) { + /* + * Only the old dest QEMU will need this sync, because each EOS + * will require one SYNC message on each channel. + */ + ret = multifd_ram_flush_and_sync(); + if (ret < 0) { + return ret; + } } if (migrate_mapped_ram()) { From patchwork Fri Dec 6 22:47:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13897907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7FB94E77179 for ; Fri, 6 Dec 2024 22:49:11 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJh7P-0004Mt-Kp; Fri, 06 Dec 2024 17:48:15 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJh7L-0004LN-JB for qemu-devel@nongnu.org; Fri, 06 Dec 2024 17:48:11 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJh7J-0002Py-Ie for qemu-devel@nongnu.org; Fri, 06 Dec 2024 17:48:11 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733525288; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NiMz2upzz1D7+xWewetQWKt2ho3rwd/SN8vyiWBwuro=; b=QpEy3mco1k06UxeHOV824/l9G8MsrdxCznacWvZ1AgQ9Kshnvse00j1g6R9lpcwHPcKOuB w42phwlwaPHMBvPUK4WGlwiKLjHhIcO51yUxvq0+RtPqKg7uXLgW5RcoqRJVx/BTCowjs2 XWcICj00y+DxtYPwu9drLqvdhk9H6fY= Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-56-CY4YyscZOUS2lNNwcuQ38Q-1; Fri, 06 Dec 2024 17:48:05 -0500 X-MC-Unique: CY4YyscZOUS2lNNwcuQ38Q-1 X-Mimecast-MFC-AGG-ID: CY4YyscZOUS2lNNwcuQ38Q Received: by mail-io1-f72.google.com with SMTP id ca18e2360f4ac-841a9ae0f26so439060139f.3 for ; Fri, 06 Dec 2024 14:48:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525282; x=1734130082; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NiMz2upzz1D7+xWewetQWKt2ho3rwd/SN8vyiWBwuro=; b=rrQD/sEMJgq/UnMr/Wwrv26Hi4YeWAU8+Kc1MA2rSpnsu4cBiqrzkP6D/my3O+TBFJ NT++0PdPeYacrKWEEU3zyDGR15EQMdZyLvwX8e3+GpLNd0HlZLGHLPJ3V23Y7fbzNoL8 u4UAGgNO4adNkwG7kQ/hzGdVd7R9ZuATxG0c5EKdB9FlqmV5+PT/UfM2yD1qvDqvaqXE fLb3W/eyPm6VQOlKLSDtUhdtRpP9YdRBncooyZWgb/3E7dssE3ZHlY0N0Uo9Y4zQz/BX 4AriAyOIW6RWKsEQB/Y1bBxHeafGBNp3J5k/A0a17l0liAGVK39a1PfFXIrm+ud8xsLK 6Mgw== X-Gm-Message-State: AOJu0Yw2O0EiHZYNZyQ0q9ututoue4XHmrfvZiqG6eWnTiw980TWXH/0 BI9sXGxguy+90kOTQPAMkiBbaEoxPf7jmxwLjAfjseoqCdU379pKzcQw1T667Xcehl8m/6D/BnT +WBFrfiaqXLXRmVJUbUTf3rpBF/X4FFFpgmnq+9WqL4MTDO6xqhaMtC7h5aCzOyX+9UdeS6wgq1 SzwVm2O0dGPsE9Yjvwxm6fNuDI+D4INO1cyw== X-Gm-Gg: ASbGncvSWi80ryZOD0ZcgvW5ijHAT3co0+SiGDOQwY0BcR3ISa5F+fpyuGcPIUe3ILe fGJ5kynlaD5Y08qD6HnnIzdypuJbNaHpuV29WuT8tNDRMP1OYBZ/3neIND+Mu2YpaDGoQ3QndHU LJdV2Irb/WNOzYkkkzggynTU0uzOxpZmCa+HeEZul0goUVI1Ly09n5YFiGXRGZzIlfiha+NBL+p /97DhoZCcuafC5KzZfg8C2ifuQE+GfUtiwT+2RzjgDGML4nPOcZypk3Bm8gGV++5vOlhN+2JOcB TiOgSUfy/FIcX3ZmWnMdMmktTw== X-Received: by 2002:a6b:6511:0:b0:842:ff2d:c3de with SMTP id ca18e2360f4ac-8447e202021mr474457739f.1.1733525282209; Fri, 06 Dec 2024 14:48:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IE04HG0VbEh/zk6xrbNLHLo2XNM2REETbK76tzH4rQJR65Sy+I6zdMe4n4G2sODT8RGh4fzRA== X-Received: by 2002:a6b:6511:0:b0:842:ff2d:c3de with SMTP id ca18e2360f4ac-8447e202021mr474455839f.1.1733525281878; Fri, 06 Dec 2024 14:48:01 -0800 (PST) Received: from x1n.redhat.com (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-84473a5bd28sm125601439f.38.2024.12.06.14.48.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Dec 2024 14:48:00 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Cc: Prasad Pandit , "Maciej S . Szmigiero" , =?utf-8?q?C=C3=A9dric_?= =?utf-8?q?Le_Goater?= , Alex Williamson , peterx@redhat.com, Avihai Horon , Fabiano Rosas Subject: [PATCH v3 2/7] migration/multifd: Allow to sync with sender threads only Date: Fri, 6 Dec 2024 17:47:50 -0500 Message-ID: <20241206224755.1108686-3-peterx@redhat.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241206224755.1108686-1-peterx@redhat.com> References: <20241206224755.1108686-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.996, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Teach multifd_send_sync_main() to sync with threads only. We already have such requests, which is when mapped-ram is enabled with multifd. In that case, no SYNC messages will be pushed to the stream when multifd syncs the sender threads because there's no destination threads waiting for that. The whole point of the sync is to make sure all threads finished their jobs. So fundamentally we have a request to do the sync in different ways: - Either to sync the threads only, - Or to sync the threads but also with the destination side. Mapped-ram did it already because of the use_packet check in the sync handler of the sender thread. It works. However it may stop working when e.g. VFIO may start to reuse multifd channels to push device states. In that case VFIO has similar request on "thread-only sync" however we can't check a flag because such sync request can still come from RAM which needs the on-wire notifications. Paving way for that by allowing the multifd_send_sync_main() to specify what kind of sync the caller needs. We can use it for mapped-ram already. No functional change intended. Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.h | 23 ++++++++++++++++++++--- migration/multifd-nocomp.c | 7 ++++++- migration/multifd.c | 17 +++++++++++------ 3 files changed, 37 insertions(+), 10 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index 50d58c0c9c..6493512305 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -19,6 +19,22 @@ typedef struct MultiFDRecvData MultiFDRecvData; typedef struct MultiFDSendData MultiFDSendData; +typedef enum { + /* No sync request */ + MULTIFD_SYNC_NONE = 0, + /* Sync locally on the sender threads without pushing messages */ + MULTIFD_SYNC_LOCAL, + /* + * Sync not only on the sender threads, but also push MULTIFD_FLAG_SYNC + * message to the wire for each iochannel (which is for a remote sync). + * + * When remote sync is used, need to be paired with a follow up + * RAM_SAVE_FLAG_EOS / RAM_SAVE_FLAG_MULTIFD_FLUSH message on the main + * channel. + */ + MULTIFD_SYNC_ALL, +} MultiFDSyncReq; + bool multifd_send_setup(void); void multifd_send_shutdown(void); void multifd_send_channel_created(void); @@ -28,7 +44,7 @@ void multifd_recv_shutdown(void); bool multifd_recv_all_channels_created(void); void multifd_recv_new_channel(QIOChannel *ioc, Error **errp); void multifd_recv_sync_main(void); -int multifd_send_sync_main(void); +int multifd_send_sync_main(MultiFDSyncReq req); bool multifd_queue_page(RAMBlock *block, ram_addr_t offset); bool multifd_recv(void); MultiFDRecvData *multifd_get_recv_data(void); @@ -143,7 +159,7 @@ typedef struct { /* multifd flags for each packet */ uint32_t flags; /* - * The sender thread has work to do if either of below boolean is set. + * The sender thread has work to do if either of below field is set. * * @pending_job: a job is pending * @pending_sync: a sync request is pending @@ -152,7 +168,8 @@ typedef struct { * cleared by the multifd sender threads. */ bool pending_job; - bool pending_sync; + MultiFDSyncReq pending_sync; + MultiFDSendData *data; /* thread local variables. No locking required */ diff --git a/migration/multifd-nocomp.c b/migration/multifd-nocomp.c index 55191152f9..219f9e58ef 100644 --- a/migration/multifd-nocomp.c +++ b/migration/multifd-nocomp.c @@ -345,6 +345,8 @@ retry: int multifd_ram_flush_and_sync(void) { + MultiFDSyncReq req; + if (!migrate_multifd()) { return 0; } @@ -356,7 +358,10 @@ int multifd_ram_flush_and_sync(void) } } - return multifd_send_sync_main(); + /* File migrations only need to sync with threads */ + req = migrate_mapped_ram() ? MULTIFD_SYNC_LOCAL : MULTIFD_SYNC_ALL; + + return multifd_send_sync_main(req); } bool multifd_send_prepare_common(MultiFDSendParams *p) diff --git a/migration/multifd.c b/migration/multifd.c index 498e71fd10..7ecc3964ee 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -523,11 +523,13 @@ static int multifd_zero_copy_flush(QIOChannel *c) return ret; } -int multifd_send_sync_main(void) +int multifd_send_sync_main(MultiFDSyncReq req) { int i; bool flush_zero_copy; + assert(req != MULTIFD_SYNC_NONE); + flush_zero_copy = migrate_zero_copy_send(); for (i = 0; i < migrate_multifd_channels(); i++) { @@ -543,8 +545,8 @@ int multifd_send_sync_main(void) * We should be the only user so far, so not possible to be set by * others concurrently. */ - assert(qatomic_read(&p->pending_sync) == false); - qatomic_set(&p->pending_sync, true); + assert(qatomic_read(&p->pending_sync) == MULTIFD_SYNC_NONE); + qatomic_set(&p->pending_sync, req); qemu_sem_post(&p->sem); } for (i = 0; i < migrate_multifd_channels(); i++) { @@ -635,14 +637,17 @@ static void *multifd_send_thread(void *opaque) */ qatomic_store_release(&p->pending_job, false); } else { + MultiFDSyncReq req = qatomic_read(&p->pending_sync); + /* * If not a normal job, must be a sync request. Note that * pending_sync is a standalone flag (unlike pending_job), so * it doesn't require explicit memory barriers. */ - assert(qatomic_read(&p->pending_sync)); + assert(req != MULTIFD_SYNC_NONE); - if (use_packets) { + /* Only push the SYNC message if it involves a remote sync */ + if (req == MULTIFD_SYNC_ALL) { p->flags = MULTIFD_FLAG_SYNC; multifd_send_fill_packet(p); ret = qio_channel_write_all(p->c, (void *)p->packet, @@ -654,7 +659,7 @@ static void *multifd_send_thread(void *opaque) stat64_add(&mig_stats.multifd_bytes, p->packet_len); } - qatomic_set(&p->pending_sync, false); + qatomic_set(&p->pending_sync, MULTIFD_SYNC_NONE); qemu_sem_post(&p->sem_sync); } } From patchwork Fri Dec 6 22:47:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13897904 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7DEF2E7717B for ; Fri, 6 Dec 2024 22:48:52 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJh7O-0004Md-NK; Fri, 06 Dec 2024 17:48:14 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJh7L-0004LS-JN for qemu-devel@nongnu.org; Fri, 06 Dec 2024 17:48:11 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJh7J-0002Pv-HV for qemu-devel@nongnu.org; Fri, 06 Dec 2024 17:48:11 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733525288; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hgUFHeik5gquqKyjKvJG+2JHFRfxmB+aOMmFGA+h4cE=; b=Fa/qhzuNLc/Kk7u18TJByRodGPEH9dhGZCw4tdllh3vaME/VRC5wTCiAskBXI3/67Iym3T acoR+8FfFZalt0hRT1HPQzRHfwEbwMm/t2vCwf9zZn299O36Q4RADrQKU9A56swFk4ieU+ XCr3I0h+iqEvAGJ27LuSu3en11tfxVo= Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-516-KvNb8RrmMPqJm0p0U_pBmg-1; Fri, 06 Dec 2024 17:48:07 -0500 X-MC-Unique: KvNb8RrmMPqJm0p0U_pBmg-1 X-Mimecast-MFC-AGG-ID: KvNb8RrmMPqJm0p0U_pBmg Received: by mail-io1-f72.google.com with SMTP id ca18e2360f4ac-843e86e0b3fso431697139f.3 for ; Fri, 06 Dec 2024 14:48:07 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525286; x=1734130086; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hgUFHeik5gquqKyjKvJG+2JHFRfxmB+aOMmFGA+h4cE=; b=A8I6/zK0syGrrzTIQGQHQPTcc/2NZ2MWwwGQ3leGwTgxAANEi3EvlGT5v+9964Mwzl KPp8TIDYekOYiNaShW75y7FROfeDscBUxXt+HuuBeBVy9H/n+/thm1gc+qsejab2BMoP r9arGhZRRf8jzY2HnM1JYErbrUqZwq63sOPbuG3scPnkqMD8+R5CJGBlxf/a20KRWU9k ZgEby4EQv24iFv96ijdlaPUj6/NSANFr2cLtxh73/avSM8rhqt1Bq+VIRGUYoSpTpyaD +Ac3hNr07dOts0CDkQj+f4oAJnTPaXzYyfKT05jDM+tkWtbdRxzZz3zTdIufKL6n48CJ bbCg== X-Gm-Message-State: AOJu0YytNPdcPETxSKId8RoKoOTiH8e8DAg9tFPeCfi16LlNjsk1exzZ xReSxZDQqh9JwuBgi6FyviytxfnETlqBi9oZCSw3WyG96cWZeu3k9jLzKISh3G8A+5EeJleyykH 0ZCg2Ld765MFZeJ1i+akd6PGUnZvqIkWfND4r1RUNsf3cZzHLZC8R6piTM5VDYJJeeVEEe+9rhL 6cHEMBqZlizSQt+bz5eqA4X8iEEk4E4zEkQg== X-Gm-Gg: ASbGncuBvsWHrhyG3eFFBgwo8C0rsdX+hgV9IO/ePcfKYWH6Z5yItkqp5T5CcImbqJ8 +vZ+sXC3dcernmC8LUqKHmZQpR9h13jyuHYOvdGjtN0sNRn43aktvcsMqDJulefNWhXfhrHK9v8 oPI+lClkhteN9us+/8Mhf2PytVm6hAkGhxnmR3irU1sbNpOno7OHBt+ktdoBdeJP/79FCORTbgc dKocgrvcty5nGnJXsTzE8DQscNtT2zubVqMX/iKi51eVuX5Hypbpr3UUS6jSfqcaTvCOw0aima6 BIUUR+j+lcLRr5b929TlQ98c2g== X-Received: by 2002:a05:6602:3402:b0:82c:e4e1:2e99 with SMTP id ca18e2360f4ac-8447e296ad5mr543146539f.11.1733525286157; Fri, 06 Dec 2024 14:48:06 -0800 (PST) X-Google-Smtp-Source: AGHT+IGfo3cpBwgAKEnRhayyIIZVeYSRjfv5pXJbzMtI2PUa/TuuVYh68EXWqS+kAaWNLXuxmfy0Tw== X-Received: by 2002:a05:6602:3402:b0:82c:e4e1:2e99 with SMTP id ca18e2360f4ac-8447e296ad5mr543144739f.11.1733525285771; Fri, 06 Dec 2024 14:48:05 -0800 (PST) Received: from x1n.redhat.com (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-84473a5bd28sm125601439f.38.2024.12.06.14.48.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Dec 2024 14:48:02 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Cc: Prasad Pandit , "Maciej S . Szmigiero" , =?utf-8?q?C=C3=A9dric_?= =?utf-8?q?Le_Goater?= , Alex Williamson , peterx@redhat.com, Avihai Horon , Fabiano Rosas Subject: [PATCH v3 3/7] migration/ram: Move RAM_SAVE_FLAG* into ram.h Date: Fri, 6 Dec 2024 17:47:51 -0500 Message-ID: <20241206224755.1108686-4-peterx@redhat.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241206224755.1108686-1-peterx@redhat.com> References: <20241206224755.1108686-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.996, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Firstly, we're going to use the multifd flag soon in multifd code, so ram.c isn't gonna work. Secondly, we have a separate RDMA flag dangling around, which is definitely not obvious. There's one comment that helps, but not too much. Put all RAM save flags altogether, so nothing will get overlooked. Add a section explain why we can't use bits over 0x200. Remove RAM_SAVE_FLAG_FULL as it's already not used in QEMU, as the comment explained. Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/ram.h | 28 ++++++++++++++++++++++++++++ migration/rdma.h | 7 ------- migration/ram.c | 21 --------------------- 3 files changed, 28 insertions(+), 28 deletions(-) diff --git a/migration/ram.h b/migration/ram.h index 0d1981f888..921c39a2c5 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -33,6 +33,34 @@ #include "exec/cpu-common.h" #include "io/channel.h" +/* + * RAM_SAVE_FLAG_ZERO used to be named RAM_SAVE_FLAG_COMPRESS, it + * worked for pages that were filled with the same char. We switched + * it to only search for the zero value. And to avoid confusion with + * RAM_SAVE_FLAG_COMPRESS_PAGE just rename it. + * + * RAM_SAVE_FLAG_FULL (0x01) was obsoleted in 2009. + * + * RAM_SAVE_FLAG_COMPRESS_PAGE (0x100) was removed in QEMU 9.1. + * + * RAM_SAVE_FLAG_HOOK is only used in RDMA. Whenever this is found in the + * data stream, the flags will be passed to rdma functions in the + * incoming-migration side. + * + * We can't use any flag that is bigger than 0x200, because the flags are + * always assumed to be encoded in a ramblock address offset, which is + * multiple of PAGE_SIZE. Here it means QEMU supports migration with any + * architecture that has PAGE_SIZE>=1K (0x400). + */ +#define RAM_SAVE_FLAG_ZERO 0x002 +#define RAM_SAVE_FLAG_MEM_SIZE 0x004 +#define RAM_SAVE_FLAG_PAGE 0x008 +#define RAM_SAVE_FLAG_EOS 0x010 +#define RAM_SAVE_FLAG_CONTINUE 0x020 +#define RAM_SAVE_FLAG_XBZRLE 0x040 +#define RAM_SAVE_FLAG_HOOK 0x080 +#define RAM_SAVE_FLAG_MULTIFD_FLUSH 0x200 + extern XBZRLECacheStats xbzrle_counters; /* Should be holding either ram_list.mutex, or the RCU lock. */ diff --git a/migration/rdma.h b/migration/rdma.h index a8d27f33b8..f55f28bbed 100644 --- a/migration/rdma.h +++ b/migration/rdma.h @@ -33,13 +33,6 @@ void rdma_start_incoming_migration(InetSocketAddress *host_port, Error **errp); #define RAM_CONTROL_ROUND 1 #define RAM_CONTROL_FINISH 3 -/* - * Whenever this is found in the data stream, the flags - * will be passed to rdma functions in the incoming-migration - * side. - */ -#define RAM_SAVE_FLAG_HOOK 0x80 - #define RAM_SAVE_CONTROL_NOT_SUPP -1000 #define RAM_SAVE_CONTROL_DELAYED -2000 diff --git a/migration/ram.c b/migration/ram.c index 7284c34bd8..44010ff325 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -71,27 +71,6 @@ /***********************************************************/ /* ram save/restore */ -/* - * RAM_SAVE_FLAG_ZERO used to be named RAM_SAVE_FLAG_COMPRESS, it - * worked for pages that were filled with the same char. We switched - * it to only search for the zero value. And to avoid confusion with - * RAM_SAVE_FLAG_COMPRESS_PAGE just rename it. - * - * RAM_SAVE_FLAG_FULL was obsoleted in 2009. - * - * RAM_SAVE_FLAG_COMPRESS_PAGE (0x100) was removed in QEMU 9.1. - */ -#define RAM_SAVE_FLAG_FULL 0x01 -#define RAM_SAVE_FLAG_ZERO 0x02 -#define RAM_SAVE_FLAG_MEM_SIZE 0x04 -#define RAM_SAVE_FLAG_PAGE 0x08 -#define RAM_SAVE_FLAG_EOS 0x10 -#define RAM_SAVE_FLAG_CONTINUE 0x20 -#define RAM_SAVE_FLAG_XBZRLE 0x40 -/* 0x80 is reserved in rdma.h for RAM_SAVE_FLAG_HOOK */ -#define RAM_SAVE_FLAG_MULTIFD_FLUSH 0x200 -/* We can't use any flag that is bigger than 0x200 */ - /* * mapped-ram migration supports O_DIRECT, so we need to make sure the * userspace buffer, the IO operation size and the file offset are From patchwork Fri Dec 6 22:47:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13897903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 695E2E77173 for ; Fri, 6 Dec 2024 22:48:52 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJh7O-0004M8-0n; Fri, 06 Dec 2024 17:48:14 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJh7N-0004Lz-B7 for qemu-devel@nongnu.org; Fri, 06 Dec 2024 17:48:13 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJh7L-0002Q7-PK for qemu-devel@nongnu.org; Fri, 06 Dec 2024 17:48:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733525291; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z8l4cpJdky+6VCEyXhHeGMSpMlUpTsGKiYvdnk77GRc=; b=dnfH0tabqFfdeYv5J3Uy6xCMO5RSXjom4iBgLYDEtsWQE0npPxYDvL5JvPt4qJlX5/vkyC L54INUk1V7q2mOYSHVGVp+JvkOBLHPeNN2JIVFZVmHLulbAWGGKABAkjaIbQopkGdbAexB dIjLBb0VlGdjq7FuYfusjxr7FZmIi+U= Received: from mail-io1-f71.google.com (mail-io1-f71.google.com [209.85.166.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-645-eIeK3V8aPKCqppoZvraFLw-1; Fri, 06 Dec 2024 17:48:10 -0500 X-MC-Unique: eIeK3V8aPKCqppoZvraFLw-1 X-Mimecast-MFC-AGG-ID: eIeK3V8aPKCqppoZvraFLw Received: by mail-io1-f71.google.com with SMTP id ca18e2360f4ac-841a54a6603so443897839f.2 for ; Fri, 06 Dec 2024 14:48:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525287; x=1734130087; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Z8l4cpJdky+6VCEyXhHeGMSpMlUpTsGKiYvdnk77GRc=; b=wbVgKd+rh/rm7zFnib7zPm1aYqJ2mtA9HRjNOrJOTQeXRRlz6xfGZCG2zqaappguG1 fNiCm8NMzRjmxOcPHOlSmCkXdZ88n69f6F+88AdWii+ehzGYDnTEjL88JTKmm8cFIUOy 4VZItpEoWIr/IhLHAorSzTcHdDqORAlNaXj7H7aOHVbfwnlBP0GOOypW8vbwp9YMoIRE 1JYRAY1+7NtohBG2aj4lxUUIolR5rYJBn5OlkUlKT42itRYBodS6aRoCND8QFt5ZOKuZ b+B82I7pJd6X23lvfQgidlxfoP6mRh6pfeBybOAz1mXQjgCnO1Vak7fP7Wz2e4WKZoN5 XoOQ== X-Gm-Message-State: AOJu0YySNV0Q2EArrEV7uDKJI++fR3weEDt771RglJpNDaHOa03J2Bnp 96AcYNVVZrKnbfWsgSZQDKIyUyFr73msXiCYkxYkJVRS+anVzs7ufgRK/XFcd9u+q04z6I3aVI2 cxt77wRJllPmTmspgUDGLox/fXWP8/vb1INYul3mV9iUKSnLZQs4ukBkgYqoLo//Z3YC9LgAutv 7XC2lTnQGT3u0FIwDo1bG4ZqPyr8ZdwRCyzA== X-Gm-Gg: ASbGncspo2jZ/9MxNuOBM68RQUkHcEaCdlaRkMrSGBS8bDpPhdO8dJe/U50we7MiZqT KqCd6Nxtl660n+tSqBDltIEYmaDCl9hoDOAqo1l1kdWe7f+j24v+iVtpZMR2wPwjjC5gaz+fdw0 zLPSNqQDTubbRdX39w4doLzas3hT++swzqgHUVsnqzWSInt4bf3m2Im+gUsorMYzs5LK0pVkSUl gx68rTN6IqmjmDgeaypwlfAScFDmonGpMEgDTtieKhTpSkWr0fNo8IT+M9ScpdypxZMjc9SO+CB FTF6syvATYOXAUHfILYd3dZ9eg== X-Received: by 2002:a05:6602:2c06:b0:841:950b:386d with SMTP id ca18e2360f4ac-8447e202016mr733309739f.3.1733525287437; Fri, 06 Dec 2024 14:48:07 -0800 (PST) X-Google-Smtp-Source: AGHT+IFmF9yRhri8h5+sffg2R5gMhZMQbNVM/Nt4jbRYRF4D2dxEKR9J5DT91XtJIR6Z2Ohr2IN8GQ== X-Received: by 2002:a05:6602:2c06:b0:841:950b:386d with SMTP id ca18e2360f4ac-8447e202016mr733307239f.3.1733525287123; Fri, 06 Dec 2024 14:48:07 -0800 (PST) Received: from x1n.redhat.com (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-84473a5bd28sm125601439f.38.2024.12.06.14.48.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Dec 2024 14:48:06 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Cc: Prasad Pandit , "Maciej S . Szmigiero" , =?utf-8?q?C=C3=A9dric_?= =?utf-8?q?Le_Goater?= , Alex Williamson , peterx@redhat.com, Avihai Horon , Fabiano Rosas Subject: [PATCH v3 4/7] migration/multifd: Unify RAM_SAVE_FLAG_MULTIFD_FLUSH messages Date: Fri, 6 Dec 2024 17:47:52 -0500 Message-ID: <20241206224755.1108686-5-peterx@redhat.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241206224755.1108686-1-peterx@redhat.com> References: <20241206224755.1108686-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.996, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org RAM_SAVE_FLAG_MULTIFD_FLUSH message should always be correlated to a sync request on src. Unify such message into one place, and conditionally send the message only if necessary. Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/multifd.h | 2 +- migration/multifd-nocomp.c | 27 +++++++++++++++++++++++++-- migration/ram.c | 18 ++++-------------- 3 files changed, 30 insertions(+), 17 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index 6493512305..0fef431f6b 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -354,7 +354,7 @@ static inline uint32_t multifd_ram_page_count(void) void multifd_ram_save_setup(void); void multifd_ram_save_cleanup(void); -int multifd_ram_flush_and_sync(void); +int multifd_ram_flush_and_sync(QEMUFile *f); size_t multifd_ram_payload_size(void); void multifd_ram_fill_packet(MultiFDSendParams *p); int multifd_ram_unfill_packet(MultiFDRecvParams *p, Error **errp); diff --git a/migration/multifd-nocomp.c b/migration/multifd-nocomp.c index 219f9e58ef..58372db0f4 100644 --- a/migration/multifd-nocomp.c +++ b/migration/multifd-nocomp.c @@ -20,6 +20,7 @@ #include "qemu/cutils.h" #include "qemu/error-report.h" #include "trace.h" +#include "qemu-file.h" static MultiFDSendData *multifd_ram_send; @@ -343,9 +344,10 @@ retry: return true; } -int multifd_ram_flush_and_sync(void) +int multifd_ram_flush_and_sync(QEMUFile *f) { MultiFDSyncReq req; + int ret; if (!migrate_multifd()) { return 0; @@ -361,7 +363,28 @@ int multifd_ram_flush_and_sync(void) /* File migrations only need to sync with threads */ req = migrate_mapped_ram() ? MULTIFD_SYNC_LOCAL : MULTIFD_SYNC_ALL; - return multifd_send_sync_main(req); + ret = multifd_send_sync_main(req); + if (ret) { + return ret; + } + + /* If we don't need to sync with remote at all, nothing else to do */ + if (req == MULTIFD_SYNC_LOCAL) { + return 0; + } + + /* + * Old QEMUs don't understand RAM_SAVE_FLAG_MULTIFD_FLUSH, it relies + * on RAM_SAVE_FLAG_EOS instead. + */ + if (migrate_multifd_flush_after_each_section()) { + return 0; + } + + qemu_put_be64(f, RAM_SAVE_FLAG_MULTIFD_FLUSH); + qemu_fflush(f); + + return 0; } bool multifd_send_prepare_common(MultiFDSendParams *p) diff --git a/migration/ram.c b/migration/ram.c index 44010ff325..90811aabd4 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1306,15 +1306,10 @@ static int find_dirty_block(RAMState *rs, PageSearchStatus *pss) (!migrate_multifd_flush_after_each_section() || migrate_mapped_ram())) { QEMUFile *f = rs->pss[RAM_CHANNEL_PRECOPY].pss_channel; - int ret = multifd_ram_flush_and_sync(); + int ret = multifd_ram_flush_and_sync(f); if (ret < 0) { return ret; } - - if (!migrate_mapped_ram()) { - qemu_put_be64(f, RAM_SAVE_FLAG_MULTIFD_FLUSH); - qemu_fflush(f); - } } /* Hit the end of the list */ @@ -3044,18 +3039,13 @@ static int ram_save_setup(QEMUFile *f, void *opaque, Error **errp) } bql_unlock(); - ret = multifd_ram_flush_and_sync(); + ret = multifd_ram_flush_and_sync(f); bql_lock(); if (ret < 0) { error_setg(errp, "%s: multifd synchronization failed", __func__); return ret; } - if (migrate_multifd() && !migrate_multifd_flush_after_each_section() - && !migrate_mapped_ram()) { - qemu_put_be64(f, RAM_SAVE_FLAG_MULTIFD_FLUSH); - } - qemu_put_be64(f, RAM_SAVE_FLAG_EOS); ret = qemu_fflush(f); if (ret < 0) { @@ -3190,7 +3180,7 @@ out: if (ret >= 0 && migration_is_running()) { if (migrate_multifd() && migrate_multifd_flush_after_each_section() && !migrate_mapped_ram()) { - ret = multifd_ram_flush_and_sync(); + ret = multifd_ram_flush_and_sync(f); if (ret < 0) { return ret; } @@ -3268,7 +3258,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque) * Only the old dest QEMU will need this sync, because each EOS * will require one SYNC message on each channel. */ - ret = multifd_ram_flush_and_sync(); + ret = multifd_ram_flush_and_sync(f); if (ret < 0) { return ret; } From patchwork Fri Dec 6 22:47:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13897908 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6453DE77173 for ; Fri, 6 Dec 2024 22:49:26 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJh7R-0004NC-0O; Fri, 06 Dec 2024 17:48:17 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJh7P-0004Mk-4L for qemu-devel@nongnu.org; Fri, 06 Dec 2024 17:48:15 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJh7N-0002QP-O4 for qemu-devel@nongnu.org; Fri, 06 Dec 2024 17:48:14 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733525293; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KaWXvIbyGkvjWCcWSsdzmNjfpjFNGapBznL6Ic3T0Zo=; b=bcVwtOXuMcMl/2Nd4oQowzgrmMxmcZPsIziopECrqmtBEDOLsL8TU1fzeTLb2t361Hcooi XF9GgPv3R9VV2EbhTLmjXahTczn+rud29ytCYxIToQNYFSWD39XQQoGyxiBuKGnYXKIRMk 3yhnL9Kl0PdglLlSWMwV8z5FeHlnB1w= Received: from mail-io1-f70.google.com (mail-io1-f70.google.com [209.85.166.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-638-nD_BDoCeN7aONcDf7mvkkQ-1; Fri, 06 Dec 2024 17:48:12 -0500 X-MC-Unique: nD_BDoCeN7aONcDf7mvkkQ-1 X-Mimecast-MFC-AGG-ID: nD_BDoCeN7aONcDf7mvkkQ Received: by mail-io1-f70.google.com with SMTP id ca18e2360f4ac-8418612c3b5so232160739f.3 for ; Fri, 06 Dec 2024 14:48:12 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525291; x=1734130091; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KaWXvIbyGkvjWCcWSsdzmNjfpjFNGapBznL6Ic3T0Zo=; b=XcvBp+vDpocvJSfu8bhroclDiwX3fD0OaJu7TXHv9AapgnineW4qwBLCoUjGEHoE6g /0hhiLTVJBhwJTYbuu+OwYOnPbW7gq1jNdXOKPy/368k6eYJB7yC59ef7SqjjipfYUdw 55K+aZyHytuOrHAUKvx7mfYPlpjhTGswVW6bGqkKpW30FO4eFIcvDiP+RVj5NbrvRThN e97HFsENeAtjYt1xTQRz8YjTCFonS0fq6jLi3XdrBPvJ+YLe6hsP0rWSy4shYiRvlGc5 cpIfgdgfMfCyoLFhYm6O7x0nevovKZKKdpREw8BUjAOyPZFQ/zjfEboRt5jeCR5gQfDE buCA== X-Gm-Message-State: AOJu0YyKLWpLI8lM+fdiqoQlCzqMuqLoU4Y4QG4/dSKd66L8FkRUkRHD aYC0mNlN6sJjGqScDU5EhM7VpVAv313aVf7L+/AIFIsDxJasygd2hkmIj6LcuAhDgbrj2/rmaf1 1WxdtB/T0LnXmB8VRxAaRSb0V9eFDDHmas4mtX8bBZM/2m6uka7Gs4dNOojZ7386AFu0Y7G8Oor FkFZfWuAmp6u0wlI3HK3EJdmIgiD6C/fhhfQ== X-Gm-Gg: ASbGncvavI6omANecod1F0K4OjOGNQscXDhvs+Us/fvC3JgReYGqLWgM/9T2qPQIXeR LwRLZrhHGRJU+bWa9VRDQdFytlOVLBnjrtgNgN12wigoTh0yJoP7tUCBL7NvjeTHwY/VrFujl6f 4Zl+oWyLJqs4jtpqFkvKS4h4TDbqOo/Jgxt833RWov8w8WtYk4VsSDi1KejQ+NhoNr6Lcvr6J15 hlBPkC3PofEuo0kI3Mq5t33BDfc3GezLKbMGPKycUGgJP9mrCXOScdMlA4CbBCTPkjtZwf3Zhrf x5vusLaXCsxnlTQpSEQbMcpupA== X-Received: by 2002:a05:6602:1610:b0:843:e89d:b62b with SMTP id ca18e2360f4ac-8447e32bdf7mr478795339f.11.1733525290890; Fri, 06 Dec 2024 14:48:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IELHrJW4urV9sPJvpj5tARdb9tBTOqrUrQ7szZmZuw1fhu+VECo1p00l9St/ZQKmg2OQikCvA== X-Received: by 2002:a05:6602:1610:b0:843:e89d:b62b with SMTP id ca18e2360f4ac-8447e32bdf7mr478793139f.11.1733525290514; Fri, 06 Dec 2024 14:48:10 -0800 (PST) Received: from x1n.redhat.com (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-84473a5bd28sm125601439f.38.2024.12.06.14.48.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Dec 2024 14:48:08 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Cc: Prasad Pandit , "Maciej S . Szmigiero" , =?utf-8?q?C=C3=A9dric_?= =?utf-8?q?Le_Goater?= , Alex Williamson , peterx@redhat.com, Avihai Horon , Fabiano Rosas Subject: [PATCH v3 5/7] migration/multifd: Remove sync processing on postcopy Date: Fri, 6 Dec 2024 17:47:53 -0500 Message-ID: <20241206224755.1108686-6-peterx@redhat.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241206224755.1108686-1-peterx@redhat.com> References: <20241206224755.1108686-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.996, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Multifd never worked with postcopy, at least yet so far. Remove the sync processing there, because it's confusing, and they should never appear. Now if RAM_SAVE_FLAG_MULTIFD_FLUSH is observed, we fail hard instead of trying to invoke multifd code. Reviewed-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/ram.c | 8 -------- 1 file changed, 8 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 90811aabd4..154ff5abd4 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3772,15 +3772,7 @@ int ram_load_postcopy(QEMUFile *f, int channel) TARGET_PAGE_SIZE); } break; - case RAM_SAVE_FLAG_MULTIFD_FLUSH: - multifd_recv_sync_main(); - break; case RAM_SAVE_FLAG_EOS: - /* normal exit */ - if (migrate_multifd() && - migrate_multifd_flush_after_each_section()) { - multifd_recv_sync_main(); - } break; default: error_report("Unknown combination of migration flags: 0x%x" From patchwork Fri Dec 6 22:47:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13897905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 64E86E77173 for ; Fri, 6 Dec 2024 22:48:55 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJh7S-0004Nn-6t; Fri, 06 Dec 2024 17:48:18 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJh7Q-0004ND-V1 for qemu-devel@nongnu.org; Fri, 06 Dec 2024 17:48:16 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJh7P-0002Qb-AK for qemu-devel@nongnu.org; Fri, 06 Dec 2024 17:48:16 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733525294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VcILogkNzClRSFChag2j5EOuO0zmv9yeRvDNl/B1JUw=; b=U3h+qZb6mhYeV1kt/DEyTV/JndJEjhH+1IcTHrOTUn786CXDRel2ilMTbklGE3+WQwCV4b 1WUlrVj/GH2pJKJSI9orYS3eOsV6yhvwqS04i64SxNBfl3ZQuooDXuuWK7FV4WzSGiRrM2 njeZ9UFERSDbWt74apCybBuahS8TTNg= Received: from mail-io1-f70.google.com (mail-io1-f70.google.com [209.85.166.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-314-Z6qYBrHZPQWJ2T50a1OJ1g-1; Fri, 06 Dec 2024 17:48:13 -0500 X-MC-Unique: Z6qYBrHZPQWJ2T50a1OJ1g-1 X-Mimecast-MFC-AGG-ID: Z6qYBrHZPQWJ2T50a1OJ1g Received: by mail-io1-f70.google.com with SMTP id ca18e2360f4ac-84181157edbso247549939f.2 for ; Fri, 06 Dec 2024 14:48:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525292; x=1734130092; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VcILogkNzClRSFChag2j5EOuO0zmv9yeRvDNl/B1JUw=; b=AqaFB92Wwd0Eu+UarxdsSaT00KshPXWX+4FIk9fZbQEHaFOK+euBh378i4Xrg06ia4 cYUGnsr+l5hbdBxfPv/PfEGPM9hvKF7N27x7tTIhJkkaoZFgO4FRQFdIlzeNHk50XhTk KF7X3rE1Agi2Nyt7TSFMYE3NvAHIfyC9enTrOC7Pdamy1pS7ZUf4xMZbk/YOsjc8ep2S 1eBWImJRJdwhAXRGI79wj9NH5L3j6wzxwxD0uT/LTCboNRb2MnFtHkftBRcUjXEXHIuB 3Ovmgby2kPKzPUSko10R46bantGDf6NZkCfOlpgX7nwGaUbBIh7Mtelx0lxqn4RMuxPt 1nrQ== X-Gm-Message-State: AOJu0Yx8e0az0BtBkbt+17qQCIrGXHYnS5rfjSZCU1MCChLE5NaIM7/9 vIL9VOzHqbWSMpQXPaoqh3cLrHZnu2RZPiKP6iKvWciliSqPCTkL0eWtgX1AJD/+Uubm/aRSz7P enRKnWxDaaC2lwxvFiuFUHtoUs1KZsGOuDR/1TdjkHYtNsgXuryKMgv4c6HSyaXoNc+b5ufbEJi LObEi6dW36H7exP5Hkgm+Z2fE5jxx+3muehA== X-Gm-Gg: ASbGncvzhpuqCLZglqul+rT0JA473HA0zYKN7TeSaquXuOYFDw2AEBYynANyWYIGvvT Bl1A6StiqqWZtxtlPdpNGUukb2IJ2xc2RjebFd0XkipAPRYonN6Rvwa3DyPlhPm00Rv10n4dZUW nkEqmIf8yx6BXOi43ErZadMU7digWz9P2TKIpBBgXs7o2bFy9zbDhP3OWUEZ74FeNaveIFwE2n1 oDpYoJyLqc5eRQyTVPsugHr2Bi/jUJn2YL4WjI+mhdUFRiIIaAb2x4bp42Z/6KzzTWHxh7oybqx ZCzrdmJJKZo+uqyUxyMCemgWmg== X-Received: by 2002:a05:6602:6002:b0:83a:c0ba:73c6 with SMTP id ca18e2360f4ac-8447e2bcd22mr522332139f.11.1733525292312; Fri, 06 Dec 2024 14:48:12 -0800 (PST) X-Google-Smtp-Source: AGHT+IFp4Tw4BsBCUsKd2ABGGRFGlSO7HAA1Q8gQ0DYuA69AGs+18LqF/peLiZcN/ssGfJOr20bvNQ== X-Received: by 2002:a05:6602:6002:b0:83a:c0ba:73c6 with SMTP id ca18e2360f4ac-8447e2bcd22mr522330039f.11.1733525291908; Fri, 06 Dec 2024 14:48:11 -0800 (PST) Received: from x1n.redhat.com (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-84473a5bd28sm125601439f.38.2024.12.06.14.48.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Dec 2024 14:48:11 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Cc: Prasad Pandit , "Maciej S . Szmigiero" , =?utf-8?q?C=C3=A9dric_?= =?utf-8?q?Le_Goater?= , Alex Williamson , peterx@redhat.com, Avihai Horon , Fabiano Rosas Subject: [PATCH v3 6/7] migration/multifd: Cleanup src flushes on condition check Date: Fri, 6 Dec 2024 17:47:54 -0500 Message-ID: <20241206224755.1108686-7-peterx@redhat.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241206224755.1108686-1-peterx@redhat.com> References: <20241206224755.1108686-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.996, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The src flush condition check is over complicated, and it's getting more out of control if postcopy will be involved. In general, we have two modes to do the sync: legacy or modern ways. Legacy uses per-section flush, modern uses per-round flush. Mapped-ram always uses the modern, which is per-round. Introduce two helpers, which can greatly simplify the code, and hopefully make it readable again. Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/multifd.h | 2 ++ migration/multifd-nocomp.c | 42 ++++++++++++++++++++++++++++++++++++++ migration/ram.c | 10 +++------ 3 files changed, 47 insertions(+), 7 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index 0fef431f6b..bd785b9873 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -355,6 +355,8 @@ static inline uint32_t multifd_ram_page_count(void) void multifd_ram_save_setup(void); void multifd_ram_save_cleanup(void); int multifd_ram_flush_and_sync(QEMUFile *f); +bool multifd_ram_sync_per_round(void); +bool multifd_ram_sync_per_section(void); size_t multifd_ram_payload_size(void); void multifd_ram_fill_packet(MultiFDSendParams *p); int multifd_ram_unfill_packet(MultiFDRecvParams *p, Error **errp); diff --git a/migration/multifd-nocomp.c b/migration/multifd-nocomp.c index 58372db0f4..c1f686c0ce 100644 --- a/migration/multifd-nocomp.c +++ b/migration/multifd-nocomp.c @@ -344,6 +344,48 @@ retry: return true; } +/* + * We have two modes for multifd flushes: + * + * - Per-section mode: this is the legacy way to flush, it requires one + * MULTIFD_FLAG_SYNC message for each RAM_SAVE_FLAG_EOS. + * + * - Per-round mode: this is the modern way to flush, it requires one + * MULTIFD_FLAG_SYNC message only for each round of RAM scan. Normally + * it's paired with a new RAM_SAVE_FLAG_MULTIFD_FLUSH message in network + * based migrations. + * + * One thing to mention is mapped-ram always use the modern way to sync. + */ + +/* Do we need a per-section multifd flush (legacy way)? */ +bool multifd_ram_sync_per_section(void) +{ + if (!migrate_multifd()) { + return false; + } + + if (migrate_mapped_ram()) { + return false; + } + + return migrate_multifd_flush_after_each_section(); +} + +/* Do we need a per-round multifd flush (modern way)? */ +bool multifd_ram_sync_per_round(void) +{ + if (!migrate_multifd()) { + return false; + } + + if (migrate_mapped_ram()) { + return true; + } + + return !migrate_multifd_flush_after_each_section(); +} + int multifd_ram_flush_and_sync(QEMUFile *f) { MultiFDSyncReq req; diff --git a/migration/ram.c b/migration/ram.c index 154ff5abd4..5d4bdefe69 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1302,9 +1302,7 @@ static int find_dirty_block(RAMState *rs, PageSearchStatus *pss) pss->page = 0; pss->block = QLIST_NEXT_RCU(pss->block, next); if (!pss->block) { - if (migrate_multifd() && - (!migrate_multifd_flush_after_each_section() || - migrate_mapped_ram())) { + if (multifd_ram_sync_per_round()) { QEMUFile *f = rs->pss[RAM_CHANNEL_PRECOPY].pss_channel; int ret = multifd_ram_flush_and_sync(f); if (ret < 0) { @@ -3178,8 +3176,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) out: if (ret >= 0 && migration_is_running()) { - if (migrate_multifd() && migrate_multifd_flush_after_each_section() && - !migrate_mapped_ram()) { + if (multifd_ram_sync_per_section()) { ret = multifd_ram_flush_and_sync(f); if (ret < 0) { return ret; @@ -3252,8 +3249,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque) } } - if (migrate_multifd() && - migrate_multifd_flush_after_each_section()) { + if (multifd_ram_sync_per_section()) { /* * Only the old dest QEMU will need this sync, because each EOS * will require one SYNC message on each channel. From patchwork Fri Dec 6 22:47:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13897906 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 75A81E77173 for ; Fri, 6 Dec 2024 22:49:11 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJh7T-0004OE-5E; Fri, 06 Dec 2024 17:48:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJh7S-0004Np-89 for qemu-devel@nongnu.org; Fri, 06 Dec 2024 17:48:18 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJh7Q-0002Qn-QT for qemu-devel@nongnu.org; Fri, 06 Dec 2024 17:48:18 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733525296; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qvHR9rHEEMJo9JZLh5TSkXPd0q/isxl0gB/rbbaO7XM=; b=E7an8IHVM1UtU/l6xkyk3OEkO5VaOHy4hjN61cR5cFb0FkqfllvX3ju6JOZF/ZUQCK194R Zx7Adcy+bCiDqnA3UZAImo4EDnH8Cxwh0DzFZQ6Sfxkn3bJCmFnh9RfEBBoU2vvc4LPtp/ rmx8UWOh/OEeMXfrM71tRF7p/81Q9XA= Received: from mail-io1-f71.google.com (mail-io1-f71.google.com [209.85.166.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-563-5R1fxT0cNsGTrEDvG5NKaQ-1; Fri, 06 Dec 2024 17:48:15 -0500 X-MC-Unique: 5R1fxT0cNsGTrEDvG5NKaQ-1 X-Mimecast-MFC-AGG-ID: 5R1fxT0cNsGTrEDvG5NKaQ Received: by mail-io1-f71.google.com with SMTP id ca18e2360f4ac-841843a9970so230784739f.3 for ; Fri, 06 Dec 2024 14:48:15 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525294; x=1734130094; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qvHR9rHEEMJo9JZLh5TSkXPd0q/isxl0gB/rbbaO7XM=; b=FceGEsGLhWvcQVnsSBBIb9Ucs16WGmrbtUU5IKZ+K3Eehr2wjun/NiZWXwuCBYtDJY LDtnH02mL5TRZQV2Dd7XX+O5tMXePCtGUqWg+WTftRZyHnrbD5bdQi/lxE1XFJnv8fer GnP947IzD9WvGgf01yEBSs0Y8bKkotoUu4XN4WVaeDvQ43qdfN8/HSFTTLkzPwp7TdBI vdEaIF9I2sqgY+JZpeKB9N+P7X0ou3Fp8Cu28w6x2J60MdfYfzxMId0AnNMJCgtXYFBe HdkeVgimD7me0nIeMvIlTXGvHO2Vre2/WQ6TRzT+rowy5BQ0rZc6pHecxaiRYUEtTtYu 97UQ== X-Gm-Message-State: AOJu0Ywy+BYjnV5urKsilS5f3tGA4DLnS+FXfkvLVH/bMU+v1d4nbVDl V364QzKky3w1yD9ZLlEcyGATgGeEz2pGvLdeZVUujyomo3GoPe/VeZfAbEHG6nT3i341BiHFTdC wbk3p5cFXlTqr+Xy/QnnY1lFUr+9uAYbJejBw0fM+4Rs6zAeaA6IXr14hMLLkiaye/ZyA6TEhbC xKD0GXfbXT2jr7OzaaCB9qTZwhAW/9M+LHIQ== X-Gm-Gg: ASbGnctbeFaLhKwlTzvaR8SlIRh5E0dvsR4jHvZ7pgbh9ufIIAGAmxUU8/KsuGUdGfO bJ0a/NQGUUOeKMv9d8BQy8A1VWyQZEtXFs5+SwPddv/An8YXYqITQbp/ywBtWBa2kGpV1hQc3VL lmjifIM7wha7VuWeZzDaXMQmtSk9joQbsjeXMQtQJcujAPXRquN1H1O94N6lOgvASzquDWWnxrU GkioCRZTwHwCkSTnwl3u8xERrR+as+CQCLFDv8UezXulofChR2gPUy6Z5aiZJpJtm0pmh1PNMG8 xfnbKmhdceJShs9Y17pcPer/jg== X-Received: by 2002:a05:6602:1681:b0:843:f220:fa34 with SMTP id ca18e2360f4ac-8447e320bb4mr593948439f.15.1733525293893; Fri, 06 Dec 2024 14:48:13 -0800 (PST) X-Google-Smtp-Source: AGHT+IEEnQbTD764YsgJU6wJvcXio05QurkaJ9etDyonwrFfQDi9mpDvJDOgRRjxXCTwsng7iuo37Q== X-Received: by 2002:a05:6602:1681:b0:843:f220:fa34 with SMTP id ca18e2360f4ac-8447e320bb4mr593945539f.15.1733525293524; Fri, 06 Dec 2024 14:48:13 -0800 (PST) Received: from x1n.redhat.com (pool-99-254-114-190.cpe.net.cable.rogers.com. [99.254.114.190]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-84473a5bd28sm125601439f.38.2024.12.06.14.48.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Dec 2024 14:48:12 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Cc: Prasad Pandit , "Maciej S . Szmigiero" , =?utf-8?q?C=C3=A9dric_?= =?utf-8?q?Le_Goater?= , Alex Williamson , peterx@redhat.com, Avihai Horon , Fabiano Rosas Subject: [PATCH v3 7/7] migration/multifd: Document the reason to sync for save_setup() Date: Fri, 6 Dec 2024 17:47:55 -0500 Message-ID: <20241206224755.1108686-8-peterx@redhat.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241206224755.1108686-1-peterx@redhat.com> References: <20241206224755.1108686-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -50 X-Spam_score: -5.1 X-Spam_bar: ----- X-Spam_report: (-5.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-2.996, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org It's not straightforward to see why src QEMU needs to sync multifd during setup() phase. After all, there's no page queued at that point. For old QEMUs, there's a solid reason: EOS requires it to work. While it's clueless on the new QEMUs which do not take EOS message as sync requests. One will figure that out only when this is conditionally removed. In fact, the author did try it out. Logically we could still avoid doing this on new machine types, however that needs a separate compat field and that can be an overkill in some trivial overhead in setup() phase. Let's instead document it completely, to avoid someone else tries this again and do the debug one more time, or anyone confused on why this ever existed. Signed-off-by: Peter Xu Reviewed-by: Fabiano Rosas --- migration/ram.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 5d4bdefe69..e5c590b259 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3036,6 +3036,31 @@ static int ram_save_setup(QEMUFile *f, void *opaque, Error **errp) migration_ops->ram_save_target_page = ram_save_target_page_legacy; } + /* + * This operation is unfortunate.. + * + * For legacy QEMUs using per-section sync + * ======================================= + * + * This must exist because the EOS below requires the SYNC messages + * per-channel to work. + * + * For modern QEMUs using per-round sync + * ===================================== + * + * Logically such sync is not needed, and recv threads should not run + * until setup ready (using things like channels_ready on src). Then + * we should be all fine. + * + * However even if we add channels_ready to recv side in new QEMUs, old + * QEMU won't have them so this sync will still be needed to make sure + * multifd recv threads won't start processing guest pages early before + * ram_load_setup() is properly done. + * + * Let's stick with this. Fortunately the overhead is low to sync + * during setup because the VM is running, so at least it's not + * accounted as part of downtime. + */ bql_unlock(); ret = multifd_ram_flush_and_sync(f); bql_lock();