From patchwork Wed Oct 16 07:56:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yong Huang X-Patchwork-Id: 13837867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 58ECAD206AC for ; Wed, 16 Oct 2024 07:57:45 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0yu3-0003Yy-K1; Wed, 16 Oct 2024 03:57:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0yu0-0003YR-L9 for qemu-devel@nongnu.org; Wed, 16 Oct 2024 03:57:04 -0400 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1t0yty-0000gr-4R for qemu-devel@nongnu.org; Wed, 16 Oct 2024 03:57:04 -0400 Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-20c767a9c50so51092195ad.1 for ; Wed, 16 Oct 2024 00:57:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1729065420; x=1729670220; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=abHFdh3eAZs73oUQ82v+IFsHjPXHr9LPOZxCIH0Luwo=; b=ze5QdY02Fd6YQAWVtZboXAeZa8Hr3XaHTffIGHHLi8n93b31+SZ28EeCdE7B8DxbvZ pCCFBNCMKxD/sZ4ypQ1QcFWREXqcPDF6kWOaF/DUMD9ZS1sq/64YSwdm74f2e2UbfuFV EB0GBjcWG9Ng0atoSsb7J2OqP0cKXyPPr08AlJxqruhLqEMzIT9sCoFMA+eSpQmMFbVv y4XjnqGCWmd/cAVtW/OvTtFMRvBLfz3xHnsfb5UfMK6gQVEP3cj1TWJcRQ+XyasswlVz 0HIfWqcqQVZ9pS5fje+iHX2FA9wfLW3NnACEmWyfOOS5Z3GfbivNA5xs+SFZ9UlRjOVn r+sQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729065420; x=1729670220; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=abHFdh3eAZs73oUQ82v+IFsHjPXHr9LPOZxCIH0Luwo=; b=LGmn6AKcE+yKCGKX5AYupH1GnfuPYLsaMwaYBDgnbPpJCXQCRaZyQV/6mvsadkgd4i DzTWIplO+uXMPSm47IsE897PkkFLQ835fh+VoZzHZpyMu5+m4/ox9a/ebWk6Ag5jn4yZ 9inTPkFSfSXu4+FKHZESR0q9uN2umRq9NBvQH7bsyyDHNjkRj99XvnHXGM87P+6HvZ2a jFXN9ULem81fmFFsLvBGE6kLGDGdgTy0/YmIT6aFMRYp0VFoXUEcRhXRwG6WnzTA0KPi S8yhg0H1n43PUTT+p1g89HqNCQmlhM0xgyDlosaHCu/ziKoQWwLWKvowsHsKtylSjmP7 rM5w== X-Gm-Message-State: AOJu0Yy68oxRRvErwisoR3zEmWhDp5to2umB+MWEGIXQugsimslWRjCM 1lbZ0OXO2hHJ748b5J5v0ZlbKMFBWAhK5K0L30G/57y4O80CSNd1bfcMSlO2UrU5ekOU0NN17Oi bAz8n3A== X-Google-Smtp-Source: AGHT+IFjVDG9pkIIlJHppr5eog06xHnTaqHgriOd2DYviD4o5nXTGJEwbnjhDUmTMmb4jXEShVxpCw== X-Received: by 2002:a17:902:fb04:b0:20c:9d9e:9049 with SMTP id d9443c01a7336-20cbb1a9196mr166211245ad.22.1729065419608; Wed, 16 Oct 2024 00:56:59 -0700 (PDT) Received: from Hyman-Dev-Euler.zelin.local ([85.208.104.155]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20d17f84dd1sm23608365ad.43.2024.10.16.00.56.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Oct 2024 00:56:59 -0700 (PDT) From: yong.huang@smartx.com To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Richard Henderson , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH v3 1/4] migration: Move cpu-throttole.c from system to migration Date: Wed, 16 Oct 2024 15:56:42 +0800 Message-Id: X-Mailer: git-send-email 2.27.0 In-Reply-To: References: MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::630; envelope-from=yong.huang@smartx.com; helo=mail-pl1-x630.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Hyman Huang Move cpu-throttle.c from system to migration since it's only used for migration; this makes us avoid exporting the util functions and variables in misc.h but export them in migration.h when implementing the background ramblock dirty sync feature in the upcoming commits. Additionally, make the two modifications below: 1. Delay the timer registering of CPU throttle until migration starts since it is only used in migration. 2. Stop CPU throttle if auto converge capability is enabled since it only happens with auto converge. 3. Remove the unused header file reference in accel/tcg/icount-common.c. Signed-off-by: Hyman Huang --- accel/tcg/icount-common.c | 1 - {system => migration}/cpu-throttle.c | 2 +- {include/sysemu => migration}/cpu-throttle.h | 0 migration/meson.build | 1 + migration/migration.c | 11 +++++++++-- migration/ram.c | 2 +- migration/trace-events | 3 +++ system/cpu-timers.c | 3 --- system/meson.build | 1 - system/trace-events | 3 --- 10 files changed, 15 insertions(+), 12 deletions(-) rename {system => migration}/cpu-throttle.c (99%) rename {include/sysemu => migration}/cpu-throttle.h (100%) diff --git a/accel/tcg/icount-common.c b/accel/tcg/icount-common.c index 8d3d3a7e9d..30bf8500dc 100644 --- a/accel/tcg/icount-common.c +++ b/accel/tcg/icount-common.c @@ -36,7 +36,6 @@ #include "sysemu/runstate.h" #include "hw/core/cpu.h" #include "sysemu/cpu-timers.h" -#include "sysemu/cpu-throttle.h" #include "sysemu/cpu-timers-internal.h" /* diff --git a/system/cpu-throttle.c b/migration/cpu-throttle.c similarity index 99% rename from system/cpu-throttle.c rename to migration/cpu-throttle.c index 7632dc6143..fa47ee2e21 100644 --- a/system/cpu-throttle.c +++ b/migration/cpu-throttle.c @@ -27,7 +27,7 @@ #include "hw/core/cpu.h" #include "qemu/main-loop.h" #include "sysemu/cpus.h" -#include "sysemu/cpu-throttle.h" +#include "cpu-throttle.h" #include "trace.h" /* vcpu throttling controls */ diff --git a/include/sysemu/cpu-throttle.h b/migration/cpu-throttle.h similarity index 100% rename from include/sysemu/cpu-throttle.h rename to migration/cpu-throttle.h diff --git a/migration/meson.build b/migration/meson.build index 66d3de86f0..d53cf3417a 100644 --- a/migration/meson.build +++ b/migration/meson.build @@ -13,6 +13,7 @@ system_ss.add(files( 'block-dirty-bitmap.c', 'channel.c', 'channel-block.c', + 'cpu-throttle.c', 'dirtyrate.c', 'exec.c', 'fd.c', diff --git a/migration/migration.c b/migration/migration.c index 021faee2f3..7e71184257 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -24,7 +24,7 @@ #include "socket.h" #include "sysemu/runstate.h" #include "sysemu/sysemu.h" -#include "sysemu/cpu-throttle.h" +#include "cpu-throttle.h" #include "rdma.h" #include "ram.h" #include "migration/global_state.h" @@ -3289,7 +3289,9 @@ static MigIterateState migration_iteration_run(MigrationState *s) static void migration_iteration_finish(MigrationState *s) { /* If we enabled cpu throttling for auto-converge, turn it off. */ - cpu_throttle_stop(); + if (migrate_auto_converge()) { + cpu_throttle_stop(); + } bql_lock(); switch (s->state) { @@ -3508,6 +3510,11 @@ static void *migration_thread(void *opaque) qemu_savevm_send_colo_enable(s->to_dst_file); } + if (migrate_auto_converge()) { + /* Start cpu throttle timers */ + cpu_throttle_init(); + } + bql_lock(); ret = qemu_savevm_state_setup(s->to_dst_file, &local_err); bql_unlock(); diff --git a/migration/ram.c b/migration/ram.c index 326ce7eb79..54d352b152 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -52,7 +52,7 @@ #include "exec/target_page.h" #include "qemu/rcu_queue.h" #include "migration/colo.h" -#include "sysemu/cpu-throttle.h" +#include "cpu-throttle.h" #include "savevm.h" #include "qemu/iov.h" #include "multifd.h" diff --git a/migration/trace-events b/migration/trace-events index c65902f042..9a19599804 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -378,3 +378,6 @@ migration_block_progression(unsigned percent) "Completed %u%%" # page_cache.c migration_pagecache_init(int64_t max_num_items) "Setting cache buckets to %" PRId64 migration_pagecache_insert(void) "Error allocating page" + +# cpu-throttle.c +cpu_throttle_set(int new_throttle_pct) "set guest CPU throttled by %d%%" diff --git a/system/cpu-timers.c b/system/cpu-timers.c index 0b31c9a1b6..856e502e34 100644 --- a/system/cpu-timers.c +++ b/system/cpu-timers.c @@ -35,7 +35,6 @@ #include "sysemu/runstate.h" #include "hw/core/cpu.h" #include "sysemu/cpu-timers.h" -#include "sysemu/cpu-throttle.h" #include "sysemu/cpu-timers-internal.h" /* clock and ticks */ @@ -272,6 +271,4 @@ void cpu_timers_init(void) seqlock_init(&timers_state.vm_clock_seqlock); qemu_spin_init(&timers_state.vm_clock_lock); vmstate_register(NULL, 0, &vmstate_timers, &timers_state); - - cpu_throttle_init(); } diff --git a/system/meson.build b/system/meson.build index a296270cb0..4952f4b2c7 100644 --- a/system/meson.build +++ b/system/meson.build @@ -10,7 +10,6 @@ system_ss.add(files( 'balloon.c', 'bootdevice.c', 'cpus.c', - 'cpu-throttle.c', 'cpu-timers.c', 'datadir.c', 'dirtylimit.c', diff --git a/system/trace-events b/system/trace-events index 074d001e90..2ed1d59b1f 100644 --- a/system/trace-events +++ b/system/trace-events @@ -44,6 +44,3 @@ dirtylimit_state_finalize(void) dirtylimit_throttle_pct(int cpu_index, uint64_t pct, int64_t time_us) "CPU[%d] throttle percent: %" PRIu64 ", throttle adjust time %"PRIi64 " us" dirtylimit_set_vcpu(int cpu_index, uint64_t quota) "CPU[%d] set dirty page rate limit %"PRIu64 dirtylimit_vcpu_execute(int cpu_index, int64_t sleep_time_us) "CPU[%d] sleep %"PRIi64 " us" - -# cpu-throttle.c -cpu_throttle_set(int new_throttle_pct) "set guest CPU throttled by %d%%" From patchwork Wed Oct 16 07:56:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yong Huang X-Patchwork-Id: 13837869 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2C4A4D1AD21 for ; Wed, 16 Oct 2024 07:58:28 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0yu4-0003Zz-RS; Wed, 16 Oct 2024 03:57:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0yu2-0003Yj-3D for qemu-devel@nongnu.org; Wed, 16 Oct 2024 03:57:06 -0400 Received: from mail-pl1-x631.google.com ([2607:f8b0:4864:20::631]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1t0ytz-0000h1-LK for qemu-devel@nongnu.org; Wed, 16 Oct 2024 03:57:05 -0400 Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-20cb7088cbcso35256205ad.0 for ; Wed, 16 Oct 2024 00:57:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1729065422; x=1729670222; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nPCILjRi6VdBAXe5YCsoUa6w4039+pCXbLXnN2JBo9s=; b=DO6G4h9HcTOgg+vP6JXLy2OeJnh0gUAV0crQLaLKH4i/AJsGyTFv9M95ahkOv4bIA/ bZ2KM7BEv8UMfc/1VjJwqEqKvBnmM5IOHXcundf4/uGs+eamSn/vCGyNK5w0Es0yz5BM OHltCM3i5s7nXc2iMTy6iMVDSlPcQsl7SiQh7a3gcD4VKL+RbPGbapptzpuChkM165u1 Y6XoGQfFeDAx3yAje4Fxc3WK9gtp4YkS4QZVUHyL9ufX0DhkJTZTN829N2SokuYfu3x5 OUNqIBP5AUpABGjR1oGLIaXJEnfFHb8iHAW7SIBVV+gn5z6allK3grvO9w/O6OTa9mGh Ii3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729065422; x=1729670222; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nPCILjRi6VdBAXe5YCsoUa6w4039+pCXbLXnN2JBo9s=; b=HVjLL2Dc4XEBXnDwur2mktGoLeAekj2sBYKI7AGP9NyVA6c4B9DQUkkPeTB1WTACXG nNVQfRPiuYNbpvBkJOwlgO5H2qSGydx1I5I8uTcJK6fNHaPfbFN7KRyuYLWtdKyl1xm8 eWzKq1HmCKlmMBNvhZZrk+B5uFapkVEGxEB+H+UkAEwn1pPxXAm5MRkJbmVpWfN5Z70L bjBnoO5wfIDUBF7slbHjFCogYyzFn/aovHFA58yHdxlkDq/ssqeOZV4UAd8gRXian359 7QkYyxWhtf4unksfFRfDBzPgJzOCYkBrte5SLDUIeHhyMPjwtsdAWvFDHZf8F9FrNkgG LjCQ== X-Gm-Message-State: AOJu0Yw1Dx1qjOwr/xgS3KgdVYNAnnXt/KaikF00APgeegih3+ph088l ObHyV6jYqQieFfrBAHTljYdFcL0+mwMVExih4bR6rQ3ummVOCz31RstFtrywdimHhiLeJtzwKwS 1QutQnw== X-Google-Smtp-Source: AGHT+IHxC9QA4Iqd3SqdUdzurOWdLLrjStHMmA4Ci6b9FoCmcFj/a6d5Os89fRePtlGUcCMr6nrf9Q== X-Received: by 2002:a17:902:db12:b0:20c:5ba1:e8e5 with SMTP id d9443c01a7336-20ca145d515mr261043765ad.19.1729065421626; Wed, 16 Oct 2024 00:57:01 -0700 (PDT) Received: from Hyman-Dev-Euler.zelin.local ([85.208.104.155]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20d17f84dd1sm23608365ad.43.2024.10.16.00.56.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Oct 2024 00:57:01 -0700 (PDT) From: yong.huang@smartx.com To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Richard Henderson , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH v3 2/4] migration: Remove "rs" parameter in migration_bitmap_sync_precopy Date: Wed, 16 Oct 2024 15:56:43 +0800 Message-Id: <63b04591ecd6fcd1ffe570e1005fc3040168ad42.1729064919.git.yong.huang@smartx.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: References: MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::631; envelope-from=yong.huang@smartx.com; helo=mail-pl1-x631.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Hyman Huang The global static variable ram_state in fact is referred to by the "rs" parameter in migration_bitmap_sync_precopy. For ease of calling by the callees, use the global variable directly in migration_bitmap_sync_precopy and remove "rs" parameter. The migration_bitmap_sync_precopy will be exported in the next commit. Signed-off-by: Hyman Huang Reviewed-by: Peter Xu --- migration/ram.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 54d352b152..9b5b350405 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1088,9 +1088,10 @@ static void migration_bitmap_sync(RAMState *rs, bool last_stage) } } -static void migration_bitmap_sync_precopy(RAMState *rs, bool last_stage) +static void migration_bitmap_sync_precopy(bool last_stage) { Error *local_err = NULL; + assert(ram_state); /* * The current notifier usage is just an optimization to migration, so we @@ -1101,7 +1102,7 @@ static void migration_bitmap_sync_precopy(RAMState *rs, bool last_stage) local_err = NULL; } - migration_bitmap_sync(rs, last_stage); + migration_bitmap_sync(ram_state, last_stage); if (precopy_notify(PRECOPY_NOTIFY_AFTER_BITMAP_SYNC, &local_err)) { error_report_err(local_err); @@ -2782,7 +2783,7 @@ static bool ram_init_bitmaps(RAMState *rs, Error **errp) if (!ret) { goto out_unlock; } - migration_bitmap_sync_precopy(rs, false); + migration_bitmap_sync_precopy(false); } } out_unlock: @@ -3248,7 +3249,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque) WITH_RCU_READ_LOCK_GUARD() { if (!migration_in_postcopy()) { - migration_bitmap_sync_precopy(rs, true); + migration_bitmap_sync_precopy(true); } ret = rdma_registration_start(f, RAM_CONTROL_FINISH); @@ -3330,7 +3331,7 @@ static void ram_state_pending_exact(void *opaque, uint64_t *must_precopy, if (!migration_in_postcopy()) { bql_lock(); WITH_RCU_READ_LOCK_GUARD() { - migration_bitmap_sync_precopy(rs, false); + migration_bitmap_sync_precopy(false); } bql_unlock(); } From patchwork Wed Oct 16 07:56:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yong Huang X-Patchwork-Id: 13837865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E5D13D20684 for ; Wed, 16 Oct 2024 07:57:30 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0yu9-0003aJ-2v; Wed, 16 Oct 2024 03:57:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0yu3-0003ZQ-JV for qemu-devel@nongnu.org; Wed, 16 Oct 2024 03:57:07 -0400 Received: from mail-pl1-x631.google.com ([2607:f8b0:4864:20::631]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1t0yu1-0000hJ-Sk for qemu-devel@nongnu.org; Wed, 16 Oct 2024 03:57:07 -0400 Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-20c8c50fdd9so4334225ad.0 for ; Wed, 16 Oct 2024 00:57:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1729065424; x=1729670224; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H9gc6WR+7vJ9WeCpYan0Utue6qlnIehIAkut2ksaHrI=; b=MqFy4XAya8KRZGGi1s+aIulfDNqM/aZmhmvf6Jm760CjJsnKzp106lNsySJwi6vHVl RyxJ/d3AH0XTWJmlKHIsBcePp6prsFxZYhZZESS6SIUJahJxGVyaE5SXigFsRWdDgJik 1xfGzw9x6LQSvjYRItH5qoZG+hzRjo2349YZ6+76YVJjpAk4Dznq49tGN1UswP6nxJxC HjaNOZ4tRh3C/IWPBN4nJON9OHLL1O7LNxfVYsjsWpyD21L9l50ND1XgEyF4/svRTOG/ PW5LIws9Ye8z/yKcdQP7b/wRlP9aRJi11Ql8YynLwgB339vnjt8J9gu0BhE52xcLhGg7 CPkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729065424; x=1729670224; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H9gc6WR+7vJ9WeCpYan0Utue6qlnIehIAkut2ksaHrI=; b=FfW3PVgFSmzjZ8OaA3lqyabAq8n73ocJefVXJtokbLNq94GWPT9bSpKRd1H1LE1OA7 Gd9iBtn0CbWMF0CHnTc5KvqwRetu4laC+fADrFJKOd3tdlROAgH6hu32JD0oCxjB45Kf ZrEF2Rk8+UejSoaf0OE7ZF3tqEFOsax+L8wHz19JgC1pR216nu1FAC/ZpfL6m3X80veU ZnLRPwiymj7uDG0osiamW0aa8ckUbUveAeCahlwIMhAAnZc7/m4ngWF4w4X9Kv+B6Gis tk9v3njp+vcofQS1AQcVeT87p7zOldKwF9jaoX0bxhNvNZRTvB1kCbP+Kv5TzO3hf0AQ RdPA== X-Gm-Message-State: AOJu0Ywzzgvj28v4GJqt+kgyYj9yF8mXnmMWIWmxGFt/cbN0m4hGutPh 5OkzyEtgfXYPovVpNdxAvIPixi6X6Ohg7a2vnYzfX7KLLlr5iZHHg8NDMBbcLd/p3L7mReYIALa 1Gcq6Dw== X-Google-Smtp-Source: AGHT+IFqq97Pzjox/ocrz8Jc/hVSVo7D5Fp55Houf3yxYVSoG16MoBaxXDO4IwAavTzUHTRU4UL5Lg== X-Received: by 2002:a17:903:2582:b0:20c:5990:897c with SMTP id d9443c01a7336-20d2708c932mr44244295ad.27.1729065423675; Wed, 16 Oct 2024 00:57:03 -0700 (PDT) Received: from Hyman-Dev-Euler.zelin.local ([85.208.104.155]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20d17f84dd1sm23608365ad.43.2024.10.16.00.57.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Oct 2024 00:57:03 -0700 (PDT) From: yong.huang@smartx.com To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Richard Henderson , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH v3 3/4] migration: Support periodic ramblock dirty sync Date: Wed, 16 Oct 2024 15:56:44 +0800 Message-Id: X-Mailer: git-send-email 2.27.0 In-Reply-To: References: MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::631; envelope-from=yong.huang@smartx.com; helo=mail-pl1-x631.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Hyman Huang When VM is configured with huge memory, the current throttle logic doesn't look like to scale, because migration_trigger_throttle() is only called for each iteration, so it won't be invoked for a long time if one iteration can take a long time. The periodic dirty sync aims to fix the above issue by synchronizing the ramblock from remote dirty bitmap and, when necessary, triggering the CPU throttle multiple times during a long iteration. This is a trade-off between synchronization overhead and CPU throttle impact. Signed-off-by: Hyman Huang --- migration/cpu-throttle.c | 70 +++++++++++++++++++++++++++++++++++++++- migration/cpu-throttle.h | 14 ++++++++ migration/migration.h | 1 + migration/ram.c | 9 ++++-- migration/trace-events | 1 + 5 files changed, 92 insertions(+), 3 deletions(-) diff --git a/migration/cpu-throttle.c b/migration/cpu-throttle.c index fa47ee2e21..784b51ab35 100644 --- a/migration/cpu-throttle.c +++ b/migration/cpu-throttle.c @@ -28,16 +28,23 @@ #include "qemu/main-loop.h" #include "sysemu/cpus.h" #include "cpu-throttle.h" +#include "migration.h" +#include "migration-stats.h" +#include "options.h" #include "trace.h" /* vcpu throttling controls */ -static QEMUTimer *throttle_timer; +static QEMUTimer *throttle_timer, *throttle_dirty_sync_timer; static unsigned int throttle_percentage; +static bool throttle_dirty_sync_timer_active; #define CPU_THROTTLE_PCT_MIN 1 #define CPU_THROTTLE_PCT_MAX 99 #define CPU_THROTTLE_TIMESLICE_NS 10000000 +/* RAMBlock dirty sync trigger every five seconds */ +#define CPU_THROTTLE_DIRTY_SYNC_TIMESLICE_MS 5000 + static void cpu_throttle_thread(CPUState *cpu, run_on_cpu_data opaque) { double pct; @@ -112,6 +119,7 @@ void cpu_throttle_set(int new_throttle_pct) void cpu_throttle_stop(void) { qatomic_set(&throttle_percentage, 0); + cpu_throttle_dirty_sync_timer(false); } bool cpu_throttle_active(void) @@ -124,8 +132,68 @@ int cpu_throttle_get_percentage(void) return qatomic_read(&throttle_percentage); } +void cpu_throttle_dirty_sync_timer_tick(void *opaque) +{ + static uint64_t prev_sync_cnt = 2; + uint64_t sync_cnt = stat64_get(&mig_stats.dirty_sync_count); + + if (!migrate_auto_converge()) { + /* Stop the timer when auto converge is disabled */ + return; + } + + /* + * The first iteration copies all memory anyhow and has no + * effect on guest performance, therefore omit it to avoid + * paying extra for the sync penalty. + */ + if (sync_cnt <= 1) { + goto end; + } + + if (sync_cnt == prev_sync_cnt) { + trace_cpu_throttle_dirty_sync(); + WITH_RCU_READ_LOCK_GUARD() { + migration_bitmap_sync_precopy(false); + } + } + +end: + prev_sync_cnt = stat64_get(&mig_stats.dirty_sync_count); + + timer_mod(throttle_dirty_sync_timer, + qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_RT) + + CPU_THROTTLE_DIRTY_SYNC_TIMESLICE_MS); +} + +static bool cpu_throttle_dirty_sync_active(void) +{ + return qatomic_read(&throttle_dirty_sync_timer_active); +} + +void cpu_throttle_dirty_sync_timer(bool enable) +{ + if (enable) { + assert(throttle_dirty_sync_timer); + if (!cpu_throttle_dirty_sync_active()) { + timer_mod(throttle_dirty_sync_timer, + qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL_RT) + + CPU_THROTTLE_DIRTY_SYNC_TIMESLICE_MS); + qatomic_set(&throttle_dirty_sync_timer_active, 1); + } + } else { + if (throttle_dirty_sync_timer != NULL) { + timer_del(throttle_dirty_sync_timer); + qatomic_set(&throttle_dirty_sync_timer_active, 0); + } + } +} + void cpu_throttle_init(void) { throttle_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL_RT, cpu_throttle_timer_tick, NULL); + throttle_dirty_sync_timer = + timer_new_ms(QEMU_CLOCK_VIRTUAL_RT, + cpu_throttle_dirty_sync_timer_tick, NULL); } diff --git a/migration/cpu-throttle.h b/migration/cpu-throttle.h index d65bdef6d0..420702b8d3 100644 --- a/migration/cpu-throttle.h +++ b/migration/cpu-throttle.h @@ -65,4 +65,18 @@ bool cpu_throttle_active(void); */ int cpu_throttle_get_percentage(void); +/** + * cpu_throttle_dirty_sync_timer_tick: + * + * Dirty sync timer hook. + */ +void cpu_throttle_dirty_sync_timer_tick(void *opaque); + +/** + * cpu_throttle_dirty_sync_timer: + * + * Start or stop the dirty sync timer. + */ +void cpu_throttle_dirty_sync_timer(bool enable); + #endif /* SYSEMU_CPU_THROTTLE_H */ diff --git a/migration/migration.h b/migration/migration.h index 38aa1402d5..fbd0d19092 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -537,4 +537,5 @@ int migration_rp_wait(MigrationState *s); */ void migration_rp_kick(MigrationState *s); +void migration_bitmap_sync_precopy(bool last_stage); #endif diff --git a/migration/ram.c b/migration/ram.c index 9b5b350405..ac34e731e2 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1020,6 +1020,11 @@ static void migration_trigger_throttle(RAMState *rs) migration_transferred_bytes() - rs->bytes_xfer_prev; uint64_t bytes_dirty_period = rs->num_dirty_pages_period * TARGET_PAGE_SIZE; uint64_t bytes_dirty_threshold = bytes_xfer_period * threshold / 100; + bool auto_converge = migrate_auto_converge(); + + if (auto_converge) { + cpu_throttle_dirty_sync_timer(true); + } /* * The following detection logic can be refined later. For now: @@ -1031,7 +1036,7 @@ static void migration_trigger_throttle(RAMState *rs) if ((bytes_dirty_period > bytes_dirty_threshold) && (++rs->dirty_rate_high_cnt >= 2)) { rs->dirty_rate_high_cnt = 0; - if (migrate_auto_converge()) { + if (auto_converge) { trace_migration_throttle(); mig_throttle_guest_down(bytes_dirty_period, bytes_dirty_threshold); @@ -1088,7 +1093,7 @@ static void migration_bitmap_sync(RAMState *rs, bool last_stage) } } -static void migration_bitmap_sync_precopy(bool last_stage) +void migration_bitmap_sync_precopy(bool last_stage) { Error *local_err = NULL; assert(ram_state); diff --git a/migration/trace-events b/migration/trace-events index 9a19599804..0638183056 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -381,3 +381,4 @@ migration_pagecache_insert(void) "Error allocating page" # cpu-throttle.c cpu_throttle_set(int new_throttle_pct) "set guest CPU throttled by %d%%" +cpu_throttle_dirty_sync(void) "" From patchwork Wed Oct 16 07:56:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yong Huang X-Patchwork-Id: 13837868 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5F6CBD1AD23 for ; Wed, 16 Oct 2024 07:58:00 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0yu9-0003b9-5J; Wed, 16 Oct 2024 03:57:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0yu5-0003a1-7b for qemu-devel@nongnu.org; Wed, 16 Oct 2024 03:57:09 -0400 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1t0yu3-0000hW-OD for qemu-devel@nongnu.org; Wed, 16 Oct 2024 03:57:09 -0400 Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-20cf3e36a76so27502685ad.0 for ; Wed, 16 Oct 2024 00:57:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1729065426; x=1729670226; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pcYOvgD8gChlYAF/Rf2Le7Pe4jCTRSW5UuA6AvRkQG8=; b=HlCqMLWwerVg7V5oD4Zma7UFsCX5VyuhevYJv2GqzoKP7MtI/9uzuZyfjicrn+9cWa OXe36AM8hXy2apyBuFTzZdlMpmXS+Gj0lvmJaYAGsj+qNcMx8WpbmL3rchs9pIyCokFt jMoULTUOaeZ92z3+uG6156jT/HY9fNeWH3Zua1b5uSG4qMCTJNSJa+wYdB4/+kY5qzum kMey3SCsQ+ScXQDFAgp46vPx0/QzTr0lbehQp6UzURz79qt7D8onkhneMinx/gVN8/+m ugIKVnUtH8UMLTX1EUJSAEE/N7m8R4Pqyq7WzNCImgeKmYx7fKmkjUBq48FGG2kQ58oA FH+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729065426; x=1729670226; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pcYOvgD8gChlYAF/Rf2Le7Pe4jCTRSW5UuA6AvRkQG8=; b=mu6bUr5IDA7epqK3VXnroxrig0yPbDzbCLS6Ct9DMAkrUe5Xa/YHGF+NUGpGIUb8GJ M/vIO1YeKvOyfo2xrEtET/ceXbqpgsgHANVRI8sVB6wMA71wqIKzF+zGaYUihBkcUuDq XxcRawJuPUt+nN4J8V7mG+iaIB1PSo7DvTuqYdhyejvLnPcfB+ERCiHJZAXcvOgQtDKh KkF/5Csvm/t1R2ewq/wL85kcKz6WZJdiQrSya5wvjA3lPTRaF8tgOOuZzaGntJ4ISGQv MAN2MUNZdevE+fmGtww8Tzj44fBb0JGoJi2mp6vI01jkn1e/hvs4qejMt7ti4IDFNfxW Kufw== X-Gm-Message-State: AOJu0YxWQHtumDQMcwdXl5/xG98T4oQQPQnakcK77eMbaNeJYHESfGof JI7z8LGCTGqhX+5Vzew4IrQwFKIGbHrESQ0NnQxpcWdTlMlvdotoGqtkR61u5RjVO5qdT3Kchei R4WItyA== X-Google-Smtp-Source: AGHT+IG2wjYJnDF1lSnEufnk35yRFrDDMqVotom4wcV/Zill1PNjw9sakq/oCxkKdV6gmoLoxlEm/w== X-Received: by 2002:a17:902:e747:b0:20c:bca5:12a8 with SMTP id d9443c01a7336-20cbca515a5mr158145675ad.60.1729065425667; Wed, 16 Oct 2024 00:57:05 -0700 (PDT) Received: from Hyman-Dev-Euler.zelin.local ([85.208.104.155]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20d17f84dd1sm23608365ad.43.2024.10.16.00.57.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Oct 2024 00:57:05 -0700 (PDT) From: yong.huang@smartx.com To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Richard Henderson , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH v3 4/4] tests/migration: Add case for periodic ramblock dirty sync Date: Wed, 16 Oct 2024 15:56:45 +0800 Message-Id: <4252161b9ec3964330d91a15295abe9fb171145f.1729064919.git.yong.huang@smartx.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: References: MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62c; envelope-from=yong.huang@smartx.com; helo=mail-pl1-x62c.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Hyman Huang Signed-off-by: Hyman Huang Reviewed-by: Peter Xu --- tests/qtest/migration-test.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 95e45b5029..e6a2803e71 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -2791,6 +2791,8 @@ static void test_migrate_auto_converge(void) * so we need to decrease a bandwidth. */ const int64_t init_pct = 5, inc_pct = 25, max_pct = 95; + uint64_t prev_dirty_sync_cnt, dirty_sync_cnt; + int max_try_count, hit = 0; if (test_migrate_start(&from, &to, uri, &args)) { return; @@ -2827,6 +2829,36 @@ static void test_migrate_auto_converge(void) } while (true); /* The first percentage of throttling should be at least init_pct */ g_assert_cmpint(percentage, >=, init_pct); + + /* + * End the loop when the dirty sync count greater than 1. + */ + while ((dirty_sync_cnt = get_migration_pass(from)) < 2) { + usleep(1000 * 1000); + } + + prev_dirty_sync_cnt = dirty_sync_cnt; + + /* + * The RAMBlock dirty sync count must changes in 5 seconds, here we set + * the timeout to 10 seconds to ensure it changes. + * + * Note that migrate_ensure_non_converge set the max-bandwidth to 3MB/s, + * while the qtest mem is >= 100MB, one iteration takes at least 33s (100/3) + * to complete; this ensures that the RAMBlock dirty sync occurs. + */ + max_try_count = 10; + while (--max_try_count) { + dirty_sync_cnt = get_migration_pass(from); + if (dirty_sync_cnt != prev_dirty_sync_cnt) { + hit = 1; + break; + } + prev_dirty_sync_cnt = dirty_sync_cnt; + sleep(1); + } + g_assert_cmpint(hit, ==, 1); + /* Now, when we tested that throttling works, let it converge */ migrate_ensure_converge(from);