From patchwork Mon Sep 9 14:25:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yong Huang X-Patchwork-Id: 13797176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1E658ECE579 for ; Mon, 9 Sep 2024 14:26:05 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snfL6-0000XU-4v; Mon, 09 Sep 2024 10:26:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snfL4-0000Qj-H4 for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:25:58 -0400 Received: from mail-pf1-x435.google.com ([2607:f8b0:4864:20::435]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1snfL2-0004Sn-9R for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:25:58 -0400 Received: by mail-pf1-x435.google.com with SMTP id d2e1a72fcca58-717929b671eso3436447b3a.0 for ; Mon, 09 Sep 2024 07:25:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1725891955; x=1726496755; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=368uvjWdieiurybOR+FslAHDldotj//VpXJQhIwe/k0=; b=2Qt13YCtr1mKLVLc7RBBP2dRtInfUur6QbFCTLpWqraaUI9mCDwE3W9AijLEbgdF0t PeIkMgId1kO9WkohLU/xX9EnERJpOZrsqpdLdhAwo0aNpUMpC04rOHAHK+nijbrrY02M tsVvuZoVtsn6vYsJ+4DdIRiKdqSssAlP+jMAnKgkoAT6UglTKJecfZp/kSUF1BZptAc3 8K5m1xBbxq26u8PruJAxUbBv90i5p00ScNpDB53OU3rysqKkI/8aj2Krf4TfGcwbI/Ua z1lDtSQGND2xb4cegsgWN8epQC2dwkOfFYOknJTX5oNQfA9zP/+Hy/groG6Wg8rhDcsW DJfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725891955; x=1726496755; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=368uvjWdieiurybOR+FslAHDldotj//VpXJQhIwe/k0=; b=TpyupWjcKsJ5aTHUe7KBqBSgCwhREqtRGXrZjsROUatiYvhdK7zuVXM8RhIDDe9+VS r2bnkQJY6FfQfg7+FFZd1hpVcir+uLVdxDvBlsfNKL5QS/CduMa4KZHEXkm+du8a0ZBq n15cK66xmbQALcFXnxag9UCG5jKOmH97Q3ioBY49UF6lpgjaEqPO6X+MDkwkDH+YCsjW 7G6RCDk75rY7McxIupQB0EoGWUQuI7KUX5S5nz7BpxIl+OiUzxW6XJmz9UNjFks+h5Ht Ef33wIlBZAvDQwxG7l6m9YADRb8smMyzS3oGvQDh4D//JejWAm7h8gGLl9Xo6KAP/prJ kNzw== X-Gm-Message-State: AOJu0Yw5jpQn4YQ95yP8yCVDsVGzNWlqeN0GgEny3mqwksaksMOShj59 LUoLHDdcm36C+C5agO05TaLxFkUwxYnaGfnxuDyPlygywYhmQxoSjPf+zTpPdmh40Os+Os38Eoe A71JYYw== X-Google-Smtp-Source: AGHT+IEtWOPpUJwtoPHbtmpNCecPdBne9wxuYxVuf9EjOqqzJPihVoNrW7Xp7CisLEgoaba63/bOyA== X-Received: by 2002:a05:6a00:179a:b0:706:5dab:83c4 with SMTP id d2e1a72fcca58-718d5e5c728mr11910498b3a.14.1725891954142; Mon, 09 Sep 2024 07:25:54 -0700 (PDT) Received: from localhost.localdomain ([118.114.94.226]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-718e58b2a88sm3561164b3a.46.2024.09.09.07.25.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 07:25:53 -0700 (PDT) From: Hyman Huang To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Eric Blake , Markus Armbruster , David Hildenbrand , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH RESEND RFC 01/10] migration: Introduce structs for periodic CPU throttle Date: Mon, 9 Sep 2024 22:25:34 +0800 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::435; envelope-from=yong.huang@smartx.com; helo=mail-pf1-x435.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org shadow_bmap, iter_bmap, iter_dirty_pages and periodic_sync_shown_up are introduced to satisfy the need for periodic CPU throttle. Meanwhile, introduce enumeration of dirty bitmap sync method. Signed-off-by: Hyman Huang --- include/exec/ramblock.h | 45 +++++++++++++++++++++++++++++++++++++++++ migration/ram.c | 6 ++++++ 2 files changed, 51 insertions(+) diff --git a/include/exec/ramblock.h b/include/exec/ramblock.h index 0babd105c0..619c52885a 100644 --- a/include/exec/ramblock.h +++ b/include/exec/ramblock.h @@ -24,6 +24,30 @@ #include "qemu/rcu.h" #include "exec/ramlist.h" +/* Possible bits for migration_bitmap_sync */ + +/* + * The old-fashioned sync method, which is, in turn, used for CPU + * throttle and memory transfer. + */ +#define RAMBLOCK_SYN_LEGACY_ITER (1U << 0) + +/* + * The modern sync method, which is, in turn, used for CPU throttle + * and memory transfer. + */ +#define RAMBLOCK_SYN_MODERN_ITER (1U << 1) + +/* The modern sync method, which is used for CPU throttle only */ +#define RAMBLOCK_SYN_MODERN_PERIOD (1U << 2) + +#define RAMBLOCK_SYN_MASK (0x7) + +typedef enum RAMBlockSynMode { + RAMBLOCK_SYN_LEGACY, /* Old-fashined mode */ + RAMBLOCK_SYN_MODERN, +} RAMBlockSynMode; + struct RAMBlock { struct rcu_head rcu; struct MemoryRegion *mr; @@ -89,6 +113,27 @@ struct RAMBlock { * could not have been valid on the source. */ ram_addr_t postcopy_length; + + /* + * Used to backup the bmap during periodic sync to see whether any dirty + * pages were sent during that time. + */ + unsigned long *shadow_bmap; + + /* + * The bitmap "bmap," which was initially used for both sync and memory + * transfer, will be replaced by two bitmaps: the previously used "bmap" + * and the recently added "iter_bmap." Only the memory transfer is + * conducted with the previously used "bmap"; the recently added + * "iter_bmap" is utilized for sync. + */ + unsigned long *iter_bmap; + + /* Number of new dirty pages during iteration */ + uint64_t iter_dirty_pages; + + /* If periodic sync has shown up during iteration */ + bool periodic_sync_shown_up; }; #endif #endif diff --git a/migration/ram.c b/migration/ram.c index 67ca3d5d51..f29faa82d6 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2362,6 +2362,10 @@ static void ram_bitmaps_destroy(void) block->bmap = NULL; g_free(block->file_bmap); block->file_bmap = NULL; + g_free(block->shadow_bmap); + block->shadow_bmap = NULL; + g_free(block->iter_bmap); + block->iter_bmap = NULL; } } @@ -2753,6 +2757,8 @@ static void ram_list_init_bitmaps(void) } block->clear_bmap_shift = shift; block->clear_bmap = bitmap_new(clear_bmap_size(pages, shift)); + block->shadow_bmap = bitmap_new(pages); + block->iter_bmap = bitmap_new(pages); } } } From patchwork Mon Sep 9 14:25:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yong Huang X-Patchwork-Id: 13797177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 18433ECE579 for ; Mon, 9 Sep 2024 14:26:10 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snfL9-0000os-V5; Mon, 09 Sep 2024 10:26:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snfL8-0000fc-3P for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:02 -0400 Received: from mail-pg1-x534.google.com ([2607:f8b0:4864:20::534]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1snfL6-0004TR-0T for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:01 -0400 Received: by mail-pg1-x534.google.com with SMTP id 41be03b00d2f7-7b8884631c4so2063179a12.2 for ; Mon, 09 Sep 2024 07:25:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1725891958; x=1726496758; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pVk80rEdIzOmBvs0atZRrFtqRDrxfejNxURYXwd1kcM=; b=XkCWhf1c1QSG2/F6I7U7UhCOlEhmETHg00CdzlFZxFuTaJYgUVP8e0ewiRtpOYC+PX MduPwTe8J8iZPoXsWvJGM+45oW0U/k/Q6dZA1qkG4FssFudxqKIPRtjr2iiMJZ7jCxby 6bcFI4xZGpFhpeUQ0jzL2/pqorJZTTVyFYWGEDzrNxxL4oUUj8WC5b5zl+LopGVAimag /bQV6FDddC/QhnbvFXInogf1udUNmhNYOcdo6dhI7UKtQAw2AP1UcSujGjacUraFrWOO OV1PGjUB4NgMk4sBlOmSb4SuKbGFWD9Ix9rFAdE514evAcChfVILn1R9JpsBxf9AdS0X 9D8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725891958; x=1726496758; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pVk80rEdIzOmBvs0atZRrFtqRDrxfejNxURYXwd1kcM=; b=AV89oCU3Mtf5hKepcyu+I1ld3nPQ8ShT/aacesHH9Lb+23qW9hOqQMy5i4S+QRwaXT 1F9YfeR39hw0HswitH0lYQLP+eGz6Yhu0jBWiGcNnUtLlXyGd5FrR9XOaxAMH5ys5gQ/ YvMqvL9BHaMT3nv8PDxp8KuLRSACagQvng5uKk6hJZiUXL792cxOkx2LkfnjKKbhGkgo fppu7IOIQ4Y91phCJ1glEm5boE0S+dsZ4BEAKfo5dt32j5dB9vYFQuWp5sI2AqFD8/5I 09/D/JNf3JOxP9Uk/uCAqvxlKqWgB6Qnye8M4Oeva2KlAuyBQ8JcFE7QOpa2oVioNjNa MeJQ== X-Gm-Message-State: AOJu0YwR11HnrRalKEjeS7hv5zwTjMqsaXDlC4hmn/c1PeboKmIkoerU S5CuCSfaYNXmlIA9g5HtX6Z5xUPUgpjm/zwuqhJIPjI4E3z9Zw/leF+jZxKv9XrdzvAnLWYsOJa jAtqsUw== X-Google-Smtp-Source: AGHT+IH0b9iAa+5uGsRzRkUT07ozgShtleqSF/hJslwjR5dy0x/GT7PDQ5UV25wQpjRaEYmjN1qZMQ== X-Received: by 2002:a05:6a21:2d84:b0:1cf:337e:9920 with SMTP id adf61e73a8af0-1cf337ecfd3mr10061661637.47.1725891957579; Mon, 09 Sep 2024 07:25:57 -0700 (PDT) Received: from localhost.localdomain ([118.114.94.226]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-718e58b2a88sm3561164b3a.46.2024.09.09.07.25.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 07:25:57 -0700 (PDT) From: Hyman Huang To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Eric Blake , Markus Armbruster , David Hildenbrand , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH RESEND RFC 02/10] migration: Refine util functions to support periodic CPU throttle Date: Mon, 9 Sep 2024 22:25:35 +0800 Message-Id: <83f9756eb21e21300ced2bf77fdb3585a918f05e.1725891841.git.yong.huang@smartx.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::534; envelope-from=yong.huang@smartx.com; helo=mail-pg1-x534.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Supply the migration_bitmap_sync function along with a periodic argument. Introduce the sync_mode global variable to track the sync mode and support periodic throttling while keeping backward compatibility. Signed-off-by: Hyman Huang --- include/exec/ram_addr.h | 107 +++++++++++++++++++++++++++++++++++++--- migration/ram.c | 49 ++++++++++++++---- 2 files changed, 140 insertions(+), 16 deletions(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index 891c44cf2d..7df926ed96 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -472,17 +472,68 @@ static inline void cpu_physical_memory_clear_dirty_range(ram_addr_t start, cpu_physical_memory_test_and_clear_dirty(start, length, DIRTY_MEMORY_CODE); } +static void ramblock_clear_iter_bmap(RAMBlock *rb, + ram_addr_t start, + ram_addr_t length) +{ + ram_addr_t addr; + unsigned long *bmap = rb->bmap; + unsigned long *shadow_bmap = rb->shadow_bmap; + unsigned long *iter_bmap = rb->iter_bmap; + + for (addr = 0; addr < length; addr += TARGET_PAGE_SIZE) { + long k = (start + addr) >> TARGET_PAGE_BITS; + if (test_bit(k, shadow_bmap) && !test_bit(k, bmap)) { + /* Page has been sent, clear the iter bmap */ + clear_bit(k, iter_bmap); + } + } +} + +static void ramblock_update_iter_bmap(RAMBlock *rb, + ram_addr_t start, + ram_addr_t length) +{ + ram_addr_t addr; + unsigned long *bmap = rb->bmap; + unsigned long *iter_bmap = rb->iter_bmap; + + for (addr = 0; addr < length; addr += TARGET_PAGE_SIZE) { + long k = (start + addr) >> TARGET_PAGE_BITS; + if (test_bit(k, iter_bmap)) { + if (!test_bit(k, bmap)) { + set_bit(k, bmap); + rb->iter_dirty_pages++; + } + } + } +} /* Called with RCU critical section */ static inline uint64_t cpu_physical_memory_sync_dirty_bitmap(RAMBlock *rb, ram_addr_t start, - ram_addr_t length) + ram_addr_t length, + unsigned int flag) { ram_addr_t addr; unsigned long word = BIT_WORD((start + rb->offset) >> TARGET_PAGE_BITS); uint64_t num_dirty = 0; unsigned long *dest = rb->bmap; + unsigned long *shadow_bmap = rb->shadow_bmap; + unsigned long *iter_bmap = rb->iter_bmap; + + assert(flag && !(flag & (~RAMBLOCK_SYN_MASK))); + + /* + * We must remove the sent dirty page from the iter_bmap in order to + * minimize redundant page transfers if periodic sync has appeared + * during this iteration. + */ + if (rb->periodic_sync_shown_up && + (flag & (RAMBLOCK_SYN_MODERN_ITER | RAMBLOCK_SYN_MODERN_PERIOD))) { + ramblock_clear_iter_bmap(rb, start, length); + } /* start address and length is aligned at the start of a word? */ if (((word * BITS_PER_LONG) << TARGET_PAGE_BITS) == @@ -503,8 +554,20 @@ uint64_t cpu_physical_memory_sync_dirty_bitmap(RAMBlock *rb, if (src[idx][offset]) { unsigned long bits = qatomic_xchg(&src[idx][offset], 0); unsigned long new_dirty; + if (flag & (RAMBLOCK_SYN_MODERN_ITER | + RAMBLOCK_SYN_MODERN_PERIOD)) { + /* Back-up bmap for the next iteration */ + iter_bmap[k] |= bits; + if (flag == RAMBLOCK_SYN_MODERN_PERIOD) { + /* Back-up bmap to detect pages has been sent */ + shadow_bmap[k] = dest[k]; + } + } new_dirty = ~dest[k]; - dest[k] |= bits; + if (flag == RAMBLOCK_SYN_LEGACY_ITER) { + dest[k] |= bits; + } + new_dirty &= bits; num_dirty += ctpopl(new_dirty); } @@ -534,18 +597,50 @@ uint64_t cpu_physical_memory_sync_dirty_bitmap(RAMBlock *rb, ram_addr_t offset = rb->offset; for (addr = 0; addr < length; addr += TARGET_PAGE_SIZE) { - if (cpu_physical_memory_test_and_clear_dirty( + bool dirty = false; + long k = (start + addr) >> TARGET_PAGE_BITS; + if (flag == RAMBLOCK_SYN_MODERN_PERIOD) { + if (test_bit(k, dest)) { + /* Back-up bmap to detect pages has been sent */ + set_bit(k, shadow_bmap); + } + } + + dirty = cpu_physical_memory_test_and_clear_dirty( start + addr + offset, TARGET_PAGE_SIZE, - DIRTY_MEMORY_MIGRATION)) { - long k = (start + addr) >> TARGET_PAGE_BITS; - if (!test_and_set_bit(k, dest)) { + DIRTY_MEMORY_MIGRATION); + + if (flag == RAMBLOCK_SYN_LEGACY_ITER) { + if (dirty && !test_and_set_bit(k, dest)) { num_dirty++; } + } else { + if (dirty) { + if (!test_bit(k, dest)) { + num_dirty++; + } + /* Back-up bmap for the next iteration */ + set_bit(k, iter_bmap); + } } } } + /* + * We have to re-fetch dirty pages from the iter_bmap one by one. + * It's possible that not all of the dirty pages that meant to + * send in the current iteration are included in the bitmap + * that the current sync retrieved from the KVM. + */ + if (flag == RAMBLOCK_SYN_MODERN_ITER) { + ramblock_update_iter_bmap(rb, start, length); + } + + if (flag == RAMBLOCK_SYN_MODERN_PERIOD) { + rb->periodic_sync_shown_up = true; + } + return num_dirty; } #endif diff --git a/migration/ram.c b/migration/ram.c index f29faa82d6..a56634eb46 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -112,6 +112,8 @@ XBZRLECacheStats xbzrle_counters; +static RAMBlockSynMode sync_mode = RAMBLOCK_SYN_LEGACY; + /* used by the search for pages to send */ struct PageSearchStatus { /* The migration channel used for a specific host page */ @@ -912,13 +914,38 @@ bool ramblock_page_is_discarded(RAMBlock *rb, ram_addr_t start) return false; } +static void ramblock_reset_iter_stats(RAMBlock *rb) +{ + bitmap_clear(rb->shadow_bmap, 0, rb->used_length >> TARGET_PAGE_BITS); + bitmap_clear(rb->iter_bmap, 0, rb->used_length >> TARGET_PAGE_BITS); + rb->iter_dirty_pages = 0; + rb->periodic_sync_shown_up = false; +} + /* Called with RCU critical section */ -static void ramblock_sync_dirty_bitmap(RAMState *rs, RAMBlock *rb) +static void ramblock_sync_dirty_bitmap(RAMState *rs, + RAMBlock *rb, + bool periodic) { - uint64_t new_dirty_pages = - cpu_physical_memory_sync_dirty_bitmap(rb, 0, rb->used_length); + uint64_t new_dirty_pages; + unsigned int flag = RAMBLOCK_SYN_LEGACY_ITER; + + if (sync_mode == RAMBLOCK_SYN_MODERN) { + flag = periodic ? RAMBLOCK_SYN_MODERN_PERIOD : RAMBLOCK_SYN_MODERN_ITER; + } + + new_dirty_pages = + cpu_physical_memory_sync_dirty_bitmap(rb, 0, rb->used_length, flag); + + if (flag & (RAMBLOCK_SYN_LEGACY_ITER | RAMBLOCK_SYN_MODERN_ITER)) { + if (flag == RAMBLOCK_SYN_LEGACY_ITER) { + rs->migration_dirty_pages += new_dirty_pages; + } else { + rs->migration_dirty_pages += rb->iter_dirty_pages; + ramblock_reset_iter_stats(rb); + } + } - rs->migration_dirty_pages += new_dirty_pages; rs->num_dirty_pages_period += new_dirty_pages; } @@ -1041,7 +1068,9 @@ static void migration_trigger_throttle(RAMState *rs) } } -static void migration_bitmap_sync(RAMState *rs, bool last_stage) +static void migration_bitmap_sync(RAMState *rs, + bool last_stage, + bool periodic) { RAMBlock *block; int64_t end_time; @@ -1058,7 +1087,7 @@ static void migration_bitmap_sync(RAMState *rs, bool last_stage) WITH_QEMU_LOCK_GUARD(&rs->bitmap_mutex) { WITH_RCU_READ_LOCK_GUARD() { RAMBLOCK_FOREACH_NOT_IGNORED(block) { - ramblock_sync_dirty_bitmap(rs, block); + ramblock_sync_dirty_bitmap(rs, block, periodic); } stat64_set(&mig_stats.dirty_bytes_last_sync, ram_bytes_remaining()); } @@ -1101,7 +1130,7 @@ static void migration_bitmap_sync_precopy(RAMState *rs, bool last_stage) local_err = NULL; } - migration_bitmap_sync(rs, last_stage); + migration_bitmap_sync(rs, last_stage, false); if (precopy_notify(PRECOPY_NOTIFY_AFTER_BITMAP_SYNC, &local_err)) { error_report_err(local_err); @@ -2594,7 +2623,7 @@ void ram_postcopy_send_discard_bitmap(MigrationState *ms) RCU_READ_LOCK_GUARD(); /* This should be our last sync, the src is now paused */ - migration_bitmap_sync(rs, false); + migration_bitmap_sync(rs, false, false); /* Easiest way to make sure we don't resume in the middle of a host-page */ rs->pss[RAM_CHANNEL_PRECOPY].last_sent_block = NULL; @@ -3581,7 +3610,7 @@ void colo_incoming_start_dirty_log(void) memory_global_dirty_log_sync(false); WITH_RCU_READ_LOCK_GUARD() { RAMBLOCK_FOREACH_NOT_IGNORED(block) { - ramblock_sync_dirty_bitmap(ram_state, block); + ramblock_sync_dirty_bitmap(ram_state, block, false); /* Discard this dirty bitmap record */ bitmap_zero(block->bmap, block->max_length >> TARGET_PAGE_BITS); } @@ -3862,7 +3891,7 @@ void colo_flush_ram_cache(void) qemu_mutex_lock(&ram_state->bitmap_mutex); WITH_RCU_READ_LOCK_GUARD() { RAMBLOCK_FOREACH_NOT_IGNORED(block) { - ramblock_sync_dirty_bitmap(ram_state, block); + ramblock_sync_dirty_bitmap(ram_state, block, false); } } From patchwork Mon Sep 9 14:25:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yong Huang X-Patchwork-Id: 13797183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BA83AECE57A for ; Mon, 9 Sep 2024 14:27:27 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snfLD-00018R-RL; Mon, 09 Sep 2024 10:26:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snfLC-00011H-10 for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:06 -0400 Received: from mail-pg1-x534.google.com ([2607:f8b0:4864:20::534]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1snfLA-0004Tq-1H for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:05 -0400 Received: by mail-pg1-x534.google.com with SMTP id 41be03b00d2f7-7cd8d2731d1so2959409a12.3 for ; Mon, 09 Sep 2024 07:26:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1725891962; x=1726496762; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sLPpduPp6Uoo3uoV/VinR4B0oEyxm97ZfJYte9O4FBw=; b=oDBxL4LjzODSBDdYoQLCrUfP0ioGD3R4iLTJUCLRIzwL+/wC5Z2bpeixePzWEbbxcA QZGBGgiop8DiYqcvdHRJn2u4sMCO6V8rkYppg8RqM6EC6aukVEMWVxp0jGBJyh/paCXG duv7g2LKOG9Lr+33uStaspd7MGF77yxfDa3eJZPWYEHE9Zf0/Ma49Ico5++m9+B3Tkz9 tunZPN49zvAfZ9C2Dk+OquACAhRhDqP+OIMluw5LiUIUAhJfNLXt/DKDLx3M96qYyyOj 4wuJGufFP4P2EdDS+gpzZd82k+CksqykC6sQIfUMCJUIdr04uZ+ofz944wUqhW43sbcC Pgjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725891962; x=1726496762; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sLPpduPp6Uoo3uoV/VinR4B0oEyxm97ZfJYte9O4FBw=; b=QvNau0kIT4k48209J0FPOg2cGRShCEfzUyWr9WOZxDZA1WXYrOP9coppM6+JLLCxpQ O2S6Y1tum2QXS5MVEhI7kmO6jQrZ3TsVsOVzDWnzU9oTQOEYEX2wOqF22iPgTwGhcVbB 4ToWxsnckyN3bp794VpBxEN7qhuun2IsZ52O215dBCZ2C/wEjFWj0EKyemfVvQzCJBpZ S6cd79iq+fnyx6K6RUtaYu3PokJ4KewEjs5WKhAmiO3WmhGLN5ZtePD1L81+YcmVPnJs 3nbGMKCUTRfDmU+gmgbxdGqlZXclXpoCYqFBRaJmrK7SXcHnOiqYEuN0JHXRexoN5fZQ YkPA== X-Gm-Message-State: AOJu0Yxa9JXgD6F9PHhgJQbuxz23wS2AKhwR7kfB9/skvWu09mbhaKWA qYThk/Ip3lACskPvIvL0r0JGHDhFwrbYHx9SUiLXWNSc2RWdsZgpfOaKHfz/UId5jQgbwL+wMRZ 46dmh+g== X-Google-Smtp-Source: AGHT+IHQjcSyCU3TNl5IMUggJXSSC14PnBXkU4skGSVoyXB2YrF93iI4EvKwqafNN3iXCCXRwU1f9A== X-Received: by 2002:a05:6a21:6e4a:b0:1cf:440a:d449 with SMTP id adf61e73a8af0-1cf440ad611mr1789675637.42.1725891961885; Mon, 09 Sep 2024 07:26:01 -0700 (PDT) Received: from localhost.localdomain ([118.114.94.226]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-718e58b2a88sm3561164b3a.46.2024.09.09.07.25.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 07:26:01 -0700 (PDT) From: Hyman Huang To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Eric Blake , Markus Armbruster , David Hildenbrand , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH RESEND RFC 03/10] qapi/migration: Introduce periodic CPU throttling parameters Date: Mon, 9 Sep 2024 22:25:36 +0800 Message-Id: <0bbcdfd86f35830e0a398220663aac5afd8b7e1e.1725891841.git.yong.huang@smartx.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::534; envelope-from=yong.huang@smartx.com; helo=mail-pg1-x534.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org To activate the periodic CPU throttleing feature, introduce the cpu-periodic-throttle. To control the frequency of throttling, introduce the cpu-periodic-throttle-interval. Signed-off-by: Hyman Huang --- migration/migration-hmp-cmds.c | 17 +++++++++++ migration/options.c | 54 ++++++++++++++++++++++++++++++++++ migration/options.h | 2 ++ qapi/migration.json | 25 +++++++++++++++- 4 files changed, 97 insertions(+), 1 deletion(-) diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index 7d608d26e1..f7b8e06bb4 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -264,6 +264,15 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) monitor_printf(mon, "%s: %s\n", MigrationParameter_str(MIGRATION_PARAMETER_CPU_THROTTLE_TAILSLOW), params->cpu_throttle_tailslow ? "on" : "off"); + assert(params->has_cpu_periodic_throttle); + monitor_printf(mon, "%s: %s\n", + MigrationParameter_str(MIGRATION_PARAMETER_CPU_PERIODIC_THROTTLE), + params->cpu_periodic_throttle ? "on" : "off"); + assert(params->has_cpu_periodic_throttle_interval); + monitor_printf(mon, "%s: %u\n", + MigrationParameter_str( + MIGRATION_PARAMETER_CPU_PERIODIC_THROTTLE_INTERVAL), + params->cpu_periodic_throttle_interval); assert(params->has_max_cpu_throttle); monitor_printf(mon, "%s: %u\n", MigrationParameter_str(MIGRATION_PARAMETER_MAX_CPU_THROTTLE), @@ -512,6 +521,14 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_cpu_throttle_tailslow = true; visit_type_bool(v, param, &p->cpu_throttle_tailslow, &err); break; + case MIGRATION_PARAMETER_CPU_PERIODIC_THROTTLE: + p->has_cpu_periodic_throttle = true; + visit_type_bool(v, param, &p->cpu_periodic_throttle, &err); + break; + case MIGRATION_PARAMETER_CPU_PERIODIC_THROTTLE_INTERVAL: + p->has_cpu_periodic_throttle_interval = true; + visit_type_uint8(v, param, &p->cpu_periodic_throttle_interval, &err); + break; case MIGRATION_PARAMETER_MAX_CPU_THROTTLE: p->has_max_cpu_throttle = true; visit_type_uint8(v, param, &p->max_cpu_throttle, &err); diff --git a/migration/options.c b/migration/options.c index 645f55003d..2dbe275ba0 100644 --- a/migration/options.c +++ b/migration/options.c @@ -44,6 +44,7 @@ #define DEFAULT_MIGRATE_THROTTLE_TRIGGER_THRESHOLD 50 #define DEFAULT_MIGRATE_CPU_THROTTLE_INITIAL 20 #define DEFAULT_MIGRATE_CPU_THROTTLE_INCREMENT 10 +#define DEFAULT_MIGRATE_CPU_PERIODIC_THROTTLE_INTERVAL 5 #define DEFAULT_MIGRATE_MAX_CPU_THROTTLE 99 /* Migration XBZRLE default cache size */ @@ -104,6 +105,11 @@ Property migration_properties[] = { DEFAULT_MIGRATE_CPU_THROTTLE_INCREMENT), DEFINE_PROP_BOOL("x-cpu-throttle-tailslow", MigrationState, parameters.cpu_throttle_tailslow, false), + DEFINE_PROP_BOOL("x-cpu-periodic-throttle", MigrationState, + parameters.cpu_periodic_throttle, false), + DEFINE_PROP_UINT8("x-cpu-periodic-throttle-interval", MigrationState, + parameters.cpu_periodic_throttle_interval, + DEFAULT_MIGRATE_CPU_PERIODIC_THROTTLE_INTERVAL), DEFINE_PROP_SIZE("x-max-bandwidth", MigrationState, parameters.max_bandwidth, MAX_THROTTLE), DEFINE_PROP_SIZE("avail-switchover-bandwidth", MigrationState, @@ -695,6 +701,20 @@ uint8_t migrate_cpu_throttle_initial(void) return s->parameters.cpu_throttle_initial; } +uint8_t migrate_periodic_throttle_interval(void) +{ + MigrationState *s = migrate_get_current(); + + return s->parameters.cpu_periodic_throttle_interval; +} + +bool migrate_periodic_throttle(void) +{ + MigrationState *s = migrate_get_current(); + + return s->parameters.cpu_periodic_throttle; +} + bool migrate_cpu_throttle_tailslow(void) { MigrationState *s = migrate_get_current(); @@ -874,6 +894,11 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->cpu_throttle_increment = s->parameters.cpu_throttle_increment; params->has_cpu_throttle_tailslow = true; params->cpu_throttle_tailslow = s->parameters.cpu_throttle_tailslow; + params->has_cpu_periodic_throttle = true; + params->cpu_periodic_throttle = s->parameters.cpu_periodic_throttle; + params->has_cpu_periodic_throttle_interval = true; + params->cpu_periodic_throttle_interval = + s->parameters.cpu_periodic_throttle_interval; params->tls_creds = g_strdup(s->parameters.tls_creds); params->tls_hostname = g_strdup(s->parameters.tls_hostname); params->tls_authz = g_strdup(s->parameters.tls_authz ? @@ -940,6 +965,8 @@ void migrate_params_init(MigrationParameters *params) params->has_cpu_throttle_initial = true; params->has_cpu_throttle_increment = true; params->has_cpu_throttle_tailslow = true; + params->has_cpu_periodic_throttle = true; + params->has_cpu_periodic_throttle_interval = true; params->has_max_bandwidth = true; params->has_downtime_limit = true; params->has_x_checkpoint_delay = true; @@ -996,6 +1023,15 @@ bool migrate_params_check(MigrationParameters *params, Error **errp) return false; } + if (params->has_cpu_periodic_throttle_interval && + (params->cpu_periodic_throttle_interval < 2 || + params->cpu_periodic_throttle_interval > 10)) { + error_setg(errp, QERR_INVALID_PARAMETER_VALUE, + "cpu_periodic_throttle_interval", + "an integer in the range of 2 to 10"); + return false; + } + if (params->has_max_bandwidth && (params->max_bandwidth > SIZE_MAX)) { error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "max_bandwidth", @@ -1163,6 +1199,15 @@ static void migrate_params_test_apply(MigrateSetParameters *params, dest->cpu_throttle_tailslow = params->cpu_throttle_tailslow; } + if (params->has_cpu_periodic_throttle) { + dest->cpu_periodic_throttle = params->cpu_periodic_throttle; + } + + if (params->has_cpu_periodic_throttle_interval) { + dest->cpu_periodic_throttle_interval = + params->cpu_periodic_throttle_interval; + } + if (params->tls_creds) { assert(params->tls_creds->type == QTYPE_QSTRING); dest->tls_creds = params->tls_creds->u.s; @@ -1271,6 +1316,15 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp) s->parameters.cpu_throttle_tailslow = params->cpu_throttle_tailslow; } + if (params->has_cpu_periodic_throttle) { + s->parameters.cpu_periodic_throttle = params->cpu_periodic_throttle; + } + + if (params->has_cpu_periodic_throttle_interval) { + s->parameters.cpu_periodic_throttle_interval = + params->cpu_periodic_throttle_interval; + } + if (params->tls_creds) { g_free(s->parameters.tls_creds); assert(params->tls_creds->type == QTYPE_QSTRING); diff --git a/migration/options.h b/migration/options.h index a2397026db..efeac01470 100644 --- a/migration/options.h +++ b/migration/options.h @@ -68,6 +68,8 @@ bool migrate_has_block_bitmap_mapping(void); uint32_t migrate_checkpoint_delay(void); uint8_t migrate_cpu_throttle_increment(void); uint8_t migrate_cpu_throttle_initial(void); +uint8_t migrate_periodic_throttle_interval(void); +bool migrate_periodic_throttle(void); bool migrate_cpu_throttle_tailslow(void); bool migrate_direct_io(void); uint64_t migrate_downtime_limit(void); diff --git a/qapi/migration.json b/qapi/migration.json index 7324571e92..8281d4a83b 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -724,6 +724,12 @@ # be excessive at tail stage. The default value is false. (Since # 5.1) # +# @cpu-periodic-throttle: Make CPU throttling periodically. +# (Since 9.1) +# +# @cpu-periodic-throttle-interval: Interval of the periodic CPU throttling. +# (Since 9.1) +# # @tls-creds: ID of the 'tls-creds' object that provides credentials # for establishing a TLS connection over the migration data # channel. On the outgoing side of the migration, the credentials @@ -844,7 +850,8 @@ 'announce-rounds', 'announce-step', 'throttle-trigger-threshold', 'cpu-throttle-initial', 'cpu-throttle-increment', - 'cpu-throttle-tailslow', + 'cpu-throttle-tailslow', 'cpu-periodic-throttle', + 'cpu-periodic-throttle-interval', 'tls-creds', 'tls-hostname', 'tls-authz', 'max-bandwidth', 'avail-switchover-bandwidth', 'downtime-limit', { 'name': 'x-checkpoint-delay', 'features': [ 'unstable' ] }, @@ -899,6 +906,12 @@ # be excessive at tail stage. The default value is false. (Since # 5.1) # +# @cpu-periodic-throttle: Make CPU throttling periodically. +# (Since 9.1) +# +# @cpu-periodic-throttle-interval: Interval of the periodic CPU throttling. +# (Since 9.1) +# # @tls-creds: ID of the 'tls-creds' object that provides credentials # for establishing a TLS connection over the migration data # channel. On the outgoing side of the migration, the credentials @@ -1026,6 +1039,8 @@ '*cpu-throttle-initial': 'uint8', '*cpu-throttle-increment': 'uint8', '*cpu-throttle-tailslow': 'bool', + '*cpu-periodic-throttle': 'bool', + '*cpu-periodic-throttle-interval': 'uint8', '*tls-creds': 'StrOrNull', '*tls-hostname': 'StrOrNull', '*tls-authz': 'StrOrNull', @@ -1107,6 +1122,12 @@ # be excessive at tail stage. The default value is false. (Since # 5.1) # +# @cpu-periodic-throttle: Make CPU throttling periodically. +# (Since 9.1) +# +# @cpu-periodic-throttle-interval: Interval of the periodic CPU throttling. +# (Since 9.1) +# # @tls-creds: ID of the 'tls-creds' object that provides credentials # for establishing a TLS connection over the migration data # channel. On the outgoing side of the migration, the credentials @@ -1227,6 +1248,8 @@ '*cpu-throttle-initial': 'uint8', '*cpu-throttle-increment': 'uint8', '*cpu-throttle-tailslow': 'bool', + '*cpu-periodic-throttle': 'bool', + '*cpu-periodic-throttle-interval': 'uint8', '*tls-creds': 'str', '*tls-hostname': 'str', '*tls-authz': 'str', From patchwork Mon Sep 9 14:25:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yong Huang X-Patchwork-Id: 13797182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DAA4BECE57E for ; Mon, 9 Sep 2024 14:27:27 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snfLG-0001M8-PX; Mon, 09 Sep 2024 10:26:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snfLF-0001Fc-DO for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:09 -0400 Received: from mail-oi1-x22f.google.com ([2607:f8b0:4864:20::22f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1snfLD-0004U1-MC for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:09 -0400 Received: by mail-oi1-x22f.google.com with SMTP id 5614622812f47-3e033e66a31so1521596b6e.0 for ; Mon, 09 Sep 2024 07:26:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1725891966; x=1726496766; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A4+6JdQYvda/1Hm0Qzqic1h3nzIQOs1z0LtM+uvLl5s=; b=uhnqUVhsAthcEiwJUVBVWETejxaOcKdm/BU33CZJzDMZcpohdMmpwamR9CziX6twRb bZj1evOzyXB1w9EJbIazl3TSHDWKZ4VPKcDkhqd79Jovlra4RxfgjWxJI+LxJD+2NYNm X+WEb9zRDdxMgQYg4krn2J6PtaI24vktYUi5s/zAFxFrzbWO8LQfSjj6MgceidchdbWr nNnuudeskXIODWaKZU0fRV7dBMa6QTycekAU8PjpemRuwKAd9dKJCpc9DCVElhb9hACu DPsnnHQbivVa5s8WTk69wSbKgRxsy9pb4bXqC3ARBVu0gt6bQS6Fmi9FZnn5udUBaxIW 4jQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725891966; x=1726496766; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A4+6JdQYvda/1Hm0Qzqic1h3nzIQOs1z0LtM+uvLl5s=; b=ar3ppsClkDErwg+I1eR7uQB/cjYjHQslr/vY+2Uiu2MZO0QDeNrtzH3s/hDa8W5f5Z dp+BArZJ9n2X3YhJnPpuD6AbIRRTO4QSB4Ayhs12Mrd/033KpsOvJtOaHLorfbed3P10 MJKELAbGRZnU0cDhmBAcBL3/4S70EV+L44rizkw+49VdgtunOqZZkmRKgEu8awAOGG7t C+33KpB2k48g3V6+OJwfNM27tuaKm6pMWjfvD3XQQ5vJH8VJV6axClbyUvaeBcSlcsaw rNbJlk/6okGrD9to5qS/pDhThbE/T6BFNCqzNqx+xJyNQDwNHk7rQoLy3XpTVf+sqp10 nQ1g== X-Gm-Message-State: AOJu0YwfTbGvharxwnmYe5U7DDLUTrzI+HQOhuBNv3S+Az6uS2lNMg6r +2uMTZwofvMOsynpm2Y8NRscIdX3kSi21WnWDp/Y+f/YeSfiNalneUSWITk/azb76gw0V7xiY37 GTf0vlQ== X-Google-Smtp-Source: AGHT+IH76pZoBbETP32Ke1fH3vmV1055l/ejb7q4fQ43imhs6kNFFnvibA9y9NCOnvzO5+Luij3Fow== X-Received: by 2002:a05:6870:9f08:b0:277:e641:5659 with SMTP id 586e51a60fabf-27b9daecd2cmr3397826fac.35.1725891965569; Mon, 09 Sep 2024 07:26:05 -0700 (PDT) Received: from localhost.localdomain ([118.114.94.226]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-718e58b2a88sm3561164b3a.46.2024.09.09.07.26.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 07:26:05 -0700 (PDT) From: Hyman Huang To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Eric Blake , Markus Armbruster , David Hildenbrand , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH RESEND RFC 04/10] qapi/migration: Introduce the iteration-count Date: Mon, 9 Sep 2024 22:25:37 +0800 Message-Id: <9f16a57bd430fd99807a57454780c7e0a90e739c.1725891841.git.yong.huang@smartx.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::22f; envelope-from=yong.huang@smartx.com; helo=mail-oi1-x22f.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The original migration information dirty-sync-count could no longer reflect iteration count due to the introduction of periodic synchronization in the next commit; add the iteration count to compensate. Signed-off-by: Hyman Huang --- migration/migration-stats.h | 4 ++++ migration/migration.c | 1 + migration/ram.c | 12 ++++++++---- qapi/migration.json | 6 +++++- tests/qtest/migration-test.c | 2 +- 5 files changed, 19 insertions(+), 6 deletions(-) diff --git a/migration/migration-stats.h b/migration/migration-stats.h index 05290ade76..43ee0f4f05 100644 --- a/migration/migration-stats.h +++ b/migration/migration-stats.h @@ -50,6 +50,10 @@ typedef struct { * Number of times we have synchronized guest bitmaps. */ Stat64 dirty_sync_count; + /* + * Number of migration iteration processed. + */ + Stat64 iteration_count; /* * Number of times zero copy failed to send any page using zero * copy. diff --git a/migration/migration.c b/migration/migration.c index 3dea06d577..055d527ff6 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1197,6 +1197,7 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s) info->ram->mbps = s->mbps; info->ram->dirty_sync_count = stat64_get(&mig_stats.dirty_sync_count); + info->ram->iteration_count = stat64_get(&mig_stats.iteration_count); info->ram->dirty_sync_missed_zero_copy = stat64_get(&mig_stats.dirty_sync_missed_zero_copy); info->ram->postcopy_requests = diff --git a/migration/ram.c b/migration/ram.c index a56634eb46..23471c9e5a 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -594,7 +594,7 @@ static void xbzrle_cache_zero_page(ram_addr_t current_addr) /* We don't care if this fails to allocate a new cache page * as long as it updated an old one */ cache_insert(XBZRLE.cache, current_addr, XBZRLE.zero_target_page, - stat64_get(&mig_stats.dirty_sync_count)); + stat64_get(&mig_stats.iteration_count)); } #define ENCODING_FLAG_XBZRLE 0x1 @@ -620,7 +620,7 @@ static int save_xbzrle_page(RAMState *rs, PageSearchStatus *pss, int encoded_len = 0, bytes_xbzrle; uint8_t *prev_cached_page; QEMUFile *file = pss->pss_channel; - uint64_t generation = stat64_get(&mig_stats.dirty_sync_count); + uint64_t generation = stat64_get(&mig_stats.iteration_count); if (!cache_is_cached(XBZRLE.cache, current_addr, generation)) { xbzrle_counters.cache_miss++; @@ -1075,6 +1075,10 @@ static void migration_bitmap_sync(RAMState *rs, RAMBlock *block; int64_t end_time; + if (!periodic) { + stat64_add(&mig_stats.iteration_count, 1); + } + stat64_add(&mig_stats.dirty_sync_count, 1); if (!rs->time_last_bitmap_sync) { @@ -1111,8 +1115,8 @@ static void migration_bitmap_sync(RAMState *rs, rs->num_dirty_pages_period = 0; rs->bytes_xfer_prev = migration_transferred_bytes(); } - if (migrate_events()) { - uint64_t generation = stat64_get(&mig_stats.dirty_sync_count); + if (!periodic && migrate_events()) { + uint64_t generation = stat64_get(&mig_stats.iteration_count); qapi_event_send_migration_pass(generation); } } diff --git a/qapi/migration.json b/qapi/migration.json index 8281d4a83b..6d8358c202 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -60,6 +60,9 @@ # between 0 and @dirty-sync-count * @multifd-channels. (since # 7.1) # +# @iteration-count: The number of iterations since migration started. +# (since 9.2) +# # Since: 0.14 ## { 'struct': 'MigrationStats', @@ -72,7 +75,8 @@ 'multifd-bytes': 'uint64', 'pages-per-second': 'uint64', 'precopy-bytes': 'uint64', 'downtime-bytes': 'uint64', 'postcopy-bytes': 'uint64', - 'dirty-sync-missed-zero-copy': 'uint64' } } + 'dirty-sync-missed-zero-copy': 'uint64', + 'iteration-count' : 'int' } } ## # @XBZRLECacheStats: diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 9d08101643..2fb10658d4 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -278,7 +278,7 @@ static int64_t read_migrate_property_int(QTestState *who, const char *property) static uint64_t get_migration_pass(QTestState *who) { - return read_ram_property_int(who, "dirty-sync-count"); + return read_ram_property_int(who, "iteration-count"); } static void read_blocktime(QTestState *who) From patchwork Mon Sep 9 14:25:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yong Huang X-Patchwork-Id: 13797178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 92457ECE579 for ; Mon, 9 Sep 2024 14:26:49 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snfLM-0001la-N8; Mon, 09 Sep 2024 10:26:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snfLK-0001eP-S4 for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:14 -0400 Received: from mail-oo1-xc2c.google.com ([2607:f8b0:4864:20::c2c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1snfLH-0004UB-Bl for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:13 -0400 Received: by mail-oo1-xc2c.google.com with SMTP id 006d021491bc7-5dc93fa5639so2879682eaf.1 for ; Mon, 09 Sep 2024 07:26:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1725891970; x=1726496770; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aPNIJP+CkLZPeSwpwgFasxHEV3wc4ZaUYexML1Q2BSA=; b=nndygecnZpNB50Rj95YihLhkJ2yXjVZ+8WGP5YiXQQdM3pDmDvvtmfgywwMWhuSZMf ynyKMnCB538AaxYm9Sf03dThai1/JDdBRgdwaRJwmFQ+fZk232aJtmtoleojia2JprJs xiuK7x8a5VStcWdWOfTaxRJdyHpUGxjEWCyUTAnk5Y4qQuhB/pw/Le8vkE4ZFmGxLMMk mszRS63Z3ONFeLHUBgwPKJE59BJrANsYgIIRhT1RF5T4rADZzgAZNW9nzpgiJ3YsKkeX gXE4E0oUvAAY3HoPpfgrMgZlzuugG+/DYUpVWykjg87eXgN1rbZKiHbdAUUnSlc0EuuU X1ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725891970; x=1726496770; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aPNIJP+CkLZPeSwpwgFasxHEV3wc4ZaUYexML1Q2BSA=; b=veDnLt1iQsu5gjfL9f3sTqlO1d1d0RlLKmAKYbO6iCGQeLDLE5RU/E3cievCdRXkoD 2SZ0uGCgN9MPl1KF1Blj2y5BNSlzf7sMrsLClO0mQmnDZSoOnltiSgk5PD8bM4wE19Hi lIVkjJVr5V6U2BlEBwB31ISPuJQux2BWTlxDPqUR0vT3AatN61MFQlcjdIdAvxMf1wo4 M2dAyFmTgBnmA2MSQ064FhwDHbEvrSBTDYxBoBcvhNbKrq9ImrmgcjTm+i6hXDODdX7Z VXImNw8z6kv2ib+wTAXOaegmk6MxQwi84FXMdjBH4GFTnvx1x5RpykvfWiMRrPbCs0JV oczw== X-Gm-Message-State: AOJu0YzoCFR/7HgN+J982+s+TAys+gah3H0I1GLKNhxi6yv51bbXAO3N 9jaBimfHplnI2LaTyr8rZd9/bw2mdvQE0km8FxhiuKtprO1OCYSB5M1YEKr7VbrfhMTv9OkFo4/ yicK8Sg== X-Google-Smtp-Source: AGHT+IEkrl+X9ifbamlulLDf67X9wSQX0DpcdQ4pAKb5XgSdoM7LmyAxtVILMUIEMoR/Bg7JbPwL+w== X-Received: by 2002:a05:6870:961a:b0:270:1295:af54 with SMTP id 586e51a60fabf-27b82ee80a9mr13207260fac.28.1725891969248; Mon, 09 Sep 2024 07:26:09 -0700 (PDT) Received: from localhost.localdomain ([118.114.94.226]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-718e58b2a88sm3561164b3a.46.2024.09.09.07.26.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 07:26:08 -0700 (PDT) From: Hyman Huang To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Eric Blake , Markus Armbruster , David Hildenbrand , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH RESEND RFC 05/10] migration: Introduce util functions for periodic CPU throttle Date: Mon, 9 Sep 2024 22:25:38 +0800 Message-Id: <26d4e848d67b14ebeb9a3e0667688056a65df4b4.1725891841.git.yong.huang@smartx.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::c2c; envelope-from=yong.huang@smartx.com; helo=mail-oo1-xc2c.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Provide useful utilities to manage the periodic_throttle_thread's lifespan. Additionally, to set up sync mode, provide periodic_throttle_setup. Signed-off-by: Hyman Huang --- migration/ram.c | 98 +++++++++++++++++++++++++++++++++++++++++- migration/ram.h | 4 ++ migration/trace-events | 3 ++ 3 files changed, 104 insertions(+), 1 deletion(-) diff --git a/migration/ram.c b/migration/ram.c index 23471c9e5a..d9d8ed0fda 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -416,6 +416,10 @@ struct RAMState { * RAM migration. */ unsigned int postcopy_bmap_sync_requested; + + /* Periodic throttle information */ + bool throttle_running; + QemuThread throttle_thread; }; typedef struct RAMState RAMState; @@ -1075,7 +1079,13 @@ static void migration_bitmap_sync(RAMState *rs, RAMBlock *block; int64_t end_time; - if (!periodic) { + if (periodic) { + /* Be careful that we don't synchronize too often */ + int64_t curr_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + if (curr_time < rs->time_last_bitmap_sync + 1000) { + return; + } + } else { stat64_add(&mig_stats.iteration_count, 1); } @@ -1121,6 +1131,92 @@ static void migration_bitmap_sync(RAMState *rs, } } +static void *periodic_throttle_thread(void *opaque) +{ + RAMState *rs = opaque; + bool skip_sleep = false; + int sleep_duration = migrate_periodic_throttle_interval(); + + rcu_register_thread(); + + while (qatomic_read(&rs->throttle_running)) { + int64_t curr_time; + /* + * The first iteration copies all memory anyhow and has no + * effect on guest performance, therefore omit it to avoid + * paying extra for the sync penalty. + */ + if (stat64_get(&mig_stats.iteration_count) <= 1) { + continue; + } + + if (!skip_sleep) { + sleep(sleep_duration); + } + + /* Be careful that we don't synchronize too often */ + curr_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + if (curr_time > rs->time_last_bitmap_sync + 1000) { + bql_lock(); + trace_migration_periodic_throttle(); + WITH_RCU_READ_LOCK_GUARD() { + migration_bitmap_sync(rs, false, true); + } + bql_unlock(); + skip_sleep = false; + } else { + skip_sleep = true; + } + } + + rcu_unregister_thread(); + + return NULL; +} + +void periodic_throttle_start(void) +{ + RAMState *rs = ram_state; + + if (!rs) { + return; + } + + if (qatomic_read(&rs->throttle_running)) { + return; + } + + trace_migration_periodic_throttle_start(); + + qatomic_set(&rs->throttle_running, 1); + qemu_thread_create(&rs->throttle_thread, + NULL, periodic_throttle_thread, + rs, QEMU_THREAD_JOINABLE); +} + +void periodic_throttle_stop(void) +{ + RAMState *rs = ram_state; + + if (!rs) { + return; + } + + if (!qatomic_read(&rs->throttle_running)) { + return; + } + + trace_migration_periodic_throttle_stop(); + + qatomic_set(&rs->throttle_running, 0); + qemu_thread_join(&rs->throttle_thread); +} + +void periodic_throttle_setup(bool enable) +{ + sync_mode = enable ? RAMBLOCK_SYN_MODERN : RAMBLOCK_SYN_LEGACY; +} + static void migration_bitmap_sync_precopy(RAMState *rs, bool last_stage) { Error *local_err = NULL; diff --git a/migration/ram.h b/migration/ram.h index bc0318b834..f7c7b2e7ad 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -93,4 +93,8 @@ void ram_write_tracking_prepare(void); int ram_write_tracking_start(void); void ram_write_tracking_stop(void); +/* Periodic throttle */ +void periodic_throttle_start(void); +void periodic_throttle_stop(void); +void periodic_throttle_setup(bool enable); #endif diff --git a/migration/trace-events b/migration/trace-events index c65902f042..5b9db57c8f 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -95,6 +95,9 @@ get_queued_page_not_dirty(const char *block_name, uint64_t tmp_offset, unsigned migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64 migration_bitmap_clear_dirty(char *str, uint64_t start, uint64_t size, unsigned long page) "rb %s start 0x%"PRIx64" size 0x%"PRIx64" page 0x%lx" +migration_periodic_throttle(void) "" +migration_periodic_throttle_start(void) "" +migration_periodic_throttle_stop(void) "" migration_throttle(void) "" migration_dirty_limit_guest(int64_t dirtyrate) "guest dirty page rate limit %" PRIi64 " MB/s" ram_discard_range(const char *rbname, uint64_t start, size_t len) "%s: start: %" PRIx64 " %zx" From patchwork Mon Sep 9 14:25:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yong Huang X-Patchwork-Id: 13797179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 798C0ECE57A for ; Mon, 9 Sep 2024 14:26:54 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snfLT-0002A9-DG; Mon, 09 Sep 2024 10:26:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snfLO-0001tD-Iv for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:21 -0400 Received: from mail-oa1-x2e.google.com ([2001:4860:4864:20::2e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1snfLK-0004US-LA for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:17 -0400 Received: by mail-oa1-x2e.google.com with SMTP id 586e51a60fabf-277e965305dso2335041fac.2 for ; Mon, 09 Sep 2024 07:26:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1725891973; x=1726496773; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fDNjzZDuWCBsPbd7tcy7/yK9DFVnmavsqu74xEof5CQ=; b=gfSQZ3VcLIFV+/1XYmrXocIaohet6ol8UWwt3qXGDvRFn42AZJb/eeRNuf+zzyf49N lEr8Fyglgp5kF1JgdH2nWgRKHQ9FzIgJr/UI8Tls5V1yu3zt5sbirGeiiU1fgZ9c2xNc xD82SMp6kiHS69kV0s07aw3Cs/Z0fsprpEmbnmj1Rsba+dQj/sQE4YH2NRBUjRDSbA+Y eK/2HJZB6sfuFuAIxqXsE9J8n8I6Ly+BMxcSS8Wslz57ASxub50N0Rn6rJ6vfyX4zKj/ GbSQZ9CdtqaFKld4LZCop6psF0wa+XkgOssU/9E6pIDCFPwMOf1OikZNilOq8j2LkEa1 5qhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725891973; x=1726496773; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fDNjzZDuWCBsPbd7tcy7/yK9DFVnmavsqu74xEof5CQ=; b=S2tKW2pp1XE9/ywR5yYJoCBhfTgncqJ4KqDZyxdPLNYo7o/2/5TXxHwQlUKx/YtBj0 JOzz+UopuVZZTtWbC2VoN4TLbtsRRfg+5l5wMWU7NLysINbbwnA+Daa0FZYGdnG0q6vZ iNhaeEpxgJFHZe0xgeicHXvf4c9ZPNTORrdlLEnbCz3DmihYDdXCMcgj5yub/E9jM7Zv GW1jrzmGIKfmdUkhGqmEp6xbNqaUfTbBeOIu8FMBH0btTPfBipElB+VHNhhafTK1yzBD N4Y1mhO2wjbz53L0EQw3YxDzHoyCf0PA2sGyPqbbnUzxA5dmvDUXNhcJn4jQ7WmGr52x ugng== X-Gm-Message-State: AOJu0YwjyKkleaB5shfrQf+Dc/1kF3kc9s9wjF3l+P/CDeIF80YxgZJV uAFF6U2aDBN3Fx8wy8VI0xx9JPTmrErfihTC/c8Gwvucvjs+PZC49Xj57j7+ReywkE+cFk8HF+Z h6RiFoQ== X-Google-Smtp-Source: AGHT+IHca8Om7eqk8602bCQnCxFF0qmMNz22RXBMDsEwvgw66/xxzlAccL+Fmk+XEvJqdv3GlK+yuA== X-Received: by 2002:a05:6870:e391:b0:270:1b79:ccbc with SMTP id 586e51a60fabf-27b82da186dmr8372667fac.5.1725891972415; Mon, 09 Sep 2024 07:26:12 -0700 (PDT) Received: from localhost.localdomain ([118.114.94.226]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-718e58b2a88sm3561164b3a.46.2024.09.09.07.26.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 07:26:12 -0700 (PDT) From: Hyman Huang To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Eric Blake , Markus Armbruster , David Hildenbrand , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH RESEND RFC 06/10] migration: Support periodic CPU throttle Date: Mon, 9 Sep 2024 22:25:39 +0800 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Received-SPF: pass client-ip=2001:4860:4864:20::2e; envelope-from=yong.huang@smartx.com; helo=mail-oa1-x2e.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org When VM is configured with huge memory, the current throttle logic doesn't look like to scale, because migration_trigger_throttle() is only called for each iteration, so it won't be invoked for a long time if one iteration can take a long time. The periodic sync and throttle aims to fix the above issue by synchronizing the remote dirty bitmap and triggering the throttle periodically. This is a trade-off between synchronization overhead and CPU throttle impact. Signed-off-by: Hyman Huang --- migration/migration.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 055d527ff6..fefd93b683 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1420,6 +1420,9 @@ static void migrate_fd_cleanup(MigrationState *s) qemu_thread_join(&s->thread); s->migration_thread_running = false; } + if (migrate_periodic_throttle()) { + periodic_throttle_stop(); + } bql_lock(); multifd_send_shutdown(); @@ -3263,6 +3266,9 @@ static MigIterateState migration_iteration_run(MigrationState *s) if ((!pending_size || pending_size < s->threshold_size) && can_switchover) { trace_migration_thread_low_pending(pending_size); + if (migrate_periodic_throttle()) { + periodic_throttle_stop(); + } migration_completion(s); return MIG_ITERATE_BREAK; } @@ -3508,6 +3514,11 @@ static void *migration_thread(void *opaque) ret = qemu_savevm_state_setup(s->to_dst_file, &local_err); bql_unlock(); + if (migrate_periodic_throttle()) { + periodic_throttle_setup(true); + periodic_throttle_start(); + } + qemu_savevm_wait_unplug(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_ACTIVE); From patchwork Mon Sep 9 14:25:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yong Huang X-Patchwork-Id: 13797181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 242C0ECE579 for ; Mon, 9 Sep 2024 14:27:19 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snfLb-0002Ya-O0; Mon, 09 Sep 2024 10:26:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snfLS-00028L-BG for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:22 -0400 Received: from mail-pg1-x535.google.com ([2607:f8b0:4864:20::535]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1snfLO-0004Un-BV for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:20 -0400 Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-7163489149eso3529633a12.1 for ; Mon, 09 Sep 2024 07:26:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1725891976; x=1726496776; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PMsMrxXIYF55uyEUpAg6oCulZa95qOnP3qJj8dOgMPY=; b=MGBDKqVf7zq8wAW3+BY7XkO/pil5AP3J+ti/4GeFhh9q5UA3zmI5gSJEQBNUApmOJb vpE/rHwKulVSGhl28tNhsDonNsNGdmdPrQsTu/Hh9NGDCMA17y0aWTcAyEBJrmFil0XK nGngOZTybIc4hb8+uxJOXrrneK7uHiWNDnCGVZssLs/MiSRG+Fkzid08hBk9kSQkefwt lRqElvgu1xFkqQmq5ihY8dfMnwNlp6hhpX8XPgD51OPaZZSO7i6J47TNmGbB2olgv/pF GS1hMkqI768SEt0r54ca0daQR+xiybs259BIKDJhLo1WhD8QP6rR434ekLz+8+ziKIIb FJZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725891976; x=1726496776; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PMsMrxXIYF55uyEUpAg6oCulZa95qOnP3qJj8dOgMPY=; b=xQm9FOJgb9U2qPkfW2OKOyhg6zlhXm3mgOp5lIirOvKeWEg8yEbIxK5FAmT0LnV2BZ S19Bjkmw2K4ISbUOW7vgRiVod+0L6wuGL51VIGMVglXZe/F0MD2buxVy67TZq/T/GOXw Nh3/uyfLkNjzyMG+5oU05Hcj7bNAg4rx8aEETreyZKr8huJDYJnVkMOnPa/AlWQEMPyu 7Ned2dNUdTyFLSricOodZSSC91kp6c/Bze3hc2rw1QsAOXUMP+B7KQP6oqXaozQC/G/9 9l4v8JdUvOPsf7adTd8KxFqxHUTBoV/j04ym1P/P2eFVAx1v/51Rp7rjr3x+prDMjfu+ q4BA== X-Gm-Message-State: AOJu0YymySucwhvn0EJ6AeDwN+zAiVWIhAmNPt7wysOkpsGV8URHvLhE 9FUW2uGUZelqyX7sFis7NB2Cfn0zpgvWS0iOx+Mg69LxzJ9M95D3l8B/qW9JwbrGuOB0yypVKSA GX0vitg== X-Google-Smtp-Source: AGHT+IEMbRYQFUUG8WXjkKSYiGzZOuXV4BZYZPDJVrOdXLDhURhSEXi6aehjLoqkl7nz0x3xD8X6sA== X-Received: by 2002:a05:6a20:4387:b0:1cf:41aa:f6a1 with SMTP id adf61e73a8af0-1cf41ab0323mr3357398637.20.1725891975541; Mon, 09 Sep 2024 07:26:15 -0700 (PDT) Received: from localhost.localdomain ([118.114.94.226]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-718e58b2a88sm3561164b3a.46.2024.09.09.07.26.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 07:26:15 -0700 (PDT) From: Hyman Huang To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Eric Blake , Markus Armbruster , David Hildenbrand , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH RESEND RFC 07/10] tests/migration-tests: Add test case for periodic throttle Date: Mon, 9 Sep 2024 22:25:40 +0800 Message-Id: <8eae7759cdef113ff2fd5a77c9374b51dcd72284.1725891841.git.yong.huang@smartx.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::535; envelope-from=yong.huang@smartx.com; helo=mail-pg1-x535.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org To make sure periodic throttle feature doesn't regression any features and functionalities, enable this feature in the auto-converge migration test. Signed-off-by: Hyman Huang --- tests/qtest/migration-test.c | 56 +++++++++++++++++++++++++++++++++++- 1 file changed, 55 insertions(+), 1 deletion(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 2fb10658d4..61d7182f88 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -281,6 +281,11 @@ static uint64_t get_migration_pass(QTestState *who) return read_ram_property_int(who, "iteration-count"); } +static uint64_t get_migration_dirty_sync_count(QTestState *who) +{ + return read_ram_property_int(who, "dirty-sync-count"); +} + static void read_blocktime(QTestState *who) { QDict *rsp_return; @@ -710,6 +715,11 @@ typedef struct { PostcopyRecoveryFailStage postcopy_recovery_fail_stage; } MigrateCommon; +typedef struct { + /* CPU throttle parameters */ + bool periodic; +} AutoConvergeArgs; + static int test_migrate_start(QTestState **from, QTestState **to, const char *uri, MigrateStart *args) { @@ -2778,12 +2788,13 @@ static void test_validate_uri_channels_none_set(void) * To make things even worse, we need to run the initial stage at * 3MB/s so we enter autoconverge even when host is (over)loaded. */ -static void test_migrate_auto_converge(void) +static void test_migrate_auto_converge_args(AutoConvergeArgs *input_args) { g_autofree char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs); MigrateStart args = {}; QTestState *from, *to; int64_t percentage; + bool periodic = (input_args && input_args->periodic); /* * We want the test to be stable and as fast as possible. @@ -2791,6 +2802,7 @@ static void test_migrate_auto_converge(void) * so we need to decrease a bandwidth. */ const int64_t init_pct = 5, inc_pct = 25, max_pct = 95; + const int64_t periodic_throttle_interval = 2; if (test_migrate_start(&from, &to, uri, &args)) { return; @@ -2801,6 +2813,12 @@ static void test_migrate_auto_converge(void) migrate_set_parameter_int(from, "cpu-throttle-increment", inc_pct); migrate_set_parameter_int(from, "max-cpu-throttle", max_pct); + if (periodic) { + migrate_set_parameter_bool(from, "cpu-periodic-throttle", true); + migrate_set_parameter_int(from, "cpu-periodic-throttle-interval", + periodic_throttle_interval); + } + /* * Set the initial parameters so that the migration could not converge * without throttling. @@ -2827,6 +2845,29 @@ static void test_migrate_auto_converge(void) } while (true); /* The first percentage of throttling should be at least init_pct */ g_assert_cmpint(percentage, >=, init_pct); + + if (periodic) { + /* + * Check if periodic sync take effect, set the timeout with 20s + * (max_try_count * 1s), if extra sync doesn't show up, fail test. + */ + uint64_t iteration_count, dirty_sync_count; + bool extra_sync = false; + int max_try_count = 20; + + /* Check if periodic sync take effect */ + while (--max_try_count) { + usleep(1000 * 1000); + iteration_count = get_migration_pass(from); + dirty_sync_count = get_migration_dirty_sync_count(from); + if (dirty_sync_count > iteration_count) { + extra_sync = true; + break; + } + } + g_assert(extra_sync); + } + /* Now, when we tested that throttling works, let it converge */ migrate_ensure_converge(from); @@ -2849,6 +2890,17 @@ static void test_migrate_auto_converge(void) test_migrate_end(from, to, true); } +static void test_migrate_auto_converge(void) +{ + test_migrate_auto_converge_args(NULL); +} + +static void test_migrate_auto_converge_periodic_throttle(void) +{ + AutoConvergeArgs args = {.periodic = true}; + test_migrate_auto_converge_args(&args); +} + static void * test_migrate_precopy_tcp_multifd_start_common(QTestState *from, QTestState *to, @@ -3900,6 +3952,8 @@ int main(int argc, char **argv) if (g_test_slow()) { migration_test_add("/migration/auto_converge", test_migrate_auto_converge); + migration_test_add("/migration/auto_converge_periodic_throttle", + test_migrate_auto_converge_periodic_throttle); if (g_str_equal(arch, "x86_64") && has_kvm && kvm_dirty_ring_supported()) { migration_test_add("/migration/dirty_limit", From patchwork Mon Sep 9 14:25:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yong Huang X-Patchwork-Id: 13797185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5191BECE57A for ; Mon, 9 Sep 2024 14:27:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snfLd-0002v4-Hr; Mon, 09 Sep 2024 10:26:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snfLW-0002Ny-43 for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:27 -0400 Received: from mail-pf1-x432.google.com ([2607:f8b0:4864:20::432]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1snfLS-0004V9-3f for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:25 -0400 Received: by mail-pf1-x432.google.com with SMTP id d2e1a72fcca58-718e5edf270so1398909b3a.1 for ; Mon, 09 Sep 2024 07:26:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1725891980; x=1726496780; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=r0+VMi25/Zc4CDone71m1lCGybIDfbvM1CcXIqSCA9s=; b=kgqMy2o6P9hg56cajR/0/6tHLfs1kvvXRHRqn7p6807DHDVqE0i+w2EePEmTCg2hor S7M6+USwF+gPkIAEvd10TDvgD+w7GOPTZwynFNfcVgdqKwyL9f8X0h2SBNcUOmgRvAEF OtuqMQQI4KiGPC/NlvPI4p48OI7bFmyioduWx2Mi01p6GNkqETKp5hqak9carTtQk11P fK1ATrISqVM2Qi929/I4WkZFhaN25wNGDJhREe/mOv1pT97nOWSAgBn9kohcTh8CKzMw sXBOb6SafJ4ohCvevW8e2Ze96q6qsTuN0gLsnUkD+6exelqUVK69kJmnhAaQw9WENLCz 4rgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725891980; x=1726496780; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=r0+VMi25/Zc4CDone71m1lCGybIDfbvM1CcXIqSCA9s=; b=TUg1cVsULnMt0umSjXFBDkGcuWrszraJ21MUVBz/oStF+UCBs7N0MWA76ECM9lHmJS YL+QmR/8nXfcVPe8IZvl8xKdtqt7D8KOF5jqs168nSLPg4QlWq5IQiHn85ARMxheYeP0 CLG+gt5G/MDJc/8gbnjxTRHZzVHQzJI10/C53Hk35DN/oRBQy5ZqHfwcE0s0tjJiycJ8 q6xU0L/yT0HRWFTwpsbOWs6ouvtRf76fLpqYoSv5a1Vue5uHiWZ1aWjuXHZVIN0+d1ln 13VLpPCDwPneWNOuJt+jDTkUJ1VbadqEdSneLk2sjtqMeG7wcB94s49Tzf2Wu0/5za+f wcgw== X-Gm-Message-State: AOJu0YyZkCzlXtE0DBgzDmkla3SzaPksaMcVerUEm+rmFN1TlCamOby4 xIQIEhYt9K8AbM6L1x/6qyNaFVMqiZkqlvOfX0w9i9MsSoosRNMEmXz438nQTvgWWewZmcJBm53 2mURkew== X-Google-Smtp-Source: AGHT+IGCoojGvWBCODJCURzuzS3/b9GX3Mwr7CJ9Lbm4C/uXKI17DKKG4k3pWJcK+ENAhCpR2fZB/w== X-Received: by 2002:a05:6a00:b42:b0:718:d5fb:2fc4 with SMTP id d2e1a72fcca58-718d5fb36c3mr12411604b3a.9.1725891979438; Mon, 09 Sep 2024 07:26:19 -0700 (PDT) Received: from localhost.localdomain ([118.114.94.226]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-718e58b2a88sm3561164b3a.46.2024.09.09.07.26.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 07:26:18 -0700 (PDT) From: Hyman Huang To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Eric Blake , Markus Armbruster , David Hildenbrand , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH RESEND RFC 08/10] migration: Introduce cpu-responsive-throttle parameter Date: Mon, 9 Sep 2024 22:25:41 +0800 Message-Id: <08d77dbc95b8836189bae049e82e790ead6c8535.1725891841.git.yong.huang@smartx.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::432; envelope-from=yong.huang@smartx.com; helo=mail-pf1-x432.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org To enable the responsive throttle that will be implemented in the next commit, introduce the cpu-responsive-throttle parameter. Signed-off-by: Hyman Huang --- migration/migration-hmp-cmds.c | 8 ++++++++ migration/options.c | 20 ++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 16 +++++++++++++++- 4 files changed, 44 insertions(+), 1 deletion(-) diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index f7b8e06bb4..a3d4d3f62f 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -273,6 +273,10 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) MigrationParameter_str( MIGRATION_PARAMETER_CPU_PERIODIC_THROTTLE_INTERVAL), params->cpu_periodic_throttle_interval); + assert(params->has_cpu_responsive_throttle); + monitor_printf(mon, "%s: %s\n", + MigrationParameter_str(MIGRATION_PARAMETER_CPU_RESPONSIVE_THROTTLE), + params->cpu_responsive_throttle ? "on" : "off"); assert(params->has_max_cpu_throttle); monitor_printf(mon, "%s: %u\n", MigrationParameter_str(MIGRATION_PARAMETER_MAX_CPU_THROTTLE), @@ -529,6 +533,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_cpu_periodic_throttle_interval = true; visit_type_uint8(v, param, &p->cpu_periodic_throttle_interval, &err); break; + case MIGRATION_PARAMETER_CPU_RESPONSIVE_THROTTLE: + p->has_cpu_responsive_throttle = true; + visit_type_bool(v, param, &p->cpu_responsive_throttle, &err); + break; case MIGRATION_PARAMETER_MAX_CPU_THROTTLE: p->has_max_cpu_throttle = true; visit_type_uint8(v, param, &p->max_cpu_throttle, &err); diff --git a/migration/options.c b/migration/options.c index 2dbe275ba0..aa233684ee 100644 --- a/migration/options.c +++ b/migration/options.c @@ -110,6 +110,8 @@ Property migration_properties[] = { DEFINE_PROP_UINT8("x-cpu-periodic-throttle-interval", MigrationState, parameters.cpu_periodic_throttle_interval, DEFAULT_MIGRATE_CPU_PERIODIC_THROTTLE_INTERVAL), + DEFINE_PROP_BOOL("x-cpu-responsive-throttle", MigrationState, + parameters.cpu_responsive_throttle, false), DEFINE_PROP_SIZE("x-max-bandwidth", MigrationState, parameters.max_bandwidth, MAX_THROTTLE), DEFINE_PROP_SIZE("avail-switchover-bandwidth", MigrationState, @@ -715,6 +717,13 @@ bool migrate_periodic_throttle(void) return s->parameters.cpu_periodic_throttle; } +bool migrate_responsive_throttle(void) +{ + MigrationState *s = migrate_get_current(); + + return s->parameters.cpu_responsive_throttle; +} + bool migrate_cpu_throttle_tailslow(void) { MigrationState *s = migrate_get_current(); @@ -899,6 +908,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->has_cpu_periodic_throttle_interval = true; params->cpu_periodic_throttle_interval = s->parameters.cpu_periodic_throttle_interval; + params->has_cpu_responsive_throttle = true; + params->cpu_responsive_throttle = s->parameters.cpu_responsive_throttle; params->tls_creds = g_strdup(s->parameters.tls_creds); params->tls_hostname = g_strdup(s->parameters.tls_hostname); params->tls_authz = g_strdup(s->parameters.tls_authz ? @@ -967,6 +978,7 @@ void migrate_params_init(MigrationParameters *params) params->has_cpu_throttle_tailslow = true; params->has_cpu_periodic_throttle = true; params->has_cpu_periodic_throttle_interval = true; + params->has_cpu_responsive_throttle = true; params->has_max_bandwidth = true; params->has_downtime_limit = true; params->has_x_checkpoint_delay = true; @@ -1208,6 +1220,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params, params->cpu_periodic_throttle_interval; } + if (params->has_cpu_responsive_throttle) { + dest->cpu_responsive_throttle = params->cpu_responsive_throttle; + } + if (params->tls_creds) { assert(params->tls_creds->type == QTYPE_QSTRING); dest->tls_creds = params->tls_creds->u.s; @@ -1325,6 +1341,10 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp) params->cpu_periodic_throttle_interval; } + if (params->has_cpu_responsive_throttle) { + s->parameters.cpu_responsive_throttle = params->cpu_responsive_throttle; + } + if (params->tls_creds) { g_free(s->parameters.tls_creds); assert(params->tls_creds->type == QTYPE_QSTRING); diff --git a/migration/options.h b/migration/options.h index efeac01470..613d675003 100644 --- a/migration/options.h +++ b/migration/options.h @@ -70,6 +70,7 @@ uint8_t migrate_cpu_throttle_increment(void); uint8_t migrate_cpu_throttle_initial(void); uint8_t migrate_periodic_throttle_interval(void); bool migrate_periodic_throttle(void); +bool migrate_responsive_throttle(void); bool migrate_cpu_throttle_tailslow(void); bool migrate_direct_io(void); uint64_t migrate_downtime_limit(void); diff --git a/qapi/migration.json b/qapi/migration.json index 6d8358c202..9f52ed1899 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -734,6 +734,10 @@ # @cpu-periodic-throttle-interval: Interval of the periodic CPU throttling. # (Since 9.1) # +# @cpu-responsive-throttle: Make CPU throttling more responsively by +# introduce an extra detection metric of +# migration convergence. (Since 9.1) +# # @tls-creds: ID of the 'tls-creds' object that provides credentials # for establishing a TLS connection over the migration data # channel. On the outgoing side of the migration, the credentials @@ -855,7 +859,7 @@ 'throttle-trigger-threshold', 'cpu-throttle-initial', 'cpu-throttle-increment', 'cpu-throttle-tailslow', 'cpu-periodic-throttle', - 'cpu-periodic-throttle-interval', + 'cpu-periodic-throttle-interval', 'cpu-responsive-throttle', 'tls-creds', 'tls-hostname', 'tls-authz', 'max-bandwidth', 'avail-switchover-bandwidth', 'downtime-limit', { 'name': 'x-checkpoint-delay', 'features': [ 'unstable' ] }, @@ -916,6 +920,10 @@ # @cpu-periodic-throttle-interval: Interval of the periodic CPU throttling. # (Since 9.1) # +# @cpu-responsive-throttle: Make CPU throttling more responsively by +# introduce an extra detection metric of +# migration convergence. (Since 9.1) +# # @tls-creds: ID of the 'tls-creds' object that provides credentials # for establishing a TLS connection over the migration data # channel. On the outgoing side of the migration, the credentials @@ -1045,6 +1053,7 @@ '*cpu-throttle-tailslow': 'bool', '*cpu-periodic-throttle': 'bool', '*cpu-periodic-throttle-interval': 'uint8', + '*cpu-responsive-throttle': 'bool', '*tls-creds': 'StrOrNull', '*tls-hostname': 'StrOrNull', '*tls-authz': 'StrOrNull', @@ -1132,6 +1141,10 @@ # @cpu-periodic-throttle-interval: Interval of the periodic CPU throttling. # (Since 9.1) # +# @cpu-responsive-throttle: Make CPU throttling more responsively by +# introduce an extra detection metric of +# migration convergence. (Since 9.1) +# # @tls-creds: ID of the 'tls-creds' object that provides credentials # for establishing a TLS connection over the migration data # channel. On the outgoing side of the migration, the credentials @@ -1254,6 +1267,7 @@ '*cpu-throttle-tailslow': 'bool', '*cpu-periodic-throttle': 'bool', '*cpu-periodic-throttle-interval': 'uint8', + '*cpu-responsive-throttle': 'bool', '*tls-creds': 'str', '*tls-hostname': 'str', '*tls-authz': 'str', From patchwork Mon Sep 9 14:25:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yong Huang X-Patchwork-Id: 13797184 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 60784ECE579 for ; Mon, 9 Sep 2024 14:27:34 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snfLd-0002v8-I3; Mon, 09 Sep 2024 10:26:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snfLX-0002WM-OX for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:30 -0400 Received: from mail-pg1-x532.google.com ([2607:f8b0:4864:20::532]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1snfLU-0004VM-Qq for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:27 -0400 Received: by mail-pg1-x532.google.com with SMTP id 41be03b00d2f7-7d4f9e39c55so2978037a12.2 for ; Mon, 09 Sep 2024 07:26:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1725891983; x=1726496783; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ff6vB6OP2BGQkLDtZ3odfZR0Boq6VYV5mOBp8pXlipM=; b=be+qagHu7OVLEHaZXtQnsKueqNXuDwiXYeeMtudS+fjIrPDMDlnJS10pu48QrQeEbz sHhbVEEfcWeqqaC90tzMDCJqDfL9agC+KpZJCmxXkqxZ/NMSYoN4XExN/o8D3teA7IRi 5bf6EgUjM1JeytaP8Xr7V8A9WzmHt4eBEH3jxfoEE0D/DcAYcU+rG6fzyVYHJIaT6yim Pgk4H6vQioE94qh0veVotpYkFvfVm/Gkw++ofZieeeqHda30xMJVGzN2lPFHbLnluzOs JvE7gu2ckFvG91TXG6HVaNDkyA/VauzHWhU4+Ku/xaUFcjSaMy8nrGBSdB2F39E8T4MJ ZBcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725891983; x=1726496783; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ff6vB6OP2BGQkLDtZ3odfZR0Boq6VYV5mOBp8pXlipM=; b=TUcz3eQzmqnlah9Q63M4aCQ2xNvjGXh70XSdjTJUEYEaGfUDBxEKChCCzv4xJnbMGG PzFq7CmYH6ygGCeWdz2EZNAHQQvd37Gfqp0uiD5tC5mEn4lao2UlGVGAMJ5w52JrGq6j wGPN7ukFhrgagiH5MfhmQhUNysa/k0I+olmjBsfroxk1oi9rY/H8d8+eZAcJFBXYUW21 WrrnZaGP5Ad0flbBk5ATmy46M3O6WFl0Ln/z0unJ6FfExxYH4GGvPZFSFHwDpWXCz03F HUAzIV+GEs5qr0fugN7lWWhE8TakPHTUa9SRaCcS4ZTjNBypKap+9DM3JPrhMwY1L8ek 5hkA== X-Gm-Message-State: AOJu0YwdyS/74h1BRm+3ZFGGtz9M+Y17RZ4IWezqn5LyHkR6dD8+pJwj nGgqG0v4ofgPTY7D+Q6c4VGO+h8Dfg9t9/amzYTAAYOEXaS1kD9krNx8+Mxuh5Jwe3UUOsvBMdd IHT+sug== X-Google-Smtp-Source: AGHT+IFUEAEMRMllagyUged6Cbye+ISYX0WPfudWCviq7kjrNAtmZxH0Q2qwq4zuAQT/XFu/gGhHYg== X-Received: by 2002:a05:6a20:d81a:b0:1cf:4596:d486 with SMTP id adf61e73a8af0-1cf4596d610mr1976199637.47.1725891982579; Mon, 09 Sep 2024 07:26:22 -0700 (PDT) Received: from localhost.localdomain ([118.114.94.226]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-718e58b2a88sm3561164b3a.46.2024.09.09.07.26.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 07:26:22 -0700 (PDT) From: Hyman Huang To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Eric Blake , Markus Armbruster , David Hildenbrand , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH RESEND RFC 09/10] migration: Support responsive CPU throttle Date: Mon, 9 Sep 2024 22:25:42 +0800 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::532; envelope-from=yong.huang@smartx.com; helo=mail-pg1-x532.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Currently, the convergence algorithm determines that the migration cannot converge according to the following principle: The dirty pages generated in current iteration exceed a specific percentage (throttle-trigger-threshold, 50 by default) of the number of transmissions. Let's refer to this criteria as the "dirty rate". If this criteria is met more than or equal to twice (dirty_rate_high_cnt >= 2), the throttle percentage increased. In most cases, above implementation is appropriate. However, for a VM with high memory overload, each iteration is time-consuming. The VM's computing performance may be throttled at a high percentage and last for a long time due to the repeated confirmation behavior. Which may be intolerable for some computationally sensitive software in the VM. As the comment mentioned in the migration_trigger_throttle function, in order to avoid erroneous detection, the original algorithm confirms the criteria repeatedly. Put differently, the criteria does not need to be validated again once the detection is more reliable. In the refinement, in order to make the detection more accurate, we introduce another criteria, called the "dirty ratio" to determine the migration convergence. The "dirty ratio" is the ratio of bytes_xfer_period and bytes_dirty_period. When the algorithm repeatedly detects that the "dirty ratio" of current sync is lower than the previous, the algorithm determines that the migration cannot converge. For the "dirty rate" and "dirty ratio", if one of the two criteria is met, the penalty percentage would be increased. This makes CPU throttle more responsively and therefor saves the time of the entire iteration and therefore reduces the time of VM performance degradation. In conclusion, this refinement significantly reduces the processing time required for the throttle percentage step to its maximum while the VM is under a high memory load. Signed-off-by: Hyman Huang --- migration/ram.c | 55 ++++++++++++++++++++++++++++++++++-- migration/trace-events | 1 + tests/qtest/migration-test.c | 1 + 3 files changed, 55 insertions(+), 2 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index d9d8ed0fda..5fba572f3e 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -420,6 +420,12 @@ struct RAMState { /* Periodic throttle information */ bool throttle_running; QemuThread throttle_thread; + + /* + * Ratio of bytes_dirty_period and bytes_xfer_period in the previous + * sync. + */ + uint64_t dirty_ratio_pct; }; typedef struct RAMState RAMState; @@ -1044,6 +1050,43 @@ static void migration_dirty_limit_guest(void) trace_migration_dirty_limit_guest(quota_dirtyrate); } +static bool migration_dirty_ratio_high(RAMState *rs) +{ + static int dirty_ratio_high_cnt; + uint64_t threshold = migrate_throttle_trigger_threshold(); + uint64_t bytes_xfer_period = + migration_transferred_bytes() - rs->bytes_xfer_prev; + uint64_t bytes_dirty_period = rs->num_dirty_pages_period * TARGET_PAGE_SIZE; + bool dirty_ratio_high = false; + uint64_t prev, curr; + + /* Calculate the dirty ratio percentage */ + curr = 100 * (bytes_dirty_period * 1.0 / bytes_xfer_period); + + prev = rs->dirty_ratio_pct; + rs->dirty_ratio_pct = curr; + + if (prev == 0) { + return false; + } + + /* + * If current dirty ratio is greater than previouse, determine + * that the migration do not converge. + */ + if (curr > threshold && curr >= prev) { + trace_migration_dirty_ratio_high(curr, prev); + dirty_ratio_high_cnt++; + } + + if (dirty_ratio_high_cnt >= 2) { + dirty_ratio_high = true; + dirty_ratio_high_cnt = 0; + } + + return dirty_ratio_high; +} + static void migration_trigger_throttle(RAMState *rs) { uint64_t threshold = migrate_throttle_trigger_threshold(); @@ -1051,6 +1094,11 @@ static void migration_trigger_throttle(RAMState *rs) migration_transferred_bytes() - rs->bytes_xfer_prev; uint64_t bytes_dirty_period = rs->num_dirty_pages_period * TARGET_PAGE_SIZE; uint64_t bytes_dirty_threshold = bytes_xfer_period * threshold / 100; + bool dirty_ratio_high = false; + + if (migrate_responsive_throttle() && (bytes_xfer_period != 0)) { + dirty_ratio_high = migration_dirty_ratio_high(rs); + } /* * The following detection logic can be refined later. For now: @@ -1060,8 +1108,11 @@ static void migration_trigger_throttle(RAMState *rs) * twice, start or increase throttling. */ if ((bytes_dirty_period > bytes_dirty_threshold) && - (++rs->dirty_rate_high_cnt >= 2)) { - rs->dirty_rate_high_cnt = 0; + ((++rs->dirty_rate_high_cnt >= 2) || dirty_ratio_high)) { + + rs->dirty_rate_high_cnt = + rs->dirty_rate_high_cnt >= 2 ? 0 : rs->dirty_rate_high_cnt; + if (migrate_auto_converge()) { trace_migration_throttle(); mig_throttle_guest_down(bytes_dirty_period, diff --git a/migration/trace-events b/migration/trace-events index 5b9db57c8f..241bbfcee9 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -95,6 +95,7 @@ get_queued_page_not_dirty(const char *block_name, uint64_t tmp_offset, unsigned migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64 migration_bitmap_clear_dirty(char *str, uint64_t start, uint64_t size, unsigned long page) "rb %s start 0x%"PRIx64" size 0x%"PRIx64" page 0x%lx" +migration_dirty_ratio_high(uint64_t cur, uint64_t prev) "current ratio: %" PRIu64 " previous ratio: %" PRIu64 migration_periodic_throttle(void) "" migration_periodic_throttle_start(void) "" migration_periodic_throttle_stop(void) "" diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 61d7182f88..4626301435 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -2812,6 +2812,7 @@ static void test_migrate_auto_converge_args(AutoConvergeArgs *input_args) migrate_set_parameter_int(from, "cpu-throttle-initial", init_pct); migrate_set_parameter_int(from, "cpu-throttle-increment", inc_pct); migrate_set_parameter_int(from, "max-cpu-throttle", max_pct); + migrate_set_parameter_bool(from, "cpu-responsive-throttle", true); if (periodic) { migrate_set_parameter_bool(from, "cpu-periodic-throttle", true); From patchwork Mon Sep 9 14:25:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yong Huang X-Patchwork-Id: 13797180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8F548ECE579 for ; Mon, 9 Sep 2024 14:26:55 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snfLd-0002vl-N2; Mon, 09 Sep 2024 10:26:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snfLZ-0002eI-OG for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:31 -0400 Received: from mail-pf1-x42d.google.com ([2607:f8b0:4864:20::42d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1snfLY-0004Vb-06 for qemu-devel@nongnu.org; Mon, 09 Sep 2024 10:26:29 -0400 Received: by mail-pf1-x42d.google.com with SMTP id d2e1a72fcca58-718d704704aso2612035b3a.3 for ; Mon, 09 Sep 2024 07:26:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1725891986; x=1726496786; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3t+rJYL+OLNJsbkNL6rt5vBSdNx9BVxn2HZsRTYOAAc=; b=Lg8Kck88Wi1Q5i9snlC3be9DG0I7i4UGVn7b63cNg9zTSkkZbazhw9ZxCNB5/OH0L9 6IjT8bRDDVk5b4HNa3L7XxE2ZNYdF+v58Zw3QKSq5IAJTD1/S+vP47tPlYTyNluhFJgp 4JdqpENwwYEOniJkkNaImR9Z6y9Ts7hyJ4lnYdh9g9hmL+zRoGth8txeO0YJ67yVwmN8 Aq44L72yGzlQG2E7bqQPre70/X+wdAUOA8dbdSqxfa5WozOZfxUmoFs11ZqwFDJ/J6Iq 3EwOqTTlbrgRC/ItueQyt3oFFTooAU1MwPbP+yQg8KmmLdZMJAxmeNEO+UL00p6lybAC hx/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725891986; x=1726496786; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3t+rJYL+OLNJsbkNL6rt5vBSdNx9BVxn2HZsRTYOAAc=; b=lmj6obVqErqdyy9jNRrEKvcayjVYCuzKHBk5OczvnSz7M+HKt7Q3hmhbCRw1k4HkRA p/01du9gLkfwGSGIrvzgWYviQ+x3WyiUSejFOA/ba7BQRqQkFRNBVSRmVPne5Diwm9qj csYsARmmPhk0nTStpf8FKtwyevRx8mjeec47aObQgsRkDg+HAtGR/8HINTRwsEsugqh5 F90h2odB29sQsK1iV7XSIiLgfivq9h+a7Dz0bnMvjIGYTIDXZKKeAJmJY41R+UOaTGYw c54NLbYzCbljwSmvrepCbfwu4RarmD1okxrdJb0W33l2c+nn/EohE/4ZBXv2Y0lEQEYa TWXw== X-Gm-Message-State: AOJu0Yz3PmQoVuRpJHW3wmh0L7sG4krpt2mgulZdkiKa/8v7Ng55u5KO LECEvaIbWt2aFE9fTu6+pxUloyjMcIpdT7BqXsBRqljieAacmkLhtSobIv4t1JVo0PmV9bX26ix 15vwxWA== X-Google-Smtp-Source: AGHT+IF20x8ePnHQeOTMWmc62GgaNvFSf4hhjn7DeEGwKPf/ahMUtNfsCr2ZKZrS/sXSWylkloJfyg== X-Received: by 2002:a05:6a21:6802:b0:1c4:23f0:9665 with SMTP id adf61e73a8af0-1cf2a0b7dbfmr8712618637.29.1725891985705; Mon, 09 Sep 2024 07:26:25 -0700 (PDT) Received: from localhost.localdomain ([118.114.94.226]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-718e58b2a88sm3561164b3a.46.2024.09.09.07.26.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 07:26:25 -0700 (PDT) From: Hyman Huang To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Eric Blake , Markus Armbruster , David Hildenbrand , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH RESEND RFC 10/10] tests/migration-tests: Add test case for responsive CPU throttle Date: Mon, 9 Sep 2024 22:25:43 +0800 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::42d; envelope-from=yong.huang@smartx.com; helo=mail-pf1-x42d.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Despite the fact that the responsive CPU throttle is enabled, the dirty sync count may not always increase because this is an optimization that might not happen in any situation. This test case just making sure it doesn't interfere with any current functionality. Signed-off-by: Hyman Huang --- tests/qtest/migration-test.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 4626301435..cf0b1dcb50 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -718,6 +718,7 @@ typedef struct { typedef struct { /* CPU throttle parameters */ bool periodic; + bool responsive; } AutoConvergeArgs; static int test_migrate_start(QTestState **from, QTestState **to, @@ -2795,6 +2796,7 @@ static void test_migrate_auto_converge_args(AutoConvergeArgs *input_args) QTestState *from, *to; int64_t percentage; bool periodic = (input_args && input_args->periodic); + bool responsive = (input_args && input_args->responsive); /* * We want the test to be stable and as fast as possible. @@ -2820,6 +2822,16 @@ static void test_migrate_auto_converge_args(AutoConvergeArgs *input_args) periodic_throttle_interval); } + if (responsive) { + /* + * The dirty-sync-count may not always go down while using responsive + * throttle because it is an optimization and may not take effect in + * any scenario. Just making sure this feature doesn't break any + * existing functionality by turning it on. + */ + migrate_set_parameter_bool(from, "cpu-responsive-throttle", true); + } + /* * Set the initial parameters so that the migration could not converge * without throttling. @@ -2902,6 +2914,12 @@ static void test_migrate_auto_converge_periodic_throttle(void) test_migrate_auto_converge_args(&args); } +static void test_migrate_auto_converge_responsive_throttle(void) +{ + AutoConvergeArgs args = {.responsive = true}; + test_migrate_auto_converge_args(&args); +} + static void * test_migrate_precopy_tcp_multifd_start_common(QTestState *from, QTestState *to, @@ -3955,6 +3973,8 @@ int main(int argc, char **argv) test_migrate_auto_converge); migration_test_add("/migration/auto_converge_periodic_throttle", test_migrate_auto_converge_periodic_throttle); + migration_test_add("/migration/auto_converge_responsive_throttle", + test_migrate_auto_converge_responsive_throttle); if (g_str_equal(arch, "x86_64") && has_kvm && kvm_dirty_ring_supported()) { migration_test_add("/migration/dirty_limit",