From patchwork Mon May 31 17:03:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hyman Huang X-Patchwork-Id: 12289679 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C302AC4708F for ; Mon, 31 May 2021 17:15:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A50F1610C9 for ; Mon, 31 May 2021 17:15:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233301AbhEaRQ5 (ORCPT ); Mon, 31 May 2021 13:16:57 -0400 Received: from prt-mail.chinatelecom.cn ([42.123.76.228]:57599 "EHLO chinatelecom.cn" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S234699AbhEaRPS (ORCPT ); Mon, 31 May 2021 13:15:18 -0400 HMM_SOURCE_IP: 172.18.0.48:58222.1682084387 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP Received: from clientip-202.80.192.39?logid-48afce8d3b6b4436b73d11f8304b4286 (unknown [172.18.0.48]) by chinatelecom.cn (HERMES) with SMTP id A730F2800E8; Tue, 1 Jun 2021 01:03:23 +0800 (CST) X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn Received: from ([172.18.0.48]) by app0024 with ESMTP id 48afce8d3b6b4436b73d11f8304b4286 for qemu-devel@nongnu.org; Tue Jun 1 01:03:30 2021 X-Transaction-ID: 48afce8d3b6b4436b73d11f8304b4286 X-filter-score: filter<0> X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 172.18.0.48 X-MEDUSA-Status: 0 Sender: huangy81@chinatelecom.cn From: huangy81@chinatelecom.cn To: Cc: Paolo Bonzini , , Juan Quintela , "Dr. David Alan Gilbert" , Peter Xu , Hyman Subject: [PATCH v1 1/6] KVM: add kvm_dirty_ring_enabled function Date: Tue, 1 Jun 2021 01:03:18 +0800 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Hyman Huang(黄勇) introduce kvm_dirty_ring_enabled to show if kvm-reaper is working. dirtyrate thread could use it to check if calculation can base on dirty ring feature. Signed-off-by: Hyman Huang(黄勇) --- accel/kvm/kvm-all.c | 5 +++++ include/sysemu/kvm.h | 1 + 2 files changed, 6 insertions(+) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index c7ec538850..2e96b77b31 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -2293,6 +2293,11 @@ bool kvm_vcpu_id_is_valid(int vcpu_id) return vcpu_id >= 0 && vcpu_id < kvm_max_vcpu_id(s); } +bool kvm_dirty_ring_enabled(void) +{ + return kvm_state->kvm_dirty_ring_size ? true : false; +} + static int kvm_init(MachineState *ms) { MachineClass *mc = MACHINE_GET_CLASS(ms); diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index a1ab1ee12d..7b22aeb6ae 100644 --- a/include/sysemu/kvm.h +++ b/include/sysemu/kvm.h @@ -547,4 +547,5 @@ bool kvm_cpu_check_are_resettable(void); bool kvm_arch_cpu_check_are_resettable(void); +bool kvm_dirty_ring_enabled(void); #endif From patchwork Mon May 31 17:04:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hyman Huang X-Patchwork-Id: 12289675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 320EEC47080 for ; Mon, 31 May 2021 17:15:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F0E64610C9 for ; Mon, 31 May 2021 17:15:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233184AbhEaRQz (ORCPT ); Mon, 31 May 2021 13:16:55 -0400 Received: from prt-mail.chinatelecom.cn ([42.123.76.228]:57600 "EHLO chinatelecom.cn" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S234694AbhEaRPQ (ORCPT ); Mon, 31 May 2021 13:15:16 -0400 HMM_SOURCE_IP: 172.18.0.218:47064.1907396775 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP Received: from clientip-202.80.192.38?logid-29ae52c8a85445e99aab72860cfd77f0 (unknown [172.18.0.218]) by chinatelecom.cn (HERMES) with SMTP id 4D8B52800F6; Tue, 1 Jun 2021 01:04:12 +0800 (CST) X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn Received: from ([172.18.0.218]) by app0025 with ESMTP id 29ae52c8a85445e99aab72860cfd77f0 for qemu-devel@nongnu.org; Tue Jun 1 01:04:30 2021 X-Transaction-ID: 29ae52c8a85445e99aab72860cfd77f0 X-filter-score: filter<0> X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 172.18.0.218 X-MEDUSA-Status: 0 Sender: huangy81@chinatelecom.cn From: huangy81@chinatelecom.cn To: Cc: Paolo Bonzini , , Juan Quintela , "Dr. David Alan Gilbert" , Peter Xu , Hyman Subject: [PATCH v1 2/6] KVM: introduce dirty_pages into CPUState Date: Tue, 1 Jun 2021 01:04:06 +0800 Message-Id: <78cc154863754a93d88070d1fae9fed6a1ec5f01.1622479161.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Hyman Huang(黄勇) dirty_pages is used to calculate dirtyrate via dirty ring, when enabled, kvm-reaper will increase the dirty pages after gfns being dirtied. Signed-off-by: Hyman Huang(黄勇) --- accel/kvm/kvm-all.c | 6 ++++++ include/hw/core/cpu.h | 2 ++ 2 files changed, 8 insertions(+) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index 2e96b77b31..52cba1b094 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -506,6 +506,9 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp) } } + cpu->dirty_pages = 0; + cpu->stat_dirty_pages = false; + ret = kvm_arch_init_vcpu(cpu); if (ret < 0) { error_setg_errno(errp, -ret, @@ -739,6 +742,9 @@ static uint32_t kvm_dirty_ring_reap_one(KVMState *s, CPUState *cpu) cur->offset); dirty_gfn_set_collected(cur); trace_kvm_dirty_ring_page(cpu->cpu_index, fetch, cur->offset); + if (cpu->stat_dirty_pages) { + cpu->dirty_pages++; + } fetch++; count++; } diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 044f668a6e..973c193501 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -375,6 +375,8 @@ struct CPUState { struct kvm_run *kvm_run; struct kvm_dirty_gfn *kvm_dirty_gfns; uint32_t kvm_fetch_index; + uint64_t dirty_pages; + bool stat_dirty_pages; /* Used for events with 'vcpu' and *without* the 'disabled' properties */ DECLARE_BITMAP(trace_dstate_delayed, CPU_TRACE_DSTATE_MAX_EVENTS); From patchwork Mon May 31 17:05:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hyman Huang X-Patchwork-Id: 12289683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A8DDC47080 for ; Mon, 31 May 2021 17:16:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0D95E610A8 for ; Mon, 31 May 2021 17:16:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232274AbhEaRSd (ORCPT ); Mon, 31 May 2021 13:18:33 -0400 Received: from prt-mail.chinatelecom.cn ([42.123.76.219]:58422 "EHLO chinatelecom.cn" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S234053AbhEaRRZ (ORCPT ); Mon, 31 May 2021 13:17:25 -0400 HMM_SOURCE_IP: 172.18.0.218:37984.253484957 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP Received: from clientip-202.80.192.39?logid-4a3d34afc9d14b97a5e6c1a5d502adc7 (unknown [172.18.0.218]) by chinatelecom.cn (HERMES) with SMTP id D220D2800B0; Tue, 1 Jun 2021 01:04:59 +0800 (CST) X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn Received: from ([172.18.0.218]) by app0025 with ESMTP id 4a3d34afc9d14b97a5e6c1a5d502adc7 for qemu-devel@nongnu.org; Tue Jun 1 01:04:59 2021 X-Transaction-ID: 4a3d34afc9d14b97a5e6c1a5d502adc7 X-filter-score: filter<0> X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 172.18.0.218 X-MEDUSA-Status: 0 Sender: huangy81@chinatelecom.cn From: huangy81@chinatelecom.cn To: Cc: Paolo Bonzini , , Juan Quintela , "Dr. David Alan Gilbert" , Peter Xu , Hyman Subject: [PATCH v1 3/6] migration/dirtyrate: add vcpu option for qmp calc-dirty-rate Date: Tue, 1 Jun 2021 01:05:00 +0800 Message-Id: <28111cd734b1b6e76f7cd8f2f6ad1d4c54f12842.1622479162.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Hyman Huang(黄勇) calculate dirtyrate for each vcpu if vcpu is true, add the dirtyrate of each vcpu to the return value also. Signed-off-by: Hyman Huang(黄勇) --- migration/dirtyrate.c | 5 ++++- migration/dirtyrate.h | 1 + qapi/migration.json | 28 ++++++++++++++++++++++++++-- 3 files changed, 31 insertions(+), 3 deletions(-) diff --git a/migration/dirtyrate.c b/migration/dirtyrate.c index ccb98147e8..3c1a824a41 100644 --- a/migration/dirtyrate.c +++ b/migration/dirtyrate.c @@ -383,7 +383,10 @@ void *get_dirtyrate_thread(void *arg) return NULL; } -void qmp_calc_dirty_rate(int64_t calc_time, Error **errp) +void qmp_calc_dirty_rate(int64_t calc_time, + bool has_vcpu, + bool vcpu, + Error **errp) { static struct DirtyRateConfig config; QemuThread thread; diff --git a/migration/dirtyrate.h b/migration/dirtyrate.h index 6ec429534d..f20dd52d77 100644 --- a/migration/dirtyrate.h +++ b/migration/dirtyrate.h @@ -38,6 +38,7 @@ struct DirtyRateConfig { uint64_t sample_pages_per_gigabytes; /* sample pages per GB */ int64_t sample_period_seconds; /* time duration between two sampling */ + bool vcpu; /* calculate dirtyrate for each vcpu using dirty ring */ }; /* diff --git a/qapi/migration.json b/qapi/migration.json index 7a5bdf9a0d..896ebcb93b 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1708,6 +1708,21 @@ { 'event': 'UNPLUG_PRIMARY', 'data': { 'device-id': 'str' } } +## +# @DirtyRateVcpu: +# +# Dirty rate of vcpu. +# +# @id: vcpu index. +# +# @dirty-rate: dirty rate. +# +# Since: 6.1 +# +## +{ 'struct': 'DirtyRateVcpu', + 'data': { 'id': 'int', 'dirty-rate': 'int64' } } + ## # @DirtyRateStatus: # @@ -1740,6 +1755,10 @@ # # @calc-time: time in units of second for sample dirty pages # +# @vcpu: calculate dirtyrate for each vcpu (Since 6.1) +# +# @vcpu-dirty-rate: dirtyrate for each vcpu (Since 6.1) +# # Since: 5.2 # ## @@ -1747,7 +1766,9 @@ 'data': {'*dirty-rate': 'int64', 'status': 'DirtyRateStatus', 'start-time': 'int64', - 'calc-time': 'int64'} } + 'calc-time': 'int64', + '*vcpu': 'bool', + '*vcpu-dirty-rate': [ 'DirtyRateVcpu' ] } } ## # @calc-dirty-rate: @@ -1756,13 +1777,16 @@ # # @calc-time: time in units of second for sample dirty pages # +# @vcpu: calculate vcpu dirty rate if true, the default value is +# false (since 6.1) +# # Since: 5.2 # # Example: # {"command": "calc-dirty-rate", "data": {"calc-time": 1} } # ## -{ 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64'} } +{ 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64', '*vcpu': 'bool'} } ## # @query-dirty-rate: From patchwork Mon May 31 17:05:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hyman Huang X-Patchwork-Id: 12289685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56B5AC47080 for ; Mon, 31 May 2021 17:17:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 37C51610C9 for ; Mon, 31 May 2021 17:17:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232842AbhEaRSi (ORCPT ); Mon, 31 May 2021 13:18:38 -0400 Received: from prt-mail.chinatelecom.cn ([42.123.76.227]:41294 "EHLO chinatelecom.cn" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S232725AbhEaRRZ (ORCPT ); Mon, 31 May 2021 13:17:25 -0400 HMM_SOURCE_IP: 172.18.0.218:60136.529866758 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP Received: from clientip-202.80.192.39?logid-262a242008784dafb9db8682f1ad8717 (unknown [172.18.0.218]) by chinatelecom.cn (HERMES) with SMTP id 525342800B2; Tue, 1 Jun 2021 01:05:28 +0800 (CST) X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn Received: from ([172.18.0.218]) by app0025 with ESMTP id 262a242008784dafb9db8682f1ad8717 for qemu-devel@nongnu.org; Tue Jun 1 01:05:27 2021 X-Transaction-ID: 262a242008784dafb9db8682f1ad8717 X-filter-score: filter<0> X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 172.18.0.218 X-MEDUSA-Status: 0 Sender: huangy81@chinatelecom.cn From: huangy81@chinatelecom.cn To: Cc: Paolo Bonzini , , Juan Quintela , "Dr. David Alan Gilbert" , Peter Xu , Hyman Subject: [PATCH v1 4/6] migration/dirtyrate: adjust struct DirtyRateStat Date: Tue, 1 Jun 2021 01:05:29 +0800 Message-Id: <16e0e8f50b3b83f809187dcfed5693026bea0caa.1622479162.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Hyman Huang(黄勇) use union to store stat data of two mutual exclusive methods. Signed-off-by: Hyman Huang(黄勇) --- migration/dirtyrate.c | 32 ++++++++++++++++++++------------ migration/dirtyrate.h | 18 +++++++++++++++--- 2 files changed, 35 insertions(+), 15 deletions(-) diff --git a/migration/dirtyrate.c b/migration/dirtyrate.c index 3c1a824a41..7952eb6117 100644 --- a/migration/dirtyrate.c +++ b/migration/dirtyrate.c @@ -78,31 +78,39 @@ static struct DirtyRateInfo *query_dirty_rate_info(void) return info; } -static void init_dirtyrate_stat(int64_t start_time, int64_t calc_time) +static void init_dirtyrate_stat(int64_t start_time, + int64_t calc_time, + struct DirtyRateConfig config) { - DirtyStat.total_dirty_samples = 0; - DirtyStat.total_sample_count = 0; - DirtyStat.total_block_mem_MB = 0; DirtyStat.dirty_rate = -1; DirtyStat.start_time = start_time; DirtyStat.calc_time = calc_time; + + if (config.vcpu) { + DirtyStat.method.vcpu.nvcpu = -1; + DirtyStat.method.vcpu.rates = NULL; + } else { + DirtyStat.method.vm.total_dirty_samples = 0; + DirtyStat.method.vm.total_sample_count = 0; + DirtyStat.method.vm.total_block_mem_MB = 0; + } } static void update_dirtyrate_stat(struct RamblockDirtyInfo *info) { - DirtyStat.total_dirty_samples += info->sample_dirty_count; - DirtyStat.total_sample_count += info->sample_pages_count; + DirtyStat.method.vm.total_dirty_samples += info->sample_dirty_count; + DirtyStat.method.vm.total_sample_count += info->sample_pages_count; /* size of total pages in MB */ - DirtyStat.total_block_mem_MB += (info->ramblock_pages * + DirtyStat.method.vm.total_block_mem_MB += (info->ramblock_pages * TARGET_PAGE_SIZE) >> 20; } static void update_dirtyrate(uint64_t msec) { uint64_t dirtyrate; - uint64_t total_dirty_samples = DirtyStat.total_dirty_samples; - uint64_t total_sample_count = DirtyStat.total_sample_count; - uint64_t total_block_mem_MB = DirtyStat.total_block_mem_MB; + uint64_t total_dirty_samples = DirtyStat.method.vm.total_dirty_samples; + uint64_t total_sample_count = DirtyStat.method.vm.total_sample_count; + uint64_t total_block_mem_MB = DirtyStat.method.vm.total_block_mem_MB; dirtyrate = total_dirty_samples * total_block_mem_MB * 1000 / (total_sample_count * msec); @@ -315,7 +323,7 @@ static bool compare_page_hash_info(struct RamblockDirtyInfo *info, update_dirtyrate_stat(block_dinfo); } - if (DirtyStat.total_sample_count == 0) { + if (DirtyStat.method.vm.total_sample_count == 0) { return false; } @@ -371,7 +379,7 @@ void *get_dirtyrate_thread(void *arg) start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) / 1000; calc_time = config.sample_period_seconds; - init_dirtyrate_stat(start_time, calc_time); + init_dirtyrate_stat(start_time, calc_time, config); calculate_dirtyrate(config); diff --git a/migration/dirtyrate.h b/migration/dirtyrate.h index f20dd52d77..3ab8e81f42 100644 --- a/migration/dirtyrate.h +++ b/migration/dirtyrate.h @@ -54,16 +54,28 @@ struct RamblockDirtyInfo { uint32_t *hash_result; /* array of hash result for sampled pages */ }; +typedef struct SampleVMStat { + uint64_t total_dirty_samples; /* total dirty sampled page */ + uint64_t total_sample_count; /* total sampled pages */ + uint64_t total_block_mem_MB; /* size of total sampled pages in MB */ +} SampleVMStat; + +typedef struct VcpuStat { + int nvcpu; /* number of vcpu */ + DirtyRateVcpu *rates; /* array of dirty rate for each vcpu */ +} VcpuStat; + /* * Store calculation statistics for each measure. */ struct DirtyRateStat { - uint64_t total_dirty_samples; /* total dirty sampled page */ - uint64_t total_sample_count; /* total sampled pages */ - uint64_t total_block_mem_MB; /* size of total sampled pages in MB */ int64_t dirty_rate; /* dirty rate in MB/s */ int64_t start_time; /* calculation start time in units of second */ int64_t calc_time; /* time duration of two sampling in units of second */ + union { + SampleVMStat vm; + VcpuStat vcpu; + } method; }; void *get_dirtyrate_thread(void *arg); From patchwork Mon May 31 17:05:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hyman Huang X-Patchwork-Id: 12289677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEA0CC47082 for ; Mon, 31 May 2021 17:15:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BCC57610C9 for ; Mon, 31 May 2021 17:15:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233276AbhEaRQ4 (ORCPT ); Mon, 31 May 2021 13:16:56 -0400 Received: from prt-mail.chinatelecom.cn ([42.123.76.228]:57597 "EHLO chinatelecom.cn" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S234697AbhEaRPQ (ORCPT ); Mon, 31 May 2021 13:15:16 -0400 HMM_SOURCE_IP: 172.18.0.218:53114.1639123736 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP Received: from clientip-202.80.192.38?logid-ed481103dcf04c7799310e542c3cbd82 (unknown [172.18.0.218]) by chinatelecom.cn (HERMES) with SMTP id ED4F72800FA; Tue, 1 Jun 2021 01:05:57 +0800 (CST) X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn Received: from ([172.18.0.218]) by app0025 with ESMTP id ed481103dcf04c7799310e542c3cbd82 for qemu-devel@nongnu.org; Tue Jun 1 01:05:58 2021 X-Transaction-ID: ed481103dcf04c7799310e542c3cbd82 X-filter-score: filter<0> X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 172.18.0.218 X-MEDUSA-Status: 0 Sender: huangy81@chinatelecom.cn From: huangy81@chinatelecom.cn To: Cc: Paolo Bonzini , , Juan Quintela , "Dr. David Alan Gilbert" , Peter Xu , Hyman Subject: [PATCH v1 5/6] migration/dirtyrate: check support of calculation for vcpu Date: Tue, 1 Jun 2021 01:05:58 +0800 Message-Id: <8c48e41e38c31827f305806704e6e23faef848c3.1622479162.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Hyman Huang(黄勇) vcpu method only works when kvm dirty ring is enabled, use kvm_dirty_ring_enabled to probe if dirty ring is enabled. Signed-off-by: Hyman Huang(黄勇) --- migration/dirtyrate.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/migration/dirtyrate.c b/migration/dirtyrate.c index 7952eb6117..da6500c8ec 100644 --- a/migration/dirtyrate.c +++ b/migration/dirtyrate.c @@ -16,6 +16,7 @@ #include "cpu.h" #include "exec/ramblock.h" #include "qemu/rcu_queue.h" +#include "sysemu/kvm.h" #include "qapi/qapi-commands-migration.h" #include "ram.h" #include "trace.h" @@ -415,6 +416,14 @@ void qmp_calc_dirty_rate(int64_t calc_time, return; } + /* + * Vcpu method only works when kvm dirty ring is enabled. + */ + if (has_vcpu && vcpu && !kvm_dirty_ring_enabled()) { + error_setg(errp, "kvm dirty ring is disabled, use sample method."); + return; + } + /* * Init calculation state as unstarted. */ @@ -427,6 +436,7 @@ void qmp_calc_dirty_rate(int64_t calc_time, config.sample_period_seconds = calc_time; config.sample_pages_per_gigabytes = DIRTYRATE_DEFAULT_SAMPLE_PAGES; + config.vcpu = has_vcpu ? vcpu : false; qemu_thread_create(&thread, "get_dirtyrate", get_dirtyrate_thread, (void *)&config, QEMU_THREAD_DETACHED); } From patchwork Mon May 31 17:06:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hyman Huang X-Patchwork-Id: 12289681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 113B9C47080 for ; Mon, 31 May 2021 17:16:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DF7DC610A8 for ; Mon, 31 May 2021 17:16:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231971AbhEaRSb (ORCPT ); Mon, 31 May 2021 13:18:31 -0400 Received: from prt-mail.chinatelecom.cn ([42.123.76.219]:58424 "EHLO chinatelecom.cn" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S233794AbhEaRRY (ORCPT ); Mon, 31 May 2021 13:17:24 -0400 HMM_SOURCE_IP: 172.18.0.218:42706.1466219582 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP Received: from clientip-202.80.192.39?logid-8790cfef0ba147a299cbd8acf210f1a0 (unknown [172.18.0.218]) by chinatelecom.cn (HERMES) with SMTP id 5A2E42800B2; Tue, 1 Jun 2021 01:06:29 +0800 (CST) X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn Received: from ([172.18.0.218]) by app0025 with ESMTP id 8790cfef0ba147a299cbd8acf210f1a0 for qemu-devel@nongnu.org; Tue Jun 1 01:06:27 2021 X-Transaction-ID: 8790cfef0ba147a299cbd8acf210f1a0 X-filter-score: filter<0> X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 172.18.0.218 X-MEDUSA-Status: 0 Sender: huangy81@chinatelecom.cn From: huangy81@chinatelecom.cn To: Cc: Paolo Bonzini , , Juan Quintela , "Dr. David Alan Gilbert" , Peter Xu , Hyman Subject: [PATCH v1 6/6] migration/dirtyrate: implement dirty-ring dirtyrate calculation Date: Tue, 1 Jun 2021 01:06:29 +0800 Message-Id: <27607e12038273706273203b2146c5d4a40ac487.1622479162.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Hyman Huang(黄勇) use dirty ring feature to implement dirtyrate calculation. to enable it, set vcpu option as true in qmp calc-dirty-rate. Signed-off-by: Hyman Huang(黄勇) --- migration/dirtyrate.c | 146 ++++++++++++++++++++++++++++++++++++++--- migration/trace-events | 1 + 2 files changed, 139 insertions(+), 8 deletions(-) diff --git a/migration/dirtyrate.c b/migration/dirtyrate.c index da6500c8ec..028c11d117 100644 --- a/migration/dirtyrate.c +++ b/migration/dirtyrate.c @@ -16,14 +16,22 @@ #include "cpu.h" #include "exec/ramblock.h" #include "qemu/rcu_queue.h" +#include "qemu/main-loop.h" #include "sysemu/kvm.h" #include "qapi/qapi-commands-migration.h" #include "ram.h" #include "trace.h" #include "dirtyrate.h" +typedef enum { + CALC_NONE = 0, + CALC_DIRTY_RING, + CALC_SAMPLE_PAGES, +} CalcMethod; + static int CalculatingState = DIRTY_RATE_STATUS_UNSTARTED; static struct DirtyRateStat DirtyStat; +static CalcMethod last_method = CALC_NONE; static int64_t set_sample_page_period(int64_t msec, int64_t initial_time) { @@ -64,6 +72,7 @@ static struct DirtyRateInfo *query_dirty_rate_info(void) { int64_t dirty_rate = DirtyStat.dirty_rate; struct DirtyRateInfo *info = g_malloc0(sizeof(DirtyRateInfo)); + DirtyRateVcpuList *head = NULL, **tail = &head; if (qatomic_read(&CalculatingState) == DIRTY_RATE_STATUS_MEASURED) { info->has_dirty_rate = true; @@ -73,6 +82,22 @@ static struct DirtyRateInfo *query_dirty_rate_info(void) info->status = CalculatingState; info->start_time = DirtyStat.start_time; info->calc_time = DirtyStat.calc_time; + info->has_vcpu = true; + + if (last_method == CALC_DIRTY_RING) { + int i = 0; + info->vcpu = true; + info->has_vcpu_dirty_rate = true; + for (i = 0; i < DirtyStat.method.vcpu.nvcpu; i++) { + DirtyRateVcpu *rate = g_malloc0(sizeof(DirtyRateVcpu)); + rate->id = DirtyStat.method.vcpu.rates[i].id; + rate->dirty_rate = DirtyStat.method.vcpu.rates[i].dirty_rate; + QAPI_LIST_APPEND(tail, rate); + } + info->vcpu_dirty_rate = head; + } else { + info->vcpu = false; + } trace_query_dirty_rate_info(DirtyRateStatus_str(CalculatingState)); @@ -87,13 +112,29 @@ static void init_dirtyrate_stat(int64_t start_time, DirtyStat.start_time = start_time; DirtyStat.calc_time = calc_time; - if (config.vcpu) { - DirtyStat.method.vcpu.nvcpu = -1; - DirtyStat.method.vcpu.rates = NULL; - } else { - DirtyStat.method.vm.total_dirty_samples = 0; - DirtyStat.method.vm.total_sample_count = 0; - DirtyStat.method.vm.total_block_mem_MB = 0; + switch (last_method) { + case CALC_NONE: + case CALC_SAMPLE_PAGES: + if (config.vcpu) { + DirtyStat.method.vcpu.nvcpu = -1; + DirtyStat.method.vcpu.rates = NULL; + } else { + DirtyStat.method.vm.total_dirty_samples = 0; + DirtyStat.method.vm.total_sample_count = 0; + DirtyStat.method.vm.total_block_mem_MB = 0; + } + break; + case CALC_DIRTY_RING: + if (!config.vcpu) { + g_free(DirtyStat.method.vcpu.rates); + DirtyStat.method.vcpu.rates = NULL; + DirtyStat.method.vm.total_dirty_samples = 0; + DirtyStat.method.vm.total_sample_count = 0; + DirtyStat.method.vm.total_block_mem_MB = 0; + } + break; + default: + break; } } @@ -331,7 +372,84 @@ static bool compare_page_hash_info(struct RamblockDirtyInfo *info, return true; } -static void calculate_dirtyrate(struct DirtyRateConfig config) +static void stat_dirtypages(CPUState *cpu, bool start) +{ + cpu->stat_dirty_pages = start; +} + +static void start_kvm_dirty_log(void) +{ + qemu_mutex_lock_iothread(); + memory_global_dirty_log_start(); + qemu_mutex_unlock_iothread(); +} + +static void stop_kvm_dirty_log(void) +{ + qemu_mutex_lock_iothread(); + memory_global_dirty_log_stop(); + qemu_mutex_unlock_iothread(); +} + +static int64_t do_calculate_dirtyrate_vcpu(CPUState *cpu) +{ + uint64_t memory_size_MB; + int64_t time_s; + + memory_size_MB = (cpu->dirty_pages * TARGET_PAGE_SIZE) >> 20; + time_s = DirtyStat.calc_time; + + return memory_size_MB / time_s; +} + +static void calculate_dirtyrate_vcpu(struct DirtyRateConfig config) +{ + CPUState *cpu; + int64_t msec = 0; + int64_t start_time; + uint64_t dirtyrate = 0; + uint64_t dirtyrate_sum = 0; + int nvcpu, i = 0; + + CPU_FOREACH(cpu) { + stat_dirtypages(cpu, true); + nvcpu++; + } + + DirtyStat.method.vcpu.nvcpu = nvcpu; + + if (last_method != CALC_DIRTY_RING) { + DirtyStat.method.vcpu.rates = + g_malloc0(sizeof(DirtyRateVcpu) * nvcpu); + } + + start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + DirtyStat.start_time = start_time / 1000; + + start_kvm_dirty_log(); + + msec = config.sample_period_seconds * 1000; + msec = set_sample_page_period(msec, start_time); + DirtyStat.calc_time = msec / 1000; + + CPU_FOREACH(cpu) { + stat_dirtypages(cpu, false); + } + + stop_kvm_dirty_log(); + + CPU_FOREACH(cpu) { + dirtyrate = do_calculate_dirtyrate_vcpu(cpu); + DirtyStat.method.vcpu.rates[i].id = cpu->cpu_index; + DirtyStat.method.vcpu.rates[i].dirty_rate = dirtyrate; + dirtyrate_sum += dirtyrate; + i++; + } + + DirtyStat.dirty_rate = dirtyrate_sum / nvcpu; +} + +static void calculate_dirtyrate_sample_vm(struct DirtyRateConfig config) { struct RamblockDirtyInfo *block_dinfo = NULL; int block_count = 0; @@ -364,6 +482,18 @@ out: rcu_unregister_thread(); } +static void calculate_dirtyrate(struct DirtyRateConfig config) +{ + if (config.vcpu) { + calculate_dirtyrate_vcpu(config); + last_method = CALC_DIRTY_RING; + } else { + calculate_dirtyrate_sample_vm(config); + last_method = CALC_SAMPLE_PAGES; + } + trace_calculate_dirtyrate(DirtyStat.dirty_rate); +} + void *get_dirtyrate_thread(void *arg) { struct DirtyRateConfig config = *(struct DirtyRateConfig *)arg; diff --git a/migration/trace-events b/migration/trace-events index 668c562fed..5a80b39a62 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -330,6 +330,7 @@ get_ramblock_vfn_hash(const char *idstr, uint64_t vfn, uint32_t crc) "ramblock n calc_page_dirty_rate(const char *idstr, uint32_t new_crc, uint32_t old_crc) "ramblock name: %s, new crc: %" PRIu32 ", old crc: %" PRIu32 skip_sample_ramblock(const char *idstr, uint64_t ramblock_size) "ramblock name: %s, ramblock size: %" PRIu64 find_page_matched(const char *idstr) "ramblock %s addr or size changed" +calculate_dirtyrate(int64_t dirtyrate) "dirty rate: %" PRIi64 # block.c migration_block_init_shared(const char *blk_device_name) "Start migration for %s with shared base image"