From patchwork Sat Mar 10 21:21:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10274381 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 3750260594 for ; Sat, 10 Mar 2018 21:36:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1DA1B285BE for ; Sat, 10 Mar 2018 21:36:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0F1D328E19; Sat, 10 Mar 2018 21:36:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 525E3285BE for ; Sat, 10 Mar 2018 21:36:29 +0000 (UTC) Received: from localhost ([::1]:52041 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eumAG-0005fo-4R for patchwork-qemu-devel@patchwork.kernel.org; Sat, 10 Mar 2018 16:36:28 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48546) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulxH-0002bW-ND for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:04 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulxG-0003Oc-L5 for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:03 -0500 Received: from mail-pg0-x244.google.com ([2607:f8b0:400e:c05::244]:45929) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulxG-0003OT-Eg for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:02 -0500 Received: by mail-pg0-x244.google.com with SMTP id s13so2025749pgn.12 for ; Sat, 10 Mar 2018 13:23:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=XqbOQdGJaaK869FsHU8cyOLMJv+HPHjhy3tE5GWcJ+0=; b=ZyQqNN3Z4z34X+UbkuU8RAH1t+FhWGaI8r4dOu6BjOLevMk6GNN0iBtq2Uonp0GjtR HuexkButXfq1jikYzR6fSUNr/lIzb/wVZhJ99xkZdjlyfVcxXGA4koJo24lfu5RWNUjT At6bNuDcC/3WmcTh8HpxgP6Mfd9QkkA/mxY/2GwUuLcS98E4fvpVuJRGRHMUv0gJQLjv DXm03u08rftGmy6/hMy0oh+t982pSI9gzLWOS2xLxWO5VuTSpVn5jvSBIqxE4g93x+8z PJa0hEbQ7bWnjurRpnIKfr0nAZepuuVvl97nkQZtBc0q41s79/ni63ph2c7Vfg/8PLGJ ddlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=XqbOQdGJaaK869FsHU8cyOLMJv+HPHjhy3tE5GWcJ+0=; b=NpwWxaVBP5lsU4jRtwPWQLMEfW5Wh7x3qVDvX6sTB0f5zsdSlKNXeyDpGccNTjLYN6 yW+BCxznL4v0MGX/p6VyawQxkBgAhoJPqLSG2SlIC+jQkpDjzUGbOqPCKdbr/o8yG0PD belViv9OKbUov3379h3rVsiKW6iKWBwPYalMcqWq6/EmN3O7M/EjYjNuHsF5qc2gfZD8 w5TF3FQXnvSjkacVODCKSjV9e2xf/QqwEHYYDlVbU9IsIl+u3FB3jgMY82Kz1cg5lBRn aLpVdqhUOy4rLPbTnKFxa+oop43vx1jyE42j/I2l8wt8HHJjuatgCmSXbHPREHi8ZErn gW1g== X-Gm-Message-State: AElRT7HIS3hOYB3R9ZhCJ0Q04sB3zdV0zkdSf5O3j/qvYX7tFDv+0DkA 1D0IesO7NTlR0e+UxgitPnK2AL+M X-Google-Smtp-Source: AG47ELtU9tOFm+WzJiz6o4vJnLECVfbQgXCmG1oV6gBgWuZ9DB/i+z08JjJIDIP2KjaUAQLdtnyfsw== X-Received: by 10.98.245.131 with SMTP id b3mr3040367pfm.20.1520716981226; Sat, 10 Mar 2018 13:23:01 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.22.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:23:00 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:21:59 +0800 Message-Id: <1520716927-17068-10-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::244 Subject: [Qemu-devel] [PATCH V5 09/17] COLO: Flush memory data from ram cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint time. So, the content of SVM's RAM cache will always be same with PVM's memory after checkpoint. Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way: Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's. Besides, we must ensure flush RAM cache before load device state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 39 +++++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 2 files changed, 41 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 4534c7a..ff7265e 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2927,6 +2927,40 @@ static bool postcopy_is_running(void) return ps >= POSTCOPY_INCOMING_LISTENING && ps < POSTCOPY_INCOMING_END; } +/* + * Flush content of RAM cache into SVM's memory. + * Only flush the pages that be dirtied by PVM or SVM or both. + */ +static void colo_flush_ram_cache(void) +{ + RAMBlock *block = NULL; + void *dst_host; + void *src_host; + unsigned long offset = 0; + + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); + rcu_read_lock(); + block = QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset = migration_bitmap_find_dirty(ram_state, block, offset); + migration_bitmap_clear_dirty(ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >= block->used_length) { + offset = 0; + block = QLIST_NEXT_RCU(block, next); + } else { + dst_host = block->host + (offset << TARGET_PAGE_BITS); + src_host = block->colo_cache + (offset << TARGET_PAGE_BITS); + memcpy(dst_host, src_host, TARGET_PAGE_SIZE); + } + } + + rcu_read_unlock(); + trace_colo_flush_ram_cache_end(); + assert(ram_state->migration_dirty_pages == 0); +} + static int ram_load(QEMUFile *f, void *opaque, int version_id) { int flags = 0, ret = 0, invalid_flags = 0; @@ -2939,6 +2973,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) bool postcopy_running = postcopy_is_running(); /* ADVISE is earlier, it shows the source has the postcopy capability on */ bool postcopy_advised = postcopy_is_advised(); + bool need_flush = false; seq_iter++; @@ -3114,6 +3149,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) wait_for_decompress_done(); rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); + + if (!ret && migration_incoming_in_colo_state() && need_flush) { + colo_flush_ram_cache(); + } return ret; } diff --git a/migration/trace-events b/migration/trace-events index b27a3a0..a3a124b 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -78,6 +78,8 @@ ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x" ram_postcopy_send_discard_bitmap(void) "" ram_save_page(const char *rbname, uint64_t offset, void *host) "%s: offset: 0x%" PRIx64 " host: %p" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: start: 0x%zx len: 0x%zx" +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" # migration/migration.c await_return_path_close_on_source_close(void) ""