From patchwork Fri Mar 4 08:41:20 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Changlong Xie X-Patchwork-Id: 8500261 Return-Path: X-Original-To: patchwork-xen-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id E53A39F659 for ; Fri, 4 Mar 2016 08:42:48 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 820AC2035D for ; Fri, 4 Mar 2016 08:42:46 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8EF6A20221 for ; Fri, 4 Mar 2016 08:42:42 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xen.org with esmtp (Exim 4.84) (envelope-from ) id 1ablHN-00070D-7F; Fri, 04 Mar 2016 08:40:09 +0000 Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.84) (envelope-from ) id 1ablHL-0006Rh-9D for xen-devel@lists.xen.org; Fri, 04 Mar 2016 08:40:07 +0000 Received: from [193.109.254.147] by server-7.bemta-14.messagelabs.com id FA/D2-04065-7E949D65; Fri, 04 Mar 2016 08:40:07 +0000 X-Env-Sender: xiecl.fnst@cn.fujitsu.com X-Msg-Ref: server-9.tower-27.messagelabs.com!1457080794!28608841!6 X-Originating-IP: [59.151.112.132] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 8.11; banners=-,-,- X-VirusChecked: Checked Received: (qmail 11428 invoked from network); 4 Mar 2016 08:40:04 -0000 Received: from cn.fujitsu.com (HELO heian.cn.fujitsu.com) (59.151.112.132) by server-9.tower-27.messagelabs.com with SMTP; 4 Mar 2016 08:40:04 -0000 X-IronPort-AV: E=Sophos;i="5.22,518,1449504000"; d="scan'208";a="4230660" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 04 Mar 2016 16:40:03 +0800 Received: from G08CNEXCHPEKD01.g08.fujitsu.local (unknown [10.167.33.80]) by cn.fujitsu.com (Postfix) with ESMTP id 04F8542B4AFB; Fri, 4 Mar 2016 16:39:58 +0800 (CST) Received: from changlox.g08.fujitsu.local (10.167.225.55) by G08CNEXCHPEKD01.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP Server (TLS) id 14.3.181.6; Fri, 4 Mar 2016 16:39:57 +0800 From: Changlong Xie To: xen devel , Konrad Rzeszutek Wilk , Andrew Cooper , Ian Campbell , Ian Jackson , Wei Liu Date: Fri, 4 Mar 2016 16:41:20 +0800 Message-ID: <1457080891-26054-17-git-send-email-xiecl.fnst@cn.fujitsu.com> X-Mailer: git-send-email 1.9.3 In-Reply-To: <1457080891-26054-1-git-send-email-xiecl.fnst@cn.fujitsu.com> References: <1457080891-26054-1-git-send-email-xiecl.fnst@cn.fujitsu.com> MIME-Version: 1.0 X-Originating-IP: [10.167.225.55] X-yoursite-MailScanner-ID: 04F8542B4AFB.A4575 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: xiecl.fnst@cn.fujitsu.com X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Cc: Lars Kurth , Changlong Xie , Wen Congyang , Gui Jianfeng , Jiang Yunhong , Dong Eddie , Anthony Perard , Shriram Rajagopalan , Yang Hongyang Subject: [Xen-devel] [PATCH v11 16/27] libxc/restore: support COLO restore X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Wen Congyang a. call callbacks resume/checkpoint/suspend while secondary vm status is consistent with primary b. send dirty pfn list to primary when checkpoint under colo c. send store gfn and console gfn to xl before resuming secondary vm Signed-off-by: Yang Hongyang Signed-off-by: Wen Congyang Signed-off-by: Changlong Xie CC: Andrew Cooper --- tools/libxc/include/xenguest.h | 8 ++ tools/libxc/xc_sr_common.h | 8 +- tools/libxc/xc_sr_restore.c | 181 +++++++++++++++++++++++++++++++++++++ tools/libxl/libxl_colo_restore.c | 5 - tools/libxl/libxl_create.c | 3 + tools/libxl/libxl_save_msgs_gen.pl | 4 +- 6 files changed, 200 insertions(+), 9 deletions(-) diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h index cb0bbcf..67cb540 100644 --- a/tools/libxc/include/xenguest.h +++ b/tools/libxc/include/xenguest.h @@ -127,6 +127,14 @@ struct restore_callbacks { */ int (*wait_checkpoint)(void* data); + /* + * callback to send store gfn and console gfn to xl + * if we want to resume vm before xc_domain_save() + * exits. + */ + void (*restore_results)(xen_pfn_t store_gfn, xen_pfn_t console_gfn, + void *data); + /* to be provided as the last argument to each callback function */ void* data; }; diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h index 2bfed64..29ab4eb 100644 --- a/tools/libxc/xc_sr_common.h +++ b/tools/libxc/xc_sr_common.h @@ -226,6 +226,10 @@ struct xc_sr_context struct xc_sr_restore_ops ops; struct restore_callbacks *callbacks; + int send_back_fd; + unsigned long p2m_size; + xc_hypercall_buffer_t dirty_bitmap_hbuf; + /* From Image Header. */ uint32_t format_version; @@ -234,13 +238,13 @@ struct xc_sr_context uint32_t guest_page_size; /* Plain VM, or checkpoints over time. */ - bool checkpointed; + int checkpointed; /* Currently buffering records between a checkpoint */ bool buffer_all_records; /* - * With Remus, we buffer the records sent by the primary at checkpoint, + * With Remus/COLO, we buffer the records sent by the primary at checkpoint, * in case the primary will fail, we can recover from the last * checkpoint state. * This should be enough for most of the cases because primary only send diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c index aef9bca..24c4719 100644 --- a/tools/libxc/xc_sr_restore.c +++ b/tools/libxc/xc_sr_restore.c @@ -411,6 +411,92 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec) return rc; } +/* + * Send checkpoint dirty pfn list to primary. + */ +static int send_checkpoint_dirty_pfn_list(struct xc_sr_context *ctx) +{ + xc_interface *xch = ctx->xch; + int rc = -1; + unsigned count, written; + uint64_t i, *pfns = NULL; + struct iovec *iov = NULL; + xc_shadow_op_stats_t stats = { 0, ctx->restore.p2m_size }; + struct xc_sr_record rec = + { + .type = REC_TYPE_CHECKPOINT_DIRTY_PFN_LIST, + }; + DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap, + &ctx->restore.dirty_bitmap_hbuf); + + if ( xc_shadow_control( + xch, ctx->domid, XEN_DOMCTL_SHADOW_OP_CLEAN, + HYPERCALL_BUFFER(dirty_bitmap), ctx->restore.p2m_size, + NULL, 0, &stats) != ctx->restore.p2m_size ) + { + PERROR("Failed to retrieve logdirty bitmap"); + goto err; + } + + for ( i = 0, count = 0; i < ctx->restore.p2m_size; i++ ) + { + if ( test_bit(i, dirty_bitmap) ) + count++; + } + + + pfns = malloc(count * sizeof(*pfns)); + if ( !pfns ) + { + ERROR("Unable to allocate %zu bytes of memory for dirty pfn list", + count * sizeof(*pfns)); + goto err; + } + + for ( i = 0, written = 0; i < ctx->restore.p2m_size; ++i ) + { + if ( !test_bit(i, dirty_bitmap) ) + continue; + + if ( written > count ) + { + ERROR("Dirty pfn list exceed"); + goto err; + } + + pfns[written++] = i; + } + + /* iovec[] for writev(). */ + iov = malloc(3 * sizeof(*iov)); + if ( !iov ) + { + ERROR("Unable to allocate memory for sending dirty bitmap"); + goto err; + } + + rec.length = count * sizeof(*pfns); + + iov[0].iov_base = &rec.type; + iov[0].iov_len = sizeof(rec.type); + + iov[1].iov_base = &rec.length; + iov[1].iov_len = sizeof(rec.length); + + iov[2].iov_base = pfns; + iov[2].iov_len = count * sizeof(*pfns); + + if ( writev_exact(ctx->restore.send_back_fd, iov, 3) ) + { + PERROR("Failed to write dirty bitmap to stream"); + goto err; + } + + rc = 0; + err: + return rc; +} + static int process_record(struct xc_sr_context *ctx, struct xc_sr_record *rec); static int handle_checkpoint(struct xc_sr_context *ctx) { @@ -460,6 +546,53 @@ static int handle_checkpoint(struct xc_sr_context *ctx) else ctx->restore.buffer_all_records = true; + if ( ctx->restore.checkpointed == MIG_STREAM_COLO ) + { +#define HANDLE_CALLBACK_RETURN_VALUE(ret) \ + do { \ + if ( ret == 1 ) \ + rc = 0; /* Success */ \ + else \ + { \ + if ( ret == 2 ) \ + rc = BROKEN_CHANNEL; \ + else \ + rc = -1; /* Some unspecified error */ \ + goto err; \ + } \ + } while (0) + + /* COLO */ + + /* We need to resume guest */ + rc = ctx->restore.ops.stream_complete(ctx); + if ( rc ) + goto err; + + ctx->restore.callbacks->restore_results(ctx->restore.xenstore_gfn, + ctx->restore.console_gfn, + ctx->restore.callbacks->data); + + /* Resume secondary vm */ + ret = ctx->restore.callbacks->postcopy(ctx->restore.callbacks->data); + HANDLE_CALLBACK_RETURN_VALUE(ret); + + /* Wait for a new checkpoint */ + ret = ctx->restore.callbacks->wait_checkpoint( + ctx->restore.callbacks->data); + HANDLE_CALLBACK_RETURN_VALUE(ret); + + /* suspend secondary vm */ + ret = ctx->restore.callbacks->suspend(ctx->restore.callbacks->data); + HANDLE_CALLBACK_RETURN_VALUE(ret); + +#undef HANDLE_CALLBACK_RETURN_VALUE + + rc = send_checkpoint_dirty_pfn_list(ctx); + if ( rc ) + goto err; + } + err: return rc; } @@ -529,6 +662,21 @@ static int setup(struct xc_sr_context *ctx) { xc_interface *xch = ctx->xch; int rc; + DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap, + &ctx->restore.dirty_bitmap_hbuf); + + if ( ctx->restore.checkpointed == MIG_STREAM_COLO ) + { + dirty_bitmap = xc_hypercall_buffer_alloc_pages(xch, dirty_bitmap, + NRPAGES(bitmap_size(ctx->restore.p2m_size))); + + if ( !dirty_bitmap ) + { + ERROR("Unable to allocate memory for dirty bitmap"); + rc = -1; + goto err; + } + } rc = ctx->restore.ops.setup(ctx); if ( rc ) @@ -562,10 +710,15 @@ static void cleanup(struct xc_sr_context *ctx) { xc_interface *xch = ctx->xch; unsigned i; + DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap, + &ctx->restore.dirty_bitmap_hbuf); for ( i = 0; i < ctx->restore.buffered_rec_num; i++ ) free(ctx->restore.buffered_records[i].data); + if ( ctx->restore.checkpointed == MIG_STREAM_COLO ) + xc_hypercall_buffer_free_pages(xch, dirty_bitmap, + NRPAGES(bitmap_size(ctx->restore.p2m_size))); free(ctx->restore.buffered_records); free(ctx->restore.populated_pfns); if ( ctx->restore.ops.cleanup(ctx) ) @@ -631,6 +784,15 @@ static int restore(struct xc_sr_context *ctx) } while ( rec.type != REC_TYPE_END ); remus_failover: + + if ( ctx->restore.checkpointed == MIG_STREAM_COLO ) + { + /* With COLO, we have already called stream_complete */ + rc = 0; + IPRINTF("COLO Failover"); + goto done; + } + /* * With Remus, if we reach here, there must be some error on primary, * failover from the last checkpoint state. @@ -667,6 +829,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, int checkpointed_stream, struct restore_callbacks *callbacks, int send_back_fd) { + xen_pfn_t nr_pfns; struct xc_sr_context ctx = { .xch = xch, @@ -680,11 +843,21 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, ctx.restore.xenstore_domid = store_domid; ctx.restore.checkpointed = checkpointed_stream; ctx.restore.callbacks = callbacks; + ctx.restore.send_back_fd = send_back_fd; /* Sanity checks for callbacks. */ if ( checkpointed_stream ) assert(callbacks->checkpoint); + if ( ctx.restore.checkpointed == MIG_STREAM_COLO ) + { + /* this is COLO restore */ + assert(callbacks->suspend && + callbacks->postcopy && + callbacks->wait_checkpoint && + callbacks->restore_results); + } + DPRINTF("fd %d, dom %u, hvm %u, pae %u, superpages %d" ", checkpointed_stream %d", io_fd, dom, hvm, pae, superpages, checkpointed_stream); @@ -706,6 +879,14 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, if ( read_headers(&ctx) ) return -1; + if ( xc_domain_nr_gpfns(xch, dom, &nr_pfns) < 0 ) + { + PERROR("Unable to obtain the guest p2m size"); + return -1; + } + + ctx.restore.p2m_size = nr_pfns; + if ( ctx.dominfo.hvm ) { ctx.restore.ops = restore_ops_x86_hvm; diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c index 829687e..b0dd52b 100644 --- a/tools/libxl/libxl_colo_restore.c +++ b/tools/libxl/libxl_colo_restore.c @@ -125,11 +125,6 @@ static void colo_resume_vm(libxl__egc *egc, return; } - /* - * TODO: get store gfn and console gfn - * We should call the callback restore_results in - * xc_domain_restore() before resuming the guest. - */ libxl__xc_domain_restore_done(egc, dcs, 0, 0, 0); return; diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c index 7301ac4..3610a39 100644 --- a/tools/libxl/libxl_create.c +++ b/tools/libxl/libxl_create.c @@ -1013,6 +1013,8 @@ static void domcreate_bootloader_done(libxl__egc *egc, const int checkpointed_stream = dcs->restore_params.checkpointed_stream; libxl__colo_restore_state *const crs = &dcs->crs; libxl_domain_build_info *const info = &d_config->b_info; + libxl__srm_restore_autogen_callbacks *const callbacks = + &dcs->srs.shs.callbacks.restore.a; if (rc) { domcreate_rebuild_done(egc, dcs, rc); @@ -1040,6 +1042,7 @@ static void domcreate_bootloader_done(libxl__egc *egc, } /* Restore */ + callbacks->restore_results = libxl__srm_callout_callback_restore_results; /* COLO only supports HVM now */ if (info->type != LIBXL_DOMAIN_TYPE_HVM && diff --git a/tools/libxl/libxl_save_msgs_gen.pl b/tools/libxl/libxl_save_msgs_gen.pl index 6016706..c2243f2 100755 --- a/tools/libxl/libxl_save_msgs_gen.pl +++ b/tools/libxl/libxl_save_msgs_gen.pl @@ -29,8 +29,8 @@ our @msgs = ( [ 6, 'srcxA', "wait_checkpoint", [] ], [ 7, 'scxA', "switch_qemu_logdirty", [qw(int domid unsigned enable)] ], - [ 8, 'r', "restore_results", ['unsigned long', 'store_mfn', - 'unsigned long', 'console_mfn'] ], + [ 8, 'rcx', "restore_results", ['unsigned long', 'store_gfn', + 'unsigned long', 'console_gfn'] ], [ 9, 'srW', "complete", [qw(int retval int errnoval)] ], );