From patchwork Thu Sep 14 15:34:01 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jennifer Herbert X-Patchwork-Id: 9953385 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A0C63603F4 for ; Thu, 14 Sep 2017 15:38:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9958629130 for ; Thu, 14 Sep 2017 15:38:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8E35529147; Thu, 14 Sep 2017 15:38:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 29CA929148 for ; Thu, 14 Sep 2017 15:38:44 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dsWBQ-0006Uj-On; Thu, 14 Sep 2017 15:36:04 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dsWBO-0006Rz-Py for xen-devel@lists.xenproject.org; Thu, 14 Sep 2017 15:36:02 +0000 Received: from [85.158.137.68] by server-14.bemta-3.messagelabs.com id 3F/16-01910-1E1AAB95; Thu, 14 Sep 2017 15:36:01 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupjkeJIrShJLcpLzFFi42JxWrohUvfhwl2 RBuc2ilt83zKZyYHR4/CHKywBjFGsmXlJ+RUJrBlrN0oW/POo2LZ4L3MD4xGzLkZODgkBf4lf 71YydjFycLAJmEi8eRDaxcjFISLQwiix+NQudpA4s4CRRNtRbpByYQFziXvvTrCA2CwCqhKfH t5iBLF5BXwknh5cwwgxUkFiysP3zCA2p4CvxP5p39lAxggB1Rxe7g0SFhLQkDi7YBErRKugxM mZT8BGMgtISBx88YJ5AiPvLCSpWUhSCxiZVjFqFKcWlaUW6RpZ6CUVZaZnlOQmZuboGhoY6+W mFhcnpqfmJCYV6yXn525iBIZNPQMD4w7G9hN+hxglOZiURHn36u6MFOJLyk+pzEgszogvKs1J LT7EKMPBoSTBywAMQyHBotT01Iq0zBxgAMOkJTh4lER4ExcApXmLCxJzizPTIVKnGHU5Om7e/ cMkxJKXn5cqJc7LCDJDAKQoozQPbgQsmi4xykoJ8zIyMDAI8RSkFuVmlqDKv2IU52BUEuZ9BL KKJzOvBG7TK6AjmICOOHN6B8gRJYkIKakGxujX2e+TvnI8X779hlTmm7jzi/6snnDHfsWp9IB Yd4O8x6+mLbjGbzKjN5rrtPGkD7VKS6SSrjWf3/N9WbD5YRbpZPc1xqs50tq7j73b/XlCTJVW 4LVfd8Od2D4+0AlL0Huya4/Dzderki032K1ZcMb4/LtKBrblCdkLUnhuLWa++rqYpcx751kll uKMREMt5qLiRADows44oQIAAA== X-Env-Sender: prvs=423118b0b=jennifer.herbert@citrix.com X-Msg-Ref: server-4.tower-31.messagelabs.com!1505403358!56759863!1 X-Originating-IP: [66.165.176.89] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni44OSA9PiAyMDMwMDc=\n, received_headers: No Received headers X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 56124 invoked from network); 14 Sep 2017 15:36:00 -0000 Received: from smtp.citrix.com (HELO SMTP.CITRIX.COM) (66.165.176.89) by server-4.tower-31.messagelabs.com with RC4-SHA encrypted SMTP; 14 Sep 2017 15:36:00 -0000 X-IronPort-AV: E=Sophos;i="5.42,393,1500940800"; d="scan'208";a="439785057" From: Jennifer Herbert To: Ian Jackson , Wei Liu , , Date: Thu, 14 Sep 2017 16:34:01 +0100 Message-ID: <1505403241-111368-6-git-send-email-Jennifer.Herbert@citrix.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1505403241-111368-1-git-send-email-Jennifer.Herbert@citrix.com> References: <1505403241-111368-1-git-send-email-Jennifer.Herbert@citrix.com> MIME-Version: 1.0 Cc: Jennifer Herbert Subject: [Xen-devel] [PATCH 2/2] Introduce migration precopy policy X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP This Patch allows a migration precopy policy to be specified. The precopy phase of the xc_domain_save() live migration algorithm has historically been implemented to run until either a) (almost) no pages are dirty or b) some fixed, hard-coded maximum number of precopy iterations has been exceeded. This policy and its implementation are less than ideal for a few reasons: - the logic of the policy is intertwined with the control flow of the mechanism of the precopy stage - it can't take into account facts external to the immediate migration context, such external state transfer state, interactive user input, or the passage of wall-clock time. - it does not permit the user to change their mind, over time, about what to do at the end of the precopy (they get an unconditional transition into the stop-and-copy phase of the migration) To permit callers to implement arbitrary higher-level policies governing when the live migration precopy phase should end, and what should be done next: - add a precopy_policy() callback to the xc_domain_save() user-supplied callbacks - during the precopy phase of live migrations, consult this policy after each batch of pages transmitted and take the dictated action, which may be to a) abort the migration entirely, b) continue with the precopy, or c) proceed to the stop-and-copy phase. - provide an implementation of the old policy, used when precopy_policy callback is not provided. Signed-off-by: Jennifer Herbert --- This is updated/modified subset of patch 7/20, part of Joshua Otto's "Add postcopy live migration support." patch, dated 27th March 2017. As indicated on the original thread, I wish to make use of this this within the XenServer product. I hope this will aid Josh in pushing the remainder of his series. --- tools/libxc/include/xenguest.h | 19 ++++++++ tools/libxc/xc_sr_common.h | 7 ++- tools/libxc/xc_sr_save.c | 102 +++++++++++++++++++++++++++++------------ 3 files changed, 94 insertions(+), 34 deletions(-) diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h index 6626f0c..d5908dc 100644 --- a/tools/libxc/include/xenguest.h +++ b/tools/libxc/include/xenguest.h @@ -39,6 +39,14 @@ */ struct xenevtchn_handle; +/* For save's precopy_policy(). */ +struct precopy_stats +{ + unsigned iteration; + unsigned total_written; + long dirty_count; /* -1 if unknown */ +}; + /* callbacks provided by xc_domain_save */ struct save_callbacks { /* Called after expiration of checkpoint interval, @@ -46,6 +54,17 @@ struct save_callbacks { */ int (*suspend)(void* data); + /* Called after every batch of page data sent during the precopy phase of a + * live migration to ask the caller what to do next based on the current + * state of the precopy migration. + */ +#define XGS_POLICY_ABORT (-1) /* Abandon the migration entirely and + * tidy up. */ +#define XGS_POLICY_CONTINUE_PRECOPY 0 /* Remain in the precopy phase. */ +#define XGS_POLICY_STOP_AND_COPY 1 /* Immediately suspend and transmit the + * remaining dirty pages. */ + int (*precopy_policy)(struct precopy_stats stats, void *data); + /* Called after the guest's dirty pages have been * copied into an output buffer. * Callback function resumes the guest & the device model, diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h index a83f22a..2bc261b 100644 --- a/tools/libxc/xc_sr_common.h +++ b/tools/libxc/xc_sr_common.h @@ -198,12 +198,11 @@ struct xc_sr_context /* Further debugging information in the stream. */ bool debug; - /* Parameters for tweaking live migration. */ - unsigned max_iterations; - unsigned dirty_threshold; - unsigned long p2m_size; + struct precopy_stats stats; + int policy_decision; + xen_pfn_t *batch_pfns; unsigned nr_batch_pfns; unsigned long *deferred_pages; diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c index 1e7502d..03dfa61 100644 --- a/tools/libxc/xc_sr_save.c +++ b/tools/libxc/xc_sr_save.c @@ -452,8 +452,7 @@ static int update_progress_string(struct xc_sr_context *ctx, xc_interface *xch = ctx->xch; char *new_str = NULL; - if ( asprintf(&new_str, "Frames iteration %u of %u", - iter, ctx->save.max_iterations) == -1 ) + if ( asprintf(&new_str, "Frames iteration %u", iter) == -1 ) { PERROR("Unable to allocate new progress string"); return -1; @@ -467,6 +466,25 @@ static int update_progress_string(struct xc_sr_context *ctx, } /* + * This is the live migration precopy policy - it's called periodically during + * the precopy phase of live migrations, and is responsible for deciding when + * the precopy phase should terminate and what should be done next. + * + * The policy implemented here behaves identically to the policy previously + * hard-coded into xc_domain_save() - it proceeds to the stop-and-copy phase of + * the live migration when there are either fewer than 50 dirty pages, or more + * than 5 precopy rounds have completed. + */ +static int simple_precopy_policy( + struct precopy_stats stats, void *user) +{ + return ((stats.dirty_count >= 0 && stats.dirty_count < 50) || + stats.iteration >= 5) + ? XGS_POLICY_STOP_AND_COPY + : XGS_POLICY_CONTINUE_PRECOPY; +} + +/* * Send memory while guest is running. */ static int send_memory_live(struct xc_sr_context *ctx) @@ -474,21 +492,62 @@ static int send_memory_live(struct xc_sr_context *ctx) xc_interface *xch = ctx->xch; xc_shadow_op_stats_t stats = { 0, ctx->save.p2m_size }; char *progress_str = NULL; - unsigned x; + unsigned int x = 0; int rc; + int policy_decision; + + DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap, + &ctx->save.dirty_bitmap_hbuf); + + int (*precopy_policy)(struct precopy_stats, void *) = + ctx->save.callbacks->precopy_policy; + void *data = ctx->save.callbacks->data; + + struct precopy_stats *policy_stats; rc = update_progress_string(ctx, &progress_str, 0); if ( rc ) goto out; - rc = send_all_pages(ctx); - if ( rc ) - goto out; + ctx->save.stats = (struct precopy_stats) + { + .iteration = x, + .total_written = 0, + .dirty_count = ctx->save.p2m_size + }; + policy_stats = &ctx->save.stats; + + if (precopy_policy == NULL) + precopy_policy = simple_precopy_policy; + + bitmap_set(dirty_bitmap, ctx->save.p2m_size); + + do { + policy_decision = precopy_policy(*policy_stats, data); + x++; + + if ( stats.dirty_count > 0 && policy_decision != XGS_POLICY_ABORT ) { + rc = update_progress_string(ctx, &progress_str, x); + if ( rc ) + goto out; + + rc = send_dirty_pages(ctx, stats.dirty_count); + if ( rc ) + goto out; + } + + if (policy_decision != XGS_POLICY_CONTINUE_PRECOPY) + break; + + policy_stats->iteration = x; + policy_stats->total_written += policy_stats->dirty_count; + policy_stats->dirty_count = -1; + + policy_decision = precopy_policy(*policy_stats, data); + + if (policy_decision != XGS_POLICY_CONTINUE_PRECOPY) + break; - for ( x = 1; - ((x < ctx->save.max_iterations) && - (stats.dirty_count > ctx->save.dirty_threshold)); ++x ) - { if ( xc_shadow_control( xch, ctx->domid, XEN_DOMCTL_SHADOW_OP_CLEAN, &ctx->save.dirty_bitmap_hbuf, ctx->save.p2m_size, @@ -499,17 +558,9 @@ static int send_memory_live(struct xc_sr_context *ctx) goto out; } - if ( stats.dirty_count == 0 ) - break; - - rc = update_progress_string(ctx, &progress_str, x); - if ( rc ) - goto out; + policy_stats->dirty_count = stats.dirty_count; - rc = send_dirty_pages(ctx, stats.dirty_count); - if ( rc ) - goto out; - } + } while (true); out: xc_set_progress_prefix(xch, NULL); @@ -601,7 +652,7 @@ static int suspend_and_send_dirty(struct xc_sr_context *ctx) if ( ctx->save.live ) { rc = update_progress_string(ctx, &progress_str, - ctx->save.max_iterations); + ctx->save.stats.iteration); if ( rc ) goto out; } @@ -937,15 +988,6 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, stream_type == XC_MIG_STREAM_REMUS || stream_type == XC_MIG_STREAM_COLO); - /* - * TODO: Find some time to better tweak the live migration algorithm. - * - * These parameters are better than the legacy algorithm especially for - * busy guests. - */ - ctx.save.max_iterations = 5; - ctx.save.dirty_threshold = 50; - /* Sanity checks for callbacks. */ if ( hvm ) assert(callbacks->switch_qemu_logdirty);