[v5,COLOPre,07/18] migration/save: pass checkpointed_stream from libxl to libxc

Commit Message

Wen Congyang Dec. 17, 2015, 7:48 a.m. UTC

From: Yang Hongyang <[e-mail]>

Pass checkpointed_stream from libxl to libxc.
It won't affact legacy migration because legacy migration
won't use this param.

Signed-off-by: Yang Hongyang <[e-mail]>
Signed-off-by: Wen Congyang <[e-mail]>
CC: Ian Campbell <[e-mail]>
CC: Ian Jackson <[e-mail]>
CC: Wei Liu <[e-mail]>
CC: Andrew Cooper <[e-mail]>
---
 tools/libxc/include/xenguest.h   |  6 ++++--
 tools/libxc/xc_nomigrate.c       |  3 ++-
 tools/libxc/xc_sr_common.h       | 12 +++++++++++-
 tools/libxc/xc_sr_save.c         | 16 +++++++++-------
 tools/libxl/libxl.c              |  2 ++
 tools/libxl/libxl_dom_save.c     | 11 ++++++++---
 tools/libxl/libxl_internal.h     |  1 +
 tools/libxl/libxl_save_callout.c |  2 +-
 tools/libxl/libxl_save_helper.c  |  3 ++-
 tools/libxl/libxl_types.idl      |  1 +
 10 files changed, 41 insertions(+), 16 deletions(-)

Comments

Konrad Rzeszutek Wilk Jan. 25, 2016, 6:38 p.m. UTC | #1

On Thu, Dec 17, 2015 at 03:48:11PM +0800, Wen Congyang wrote:
> From: Yang Hongyang <[e-mail]>
> 
> Pass checkpointed_stream from libxl to libxc.
> It won't affact legacy migration because legacy migration
> won't use this param.

Ah, that is why you wanted to keep it as in 'int' in the struct
domain_create. You may want to update patch #6 to mention that
you need it to pass it on libxc.


..snip..
> @@ -834,7 +836,7 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
>      ctx.save.callbacks = callbacks;
>      ctx.save.live  = !!(flags & XCFLAGS_LIVE);
>      ctx.save.debug = !!(flags & XCFLAGS_DEBUG);
> -    ctx.save.checkpointed = !!(flags & XCFLAGS_CHECKPOINTED);
> +    ctx.save.checkpointed = checkpointed_stream;

Should you have an check to make sure they are the right enums?

/* If altering migration_stream update this assert too. */
assert(checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_NONE || 
	checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_REMUS);


.. snip..

> diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
> index 3ef11aa..9aa94be 100644
> --- a/tools/libxl/libxl_types.idl
> +++ b/tools/libxl/libxl_types.idl
> @@ -228,6 +228,7 @@ libxl_hdtype = Enumeration("hdtype", [
>      (2, "AHCI"),
>      ], init_val = "LIBXL_HDTYPE_IDE")
>  
> +# Consistent with the values defined for migration_stream

s/stream/stream./

>  libxl_checkpointed_stream = Enumeration("checkpointed_stream", [
>      (0, "NONE"),
>      (1, "REMUS"),
> -- 
> 2.5.0
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> [e-mail]
> http://lists.xen.org/xen-devel

Konrad Rzeszutek Wilk Jan. 25, 2016, 7:14 p.m. UTC | #2

On Mon, Jan 25, 2016 at 01:38:36PM -0500, Konrad Rzeszutek Wilk wrote:
> On Thu, Dec 17, 2015 at 03:48:11PM +0800, Wen Congyang wrote:
> > From: Yang Hongyang <[e-mail]>
> > 
> > Pass checkpointed_stream from libxl to libxc.
> > It won't affact legacy migration because legacy migration
> > won't use this param.
> 
> Ah, that is why you wanted to keep it as in 'int' in the struct
> domain_create. You may want to update patch #6 to mention that
> you need it to pass it on libxc.
> 
> 
> ..snip..
> > @@ -834,7 +836,7 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
> >      ctx.save.callbacks = callbacks;
> >      ctx.save.live  = !!(flags & XCFLAGS_LIVE);
> >      ctx.save.debug = !!(flags & XCFLAGS_DEBUG);
> > -    ctx.save.checkpointed = !!(flags & XCFLAGS_CHECKPOINTED);
> > +    ctx.save.checkpointed = checkpointed_stream;
> 
> Should you have an check to make sure they are the right enums?
> 
> /* If altering migration_stream update this assert too. */
> assert(checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_NONE || 
> 	checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_REMUS);

Oops.

That should have been MIG_STREAM_REMUS and MIG_STREAM_NONE.

And sorry about responding under the wrong patchset (v5) instead
of the (v6)! They look the same so it ought to be ok?
> 
> 
> .. snip..
> 
> > diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
> > index 3ef11aa..9aa94be 100644
> > --- a/tools/libxl/libxl_types.idl
> > +++ b/tools/libxl/libxl_types.idl
> > @@ -228,6 +228,7 @@ libxl_hdtype = Enumeration("hdtype", [
> >      (2, "AHCI"),
> >      ], init_val = "LIBXL_HDTYPE_IDE")
> >  
> > +# Consistent with the values defined for migration_stream
> 
> s/stream/stream./
> 
> >  libxl_checkpointed_stream = Enumeration("checkpointed_stream", [
> >      (0, "NONE"),
> >      (1, "REMUS"),
> > -- 
> > 2.5.0
> > 
> > 
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > [e-mail]
> > http://lists.xen.org/xen-devel

Wen Congyang Jan. 26, 2016, 5:42 a.m. UTC | #3

On 01/26/2016 03:14 AM, Konrad Rzeszutek Wilk wrote:
> On Mon, Jan 25, 2016 at 01:38:36PM -0500, Konrad Rzeszutek Wilk wrote:
>> On Thu, Dec 17, 2015 at 03:48:11PM +0800, Wen Congyang wrote:
>>> From: Yang Hongyang <[e-mail]>
>>>
>>> Pass checkpointed_stream from libxl to libxc.
>>> It won't affact legacy migration because legacy migration
>>> won't use this param.
>>
>> Ah, that is why you wanted to keep it as in 'int' in the struct
>> domain_create. You may want to update patch #6 to mention that
>> you need it to pass it on libxc.
>>
>>
>> ..snip..
>>> @@ -834,7 +836,7 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
>>>      ctx.save.callbacks = callbacks;
>>>      ctx.save.live  = !!(flags & XCFLAGS_LIVE);
>>>      ctx.save.debug = !!(flags & XCFLAGS_DEBUG);
>>> -    ctx.save.checkpointed = !!(flags & XCFLAGS_CHECKPOINTED);
>>> +    ctx.save.checkpointed = checkpointed_stream;
>>
>> Should you have an check to make sure they are the right enums?
>>
>> /* If altering migration_stream update this assert too. */
>> assert(checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_NONE || 
>> 	checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_REMUS);
> 
> Oops.
> 
> That should have been MIG_STREAM_REMUS and MIG_STREAM_NONE.
> 
> And sorry about responding under the wrong patchset (v5) instead
> of the (v6)! They look the same so it ought to be ok?

It is OK. I will update it in the next version.

Thanks
Wen Congyang

>>
>>
>> .. snip..
>>
>>> diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
>>> index 3ef11aa..9aa94be 100644
>>> --- a/tools/libxl/libxl_types.idl
>>> +++ b/tools/libxl/libxl_types.idl
>>> @@ -228,6 +228,7 @@ libxl_hdtype = Enumeration("hdtype", [
>>>      (2, "AHCI"),
>>>      ], init_val = "LIBXL_HDTYPE_IDE")
>>>  
>>> +# Consistent with the values defined for migration_stream
>>
>> s/stream/stream./
>>
>>>  libxl_checkpointed_stream = Enumeration("checkpointed_stream", [
>>>      (0, "NONE"),
>>>      (1, "REMUS"),
>>> -- 
>>> 2.5.0
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> [e-mail]
>>> http://lists.xen.org/xen-devel
> 
> 
> .
>

Patch

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index 8f918b1..e8bc46d 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -29,7 +29,6 @@ 
 #define XCFLAGS_HVM       (1 << 2)
 #define XCFLAGS_STDVGA    (1 << 3)
 #define XCFLAGS_CHECKPOINT_COMPRESS    (1 << 4)
-#define XCFLAGS_CHECKPOINTED    (1 << 5)
 
 #define X86_64_B_SIZE   64 
 #define X86_32_B_SIZE   32
@@ -76,11 +75,14 @@  struct save_callbacks {
  * @parm xch a handle to an open hypervisor interface
  * @parm fd the file descriptor to save a domain to
  * @parm dom the id of the domain
+ * @param checkpointed_stream MIG_STREAM_NONE if the far end of the stream
+ *        doesn't use checkpointing
  * @return 0 on success, -1 on failure
  */
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iters,
                    uint32_t max_factor, uint32_t flags /* XCFLAGS_xxx */,
-                   struct save_callbacks* callbacks, int hvm);
+                   struct save_callbacks* callbacks, int hvm,
+                   int checkpointed_stream);
 
 /* callbacks provided by xc_domain_restore */
 struct restore_callbacks {
diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c
index 902429e..c9124df 100644
--- a/tools/libxc/xc_nomigrate.c
+++ b/tools/libxc/xc_nomigrate.c
@@ -22,7 +22,8 @@ 
 
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iters,
                    uint32_t max_factor, uint32_t flags,
-                   struct save_callbacks* callbacks, int hvm)
+                   struct save_callbacks* callbacks, int hvm,
+                   int checkpointed_stream)
 {
     errno = ENOSYS;
     return -1;
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 9aecde2..bc99e9a 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -171,6 +171,16 @@  struct xc_sr_context
 
     xc_dominfo_t dominfo;
 
+    /*
+     * migration stream
+     * 0: Plain VM
+     * 1: Remus
+     */
+    enum {
+        MIG_STREAM_NONE, /* plain stream */
+        MIG_STREAM_REMUS,
+    } migration_stream;
+
     union /* Common save or restore data. */
     {
         struct /* Save data. */
@@ -182,7 +192,7 @@  struct xc_sr_context
             bool live;
 
             /* Plain VM, or checkpoints over time. */
-            bool checkpointed;
+            int checkpointed;
 
             /* Further debugging information in the stream. */
             bool debug;
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index cefcef5..a934187 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -628,7 +628,8 @@  static int send_domain_memory_live(struct xc_sr_context *ctx)
     if ( rc )
         goto out;
 
-    if ( ctx->save.debug && !ctx->save.checkpointed )
+    if ( ctx->save.debug &&
+         ctx->save.checkpointed != MIG_STREAM_NONE )
     {
         rc = verify_frames(ctx);
         if ( rc )
@@ -753,7 +754,7 @@  static int save(struct xc_sr_context *ctx, uint16_t guest_type)
 
         if ( ctx->save.live )
             rc = send_domain_memory_live(ctx);
-        else if ( ctx->save.checkpointed )
+        else if ( ctx->save.checkpointed != MIG_STREAM_NONE )
             rc = send_domain_memory_checkpointed(ctx);
         else
             rc = send_domain_memory_nonlive(ctx);
@@ -773,7 +774,7 @@  static int save(struct xc_sr_context *ctx, uint16_t guest_type)
         if ( rc )
             goto err;
 
-        if ( ctx->save.checkpointed )
+        if ( ctx->save.checkpointed != MIG_STREAM_NONE )
         {
             /*
              * We have now completed the initial live portion of the checkpoint
@@ -790,9 +791,9 @@  static int save(struct xc_sr_context *ctx, uint16_t guest_type)
 
             rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
             if ( rc <= 0 )
-                ctx->save.checkpointed = false;
+                ctx->save.checkpointed = MIG_STREAM_NONE;
         }
-    } while ( ctx->save.checkpointed );
+    } while ( ctx->save.checkpointed != MIG_STREAM_NONE );
 
     xc_report_progress_single(xch, "End of stream");
 
@@ -822,7 +823,8 @@  static int save(struct xc_sr_context *ctx, uint16_t guest_type)
 
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
                    uint32_t max_iters, uint32_t max_factor, uint32_t flags,
-                   struct save_callbacks* callbacks, int hvm)
+                   struct save_callbacks* callbacks, int hvm,
+                   int checkpointed_stream)
 {
     struct xc_sr_context ctx =
         {
@@ -834,7 +836,7 @@  int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
     ctx.save.callbacks = callbacks;
     ctx.save.live  = !!(flags & XCFLAGS_LIVE);
     ctx.save.debug = !!(flags & XCFLAGS_DEBUG);
-    ctx.save.checkpointed = !!(flags & XCFLAGS_CHECKPOINTED);
+    ctx.save.checkpointed = checkpointed_stream;
 
     /*
      * TODO: Find some time to better tweak the live migration algorithm.
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index f8f7158..2faea4d 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -877,6 +877,7 @@  int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
     dss->live = 1;
     dss->debug = 0;
     dss->remus = info;
+    dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
 
     assert(info);
 
@@ -937,6 +938,7 @@  int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, int flags,
     dss->type = type;
     dss->live = flags & LIBXL_SUSPEND_LIVE;
     dss->debug = flags & LIBXL_SUSPEND_DEBUG;
+    dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_NONE;
 
     rc = libxl__fd_flags_modify_save(gc, dss->fd,
                                      ~(O_NONBLOCK|O_NDELAY), 0,
diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index f185f82..0bfa5fc 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -338,6 +338,12 @@  void libxl__domain_save(libxl__egc *egc, libxl__domain_save_state *dss)
     unsigned int nr_vnodes = 0, nr_vmemranges = 0, nr_vcpus = 0;
     libxl__domain_suspend_state *dsps = &dss->dsps;
 
+    if (dss->checkpointed_stream != LIBXL_CHECKPOINTED_STREAM_NONE && !r_info) {
+        LOG(ERROR, "Migration stream is checkpointed, but there's no "
+                   "checkpoint info!");
+        goto out;
+    }
+
     dss->rc = 0;
     logdirty_init(&dss->logdirty);
     dsps->ao = ao;
@@ -376,15 +382,14 @@  void libxl__domain_save(libxl__egc *egc, libxl__domain_save_state *dss)
         goto out;
     }
 
-    if (r_info != NULL) {
+    if (dss->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_REMUS) {
         dss->interval = r_info->interval;
-        dss->xcflags |= XCFLAGS_CHECKPOINTED;
         if (libxl_defbool_val(r_info->compression))
             dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS;
     }
 
     memset(callbacks, 0, sizeof(*callbacks));
-    if (r_info != NULL) {
+    if (dss->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_REMUS) {
         callbacks->suspend = libxl__remus_domain_suspend_callback;
         callbacks->postcopy = libxl__remus_domain_resume_callback;
         callbacks->checkpoint = libxl__remus_domain_save_checkpoint_callback;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 20458ef..9470d96 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3108,6 +3108,7 @@  struct libxl__domain_save_state {
     libxl_domain_type type;
     int live;
     int debug;
+    int checkpointed_stream;
     const libxl_domain_remus_info *remus;
     /* private */
     int rc;
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 2d06b42..416b318 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -85,7 +85,7 @@  void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_save_state *dss,
 
     const unsigned long argnums[] = {
         dss->domid, 0, 0, dss->xcflags, dss->hvm,
-        cbflags,
+        cbflags, dss->checkpointed_stream,
     };
 
     shs->ao = ao;
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
index 39038f9..6bdcf13 100644
--- a/tools/libxl/libxl_save_helper.c
+++ b/tools/libxl/libxl_save_helper.c
@@ -253,6 +253,7 @@  int main(int argc, char **argv)
         uint32_t flags =           strtoul(NEXTARG,0,10);
         int hvm =                  atoi(NEXTARG);
         unsigned cbflags =         strtoul(NEXTARG,0,10);
+        int checkpointed_stream =  strtoul(NEXTARG,0,10);
         assert(!*++argv);
 
         helper_setcallbacks_save(&helper_save_callbacks, cbflags);
@@ -261,7 +262,7 @@  int main(int argc, char **argv)
         setup_signals(save_signal_handler);
 
         r = xc_domain_save(xch, io_fd, dom, max_iters, max_factor, flags,
-                           &helper_save_callbacks, hvm);
+                           &helper_save_callbacks, hvm, checkpointed_stream);
         complete(r);
 
     } else if (!strcmp(mode,"--restore-domain")) {
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 3ef11aa..9aa94be 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -228,6 +228,7 @@  libxl_hdtype = Enumeration("hdtype", [
     (2, "AHCI"),
     ], init_val = "LIBXL_HDTYPE_IDE")
 
+# Consistent with the values defined for migration_stream
 libxl_checkpointed_stream = Enumeration("checkpointed_stream", [
     (0, "NONE"),
     (1, "REMUS"),