diff mbox series

migration: cpr-reboot documentation

Message ID 1710338119-330923-1-git-send-email-steven.sistare@oracle.com (mailing list archive)
State New, archived
Headers show
Series migration: cpr-reboot documentation | expand

Commit Message

Steven Sistare March 13, 2024, 1:55 p.m. UTC 
Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
 docs/devel/migration/CPR.rst      | 147 ++++++++++++++++++++++++++++++++++++++
 docs/devel/migration/features.rst |   1 +
 2 files changed, 148 insertions(+)
 create mode 100644 docs/devel/migration/CPR.rst

Comments

Cédric Le Goater March 13, 2024, 2:15 p.m. UTC | #1 
On 3/13/24 14:55, Steve Sistare wrote:
> Signed-off-by: Steve Sistare <steven.sistare@oracle.com>

This would be good to have for 9.0. Anyhow,

Reviewed-by: Cédric Le Goater <clg@redhat.com>

Thanks,

C.


> ---
>   docs/devel/migration/CPR.rst      | 147 ++++++++++++++++++++++++++++++++++++++
>   docs/devel/migration/features.rst |   1 +
>   2 files changed, 148 insertions(+)
>   create mode 100644 docs/devel/migration/CPR.rst
> 
> diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst
> new file mode 100644
> index 0000000..63c3647
> --- /dev/null
> +++ b/docs/devel/migration/CPR.rst
> @@ -0,0 +1,147 @@
> +CheckPoint and Restart (CPR)
> +============================
> +
> +CPR is the umbrella name for a set of migration modes in which the
> +VM is migrated to a new QEMU instance on the same host.  It is
> +intended for use when the goal is to update host software components
> +that run the VM, such as QEMU or even the host kernel.  At this time,
> +cpr-reboot is the only available mode.
> +
> +Because QEMU is restarted on the same host, with access to the same
> +local devices, CPR is allowed in certain cases where normal migration
> +would be blocked.  However, the user must not modify the contents of
> +guest block devices between quitting old QEMU and starting new QEMU.
> +
> +CPR unconditionally stops VM execution before memory is saved, and
> +thus does not depend on any form of dirty page tracking.
> +
> +cpr-reboot mode
> +---------------
> +
> +In this mode, QEMU stops the VM, and writes VM state to the migration
> +URI, which will typically be a file.  After quitting QEMU, the user
> +resumes by running QEMU with the ``-incoming`` option.  Because the
> +old and new QEMU instances are not active concurrently, the URI cannot
> +be a type that streams data from one instance to the other.
> +
> +Guest RAM can be saved in place if backed by shared memory, or can be
> +copied to a file.  The former is more efficient and is therefore
> +preferred.
> +
> +After state and memory are saved, the user may update userland host
> +software before restarting QEMU and resuming the VM.  Further, if
> +the RAM is backed by persistent shared memory, such as a DAX device,
> +then the user may reboot to a new host kernel before restarting QEMU.
> +
> +This mode supports VFIO devices provided the user first puts the
> +guest in the suspended runstate, such as by issuing the
> +``guest-suspend-ram`` command to the QEMU guest agent.  The agent
> +must be pre-installed in the guest, and the guest must support
> +suspend to RAM.  Beware that suspension can take a few seconds, so
> +the user should poll to see the suspended state before proceeding
> +with the CPR operation.
> +
> +Usage
> +^^^^^
> +
> +It is recommended that guest RAM be backed with some type of shared
> +memory, such as ``memory-backend-file,share=on``, and that the
> +``x-ignore-shared`` capability be set.  This combination allows memory
> +to be saved in place.  Otherwise, after QEMU stops the VM, all guest
> +RAM is copied to the migration URI.
> +
> +Outgoing:
> +  * Set the migration mode parameter to ``cpr-reboot``.
> +  * Set the ``x-ignore-shared`` capability if desired.
> +  * Issue the ``migrate`` command.  It is recommended the the URI be a
> +    ``file`` type, but one can use other types such as ``exec``,
> +    provided the command captures all the data from the outgoing side,
> +    and provides all the data to the incoming side.
> +  * Quit when QEMU reaches the postmigrate state.
> +
> +Incoming:
> +  * Start QEMU with the ``-incoming defer`` option.
> +  * Set the migration mode parameter to ``cpr-reboot``.
> +  * Set the ``x-ignore-shared`` capability if desired.
> +  * Issue the ``migrate-incoming`` command.
> +  * If the VM was running when the outgoing ``migrate`` command was
> +    issued, then QEMU automatically resumes VM execution.
> +
> +Example 1
> +^^^^^^^^^
> +::
> +
> +  # qemu-kvm -monitor stdio
> +  -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G
> +  ...
> +
> +  (qemu) info status
> +  VM status: running
> +  (qemu) migrate_set_parameter mode cpr-reboot
> +  (qemu) migrate_set_capability x-ignore-shared on
> +  (qemu) migrate -d file:vm.state
> +  (qemu) info status
> +  VM status: paused (postmigrate)
> +  (qemu) quit
> +
> +  ### optionally update kernel and reboot
> +  # systemctl kexec
> +  kexec_core: Starting new kernel
> +  ...
> +
> +  # qemu-kvm ... -incoming defer
> +  (qemu) info status
> +  VM status: paused (inmigrate)
> +  (qemu) migrate_set_parameter mode cpr-reboot
> +  (qemu) migrate_set_capability x-ignore-shared on
> +  (qemu) migrate_incoming file:vm.state
> +  (qemu) info status
> +  VM status: running
> +
> +Example 2: VFIO
> +^^^^^^^^^^^^^^^
> +::
> +
> +  # qemu-kvm -monitor stdio
> +  -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G
> +  -device vfio-pci, ...
> +  -chardev socket,id=qga0,path=qga.sock,server=on,wait=off
> +  -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0
> +  ...
> +
> +  (qemu) info status
> +  VM status: running
> +
> +  # echo '{"execute":"guest-suspend-ram"}' | ncat --send-only -U qga.sock
> +
> +  (qemu) info status
> +  VM status: paused (suspended)
> +  (qemu) migrate_set_parameter mode cpr-reboot
> +  (qemu) migrate_set_capability x-ignore-shared on
> +  (qemu) migrate -d file:vm.state
> +  (qemu) info status
> +  VM status: paused (postmigrate)
> +  (qemu) quit
> +
> +  ### optionally update kernel and reboot
> +  # systemctl kexec
> +  kexec_core: Starting new kernel
> +  ...
> +
> +  # qemu-kvm ... -incoming defer
> +  (qemu) info status
> +  VM status: paused (inmigrate)
> +  (qemu) migrate_set_parameter mode cpr-reboot
> +  (qemu) migrate_set_capability x-ignore-shared on
> +  (qemu) migrate_incoming file:vm.state
> +  (qemu) info status
> +  VM status: paused (suspended)
> +  (qemu) system_wakeup
> +  (qemu) info status
> +  VM status: running
> +
> +Caveats
> +^^^^^^^
> +
> +cpr-reboot mode may not be used with postcopy, background-snapshot,
> +or COLO.
> diff --git a/docs/devel/migration/features.rst b/docs/devel/migration/features.rst
> index 9d1abd2..d5ca7b8 100644
> --- a/docs/devel/migration/features.rst
> +++ b/docs/devel/migration/features.rst
> @@ -11,3 +11,4 @@ Migration has plenty of features to support different use cases.
>      vfio
>      virtio
>      mapped-ram
> +   CPR
Fabiano Rosas March 13, 2024, 2:27 p.m. UTC | #2 
Steve Sistare <steven.sistare@oracle.com> writes:

> Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
> ---
>  docs/devel/migration/CPR.rst      | 147 ++++++++++++++++++++++++++++++++++++++
>  docs/devel/migration/features.rst |   1 +
>  2 files changed, 148 insertions(+)
>  create mode 100644 docs/devel/migration/CPR.rst
>
> diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst
> new file mode 100644
> index 0000000..63c3647
> --- /dev/null
> +++ b/docs/devel/migration/CPR.rst
> @@ -0,0 +1,147 @@
> +CheckPoint and Restart (CPR)
> +============================
> +
> +CPR is the umbrella name for a set of migration modes in which the
> +VM is migrated to a new QEMU instance on the same host.  It is
> +intended for use when the goal is to update host software components
> +that run the VM, such as QEMU or even the host kernel.  At this time,
> +cpr-reboot is the only available mode.
> +
> +Because QEMU is restarted on the same host, with access to the same
> +local devices, CPR is allowed in certain cases where normal migration
> +would be blocked.  However, the user must not modify the contents of
> +guest block devices between quitting old QEMU and starting new QEMU.
> +
> +CPR unconditionally stops VM execution before memory is saved, and
> +thus does not depend on any form of dirty page tracking.
> +
> +cpr-reboot mode
> +---------------
> +
> +In this mode, QEMU stops the VM, and writes VM state to the migration
> +URI, which will typically be a file.  After quitting QEMU, the user
> +resumes by running QEMU with the ``-incoming`` option.  Because the
> +old and new QEMU instances are not active concurrently, the URI cannot
> +be a type that streams data from one instance to the other.
> +
> +Guest RAM can be saved in place if backed by shared memory, or can be
> +copied to a file.  The former is more efficient and is therefore
> +preferred.
> +
> +After state and memory are saved, the user may update userland host
> +software before restarting QEMU and resuming the VM.  Further, if
> +the RAM is backed by persistent shared memory, such as a DAX device,
> +then the user may reboot to a new host kernel before restarting QEMU.
> +
> +This mode supports VFIO devices provided the user first puts the
> +guest in the suspended runstate, such as by issuing the
> +``guest-suspend-ram`` command to the QEMU guest agent.  The agent
> +must be pre-installed in the guest, and the guest must support
> +suspend to RAM.  Beware that suspension can take a few seconds, so
> +the user should poll to see the suspended state before proceeding
> +with the CPR operation.
> +
> +Usage
> +^^^^^
> +
> +It is recommended that guest RAM be backed with some type of shared
> +memory, such as ``memory-backend-file,share=on``, and that the
> +``x-ignore-shared`` capability be set.  This combination allows memory
> +to be saved in place.  Otherwise, after QEMU stops the VM, all guest
> +RAM is copied to the migration URI.
> +
> +Outgoing:
> +  * Set the migration mode parameter to ``cpr-reboot``.
> +  * Set the ``x-ignore-shared`` capability if desired.
> +  * Issue the ``migrate`` command.  It is recommended the the URI be a
> +    ``file`` type, but one can use other types such as ``exec``,
> +    provided the command captures all the data from the outgoing side,
> +    and provides all the data to the incoming side.
> +  * Quit when QEMU reaches the postmigrate state.
> +
> +Incoming:
> +  * Start QEMU with the ``-incoming defer`` option.
> +  * Set the migration mode parameter to ``cpr-reboot``.
> +  * Set the ``x-ignore-shared`` capability if desired.
> +  * Issue the ``migrate-incoming`` command.
> +  * If the VM was running when the outgoing ``migrate`` command was
> +    issued, then QEMU automatically resumes VM execution.
> +
> +Example 1
> +^^^^^^^^^
> +::
> +
> +  # qemu-kvm -monitor stdio
> +  -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G
> +  ...
> +
> +  (qemu) info status
> +  VM status: running
> +  (qemu) migrate_set_parameter mode cpr-reboot
> +  (qemu) migrate_set_capability x-ignore-shared on
> +  (qemu) migrate -d file:vm.state
> +  (qemu) info status
> +  VM status: paused (postmigrate)
> +  (qemu) quit
> +
> +  ### optionally update kernel and reboot
> +  # systemctl kexec
> +  kexec_core: Starting new kernel
> +  ...
> +
> +  # qemu-kvm ... -incoming defer
> +  (qemu) info status
> +  VM status: paused (inmigrate)
> +  (qemu) migrate_set_parameter mode cpr-reboot
> +  (qemu) migrate_set_capability x-ignore-shared on
> +  (qemu) migrate_incoming file:vm.state
> +  (qemu) info status
> +  VM status: running
> +
> +Example 2: VFIO
> +^^^^^^^^^^^^^^^
> +::
> +
> +  # qemu-kvm -monitor stdio
> +  -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G
> +  -device vfio-pci, ...
> +  -chardev socket,id=qga0,path=qga.sock,server=on,wait=off
> +  -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0
> +  ...
> +
> +  (qemu) info status
> +  VM status: running
> +
> +  # echo '{"execute":"guest-suspend-ram"}' | ncat --send-only -U qga.sock
> +
> +  (qemu) info status
> +  VM status: paused (suspended)
> +  (qemu) migrate_set_parameter mode cpr-reboot
> +  (qemu) migrate_set_capability x-ignore-shared on
> +  (qemu) migrate -d file:vm.state
> +  (qemu) info status
> +  VM status: paused (postmigrate)
> +  (qemu) quit
> +
> +  ### optionally update kernel and reboot
> +  # systemctl kexec
> +  kexec_core: Starting new kernel
> +  ...
> +
> +  # qemu-kvm ... -incoming defer
> +  (qemu) info status
> +  VM status: paused (inmigrate)
> +  (qemu) migrate_set_parameter mode cpr-reboot
> +  (qemu) migrate_set_capability x-ignore-shared on
> +  (qemu) migrate_incoming file:vm.state
> +  (qemu) info status
> +  VM status: paused (suspended)
> +  (qemu) system_wakeup
> +  (qemu) info status
> +  VM status: running
> +
> +Caveats
> +^^^^^^^
> +
> +cpr-reboot mode may not be used with postcopy, background-snapshot,
> +or COLO.
> diff --git a/docs/devel/migration/features.rst b/docs/devel/migration/features.rst
> index 9d1abd2..d5ca7b8 100644
> --- a/docs/devel/migration/features.rst
> +++ b/docs/devel/migration/features.rst
> @@ -11,3 +11,4 @@ Migration has plenty of features to support different use cases.
>     vfio
>     virtio
>     mapped-ram
> +   CPR

Reviewed-by: Fabiano Rosas <farosas@suse.de>
Peter Xu March 13, 2024, 3:11 p.m. UTC | #3 
On Wed, Mar 13, 2024 at 06:55:19AM -0700, Steve Sistare wrote:
> Signed-off-by: Steve Sistare <steven.sistare@oracle.com>

Thanks a lot.  This looks much better..  queued it for 9.0-rc0.

One note below.

> ---
>  docs/devel/migration/CPR.rst      | 147 ++++++++++++++++++++++++++++++++++++++
>  docs/devel/migration/features.rst |   1 +
>  2 files changed, 148 insertions(+)
>  create mode 100644 docs/devel/migration/CPR.rst
> 
> diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst
> new file mode 100644
> index 0000000..63c3647
> --- /dev/null
> +++ b/docs/devel/migration/CPR.rst
> @@ -0,0 +1,147 @@
> +CheckPoint and Restart (CPR)
> +============================
> +
> +CPR is the umbrella name for a set of migration modes in which the
> +VM is migrated to a new QEMU instance on the same host.  It is
> +intended for use when the goal is to update host software components
> +that run the VM, such as QEMU or even the host kernel.  At this time,
> +cpr-reboot is the only available mode.
> +
> +Because QEMU is restarted on the same host, with access to the same
> +local devices, CPR is allowed in certain cases where normal migration
> +would be blocked.  However, the user must not modify the contents of
> +guest block devices between quitting old QEMU and starting new QEMU.
> +
> +CPR unconditionally stops VM execution before memory is saved, and
> +thus does not depend on any form of dirty page tracking.

Note that this is not completely true in reality even if conceptally true..

static void ram_init_bitmaps(RAMState *rs)
{
    qemu_mutex_lock_ramlist();

    WITH_RCU_READ_LOCK_GUARD() {
        ram_list_init_bitmaps();
        /* We don't use dirty log with background snapshots */
        if (!migrate_background_snapshot()) {
            memory_global_dirty_log_start(GLOBAL_DIRTY_MIGRATION);
            migration_bitmap_sync_precopy(rs, false);
        }
    }
    ...
}

If you want we can make it true in reality with some further changes.  I
don't think this matters a huge deal in reality as of now, but note that
Cedric is planning to add support to allow ram setup() to even fail, e.g.,
when there's no enough VFIO dirty tracking resources on the device.

It means it might still be ideal this can be fixed some day so that cpr
will bypass the tracking operations in code when attached with a VFIO
device that is capable of precopy live migration (but in this case
HW/driver support will be ignored for a cpr upgrade), because there's a
chance memory_global_dirty_log_start() will fail then even if the tracking
unit won't be used.

> +
> +cpr-reboot mode
> +---------------
> +
> +In this mode, QEMU stops the VM, and writes VM state to the migration
> +URI, which will typically be a file.  After quitting QEMU, the user
> +resumes by running QEMU with the ``-incoming`` option.  Because the
> +old and new QEMU instances are not active concurrently, the URI cannot
> +be a type that streams data from one instance to the other.
> +
> +Guest RAM can be saved in place if backed by shared memory, or can be
> +copied to a file.  The former is more efficient and is therefore
> +preferred.
> +
> +After state and memory are saved, the user may update userland host
> +software before restarting QEMU and resuming the VM.  Further, if
> +the RAM is backed by persistent shared memory, such as a DAX device,
> +then the user may reboot to a new host kernel before restarting QEMU.
> +
> +This mode supports VFIO devices provided the user first puts the
> +guest in the suspended runstate, such as by issuing the
> +``guest-suspend-ram`` command to the QEMU guest agent.  The agent
> +must be pre-installed in the guest, and the guest must support
> +suspend to RAM.  Beware that suspension can take a few seconds, so
> +the user should poll to see the suspended state before proceeding
> +with the CPR operation.
> +
> +Usage
> +^^^^^
> +
> +It is recommended that guest RAM be backed with some type of shared
> +memory, such as ``memory-backend-file,share=on``, and that the
> +``x-ignore-shared`` capability be set.  This combination allows memory
> +to be saved in place.  Otherwise, after QEMU stops the VM, all guest
> +RAM is copied to the migration URI.
> +
> +Outgoing:
> +  * Set the migration mode parameter to ``cpr-reboot``.
> +  * Set the ``x-ignore-shared`` capability if desired.
> +  * Issue the ``migrate`` command.  It is recommended the the URI be a
> +    ``file`` type, but one can use other types such as ``exec``,
> +    provided the command captures all the data from the outgoing side,
> +    and provides all the data to the incoming side.
> +  * Quit when QEMU reaches the postmigrate state.
> +
> +Incoming:
> +  * Start QEMU with the ``-incoming defer`` option.
> +  * Set the migration mode parameter to ``cpr-reboot``.
> +  * Set the ``x-ignore-shared`` capability if desired.
> +  * Issue the ``migrate-incoming`` command.
> +  * If the VM was running when the outgoing ``migrate`` command was
> +    issued, then QEMU automatically resumes VM execution.
> +
> +Example 1
> +^^^^^^^^^
> +::
> +
> +  # qemu-kvm -monitor stdio
> +  -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G
> +  ...
> +
> +  (qemu) info status
> +  VM status: running
> +  (qemu) migrate_set_parameter mode cpr-reboot
> +  (qemu) migrate_set_capability x-ignore-shared on
> +  (qemu) migrate -d file:vm.state
> +  (qemu) info status
> +  VM status: paused (postmigrate)
> +  (qemu) quit
> +
> +  ### optionally update kernel and reboot
> +  # systemctl kexec
> +  kexec_core: Starting new kernel
> +  ...
> +
> +  # qemu-kvm ... -incoming defer
> +  (qemu) info status
> +  VM status: paused (inmigrate)
> +  (qemu) migrate_set_parameter mode cpr-reboot
> +  (qemu) migrate_set_capability x-ignore-shared on
> +  (qemu) migrate_incoming file:vm.state
> +  (qemu) info status
> +  VM status: running
> +
> +Example 2: VFIO
> +^^^^^^^^^^^^^^^
> +::
> +
> +  # qemu-kvm -monitor stdio
> +  -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G
> +  -device vfio-pci, ...
> +  -chardev socket,id=qga0,path=qga.sock,server=on,wait=off
> +  -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0
> +  ...
> +
> +  (qemu) info status
> +  VM status: running
> +
> +  # echo '{"execute":"guest-suspend-ram"}' | ncat --send-only -U qga.sock
> +
> +  (qemu) info status
> +  VM status: paused (suspended)
> +  (qemu) migrate_set_parameter mode cpr-reboot
> +  (qemu) migrate_set_capability x-ignore-shared on
> +  (qemu) migrate -d file:vm.state
> +  (qemu) info status
> +  VM status: paused (postmigrate)
> +  (qemu) quit
> +
> +  ### optionally update kernel and reboot
> +  # systemctl kexec
> +  kexec_core: Starting new kernel
> +  ...
> +
> +  # qemu-kvm ... -incoming defer
> +  (qemu) info status
> +  VM status: paused (inmigrate)
> +  (qemu) migrate_set_parameter mode cpr-reboot
> +  (qemu) migrate_set_capability x-ignore-shared on
> +  (qemu) migrate_incoming file:vm.state
> +  (qemu) info status
> +  VM status: paused (suspended)
> +  (qemu) system_wakeup
> +  (qemu) info status
> +  VM status: running
> +
> +Caveats
> +^^^^^^^
> +
> +cpr-reboot mode may not be used with postcopy, background-snapshot,
> +or COLO.
> diff --git a/docs/devel/migration/features.rst b/docs/devel/migration/features.rst
> index 9d1abd2..d5ca7b8 100644
> --- a/docs/devel/migration/features.rst
> +++ b/docs/devel/migration/features.rst
> @@ -11,3 +11,4 @@ Migration has plenty of features to support different use cases.
>     vfio
>     virtio
>     mapped-ram
> +   CPR
> -- 
> 1.8.3.1
>
diff mbox series

Patch

diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst
new file mode 100644
index 0000000..63c3647
--- /dev/null
+++ b/docs/devel/migration/CPR.rst
@@ -0,0 +1,147 @@ 
+CheckPoint and Restart (CPR)
+============================
+
+CPR is the umbrella name for a set of migration modes in which the
+VM is migrated to a new QEMU instance on the same host.  It is
+intended for use when the goal is to update host software components
+that run the VM, such as QEMU or even the host kernel.  At this time,
+cpr-reboot is the only available mode.
+
+Because QEMU is restarted on the same host, with access to the same
+local devices, CPR is allowed in certain cases where normal migration
+would be blocked.  However, the user must not modify the contents of
+guest block devices between quitting old QEMU and starting new QEMU.
+
+CPR unconditionally stops VM execution before memory is saved, and
+thus does not depend on any form of dirty page tracking.
+
+cpr-reboot mode
+---------------
+
+In this mode, QEMU stops the VM, and writes VM state to the migration
+URI, which will typically be a file.  After quitting QEMU, the user
+resumes by running QEMU with the ``-incoming`` option.  Because the
+old and new QEMU instances are not active concurrently, the URI cannot
+be a type that streams data from one instance to the other.
+
+Guest RAM can be saved in place if backed by shared memory, or can be
+copied to a file.  The former is more efficient and is therefore
+preferred.
+
+After state and memory are saved, the user may update userland host
+software before restarting QEMU and resuming the VM.  Further, if
+the RAM is backed by persistent shared memory, such as a DAX device,
+then the user may reboot to a new host kernel before restarting QEMU.
+
+This mode supports VFIO devices provided the user first puts the
+guest in the suspended runstate, such as by issuing the
+``guest-suspend-ram`` command to the QEMU guest agent.  The agent
+must be pre-installed in the guest, and the guest must support
+suspend to RAM.  Beware that suspension can take a few seconds, so
+the user should poll to see the suspended state before proceeding
+with the CPR operation.
+
+Usage
+^^^^^
+
+It is recommended that guest RAM be backed with some type of shared
+memory, such as ``memory-backend-file,share=on``, and that the
+``x-ignore-shared`` capability be set.  This combination allows memory
+to be saved in place.  Otherwise, after QEMU stops the VM, all guest
+RAM is copied to the migration URI.
+
+Outgoing:
+  * Set the migration mode parameter to ``cpr-reboot``.
+  * Set the ``x-ignore-shared`` capability if desired.
+  * Issue the ``migrate`` command.  It is recommended the the URI be a
+    ``file`` type, but one can use other types such as ``exec``,
+    provided the command captures all the data from the outgoing side,
+    and provides all the data to the incoming side.
+  * Quit when QEMU reaches the postmigrate state.
+
+Incoming:
+  * Start QEMU with the ``-incoming defer`` option.
+  * Set the migration mode parameter to ``cpr-reboot``.
+  * Set the ``x-ignore-shared`` capability if desired.
+  * Issue the ``migrate-incoming`` command.
+  * If the VM was running when the outgoing ``migrate`` command was
+    issued, then QEMU automatically resumes VM execution.
+
+Example 1
+^^^^^^^^^
+::
+
+  # qemu-kvm -monitor stdio
+  -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G
+  ...
+
+  (qemu) info status
+  VM status: running
+  (qemu) migrate_set_parameter mode cpr-reboot
+  (qemu) migrate_set_capability x-ignore-shared on
+  (qemu) migrate -d file:vm.state
+  (qemu) info status
+  VM status: paused (postmigrate)
+  (qemu) quit
+
+  ### optionally update kernel and reboot
+  # systemctl kexec
+  kexec_core: Starting new kernel
+  ...
+
+  # qemu-kvm ... -incoming defer
+  (qemu) info status
+  VM status: paused (inmigrate)
+  (qemu) migrate_set_parameter mode cpr-reboot
+  (qemu) migrate_set_capability x-ignore-shared on
+  (qemu) migrate_incoming file:vm.state
+  (qemu) info status
+  VM status: running
+
+Example 2: VFIO
+^^^^^^^^^^^^^^^
+::
+
+  # qemu-kvm -monitor stdio
+  -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G
+  -device vfio-pci, ...
+  -chardev socket,id=qga0,path=qga.sock,server=on,wait=off
+  -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0
+  ...
+
+  (qemu) info status
+  VM status: running
+
+  # echo '{"execute":"guest-suspend-ram"}' | ncat --send-only -U qga.sock
+
+  (qemu) info status
+  VM status: paused (suspended)
+  (qemu) migrate_set_parameter mode cpr-reboot
+  (qemu) migrate_set_capability x-ignore-shared on
+  (qemu) migrate -d file:vm.state
+  (qemu) info status
+  VM status: paused (postmigrate)
+  (qemu) quit
+
+  ### optionally update kernel and reboot
+  # systemctl kexec
+  kexec_core: Starting new kernel
+  ...
+
+  # qemu-kvm ... -incoming defer
+  (qemu) info status
+  VM status: paused (inmigrate)
+  (qemu) migrate_set_parameter mode cpr-reboot
+  (qemu) migrate_set_capability x-ignore-shared on
+  (qemu) migrate_incoming file:vm.state
+  (qemu) info status
+  VM status: paused (suspended)
+  (qemu) system_wakeup
+  (qemu) info status
+  VM status: running
+
+Caveats
+^^^^^^^
+
+cpr-reboot mode may not be used with postcopy, background-snapshot,
+or COLO.
diff --git a/docs/devel/migration/features.rst b/docs/devel/migration/features.rst
index 9d1abd2..d5ca7b8 100644
--- a/docs/devel/migration/features.rst
+++ b/docs/devel/migration/features.rst
@@ -11,3 +11,4 @@  Migration has plenty of features to support different use cases.
    vfio
    virtio
    mapped-ram
+   CPR