mbox series

[RFC,v2,00/12] Confidential guest-assisted live migration

Message ID 20210823141636.65975-1-dovmurik@linux.ibm.com (mailing list archive)
Headers show
Series Confidential guest-assisted live migration | expand


Dov Murik Aug. 23, 2021, 2:16 p.m. UTC
This is an RFC series for fast migration of confidential guests using an
in-guest migration helper that lives in OVMF.  QEMU VM live migration
needs to read source VM's RAM and write it in the target VM; this
mechanism doesn't work when the guest memory is encrypted or QEMU is
prevented from reading it in another way.  In order to support live
migration in such scenarios, we introduce an in-guest migration helper
which can securely extract RAM content from the guest in order to send
it to the target.  The migration helper is implemented as part of the
VM's firmware in OVMF.

We've implemented and tested this on AMD SEV, but expect most of the
processes can be used with other technologies that prevent direct access
of hypervisor to the guest's memory.  Specifically, we don't use SEV's
PSP migration commands (SEV_SEND_START, SEV_RECEIVE_START, etc) at all;
but note that the mirror VM relies on KVM_CAP_VM_COPY_ENC_CONTEXT_FROM
to shared the SEV ASID with the main VM.

Corresponding RFC patches for OVMF have been posted by Tobin
Feldman-Fitzthum on edk2-devel [1].  Those include the crux of the
migration helper: a mailbox protocol over a shared memory page which
allows communication between QEMU and the migration helper.  In the
source VM this is used to read a page and encrypt it for transport; in
the target it is used to decrypt the incoming page and storing the
content in the correct address in the guest memory.  All encryption and
decryption operations occur inside the trusted context in the VM, and
therefore the VM's memory plaintext content is never accessible to the
hosts participating in the migration.

In order to allow OVMF to run the migration helper in parallel to the
guest OS, we use a mirror VM [3], which shares the same memory mapping
and SEV ASID as the main VM but has its own run loop.  To start the
mirror vcpu and the migration handler, we added a temporary
start-migration-handler QMP command; this will be removed in a future
version to run as part of the migrate QMP command.

In the target VM we need the migration handler running to receive
incoming RAM pages; to achieve that, we boot the VM into OVMF with a
special fw_cfg value that causes OVMF to not boot the guest OS; we then
allow QEMU to receive an incoming migration by issuing a new
start-migrate-incoming QMP command.

The confidential RAM migration requires checking whether a given guest
RAM page is encrypted or not.  This is achieved using SEV shared regions
list tracking, which is implemented as part the SEV live migration patch
series [2].  This feature tracks hypercalls from OVMF and guest Linux to
report changes of page encryption status so that QEMU has an up-to-date
view of which memory regions are shared and which are encrypted.

We left a few unfinished edges in this RFC but decided to publish it to
start the commmunity discussion.  TODOs:

1. QMP commands start-migration-handler and start-migrate-incoming are
   developer tools and should be performed automatically.
2. The entry point address of the in-guest migration handler and its GDT
   are currently hard-coded in QEMU (patch 8); instead they should be
   discovered using pc_system_ovmf_table_find.  Same applies for the
   mailbox address (patch 1).
3. For simplicity, this patch series forces the use of the 
   guest-assisted migration instead of the SEV PSP-based migration. 
   Ideally we might want the user to choose the desired mode using
   migrate-set-parameters or a similar mechanism.
4. There is currently no discovery protocol between QEMU and OVMF to
   verify that OVMF indeed supports in-guest migration handler.

List of patches in this series:

1-3: introduce new confidtial RAM migration functions which communicate
     with the migration helper.
4-6: use the new MH communication functions when migrating encrypted RAM
7-9: allow starting migration handler on mirror vcpu with QMP command 
10:  introduce the start-migrate-incoming QMP command to switch the
     target into accepting the incoming migration.
11:  fix devices issues when loading state into a live VM
12:  add documentation

This patch series is based on top of:

1. Add SEV guest live migration support, from Ashish Kalra [2]
2. Support for mirror VM, from Ashish Kalra [3]

[1] https://edk2.groups.io/g/devel/message/79517
[2] https://lore.kernel.org/qemu-devel/cover.1628076205.git.ashish.kalra@amd.com/
[3] https://lore.kernel.org/qemu-devel/cover.1629118207.git.ashish.kalra@amd.com/

Changes from RFC v1:
 - Use the an SEV mirror VM for the migation handler (instead of
   auxilliary vcpus)

RFC v1:

Dov Murik (12):
  migration: Add helpers to save confidential RAM
  migration: Add helpers to load confidential RAM
  migration: Introduce gpa_inside_migration_helper_shared_area
  migration: Save confidential guest RAM using migration helper
  migration: Load confidential guest RAM using migration helper
  migration: Skip ROM, non-RAM, and vga.vram memory region during RAM
  i386/kvm: Exclude mirror vcpu in kvm_synchronize_all_tsc
  migration: Allow resetting the mirror vcpu to the MH entry point
  migration: Add QMP command start-migration-handler
  migration: Add start-migrate-incoming QMP command
  hw/isa/lpc_ich9: Allow updating an already-running VM
  docs: Add confidential guest live migration documentation

 docs/confidential-guest-live-migration.rst | 145 +++++++++
 docs/confidential-guest-support.txt        |   5 +
 docs/index.rst                             |   1 +
 qapi/migration.json                        |  38 +++
 include/sysemu/sev.h                       |   1 +
 migration/confidential-ram.h               |  23 ++
 hw/isa/lpc_ich9.c                          |   3 +-
 migration/confidential-ram.c               | 339 +++++++++++++++++++++
 migration/migration.c                      |  29 ++
 migration/ram.c                            | 133 +++++++-
 target/i386/kvm/kvm.c                      |   4 +-
 migration/meson.build                      |   2 +-
 migration/trace-events                     |   4 +
 13 files changed, 714 insertions(+), 13 deletions(-)
 create mode 100644 docs/confidential-guest-live-migration.rst
 create mode 100644 migration/confidential-ram.h
 create mode 100644 migration/confidential-ram.c