| Message ID | 1736875434-106563-1-git-send-email-steven.sistare@oracle.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Live update: cpr-transfer | expand |
Hi Peter -- These still need RB. You said the first one "looks fine" in V5, and I only needed to split the qemu_ram_alloc_from_fd size calculation bug to a separate patch, which I did in V6: * physmem: qemu_ram_alloc_from_fd extensions * tests/qtest: assert qmp connected Hi Markus -- Patch "migration: incoming channel" has an Acked-by from Peter, but it would be good to also get your ack or RB, as I updated this patch in response to the comments you posted for V4. We are inches from the goal line ... - Steve On 1/14/2025 12:23 PM, Steve Sistare wrote: > What? > > This patch series adds the live migration cpr-transfer mode, which > allows the user to transfer a guest to a new QEMU instance on the same > host with minimal guest pause time, by preserving guest RAM in place, > albeit with new virtual addresses in new QEMU, and by preserving device > file descriptors. > > The new user-visible interfaces are: > * cpr-transfer (MigMode migration parameter) > * cpr (MigrationChannelType) > * incoming MigrationChannel (command-line argument) > * aux-ram-share (machine option) > > The user sets the mode parameter before invoking the migrate command. > In this mode, the user starts new QEMU on the same host as old QEMU, with > the same arguments as old QEMU, plus two -incoming options; one for the main > channel, and one for the CPR channel. The user issues the migrate command to > old QEMU, which stops the VM, saves state to the migration channels, and > enters the postmigrate state. Execution resumes in new QEMU. > > Memory-backend objects must have the share=on attribute, but memory-backend-epc > is not supported. The VM must be started with the '-machine aux-ram-share=on' > option, which allows auxilliary guest memory to be transferred in place to the > new process. > > This mode requires a second migration channel of type "cpr", in the channel > arguments on the outgoing side, and in a second -incoming command-line > parameter on the incoming side. This CPR channel must support file descriptor > transfer with SCM_RIGHTS, i.e. it must be a UNIX domain socket. > > Why? > > This mode has less impact on the guest than any other method of updating > in place. The pause time is much lower, because devices need not be torn > down and recreated, DMA does not need to be drained and quiesced, and minimal > state is copied to new QEMU. Further, there are no constraints on the guest. > By contrast, cpr-reboot mode requires the guest to support S3 suspend-to-ram, > and suspending plus resuming vfio devices adds multiple seconds to the > guest pause time. > > These benefits all derive from the core design principle of this mode, > which is preserving open descriptors. This approach is very general and > can be used to support a wide variety of devices that do not have hardware > support for live migration, including but not limited to: vfio, chardev, > vhost, vdpa, and iommufd. Some devices need new kernel software interfaces > to allow a descriptor to be used in a process that did not originally open it. > > How? > > All memory that is mapped by the guest is preserved in place. Indeed, > it must be, because it may be the target of DMA requests, which are not > quiesced during cpr-transfer. All such memory must be mmap'able in new QEMU. > This is easy for named memory-backend objects, as long as they are mapped > shared, because they are visible in the file system in both old and new QEMU. > Anonymous memory must be allocated using memfd_create rather than MAP_ANON, > so the memfd's can be sent to new QEMU. Pages that were locked in memory > for DMA in old QEMU remain locked in new QEMU, because the descriptor of > the device that locked them remains open. > > cpr-transfer preserves descriptors by sending them to new QEMU via the CPR > channel, which must support SCM_RIGHTS, and by sending the unique name of > each descriptor to new QEMU via CPR state. > > For device descriptors, new QEMU reuses the descriptor when creating the > device, rather than opening it again. For memfd descriptors, new QEMU > mmap's the preserved memfd when a ramblock is created. > > CPR state cannot be sent over the normal migration channel, because devices > and backends are created prior to reading the channel, so this mode sends > CPR state over a second "cpr" migration channel. New QEMU reads the second > channel prior to creating devices or backends. > > Example: > > In this example, we simply restart the same version of QEMU, but in > a real scenario one would use a new QEMU binary path in terminal 2. > > Terminal 1: start old QEMU > # qemu-kvm -qmp stdio -object > memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on > -m 4G -machine aux-ram-share=on ... > > Terminal 2: start new QEMU > # qemu-kvm -monitor stdio ... -incoming tcp:0:44444 > -incoming '{"channel-type": "cpr", > "addr": { "transport": "socket", "type": "unix", > "path": "cpr.sock"}}' > > Terminal 1: > {"execute":"qmp_capabilities"} > > {"execute": "query-status"} > {"return": {"status": "running", > "running": true}} > > {"execute":"migrate-set-parameters", > "arguments":{"mode":"cpr-transfer"}} > > {"execute": "migrate", "arguments": { "channels": [ > {"channel-type": "main", > "addr": { "transport": "socket", "type": "inet", > "host": "0", "port": "44444" }}, > {"channel-type": "cpr", > "addr": { "transport": "socket", "type": "unix", > "path": "cpr.sock" }}]}} > > {"execute": "query-status"} > {"return": {"status": "postmigrate", > "running": false}} > > Terminal 2: > QEMU 10.0.50 monitor - type 'help' for more information > (qemu) info status > VM status: running > > This patch series implements a minimal version of cpr-transfer. Additional > series are ready to be posted to deliver the complete vision described > above, including > * vfio > * chardev > * vhost and tap > * blockers > * cpr-exec mode > * iommufd > > Changes in V2: > * cpr-transfer is the first new mode proposed, and cpr-exec is deferred > * anon-alloc does not apply to memory-backend-object > * replaced hack with proper synchronization between source and target > * defined QEMU_CPR_FILE_MAGIC > * addressed misc review comments > > Changes in V3: > * added cpr-transfer to migration-test > * documented cpr-transfer in CPR.rst > * fix size_t trace format for 32-bit build > * drop explicit fd value in VMSTATE_FD > * defer cpr_walk_fd() and cpr_resave_fd() to later series > * drop "migration: save cpr mode". > delete mode from cpr state, and use cpr_uri to infer transfer mode. > * drop "migration: stop vm earlier for cpr" > * dropped cpr helpers, to be re-added later when needed > * fixed an unreported bug for cpr-transfer and migrate cancel > * documented cpr-transfer restrictions in qapi > * added trace for cpr_state_save and cpr_state_load > * added ftruncate to "preserve ram blocks" > > Changes in V4: > * cleaned up qtest deferred connection code > * renamed pass_fd -> can_pass_fd > * squashed patch "split qmp_migrate" > * deleted cpr-uri and its patches > * added cpr channel and its patches > * added patch "hostmem-shm: preserve for cpr" > * added patch "fd-based shared memory" > * added patch "factor out allocation of anonymous shared memory" > * added RAM_PRIVATE and its patch > * added aux-ram-share and its patch > > Changes in V5: > * added patch 'enhance migrate_uri_parse' > * supported dotted keys for -incoming channel, > and rewrote incoming_option_parse > * moved migrate_fd_cancel -> vm_resume to "stop vm earlier for cpr" > in a future series. > * updated command-line definition for aux-ram-share > * added patch "resizable qemu_ram_alloc_from_fd" > * rewrote patch "fd-based shared memory" > * fixed error message in qemu_shm_alloc > * added patch 'tests/qtest: optimize migrate_set_ports' > * added patch 'tests/qtest: enhance migration channels' > * added patch 'tests/qtest: assert qmp_ready' > * modified patch 'migration-test: cpr-transfer' > * polished the documentation in CPR.rst, qapi, and the > cpr-transfer mode commit message > * updated to master, and resolved massive context diffs for migration tests > > Changes in V6: > * added RB's and Acks. > * in patch "assert qmp_ready", deleted qmp_ready and checked qmp_fd instead. > renamed patch to ""assert qmp connected" > * factored out fix into new patch > "fix qemu_ram_alloc_from_fd size calculation" > * deleted a redundant call to migrate_hup_delete > * added commit message to "migration: cpr-transfer documentation" > * polished the text of cpr-transfer mode in qapi > > The first 10 patches below are foundational and are needed for both cpr-transfer > mode and the proposed cpr-exec mode. The next 6 patches are specific to > cpr-transfer and implement the mechanisms for sharing state across a socket > using SCM_RIGHTS. The last 8 patches supply tests and documentation. > > Steve Sistare (24): > backends/hostmem-shm: factor out allocation of "anonymous shared > memory with an fd" > physmem: fix qemu_ram_alloc_from_fd size calculation > physmem: qemu_ram_alloc_from_fd extensions > physmem: fd-based shared memory > memory: add RAM_PRIVATE > machine: aux-ram-share option > migration: cpr-state > physmem: preserve ram blocks for cpr > hostmem-memfd: preserve for cpr > hostmem-shm: preserve for cpr > migration: enhance migrate_uri_parse > migration: incoming channel > migration: SCM_RIGHTS for QEMUFile > migration: VMSTATE_FD > migration: cpr-transfer save and load > migration: cpr-transfer mode > migration-test: memory_backend > tests/qtest: optimize migrate_set_ports > tests/qtest: defer connection > migration-test: defer connection > tests/qtest: enhance migration channels > tests/qtest: assert qmp connected > migration-test: cpr-transfer > migration: cpr-transfer documentation > > backends/hostmem-epc.c | 2 +- > backends/hostmem-file.c | 2 +- > backends/hostmem-memfd.c | 14 ++- > backends/hostmem-ram.c | 2 +- > backends/hostmem-shm.c | 51 ++------ > docs/devel/migration/CPR.rst | 182 ++++++++++++++++++++++++++- > hw/core/machine.c | 20 +++ > include/exec/memory.h | 10 ++ > include/exec/ram_addr.h | 13 +- > include/hw/boards.h | 1 + > include/migration/cpr.h | 33 +++++ > include/migration/misc.h | 7 ++ > include/migration/vmstate.h | 9 ++ > include/qemu/osdep.h | 1 + > meson.build | 8 +- > migration/cpr-transfer.c | 76 +++++++++++ > migration/cpr.c | 224 +++++++++++++++++++++++++++++++++ > migration/meson.build | 2 + > migration/migration.c | 139 +++++++++++++++++++- > migration/migration.h | 4 +- > migration/options.c | 8 +- > migration/qemu-file.c | 83 +++++++++++- > migration/qemu-file.h | 2 + > migration/ram.c | 2 + > migration/trace-events | 11 ++ > migration/vmstate-types.c | 24 ++++ > qapi/migration.json | 44 ++++++- > qemu-options.hx | 34 +++++ > stubs/vmstate.c | 7 ++ > system/memory.c | 4 +- > system/physmem.c | 150 ++++++++++++++++++---- > system/trace-events | 1 + > system/vl.c | 43 ++++++- > tests/qtest/libqtest.c | 86 ++++++++----- > tests/qtest/libqtest.h | 19 ++- > tests/qtest/migration/cpr-tests.c | 60 +++++++++ > tests/qtest/migration/framework.c | 74 +++++++++-- > tests/qtest/migration/framework.h | 11 ++ > tests/qtest/migration/migration-qmp.c | 53 ++++++-- > tests/qtest/migration/migration-qmp.h | 10 +- > tests/qtest/migration/migration-util.c | 23 ++-- > tests/qtest/migration/misc-tests.c | 9 +- > tests/qtest/migration/precopy-tests.c | 6 +- > tests/qtest/virtio-net-failover.c | 8 +- > util/memfd.c | 16 ++- > util/oslib-posix.c | 53 ++++++++ > util/oslib-win32.c | 6 + > 47 files changed, 1473 insertions(+), 174 deletions(-) > create mode 100644 include/migration/cpr.h > create mode 100644 migration/cpr-transfer.c > create mode 100644 migration/cpr.c > > base-commit: e8aa7fdcddfc8589bdc7c973a052e76e8f999455 >
On 1/14/2025 12:38 PM, Steven Sistare wrote: > Hi Peter -- > > These still need RB. You said the first one "looks fine" in V5, > and I only needed to split the qemu_ram_alloc_from_fd size calculation bug > to a separate patch, which I did in V6: > * physmem: qemu_ram_alloc_from_fd extensions > * tests/qtest: assert qmp connected Thanks Peter, you were quick on the draw! Our emails crossed. - Steve > > Hi Markus -- > > Patch "migration: incoming channel" has an Acked-by from Peter, but it would > be good to also get your ack or RB, as I updated this patch in response to > the comments you posted for V4. > > We are inches from the goal line ... > > - Steve > > On 1/14/2025 12:23 PM, Steve Sistare wrote: >> What? >> >> This patch series adds the live migration cpr-transfer mode, which >> allows the user to transfer a guest to a new QEMU instance on the same >> host with minimal guest pause time, by preserving guest RAM in place, >> albeit with new virtual addresses in new QEMU, and by preserving device >> file descriptors. >> >> The new user-visible interfaces are: >> * cpr-transfer (MigMode migration parameter) >> * cpr (MigrationChannelType) >> * incoming MigrationChannel (command-line argument) >> * aux-ram-share (machine option) >> >> The user sets the mode parameter before invoking the migrate command. >> In this mode, the user starts new QEMU on the same host as old QEMU, with >> the same arguments as old QEMU, plus two -incoming options; one for the main >> channel, and one for the CPR channel. The user issues the migrate command to >> old QEMU, which stops the VM, saves state to the migration channels, and >> enters the postmigrate state. Execution resumes in new QEMU. >> >> Memory-backend objects must have the share=on attribute, but memory-backend-epc >> is not supported. The VM must be started with the '-machine aux-ram-share=on' >> option, which allows auxilliary guest memory to be transferred in place to the >> new process. >> >> This mode requires a second migration channel of type "cpr", in the channel >> arguments on the outgoing side, and in a second -incoming command-line >> parameter on the incoming side. This CPR channel must support file descriptor >> transfer with SCM_RIGHTS, i.e. it must be a UNIX domain socket. >> >> Why? >> >> This mode has less impact on the guest than any other method of updating >> in place. The pause time is much lower, because devices need not be torn >> down and recreated, DMA does not need to be drained and quiesced, and minimal >> state is copied to new QEMU. Further, there are no constraints on the guest. >> By contrast, cpr-reboot mode requires the guest to support S3 suspend-to-ram, >> and suspending plus resuming vfio devices adds multiple seconds to the >> guest pause time. >> >> These benefits all derive from the core design principle of this mode, >> which is preserving open descriptors. This approach is very general and >> can be used to support a wide variety of devices that do not have hardware >> support for live migration, including but not limited to: vfio, chardev, >> vhost, vdpa, and iommufd. Some devices need new kernel software interfaces >> to allow a descriptor to be used in a process that did not originally open it. >> >> How? >> >> All memory that is mapped by the guest is preserved in place. Indeed, >> it must be, because it may be the target of DMA requests, which are not >> quiesced during cpr-transfer. All such memory must be mmap'able in new QEMU. >> This is easy for named memory-backend objects, as long as they are mapped >> shared, because they are visible in the file system in both old and new QEMU. >> Anonymous memory must be allocated using memfd_create rather than MAP_ANON, >> so the memfd's can be sent to new QEMU. Pages that were locked in memory >> for DMA in old QEMU remain locked in new QEMU, because the descriptor of >> the device that locked them remains open. >> >> cpr-transfer preserves descriptors by sending them to new QEMU via the CPR >> channel, which must support SCM_RIGHTS, and by sending the unique name of >> each descriptor to new QEMU via CPR state. >> >> For device descriptors, new QEMU reuses the descriptor when creating the >> device, rather than opening it again. For memfd descriptors, new QEMU >> mmap's the preserved memfd when a ramblock is created. >> >> CPR state cannot be sent over the normal migration channel, because devices >> and backends are created prior to reading the channel, so this mode sends >> CPR state over a second "cpr" migration channel. New QEMU reads the second >> channel prior to creating devices or backends. >> >> Example: >> >> In this example, we simply restart the same version of QEMU, but in >> a real scenario one would use a new QEMU binary path in terminal 2. >> >> Terminal 1: start old QEMU >> # qemu-kvm -qmp stdio -object >> memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on >> -m 4G -machine aux-ram-share=on ... >> >> Terminal 2: start new QEMU >> # qemu-kvm -monitor stdio ... -incoming tcp:0:44444 >> -incoming '{"channel-type": "cpr", >> "addr": { "transport": "socket", "type": "unix", >> "path": "cpr.sock"}}' >> >> Terminal 1: >> {"execute":"qmp_capabilities"} >> >> {"execute": "query-status"} >> {"return": {"status": "running", >> "running": true}} >> >> {"execute":"migrate-set-parameters", >> "arguments":{"mode":"cpr-transfer"}} >> >> {"execute": "migrate", "arguments": { "channels": [ >> {"channel-type": "main", >> "addr": { "transport": "socket", "type": "inet", >> "host": "0", "port": "44444" }}, >> {"channel-type": "cpr", >> "addr": { "transport": "socket", "type": "unix", >> "path": "cpr.sock" }}]}} >> >> {"execute": "query-status"} >> {"return": {"status": "postmigrate", >> "running": false}} >> >> Terminal 2: >> QEMU 10.0.50 monitor - type 'help' for more information >> (qemu) info status >> VM status: running >> >> This patch series implements a minimal version of cpr-transfer. Additional >> series are ready to be posted to deliver the complete vision described >> above, including >> * vfio >> * chardev >> * vhost and tap >> * blockers >> * cpr-exec mode >> * iommufd >> >> Changes in V2: >> * cpr-transfer is the first new mode proposed, and cpr-exec is deferred >> * anon-alloc does not apply to memory-backend-object >> * replaced hack with proper synchronization between source and target >> * defined QEMU_CPR_FILE_MAGIC >> * addressed misc review comments >> >> Changes in V3: >> * added cpr-transfer to migration-test >> * documented cpr-transfer in CPR.rst >> * fix size_t trace format for 32-bit build >> * drop explicit fd value in VMSTATE_FD >> * defer cpr_walk_fd() and cpr_resave_fd() to later series >> * drop "migration: save cpr mode". >> delete mode from cpr state, and use cpr_uri to infer transfer mode. >> * drop "migration: stop vm earlier for cpr" >> * dropped cpr helpers, to be re-added later when needed >> * fixed an unreported bug for cpr-transfer and migrate cancel >> * documented cpr-transfer restrictions in qapi >> * added trace for cpr_state_save and cpr_state_load >> * added ftruncate to "preserve ram blocks" >> >> Changes in V4: >> * cleaned up qtest deferred connection code >> * renamed pass_fd -> can_pass_fd >> * squashed patch "split qmp_migrate" >> * deleted cpr-uri and its patches >> * added cpr channel and its patches >> * added patch "hostmem-shm: preserve for cpr" >> * added patch "fd-based shared memory" >> * added patch "factor out allocation of anonymous shared memory" >> * added RAM_PRIVATE and its patch >> * added aux-ram-share and its patch >> >> Changes in V5: >> * added patch 'enhance migrate_uri_parse' >> * supported dotted keys for -incoming channel, >> and rewrote incoming_option_parse >> * moved migrate_fd_cancel -> vm_resume to "stop vm earlier for cpr" >> in a future series. >> * updated command-line definition for aux-ram-share >> * added patch "resizable qemu_ram_alloc_from_fd" >> * rewrote patch "fd-based shared memory" >> * fixed error message in qemu_shm_alloc >> * added patch 'tests/qtest: optimize migrate_set_ports' >> * added patch 'tests/qtest: enhance migration channels' >> * added patch 'tests/qtest: assert qmp_ready' >> * modified patch 'migration-test: cpr-transfer' >> * polished the documentation in CPR.rst, qapi, and the >> cpr-transfer mode commit message >> * updated to master, and resolved massive context diffs for migration tests >> >> Changes in V6: >> * added RB's and Acks. >> * in patch "assert qmp_ready", deleted qmp_ready and checked qmp_fd instead. >> renamed patch to ""assert qmp connected" >> * factored out fix into new patch >> "fix qemu_ram_alloc_from_fd size calculation" >> * deleted a redundant call to migrate_hup_delete >> * added commit message to "migration: cpr-transfer documentation" >> * polished the text of cpr-transfer mode in qapi >> >> The first 10 patches below are foundational and are needed for both cpr-transfer >> mode and the proposed cpr-exec mode. The next 6 patches are specific to >> cpr-transfer and implement the mechanisms for sharing state across a socket >> using SCM_RIGHTS. The last 8 patches supply tests and documentation. >> >> Steve Sistare (24): >> backends/hostmem-shm: factor out allocation of "anonymous shared >> memory with an fd" >> physmem: fix qemu_ram_alloc_from_fd size calculation >> physmem: qemu_ram_alloc_from_fd extensions >> physmem: fd-based shared memory >> memory: add RAM_PRIVATE >> machine: aux-ram-share option >> migration: cpr-state >> physmem: preserve ram blocks for cpr >> hostmem-memfd: preserve for cpr >> hostmem-shm: preserve for cpr >> migration: enhance migrate_uri_parse >> migration: incoming channel >> migration: SCM_RIGHTS for QEMUFile >> migration: VMSTATE_FD >> migration: cpr-transfer save and load >> migration: cpr-transfer mode >> migration-test: memory_backend >> tests/qtest: optimize migrate_set_ports >> tests/qtest: defer connection >> migration-test: defer connection >> tests/qtest: enhance migration channels >> tests/qtest: assert qmp connected >> migration-test: cpr-transfer >> migration: cpr-transfer documentation >> >> backends/hostmem-epc.c | 2 +- >> backends/hostmem-file.c | 2 +- >> backends/hostmem-memfd.c | 14 ++- >> backends/hostmem-ram.c | 2 +- >> backends/hostmem-shm.c | 51 ++------ >> docs/devel/migration/CPR.rst | 182 ++++++++++++++++++++++++++- >> hw/core/machine.c | 20 +++ >> include/exec/memory.h | 10 ++ >> include/exec/ram_addr.h | 13 +- >> include/hw/boards.h | 1 + >> include/migration/cpr.h | 33 +++++ >> include/migration/misc.h | 7 ++ >> include/migration/vmstate.h | 9 ++ >> include/qemu/osdep.h | 1 + >> meson.build | 8 +- >> migration/cpr-transfer.c | 76 +++++++++++ >> migration/cpr.c | 224 +++++++++++++++++++++++++++++++++ >> migration/meson.build | 2 + >> migration/migration.c | 139 +++++++++++++++++++- >> migration/migration.h | 4 +- >> migration/options.c | 8 +- >> migration/qemu-file.c | 83 +++++++++++- >> migration/qemu-file.h | 2 + >> migration/ram.c | 2 + >> migration/trace-events | 11 ++ >> migration/vmstate-types.c | 24 ++++ >> qapi/migration.json | 44 ++++++- >> qemu-options.hx | 34 +++++ >> stubs/vmstate.c | 7 ++ >> system/memory.c | 4 +- >> system/physmem.c | 150 ++++++++++++++++++---- >> system/trace-events | 1 + >> system/vl.c | 43 ++++++- >> tests/qtest/libqtest.c | 86 ++++++++----- >> tests/qtest/libqtest.h | 19 ++- >> tests/qtest/migration/cpr-tests.c | 60 +++++++++ >> tests/qtest/migration/framework.c | 74 +++++++++-- >> tests/qtest/migration/framework.h | 11 ++ >> tests/qtest/migration/migration-qmp.c | 53 ++++++-- >> tests/qtest/migration/migration-qmp.h | 10 +- >> tests/qtest/migration/migration-util.c | 23 ++-- >> tests/qtest/migration/misc-tests.c | 9 +- >> tests/qtest/migration/precopy-tests.c | 6 +- >> tests/qtest/virtio-net-failover.c | 8 +- >> util/memfd.c | 16 ++- >> util/oslib-posix.c | 53 ++++++++ >> util/oslib-win32.c | 6 + >> 47 files changed, 1473 insertions(+), 174 deletions(-) >> create mode 100644 include/migration/cpr.h >> create mode 100644 migration/cpr-transfer.c >> create mode 100644 migration/cpr.c >> >> base-commit: e8aa7fdcddfc8589bdc7c973a052e76e8f999455 >> >
Hi Steve, The CI shows some issues, please take a look: https://gitlab.com/farosas/qemu/-/pipelines/1624984802
On 1/14/2025 3:29 PM, Fabiano Rosas wrote: > Hi Steve, > > The CI shows some issues, please take a look: > > https://gitlab.com/farosas/qemu/-/pipelines/1624984802 ------------------ I will fix this bug in V7 tomorrow. Same failure for each. cross-armhf-user cross-i686-system cross-i686-tci cross-i686-user cross-mipsel-system cross-mipsel-user In file included from ../util/oslib-posix.c:36: ../util/oslib-posix.c: In function qemu_shm_alloc: /builds/farosas/qemu/include/qapi/error.h:335:43: error: format %llu expects argument of type long long unsigned int, but argument 7 has type size_t {aka unsigned int} [-Werror=format=] 335 | (os_error), (fmt), ## __VA_ARGS__) --------------------------------------------- I will fix this bug in V7 tomorrow. cross-win64-system hw/core/machine.c:467:13: error: 'machine_set_aux_ram_share' defined but not used [-Werror=unused-function] hw/core/machine.c:460:13: error: 'machine_get_aux_ram_share' defined but not used [-Werror=unused-function] ------------------------------------------------- This bug could be mine, but I need to investigate further and reproduce it. It occurs on s390x, but all migration tests pass for me on x86_64. clang-system s390x/migration/mode/transfer - ERROR:../tests/qtest/libqtest.c:1362:qtest_vqmp_assert_success_ref: assertion failed: (qdict_haskey(response, "return")) FAIL 23/109 qemu:qtest+qtest-s390x / qtest-s390x/migration-test ERROR 199.38s killed by signal 6 SIGABRT >>> G_TEST_DBUS_DAEMON=/builds/farosas/qemu/tests/dbus-vmstate-daemon.sh ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 QTEST_QEMU_IMG=./qemu-img UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 PYTHON=/builds/farosas/qemu/build/pyvenv/bin/python3 MESON_TEST_ITERATION=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon MALLOC_PERTURB_=123 QTEST_QEMU_BINARY=./qemu-system-s390x /builds/farosas/qemu/build/tests/qtest/migration-test --tap -k stderr: ERROR:../tests/qtest/libqtest.c:1362:qtest_vqmp_assert_success_ref: assertion failed: (qdict_haskey(response, "return")) ../tests/qtest/libqtest.c:207: kill_qemu() detected QEMU death from signal 15 (Terminated) (test program exited with status code -6) Full log written to /builds/farosas/qemu/build/meson-logs/testlog.txt make: *** [Makefile.mtest:26: do-meson-check] Error 1 Saving cache for failed job 02:46 Creating cache clang-system-non_protected... ccache: found 15435 matching artifact files and directories Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/43309410/clang-system-non_protected -------------------------------------------------------------------- I do not recognize the RUST test failures. - Steve
On 1/14/2025 4:14 PM, Steven Sistare wrote: > On 1/14/2025 3:29 PM, Fabiano Rosas wrote: >> Hi Steve, >> >> The CI shows some issues, please take a look: >> >> https://gitlab.com/farosas/qemu/-/pipelines/1624984802 > > ------------------ > I will fix this bug in V7 tomorrow. Same failure for each. > > cross-armhf-user > cross-i686-system > cross-i686-tci > cross-i686-user > cross-mipsel-system > cross-mipsel-user > > In file included from ../util/oslib-posix.c:36: > ../util/oslib-posix.c: In function qemu_shm_alloc: > /builds/farosas/qemu/include/qapi/error.h:335:43: error: format %llu expects argument of type long long unsigned int, but argument 7 has type size_t {aka unsigned int} [-Werror=format=] > 335 | (os_error), (fmt), ## __VA_ARGS__) > --------------------------------------------- > > I will fix this bug in V7 tomorrow. > > cross-win64-system > > hw/core/machine.c:467:13: error: 'machine_set_aux_ram_share' defined but not used [-Werror=unused-function] > hw/core/machine.c:460:13: error: 'machine_get_aux_ram_share' defined but not used [-Werror=unused-function] > > ------------------------------------------------- > > This bug could be mine, but I need to investigate further and reproduce it. > It occurs on s390x, but all migration tests pass for me on x86_64. Duh, "s390x/migration/mode/transfer", definitely mine :) I will skip this test for unsupported architectures. - Steve > clang-system > > s390x/migration/mode/transfer - ERROR:../tests/qtest/libqtest.c:1362:qtest_vqmp_assert_success_ref: assertion failed: (qdict_haskey(response, "return")) FAIL > 23/109 qemu:qtest+qtest-s390x / qtest-s390x/migration-test ERROR 199.38s killed by signal 6 SIGABRT > >>> G_TEST_DBUS_DAEMON=/builds/farosas/qemu/tests/dbus-vmstate-daemon.sh ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 QTEST_QEMU_IMG=./qemu-img UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 PYTHON=/builds/farosas/qemu/build/pyvenv/bin/python3 MESON_TEST_ITERATION=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon MALLOC_PERTURB_=123 QTEST_QEMU_BINARY=./qemu-system-s390x /builds/farosas/qemu/build/tests/qtest/migration-test --tap -k > > stderr: > ERROR:../tests/qtest/libqtest.c:1362:qtest_vqmp_assert_success_ref: assertion failed: (qdict_haskey(response, "return")) > ../tests/qtest/libqtest.c:207: kill_qemu() detected QEMU death from signal 15 (Terminated) > (test program exited with status code -6) > > Full log written to /builds/farosas/qemu/build/meson-logs/testlog.txt > make: *** [Makefile.mtest:26: do-meson-check] Error 1 > Saving cache for failed job 02:46 > Creating cache clang-system-non_protected... > ccache: found 15435 matching artifact files and directories > Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/43309410/clang-system-non_protected > > -------------------------------------------------------------------- > > I do not recognize the RUST test failures. > > - Steve > >
What? This patch series adds the live migration cpr-transfer mode, which allows the user to transfer a guest to a new QEMU instance on the same host with minimal guest pause time, by preserving guest RAM in place, albeit with new virtual addresses in new QEMU, and by preserving device file descriptors. The new user-visible interfaces are: * cpr-transfer (MigMode migration parameter) * cpr (MigrationChannelType) * incoming MigrationChannel (command-line argument) * aux-ram-share (machine option) The user sets the mode parameter before invoking the migrate command. In this mode, the user starts new QEMU on the same host as old QEMU, with the same arguments as old QEMU, plus two -incoming options; one for the main channel, and one for the CPR channel. The user issues the migrate command to old QEMU, which stops the VM, saves state to the migration channels, and enters the postmigrate state. Execution resumes in new QEMU. Memory-backend objects must have the share=on attribute, but memory-backend-epc is not supported. The VM must be started with the '-machine aux-ram-share=on' option, which allows auxilliary guest memory to be transferred in place to the new process. This mode requires a second migration channel of type "cpr", in the channel arguments on the outgoing side, and in a second -incoming command-line parameter on the incoming side. This CPR channel must support file descriptor transfer with SCM_RIGHTS, i.e. it must be a UNIX domain socket. Why? This mode has less impact on the guest than any other method of updating in place. The pause time is much lower, because devices need not be torn down and recreated, DMA does not need to be drained and quiesced, and minimal state is copied to new QEMU. Further, there are no constraints on the guest. By contrast, cpr-reboot mode requires the guest to support S3 suspend-to-ram, and suspending plus resuming vfio devices adds multiple seconds to the guest pause time. These benefits all derive from the core design principle of this mode, which is preserving open descriptors. This approach is very general and can be used to support a wide variety of devices that do not have hardware support for live migration, including but not limited to: vfio, chardev, vhost, vdpa, and iommufd. Some devices need new kernel software interfaces to allow a descriptor to be used in a process that did not originally open it. How? All memory that is mapped by the guest is preserved in place. Indeed, it must be, because it may be the target of DMA requests, which are not quiesced during cpr-transfer. All such memory must be mmap'able in new QEMU. This is easy for named memory-backend objects, as long as they are mapped shared, because they are visible in the file system in both old and new QEMU. Anonymous memory must be allocated using memfd_create rather than MAP_ANON, so the memfd's can be sent to new QEMU. Pages that were locked in memory for DMA in old QEMU remain locked in new QEMU, because the descriptor of the device that locked them remains open. cpr-transfer preserves descriptors by sending them to new QEMU via the CPR channel, which must support SCM_RIGHTS, and by sending the unique name of each descriptor to new QEMU via CPR state. For device descriptors, new QEMU reuses the descriptor when creating the device, rather than opening it again. For memfd descriptors, new QEMU mmap's the preserved memfd when a ramblock is created. CPR state cannot be sent over the normal migration channel, because devices and backends are created prior to reading the channel, so this mode sends CPR state over a second "cpr" migration channel. New QEMU reads the second channel prior to creating devices or backends. Example: In this example, we simply restart the same version of QEMU, but in a real scenario one would use a new QEMU binary path in terminal 2. Terminal 1: start old QEMU # qemu-kvm -qmp stdio -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m 4G -machine aux-ram-share=on ... Terminal 2: start new QEMU # qemu-kvm -monitor stdio ... -incoming tcp:0:44444 -incoming '{"channel-type": "cpr", "addr": { "transport": "socket", "type": "unix", "path": "cpr.sock"}}' Terminal 1: {"execute":"qmp_capabilities"} {"execute": "query-status"} {"return": {"status": "running", "running": true}} {"execute":"migrate-set-parameters", "arguments":{"mode":"cpr-transfer"}} {"execute": "migrate", "arguments": { "channels": [ {"channel-type": "main", "addr": { "transport": "socket", "type": "inet", "host": "0", "port": "44444" }}, {"channel-type": "cpr", "addr": { "transport": "socket", "type": "unix", "path": "cpr.sock" }}]}} {"execute": "query-status"} {"return": {"status": "postmigrate", "running": false}} Terminal 2: QEMU 10.0.50 monitor - type 'help' for more information (qemu) info status VM status: running This patch series implements a minimal version of cpr-transfer. Additional series are ready to be posted to deliver the complete vision described above, including * vfio * chardev * vhost and tap * blockers * cpr-exec mode * iommufd Changes in V2: * cpr-transfer is the first new mode proposed, and cpr-exec is deferred * anon-alloc does not apply to memory-backend-object * replaced hack with proper synchronization between source and target * defined QEMU_CPR_FILE_MAGIC * addressed misc review comments Changes in V3: * added cpr-transfer to migration-test * documented cpr-transfer in CPR.rst * fix size_t trace format for 32-bit build * drop explicit fd value in VMSTATE_FD * defer cpr_walk_fd() and cpr_resave_fd() to later series * drop "migration: save cpr mode". delete mode from cpr state, and use cpr_uri to infer transfer mode. * drop "migration: stop vm earlier for cpr" * dropped cpr helpers, to be re-added later when needed * fixed an unreported bug for cpr-transfer and migrate cancel * documented cpr-transfer restrictions in qapi * added trace for cpr_state_save and cpr_state_load * added ftruncate to "preserve ram blocks" Changes in V4: * cleaned up qtest deferred connection code * renamed pass_fd -> can_pass_fd * squashed patch "split qmp_migrate" * deleted cpr-uri and its patches * added cpr channel and its patches * added patch "hostmem-shm: preserve for cpr" * added patch "fd-based shared memory" * added patch "factor out allocation of anonymous shared memory" * added RAM_PRIVATE and its patch * added aux-ram-share and its patch Changes in V5: * added patch 'enhance migrate_uri_parse' * supported dotted keys for -incoming channel, and rewrote incoming_option_parse * moved migrate_fd_cancel -> vm_resume to "stop vm earlier for cpr" in a future series. * updated command-line definition for aux-ram-share * added patch "resizable qemu_ram_alloc_from_fd" * rewrote patch "fd-based shared memory" * fixed error message in qemu_shm_alloc * added patch 'tests/qtest: optimize migrate_set_ports' * added patch 'tests/qtest: enhance migration channels' * added patch 'tests/qtest: assert qmp_ready' * modified patch 'migration-test: cpr-transfer' * polished the documentation in CPR.rst, qapi, and the cpr-transfer mode commit message * updated to master, and resolved massive context diffs for migration tests Changes in V6: * added RB's and Acks. * in patch "assert qmp_ready", deleted qmp_ready and checked qmp_fd instead. renamed patch to ""assert qmp connected" * factored out fix into new patch "fix qemu_ram_alloc_from_fd size calculation" * deleted a redundant call to migrate_hup_delete * added commit message to "migration: cpr-transfer documentation" * polished the text of cpr-transfer mode in qapi The first 10 patches below are foundational and are needed for both cpr-transfer mode and the proposed cpr-exec mode. The next 6 patches are specific to cpr-transfer and implement the mechanisms for sharing state across a socket using SCM_RIGHTS. The last 8 patches supply tests and documentation. Steve Sistare (24): backends/hostmem-shm: factor out allocation of "anonymous shared memory with an fd" physmem: fix qemu_ram_alloc_from_fd size calculation physmem: qemu_ram_alloc_from_fd extensions physmem: fd-based shared memory memory: add RAM_PRIVATE machine: aux-ram-share option migration: cpr-state physmem: preserve ram blocks for cpr hostmem-memfd: preserve for cpr hostmem-shm: preserve for cpr migration: enhance migrate_uri_parse migration: incoming channel migration: SCM_RIGHTS for QEMUFile migration: VMSTATE_FD migration: cpr-transfer save and load migration: cpr-transfer mode migration-test: memory_backend tests/qtest: optimize migrate_set_ports tests/qtest: defer connection migration-test: defer connection tests/qtest: enhance migration channels tests/qtest: assert qmp connected migration-test: cpr-transfer migration: cpr-transfer documentation backends/hostmem-epc.c | 2 +- backends/hostmem-file.c | 2 +- backends/hostmem-memfd.c | 14 ++- backends/hostmem-ram.c | 2 +- backends/hostmem-shm.c | 51 ++------ docs/devel/migration/CPR.rst | 182 ++++++++++++++++++++++++++- hw/core/machine.c | 20 +++ include/exec/memory.h | 10 ++ include/exec/ram_addr.h | 13 +- include/hw/boards.h | 1 + include/migration/cpr.h | 33 +++++ include/migration/misc.h | 7 ++ include/migration/vmstate.h | 9 ++ include/qemu/osdep.h | 1 + meson.build | 8 +- migration/cpr-transfer.c | 76 +++++++++++ migration/cpr.c | 224 +++++++++++++++++++++++++++++++++ migration/meson.build | 2 + migration/migration.c | 139 +++++++++++++++++++- migration/migration.h | 4 +- migration/options.c | 8 +- migration/qemu-file.c | 83 +++++++++++- migration/qemu-file.h | 2 + migration/ram.c | 2 + migration/trace-events | 11 ++ migration/vmstate-types.c | 24 ++++ qapi/migration.json | 44 ++++++- qemu-options.hx | 34 +++++ stubs/vmstate.c | 7 ++ system/memory.c | 4 +- system/physmem.c | 150 ++++++++++++++++++---- system/trace-events | 1 + system/vl.c | 43 ++++++- tests/qtest/libqtest.c | 86 ++++++++----- tests/qtest/libqtest.h | 19 ++- tests/qtest/migration/cpr-tests.c | 60 +++++++++ tests/qtest/migration/framework.c | 74 +++++++++-- tests/qtest/migration/framework.h | 11 ++ tests/qtest/migration/migration-qmp.c | 53 ++++++-- tests/qtest/migration/migration-qmp.h | 10 +- tests/qtest/migration/migration-util.c | 23 ++-- tests/qtest/migration/misc-tests.c | 9 +- tests/qtest/migration/precopy-tests.c | 6 +- tests/qtest/virtio-net-failover.c | 8 +- util/memfd.c | 16 ++- util/oslib-posix.c | 53 ++++++++ util/oslib-win32.c | 6 + 47 files changed, 1473 insertions(+), 174 deletions(-) create mode 100644 include/migration/cpr.h create mode 100644 migration/cpr-transfer.c create mode 100644 migration/cpr.c base-commit: e8aa7fdcddfc8589bdc7c973a052e76e8f999455