| Message ID | 1562613452-24969-1-git-send-email-kwankhede@nvidia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Add migration support for VFIO device | expand |
Patchew URL: https://patchew.org/QEMU/1562613452-24969-1-git-send-email-kwankhede@nvidia.com/
Hi,
This series failed build test on s390x host. Please find the details below.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with
# HEAD pointing to a commit that has the patches applied on top of "base"
# branch
set -e
echo
echo "=== ENV ==="
env
echo
echo "=== PACKAGES ==="
rpm -qa
echo
echo "=== UNAME ==="
uname -a
CC=$HOME/bin/cc
INSTALL=$PWD/install
BUILD=$PWD/build
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --cc=$CC --prefix=$INSTALL
make -j4
# XXX: we need reliable clean up
# make check -j4 V=1
make install
=== TEST SCRIPT END ===
CC hw/virtio/trace.o
In file included from hw/vfio/trace.c:4:
hw/vfio/trace.h: In function ‘_nocheck__trace_vfio_save_pending’:
hw/vfio/trace.h:3655:126: error: stray ‘\’ in program
3655 | qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
| ^
hw/vfio/trace.h:3655:51: error: expected ‘)’ before ‘n’
3655 | qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
| ^ ~
| )
hw/vfio/trace.h:3655:128: error: missing terminating " character [-Werror]
3655 | qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
| ^
hw/vfio/trace.h:3655:128: error: missing terminating " character
3655 | qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
| ^~
hw/vfio/trace.h:3655:20: error: format ‘%d’ expects a matching ‘int’ argument [-Werror=format=]
3655 | qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
| ~^
| |
| int
hw/vfio/trace.h:3655:24: error: format ‘%zu’ expects a matching ‘size_t’ argument [-Werror=format=]
3655 | qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
| ~~^
| |
| long unsigned int
hw/vfio/trace.h:3655:30: error: format ‘%zu’ expects a matching ‘size_t’ argument [-Werror=format=]
3655 | qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
| ~~~~^
| |
| long unsigned int
hw/vfio/trace.h:3655:18: error: format ‘%s’ expects a matching ‘char *’ argument [-Werror=format=]
3655 | qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
hw/vfio/trace.h:3655:56: note: format string is defined here
---
| ~^
| |
| char *
hw/vfio/trace.h:3655:18: error: format ‘%lx’ expects a matching ‘long unsigned int’ argument [-Werror=format=]
3655 | qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /var/tmp/patchew-tester-tmp-gaz1sa3c/src/include/qemu/osdep.h:103,
---
/usr/include/inttypes.h:121:34: note: format string is defined here
121 | # define PRIx64 __PRI64_PREFIX "x"
In file included from hw/vfio/trace.c:4:
hw/vfio/trace.h:3655:18: error: format ‘%x’ expects a matching ‘unsigned int’ argument [-Werror=format=]
3655 | qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
hw/vfio/trace.h:3655:92: note: format string is defined here
---
| ~^
| |
| unsigned int
hw/vfio/trace.h:3655:18: error: unknown conversion type character ‘P’ in format [-Werror=format=]
3655 | qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
hw/vfio/trace.h:3655:118: note: format string is defined here
3655 | qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
| ^
hw/vfio/trace.h: In function ‘_nocheck__trace_vfio_load_state_device_data’:
hw/vfio/trace.h:3770:61: error: expected ‘)’ before ‘PRIx6’
3770 | qemu_log("%d@%zu.%06zu:vfio_load_state_device_data " " (%s), Offset 0x%"PRIx64" size 0x%"PRIx6 "\n",
| ^ ~~~~~
| )
hw/vfio/trace.h:3770:20: error: format ‘%d’ expects a matching ‘int’ argument [-Werror=format=]
3770 | qemu_log("%d@%zu.%06zu:vfio_load_state_device_data " " (%s), Offset 0x%"PRIx64" size 0x%"PRIx6 "\n",
| ~^
| |
| int
hw/vfio/trace.h:3770:24: error: format ‘%zu’ expects a matching ‘size_t’ argument [-Werror=format=]
3770 | qemu_log("%d@%zu.%06zu:vfio_load_state_device_data " " (%s), Offset 0x%"PRIx64" size 0x%"PRIx6 "\n",
| ~~^
| |
| long unsigned int
hw/vfio/trace.h:3770:30: error: format ‘%zu’ expects a matching ‘size_t’ argument [-Werror=format=]
3770 | qemu_log("%d@%zu.%06zu:vfio_load_state_device_data " " (%s), Offset 0x%"PRIx64" size 0x%"PRIx6 "\n",
| ~~~~^
| |
| long unsigned int
hw/vfio/trace.h:3770:18: error: format ‘%s’ expects a matching ‘char *’ argument [-Werror=format=]
3770 | qemu_log("%d@%zu.%06zu:vfio_load_state_device_data " " (%s), Offset 0x%"PRIx64" size 0x%"PRIx6 "\n",
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
hw/vfio/trace.h:3770:66: note: format string is defined here
---
| ~^
| |
| char *
hw/vfio/trace.h:3770:18: error: format ‘%lx’ expects a matching ‘long unsigned int’ argument [-Werror=format=]
3770 | qemu_log("%d@%zu.%06zu:vfio_load_state_device_data " " (%s), Offset 0x%"PRIx64" size 0x%"PRIx6 "\n",
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /var/tmp/patchew-tester-tmp-gaz1sa3c/src/include/qemu/osdep.h:103,
---
/usr/include/inttypes.h:121:34: note: format string is defined here
121 | # define PRIx64 __PRI64_PREFIX "x"
In file included from hw/vfio/trace.c:4:
hw/vfio/trace.h:3770:18: error: spurious trailing ‘%’ in format [-Werror=format=]
3770 | qemu_log("%d@%zu.%06zu:vfio_load_state_device_data " " (%s), Offset 0x%"PRIx64" size 0x%"PRIx6 "\n",
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
hw/vfio/trace.h:3770:96: note: format string is defined here
The full log is available at
http://patchew.org/logs/1562613452-24969-1-git-send-email-kwankhede@nvidia.com/testing.s390x/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Patchew URL: https://patchew.org/QEMU/1562613452-24969-1-git-send-email-kwankhede@nvidia.com/
Hi,
This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===
CC block/qed-table.o
CC block/qed-cluster.o
In file included from hw/vfio/trace.c:4:
/tmp/qemu-test/build/hw/vfio/trace.h:3655:126: error: expected ')'
qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
^
/tmp/qemu-test/build/hw/vfio/trace.h:3655:17: note: to match this '('
qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
^
/tmp/qemu-test/build/hw/vfio/trace.h:3655:128: error: missing terminating '"' character [-Werror,-Winvalid-pp-token]
qemu_log("%d@%zu.%06zu:vfio_save_pending " " (%s), precopy 0x%"PRIx64" postcopy 0x%x"PRIx64" compatible 0x%""PRIx64 "\n",
^
/tmp/qemu-test/build/hw/vfio/trace.h:3770:98: error: expected ')'
qemu_log("%d@%zu.%06zu:vfio_load_state_device_data " " (%s), Offset 0x%"PRIx64" size 0x%"PRIx6 "\n",
^
/tmp/qemu-test/build/hw/vfio/trace.h:3770:17: note: to match this '('
The full log is available at
http://patchew.org/logs/1562613452-24969-1-git-send-email-kwankhede@nvidia.com/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Add migration support for VFIO device This Patch set include patches as below: - Define KABI for VFIO device for migration support. - Added save and restore functions for PCI configuration space - Generic migration functionality for VFIO device. * This patch set adds functionality only for PCI devices, but can be extended to other VFIO devices. * Added all the basic functions required for pre-copy, stop-and-copy and resume phases of migration. * Added state change notifier and from that notifier function, VFIO device's state changed is conveyed to VFIO device driver. * During save setup phase and resume/load setup phase, migration region is queried and is used to read/write VFIO device data. * .save_live_pending and .save_live_iterate are implemented to use QEMU's functionality of iteration during pre-copy phase. * In .save_live_complete_precopy, that is in stop-and-copy phase, iteration to read data from VFIO device driver is implemented till pending bytes returned by driver are not zero. * Added function to get dirty pages bitmap for the pages which are used by driver. - Add vfio_listerner_log_sync to mark dirty pages. - Make VFIO PCI device migration capable. If migration region is not provided by driver, migration is blocked. Below is the flow of state change for live migration where states in brackets represent VM state, migration state and VFIO device state as: (VM state, MIGRATION_STATUS, VFIO_DEVICE_STATE) Live migration save path: QEMU normal running state (RUNNING, _NONE, _RUNNING) | migrate_init spawns migration_thread. (RUNNING, _SETUP, _RUNNING|_SAVING) Migration thread then calls each device's .save_setup() | (RUNNING, _ACTIVE, _RUNNING|_SAVING) If device is active, get pending bytes by .save_live_pending() if pending bytes >= threshold_size, call save_live_iterate() Data of VFIO device for pre-copy phase is copied. Iterate till pending bytes converge and are less than threshold | On migration completion, vCPUs stops and calls .save_live_complete_precopy for each active device. VFIO device is then transitioned in _SAVING state. (FINISH_MIGRATE, _DEVICE, _SAVING) For VFIO device, iterate in .save_live_complete_precopy until pending data is 0. (FINISH_MIGRATE, _DEVICE, _STOPPED) | (FINISH_MIGRATE, _COMPLETED, STOPPED) Migraton thread schedule cleanup bottom half and exit Live migration resume path: Incomming migration calls .load_setup for each device (RESTORE_VM, _ACTIVE, STOPPED) | For each device, .load_state is called for that device section data | At the end, called .load_cleanup for each device and vCPUs are started. | (RUNNING, _NONE, _RUNNING) Note that: - Migration post copy is not supported. v4 -> v5: - Added decriptive comment about the sequence of access of members of structure vfio_device_migration_info to be followed based on Alex's suggestion - Updated get dirty pages sequence. - As per Cornelia Huck's suggestion, added callbacks to VFIODeviceOps to get_object, save_config and load_config. - Fixed multiple nit picks. - Tested live migration with multiple vfio device assigned to a VM. v3 -> v4: - Added one more bit for _RESUMING flag to be set explicitly. - data_offset field is read-only for user space application. - data_size is read for every iteration before reading data from migration, that is removed assumption that data will be till end of migration region. - If vendor driver supports mappable sparsed region, map those region during setup state of save/load, similarly unmap those from cleanup routines. - Handles race condition that causes data corruption in migration region during save device state by adding mutex and serialiaing save_buffer and get_dirty_pages routines. - Skip called get_dirty_pages routine for mapped MMIO region of device. - Added trace events. - Splitted into multiple functional patches. v2 -> v3: - Removed enum of VFIO device states. Defined VFIO device state with 2 bits. - Re-structured vfio_device_migration_info to keep it minimal and defined action on read and write access on its members. v1 -> v2: - Defined MIGRATION region type and sub-type which should be used with region type capability. - Re-structured vfio_device_migration_info. This structure will be placed at 0th offset of migration region. - Replaced ioctl with read/write for trapped part of migration region. - Added both type of access support, trapped or mmapped, for data section of the region. - Moved PCI device functions to pci file. - Added iteration to get dirty page bitmap until bitmap for all requested pages are copied. Thanks, Kirti Kirti Wankhede (13): vfio: KABI for migration interface vfio: Add function to unmap VFIO region vfio: Add vfio_get_object callback to VFIODeviceOps vfio: Add save and load functions for VFIO PCI devices vfio: Add migration region initialization and finalize function vfio: Add VM state change handler to know state of VM vfio: Add migration state change notifier vfio: Register SaveVMHandlers for VFIO device vfio: Add save state functions to SaveVMHandlers vfio: Add load state functions to SaveVMHandlers vfio: Add function to get dirty page list vfio: Add vfio_listerner_log_sync to mark dirty pages vfio: Make vfio-pci device migration capable. hw/vfio/Makefile.objs | 2 +- hw/vfio/common.c | 55 +++ hw/vfio/migration.c | 874 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 137 ++++++- hw/vfio/trace-events | 19 + include/hw/vfio/vfio-common.h | 25 ++ linux-headers/linux/vfio.h | 166 ++++++++ 7 files changed, 1271 insertions(+), 7 deletions(-) create mode 100644 hw/vfio/migration.c