mbox series

[v2,0/4] hw/arm/virt: Support dirty ring

Message ID 20230227042629.339747-1-gshan@redhat.com (mailing list archive)
Headers show
Series hw/arm/virt: Support dirty ring | expand

Message

Gavin Shan Feb. 27, 2023, 4:26 a.m. UTC
This series intends to support dirty ring for live migration for arm64. The
dirty ring use discrete buffer to track dirty pages. For arm64, the speciality
is to use backup bitmap to track dirty pages when there is no-running-vcpu
context. It's known that the backup bitmap needs to be synchronized when
KVM device "kvm-arm-gicv3" or "arm-its-kvm" has been enabled. The backup
bitmap is collected in the last stage of migration. The policy here is to
always enable the backup bitmap extension. The overhead to synchronize the
backup bitmap in the last stage of migration, when those two devices aren't
used, is introduced. However, the overhead should be very small and acceptable.
The benefit is to support future cases where those two devices are used without
modifying the code.

PATCH[1] add migration last stage indicator
PATCH[2] synchronize the backup bitmap in the last stage of migration
PATCH[3] add helper kvm_dirty_ring_init() to enable dirty ring
PATCH[4] enable dirty ring for arm64

   v1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00434.html
RFCv1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00171.html

Testing
=======
(1) kvm-unit-tests/its-pending-migration and kvm-unit-tests/its-migration with
    dirty ring or normal dirty page tracking mechanism. All test cases passed.

    QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm \
    ./its-pending-migration

    QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm \
    ./its-migration

    QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm,dirty-ring-size=65536 \
    ./its-pending-migration

    QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm,dirty-ring-size=65536 \
    ./its-migration

(2) Combinations of migration, post-copy migration, e1000e and virtio-net
    devices. All test cases passed.

    -netdev tap,id=net0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown  \
    -device e1000e,bus=pcie.5,netdev=net0,mac=52:54:00:f1:26:a0

    -netdev tap,id=vnet0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
    -device virtio-net-pci,bus=pcie.6,netdev=vnet0,mac=52:54:00:f1:26:b0

Changelog
=========
v2:
  * Drop PATCH[v1 1/6] to synchronize linux-headers                            (Gavin)
  * More restrictive comments about struct MemoryListener::log_sync_global     (PeterX)
  * Always enable the backup bitmap extension                                  (PeterM)
v1:
  * Combine two patches into one PATCH[v1 2/6] for the last stage indicator    (PeterX)
  * Drop the secondary bitmap and use the original one directly                (Juan)
  * Avoid "goto out" in helper kvm_dirty_ring_init()                           (Juan)


Gavin Shan (4):
  migration: Add last stage indicator to global dirty log
    synchronization
  kvm: Synchronize the backup bitmap in the last stage
  kvm: Add helper kvm_dirty_ring_init()
  kvm: Enable dirty ring for arm64

 accel/kvm/kvm-all.c      | 108 ++++++++++++++++++++++++++++-----------
 include/exec/memory.h    |   7 ++-
 include/sysemu/kvm_int.h |   1 +
 migration/dirtyrate.c    |   4 +-
 migration/ram.c          |  20 ++++----
 softmmu/memory.c         |  10 ++--
 6 files changed, 101 insertions(+), 49 deletions(-)

Comments

Gavin Shan March 13, 2023, 7:13 a.m. UTC | #1
On 2/27/23 12:26 PM, Gavin Shan wrote:
> This series intends to support dirty ring for live migration for arm64. The
> dirty ring use discrete buffer to track dirty pages. For arm64, the speciality
> is to use backup bitmap to track dirty pages when there is no-running-vcpu
> context. It's known that the backup bitmap needs to be synchronized when
> KVM device "kvm-arm-gicv3" or "arm-its-kvm" has been enabled. The backup
> bitmap is collected in the last stage of migration. The policy here is to
> always enable the backup bitmap extension. The overhead to synchronize the
> backup bitmap in the last stage of migration, when those two devices aren't
> used, is introduced. However, the overhead should be very small and acceptable.
> The benefit is to support future cases where those two devices are used without
> modifying the code.
> 
> PATCH[1] add migration last stage indicator
> PATCH[2] synchronize the backup bitmap in the last stage of migration
> PATCH[3] add helper kvm_dirty_ring_init() to enable dirty ring
> PATCH[4] enable dirty ring for arm64
> 
>     v1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00434.html
> RFCv1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00171.html
> 
> Testing
> =======
> (1) kvm-unit-tests/its-pending-migration and kvm-unit-tests/its-migration with
>      dirty ring or normal dirty page tracking mechanism. All test cases passed.
> 
>      QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm \
>      ./its-pending-migration
> 
>      QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm \
>      ./its-migration
> 
>      QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm,dirty-ring-size=65536 \
>      ./its-pending-migration
> 
>      QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm,dirty-ring-size=65536 \
>      ./its-migration
> 
> (2) Combinations of migration, post-copy migration, e1000e and virtio-net
>      devices. All test cases passed.
> 
>      -netdev tap,id=net0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown  \
>      -device e1000e,bus=pcie.5,netdev=net0,mac=52:54:00:f1:26:a0
> 
>      -netdev tap,id=vnet0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
>      -device virtio-net-pci,bus=pcie.6,netdev=vnet0,mac=52:54:00:f1:26:b0
> 
> Changelog
> =========
> v2:
>    * Drop PATCH[v1 1/6] to synchronize linux-headers                            (Gavin)
>    * More restrictive comments about struct MemoryListener::log_sync_global     (PeterX)
>    * Always enable the backup bitmap extension                                  (PeterM)
> v1:
>    * Combine two patches into one PATCH[v1 2/6] for the last stage indicator    (PeterX)
>    * Drop the secondary bitmap and use the original one directly                (Juan)
>    * Avoid "goto out" in helper kvm_dirty_ring_init()                           (Juan)
> 

Ping, Paolo and Peter Maydell. Please take a look to see if it can be
merged, thanks!

> 
> Gavin Shan (4):
>    migration: Add last stage indicator to global dirty log
>      synchronization
>    kvm: Synchronize the backup bitmap in the last stage
>    kvm: Add helper kvm_dirty_ring_init()
>    kvm: Enable dirty ring for arm64
> 
>   accel/kvm/kvm-all.c      | 108 ++++++++++++++++++++++++++++-----------
>   include/exec/memory.h    |   7 ++-
>   include/sysemu/kvm_int.h |   1 +
>   migration/dirtyrate.c    |   4 +-
>   migration/ram.c          |  20 ++++----
>   softmmu/memory.c         |  10 ++--
>   6 files changed, 101 insertions(+), 49 deletions(-)
>
Peter Maydell March 14, 2023, 5:05 p.m. UTC | #2
On Mon, 13 Mar 2023 at 07:13, Gavin Shan <gshan@redhat.com> wrote:
>
> On 2/27/23 12:26 PM, Gavin Shan wrote:
> > This series intends to support dirty ring for live migration for arm64. The
> > dirty ring use discrete buffer to track dirty pages. For arm64, the speciality
> > is to use backup bitmap to track dirty pages when there is no-running-vcpu
> > context. It's known that the backup bitmap needs to be synchronized when
> > KVM device "kvm-arm-gicv3" or "arm-its-kvm" has been enabled. The backup
> > bitmap is collected in the last stage of migration. The policy here is to
> > always enable the backup bitmap extension. The overhead to synchronize the
> > backup bitmap in the last stage of migration, when those two devices aren't
> > used, is introduced. However, the overhead should be very small and acceptable.
> > The benefit is to support future cases where those two devices are used without
> > modifying the code.
> >
> > PATCH[1] add migration last stage indicator
> > PATCH[2] synchronize the backup bitmap in the last stage of migration
> > PATCH[3] add helper kvm_dirty_ring_init() to enable dirty ring
> > PATCH[4] enable dirty ring for arm64
> >
> >     v1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00434.html
> > RFCv1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00171.html
> >
> > Testing
> > =======
> > (1) kvm-unit-tests/its-pending-migration and kvm-unit-tests/its-migration with
> >      dirty ring or normal dirty page tracking mechanism. All test cases passed.
> >
> >      QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm \
> >      ./its-pending-migration
> >
> >      QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm \
> >      ./its-migration
> >
> >      QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm,dirty-ring-size=65536 \
> >      ./its-pending-migration
> >
> >      QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm,dirty-ring-size=65536 \
> >      ./its-migration
> >
> > (2) Combinations of migration, post-copy migration, e1000e and virtio-net
> >      devices. All test cases passed.
> >
> >      -netdev tap,id=net0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown  \
> >      -device e1000e,bus=pcie.5,netdev=net0,mac=52:54:00:f1:26:a0
> >
> >      -netdev tap,id=vnet0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
> >      -device virtio-net-pci,bus=pcie.6,netdev=vnet0,mac=52:54:00:f1:26:b0
> >
> > Changelog
> > =========
> > v2:
> >    * Drop PATCH[v1 1/6] to synchronize linux-headers                            (Gavin)
> >    * More restrictive comments about struct MemoryListener::log_sync_global     (PeterX)
> >    * Always enable the backup bitmap extension                                  (PeterM)
> > v1:
> >    * Combine two patches into one PATCH[v1 2/6] for the last stage indicator    (PeterX)
> >    * Drop the secondary bitmap and use the original one directly                (Juan)
> >    * Avoid "goto out" in helper kvm_dirty_ring_init()                           (Juan)
> >
>
> Ping, Paolo and Peter Maydell. Please take a look to see if it can be
> merged, thanks!

No objections here; I'm assuming that since it's only touching
KVM core code that it would go via Paolo. However, as a new
feature it has missed softfreeze for 8.0, so it'll have to
wait until the tree re-opens for 8.1.

thanks
-- PMM