| Message ID | 20230213003925.40158-1-gshan@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | hw/arm/virt: Support dirty ring | expand |
[PATCH v1 0/6] hw/arm/virt: Support dirty ring
The patches work well on my arm Ampere host.
The test results are as expected.
Testing
=======
(1) kvm-unit-tests/its-pending-migration,
kvm-unit-tests/its-migrate-unmapped-collection
and kvm-unit-tests/its-migration with dirty ring or normal dirty
page tracking
mechanism. All test cases passed.
QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64 ACCEL=kvm \
PROCESSOR=host ./its-migration
QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64 ACCEL=kvm \
PROCESSOR=host ./its-migrate-unmapped-collection
QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64 ACCEL=kvm \
PROCESSOR=host ./its-pending-migration
QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64
ACCEL=kvm,dirty-ring-size=65536 \
PROCESSOR=host ./its-migration
QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64
ACCEL=kvm,dirty-ring-size=65536 \
PROCESSOR=host ./its-migrate-unmapped-collection
QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64
ACCEL=kvm,dirty-ring-size=65536 \
PROCESSOR=host ./its-pending-migration
(2) Combinations of migration, post-copy migration, e1000e and virtio-net
devices. All test cases passed.
-device '{"driver": "virtio-net-pci", "mac": "9a:97:8f:c7:cc:a6",
"rombar": 0, "netdev": "idDGdh30", "bus": "pcie-root-port-4", "addr":
"0x0"}' \
-netdev tap,id=idDGdh30,vhost=on
-device '{"driver": "e1000e", "mac": "9a:fd:93:f1:97:b1",
"netdev": "idXDOtMA", "bus": "pcie-root-port-4", "addr": "0x0"}' \
-netdev tap,id=idXDOtMA,vhost=on \
(3) Simulate heavy memory pressure scenarios and compare the migration
performance difference between dirty ring and dirty logging.
I gave with a 200G memory guest, 40 vcpus, using 10g NIC as migration
channel. When idle or dirty workload small, I don't observe major
difference on total migration time. When with higher random dirty
workload (Anonymous mapping 180G memory, 256MB/s dirty rate upon).
Total migration time is (in seconds):
|-------------------------+|-------------------------|
| dirty ring (4k entries) | dirty logging |
|-------------------------+|-------------------------|
| 67 | 74 |
| 67 | 74 |
| 66 | 76 |
| 66 | 73 |
| 67 | 76 |
| 67 | 76 |
| 66 | 73 |
| 67 | 74 |
|-------------------------+|-------------------------|
Summary:
dirty ring average: 67s
dirty logging average: 75s
Tested-by: Zhenyu Zhang <zhenzha@redhat.com>
On Mon, Feb 13, 2023 at 8:39 AM Gavin Shan <gshan@redhat.com> wrote:
>
> This series intends to support dirty ring for live migration. The dirty
> ring use discrete buffer to track dirty pages. For ARM64, the speciality
> is to use backup bitmap to track dirty pages when there is no-running-vcpu
> context. It's known that the backup bitmap needs to be synchronized when
> KVM device "kvm-arm-gicv3" or "arm-its-kvm" has been enabled. The backup
> bitmap is collected in the last stage of migration.
>
> PATCH[1] Synchronize linux-headers for dirty ring
> PATCH[2] Introduce indicator of the last stage migration and pass it
> all the way down
> PATCH[3] Synchronize the backup bitmap in the last stage of live migration
> PATCH[4] Introduce helper kvm_dirty_ring_init() to enable the dirty ring
> PATCH[5-6] Enable dirty ring for hw/arm/virt
>
> RFCv1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00171.html
>
> Testing
> =======
> (1) kvm-unit-tests/its-pending-migration and kvm-unit-tests/its-migration with
> dirty ring or normal dirty page tracking mechanism. All test cases passed.
>
> QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm \
> ./its-pending-migration
>
> QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm \
> ./its-migration
>
> QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm,dirty-ring-size=65536 \
> ./its-pending-migration
>
> QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm,dirty-ring-size=65536 \
> ./its-migration
>
> (2) Combinations of migration, post-copy migration, e1000e and virtio-net
> devices. All test cases passed.
>
> -netdev tap,id=net0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
> -device e1000e,bus=pcie.5,netdev=net0,mac=52:54:00:f1:26:a0
>
> -netdev tap,id=vnet0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
> -device virtio-net-pci,bus=pcie.6,netdev=vnet0,mac=52:54:00:f1:26:b0
>
> Changelog
> =========
> v1:
> * Combine two patches into one PATCH[v1 2/6] for the last stage indicator (Peter)
> * Drop the secondary bitmap and use the original one directly (Juan)
> * Avoid "goto out" in helper kvm_dirty_ring_init() (Juan)
>
>
> Gavin Shan (6):
> linux-headers: Update for dirty ring
> migration: Add last stage indicator to global dirty log
> synchronization
> kvm: Synchronize the backup bitmap in the last stage
> kvm: Add helper kvm_dirty_ring_init()
> hw/arm/virt: Enable backup bitmap for dirty ring
> kvm: Enable dirty ring for arm64
>
> accel/kvm/kvm-all.c | 95 ++++++++++++++++++++++++-----------
> hw/arm/virt.c | 26 ++++++++++
> include/exec/memory.h | 5 +-
> include/sysemu/kvm_int.h | 1 +
> linux-headers/asm-arm64/kvm.h | 1 +
> linux-headers/linux/kvm.h | 2 +
> migration/dirtyrate.c | 4 +-
> migration/ram.c | 20 ++++----
> softmmu/memory.c | 10 ++--
> target/arm/kvm64.c | 25 +++++++++
> target/arm/kvm_arm.h | 12 +++++
> 11 files changed, 152 insertions(+), 49 deletions(-)
>
> --
> 2.23.0
>
This series intends to support dirty ring for live migration. The dirty ring use discrete buffer to track dirty pages. For ARM64, the speciality is to use backup bitmap to track dirty pages when there is no-running-vcpu context. It's known that the backup bitmap needs to be synchronized when KVM device "kvm-arm-gicv3" or "arm-its-kvm" has been enabled. The backup bitmap is collected in the last stage of migration. PATCH[1] Synchronize linux-headers for dirty ring PATCH[2] Introduce indicator of the last stage migration and pass it all the way down PATCH[3] Synchronize the backup bitmap in the last stage of live migration PATCH[4] Introduce helper kvm_dirty_ring_init() to enable the dirty ring PATCH[5-6] Enable dirty ring for hw/arm/virt RFCv1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00171.html Testing ======= (1) kvm-unit-tests/its-pending-migration and kvm-unit-tests/its-migration with dirty ring or normal dirty page tracking mechanism. All test cases passed. QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm \ ./its-pending-migration QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm \ ./its-migration QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm,dirty-ring-size=65536 \ ./its-pending-migration QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm,dirty-ring-size=65536 \ ./its-migration (2) Combinations of migration, post-copy migration, e1000e and virtio-net devices. All test cases passed. -netdev tap,id=net0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \ -device e1000e,bus=pcie.5,netdev=net0,mac=52:54:00:f1:26:a0 -netdev tap,id=vnet0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \ -device virtio-net-pci,bus=pcie.6,netdev=vnet0,mac=52:54:00:f1:26:b0 Changelog ========= v1: * Combine two patches into one PATCH[v1 2/6] for the last stage indicator (Peter) * Drop the secondary bitmap and use the original one directly (Juan) * Avoid "goto out" in helper kvm_dirty_ring_init() (Juan) Gavin Shan (6): linux-headers: Update for dirty ring migration: Add last stage indicator to global dirty log synchronization kvm: Synchronize the backup bitmap in the last stage kvm: Add helper kvm_dirty_ring_init() hw/arm/virt: Enable backup bitmap for dirty ring kvm: Enable dirty ring for arm64 accel/kvm/kvm-all.c | 95 ++++++++++++++++++++++++----------- hw/arm/virt.c | 26 ++++++++++ include/exec/memory.h | 5 +- include/sysemu/kvm_int.h | 1 + linux-headers/asm-arm64/kvm.h | 1 + linux-headers/linux/kvm.h | 2 + migration/dirtyrate.c | 4 +- migration/ram.c | 20 ++++---- softmmu/memory.c | 10 ++-- target/arm/kvm64.c | 25 +++++++++ target/arm/kvm_arm.h | 12 +++++ 11 files changed, 152 insertions(+), 49 deletions(-)