[v5,00/12] *** A Method for evaluating dirty page rate ***

Message

Zheng Chuan Aug. 24, 2020, 9:14 a.m. UTC

v4 -> v5:
    fix git apply failed due to meson-build
    add review-by for patches in v3

v3 -> v4:
    use crc32 to get hash result instead of md5
    add DirtyRateStatus to denote calculation status
    add some trace_calls to make it easier to debug
    fix some comments accroding to review

v2 -> v3:
    fix size_t compile warning
    fix codestyle checked by checkpatch.pl

v1 -> v2:
    use g_rand_new() to generate rand_buf
    move RAMBLOCK_FOREACH_MIGRATABLE into migration/ram.h
    add skip_sample_ramblock to filter sampled ramblock
    fix multi-numa vm coredump when query dirtyrate
    rename qapi interface and rename some structures and functions
    succeed to compile by appling each patch
    add test for migrating vm

Sometimes it is neccessary to evaluate dirty page rate before migration.
Users could decide whether to proceed migration based on the evaluation
in case of vm performance loss due to heavy workload.
Unlikey simulating dirtylog sync which could do harm on runnning vm,
we provide a sample-hash method to compare hash results for samping page.
In this way, it would have hardly no impact on vm performance.

Evaluate the dirtypage rate both on running and migration vm.
The VM specifications for migration are as follows:
- VM use 4-K page;
- the number of VCPU is 32;
- the total memory is 32Gigabit;
- use 'mempress' tool to pressurize VM(mempress 4096 1024);
- migration bandwidth is 1GB/s

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|                      |  running  |                  migrating                           |
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
| no mempress          |   4MB/s   |          8MB/s      (migrated success)               |
-------------------------------------------------------------------------------------------
| mempress 4096 1024   |  1060MB/s |     456MB/s ~ 1142MB/s (cpu throttle triggered)      |
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
| mempress 4096 4096   |  4114MB/s |     688MB/s ~ 4132MB/s (cpu throttle triggered)      |
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Test dirtyrate by qmp command like this:
1.  virsh qemu-monitor-command [vmname] '{"execute":"calc-dirty-rate", "arguments": {"calc-time": [sleep-time]}}'; 
2.  sleep specific time which is a bit larger than sleep-time
3.  virsh qemu-monitor-command [vmname] '{"execute":"query-dirty-rate"}'

Further test dirtyrate by libvirt api like this:
virsh getdirtyrate [vmname] [sleep-time]

Chuan Zheng (12):
  migration/dirtyrate: setup up query-dirtyrate framwork
  migration/dirtyrate: add DirtyRateStatus to denote calculation status
  migration/dirtyrate: Add RamlockDirtyInfo to store sampled page info
  migration/dirtyrate: Add dirtyrate statistics series functions
  migration/dirtyrate: move RAMBLOCK_FOREACH_MIGRATABLE into ram.h
  migration/dirtyrate: Record hash results for each sampled page
  migration/dirtyrate: Compare page hash results for recorded sampled
    page
  migration/dirtyrate: skip sampling ramblock with size below
    MIN_RAMBLOCK_SIZE
  migration/dirtyrate: Implement get_sample_page_period() and
    block_sample_page_period()
  migration/dirtyrate: Implement calculate_dirtyrate() function
  migration/dirtyrate: Implement
    qmp_cal_dirty_rate()/qmp_get_dirty_rate() function
  migration/dirtyrate: Add trace_calls to make it easier to debug

 migration/dirtyrate.c  | 432 +++++++++++++++++++++++++++++++++++++++++++++++++
 migration/dirtyrate.h  |  87 ++++++++++
 migration/meson.build  |   1 +
 migration/ram.c        |  11 +-
 migration/ram.h        |  10 ++
 migration/trace-events |   8 +
 qapi/migration.json    |  61 +++++++
 7 files changed, 600 insertions(+), 10 deletions(-)
 create mode 100644 migration/dirtyrate.c
 create mode 100644 migration/dirtyrate.h

Comments

[e-mail] Aug. 24, 2020, 4:50 p.m. UTC | #1

Patchew URL: https://patchew.org/QEMU/[e-mail]/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: [e-mail]
Subject: [PATCH v5 00/12] *** A Method for evaluating dirty page rate ***

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
   dd8014e..df82aa7  master     -> master
 - [tag update]      patchew/[e-mail] -> patchew/[e-mail]
 - [tag update]      patchew/[e-mail] -> patchew/[e-mail]
 - [tag update]      patchew/[e-mail] -> patchew/[e-mail]
 - [tag update]      patchew/[e-mail] -> patchew/[e-mail]
 - [tag update]      patchew/[e-mail] -> patchew/[e-mail]
 - [tag update]      patchew/[e-mail] -> patchew/[e-mail]
 - [tag update]      patchew/[e-mail] -> patchew/[e-mail]
 - [tag update]      patchew/[e-mail] -> patchew/[e-mail]
 * [new tag]         patchew/[e-mail] -> patchew/[e-mail]
 * [new tag]         patchew/[e-mail] -> patchew/[e-mail]
 * [new tag]         patchew/[e-mail] -> patchew/[e-mail]
 * [new tag]         patchew/[e-mail] -> patchew/[e-mail]
 * [new tag]         patchew/[e-mail] -> patchew/[e-mail]
 * [new tag]         patchew/[e-mail] -> patchew/[e-mail]
 * [new tag]         patchew/[e-mail] -> patchew/[e-mail]
Switched to a new branch 'test'
e422c48 migration/dirtyrate: Add trace_calls to make it easier to debug
d544358 migration/dirtyrate: Implement qmp_cal_dirty_rate()/qmp_get_dirty_rate() function
b245e3e migration/dirtyrate: Implement calculate_dirtyrate() function
f634d60 migration/dirtyrate: Implement get_sample_page_period() and block_sample_page_period()
039884c migration/dirtyrate: skip sampling ramblock with size below MIN_RAMBLOCK_SIZE
b62b0f3 migration/dirtyrate: Compare page hash results for recorded sampled page
a0c77f6 migration/dirtyrate: Record hash results for each sampled page
7e4c941 migration/dirtyrate: move RAMBLOCK_FOREACH_MIGRATABLE into ram.h
5bd531b migration/dirtyrate: Add dirtyrate statistics series functions
af4f569 migration/dirtyrate: Add RamlockDirtyInfo to store sampled page info
9d2abe8 migration/dirtyrate: add DirtyRateStatus to denote calculation status
9da875e migration/dirtyrate: setup up query-dirtyrate framwork

=== OUTPUT BEGIN ===
1/12 Checking commit 9da875ef425e (migration/dirtyrate: setup up query-dirtyrate framwork)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#15: 
new file mode 100644

total: 0 errors, 1 warnings, 78 lines checked

Patch 1/12 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
2/12 Checking commit 9d2abe8b480a (migration/dirtyrate: add DirtyRateStatus to denote calculation status)
3/12 Checking commit af4f56975987 (migration/dirtyrate: Add RamlockDirtyInfo to store sampled page info)
4/12 Checking commit 5bd531bb00ce (migration/dirtyrate: Add dirtyrate statistics series functions)
5/12 Checking commit 7e4c941b32bc (migration/dirtyrate: move RAMBLOCK_FOREACH_MIGRATABLE into ram.h)
ERROR: Macros with multiple statements should be enclosed in a do - while loop
#62: FILE: migration/ram.h:42:
+#define RAMBLOCK_FOREACH_NOT_IGNORED(block)            \
+    INTERNAL_RAMBLOCK_FOREACH(block)                   \
+        if (ramblock_is_ignored(block)) {} else

ERROR: trailing statements should be on next line
#64: FILE: migration/ram.h:44:
+        if (ramblock_is_ignored(block)) {} else

ERROR: Macros with multiple statements should be enclosed in a do - while loop
#66: FILE: migration/ram.h:46:
+#define RAMBLOCK_FOREACH_MIGRATABLE(block)             \
+    INTERNAL_RAMBLOCK_FOREACH(block)                   \
+        if (!qemu_ram_is_migratable(block)) {} else

ERROR: trailing statements should be on next line
#68: FILE: migration/ram.h:48:
+        if (!qemu_ram_is_migratable(block)) {} else

ERROR: braces {} are necessary for all arms of this statement
#68: FILE: migration/ram.h:48:
+        if (!qemu_ram_is_migratable(block)) {} else
[...]
+        if (!qemu_ram_is_migratable(block)) {} else
[...]

total: 5 errors, 0 warnings, 45 lines checked

Patch 5/12 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

6/12 Checking commit a0c77f6034fc (migration/dirtyrate: Record hash results for each sampled page)
7/12 Checking commit b62b0f3c7033 (migration/dirtyrate: Compare page hash results for recorded sampled page)
8/12 Checking commit 039884cf410e (migration/dirtyrate: skip sampling ramblock with size below MIN_RAMBLOCK_SIZE)
9/12 Checking commit f634d60df875 (migration/dirtyrate: Implement get_sample_page_period() and block_sample_page_period())
10/12 Checking commit b245e3e9cd0d (migration/dirtyrate: Implement calculate_dirtyrate() function)
11/12 Checking commit d544358ff2b0 (migration/dirtyrate: Implement qmp_cal_dirty_rate()/qmp_get_dirty_rate() function)
12/12 Checking commit e422c48502c8 (migration/dirtyrate: Add trace_calls to make it easier to debug)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/[e-mail]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [e-mail]