| Message ID | 20190111063732.10484-1-xiaoguangrong@tencent.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | optimize waiting for free thread to do compression | expand |
guangrong.xiao@gmail.com writes: > From: Xiao Guangrong <xiaoguangrong@tencent.com> > > Changelog in v2: > squash 'compress-wait-thread-adaptive' into 'compress-wait-thread' based > on peter's suggestion > > > Currently we have two behaviors if all threads are busy to do compression, > the main thread mush wait one of them becoming free if @compress-wait-thread > set to on or the main thread can directly return without wait and post > the page out as normal one > > Both of them have its profits and short-comes, however, if the bandwidth is > not limited extremely so that compression can not use out all of it bandwidth, > at the same time, the migration process is easily throttled if we posted too > may pages as normal pages. None of them can work properly under this case > > In order to use the bandwidth more effectively, we introduce the third > behavior, adaptive, which make the main thread wait > if there is no bandwidth left or let the page go out as normal page if there > has enough bandwidth to make sure the migration process will not be > throttled > > Another patch introduces a new statistic, pages-per-second, as bandwidth > or mbps is not enough to measure the performance of posting pages out as > we have compression, xbzrle, which can significantly reduce the amount of > the data size, instead, pages-per-second is the one we want > > Performance data > ================ > We have limited the bandwidth to 300 > > Used Bandwidth Pages-per-Second > compress-wait-thread = on 951.66 mbps 131784 > > compress-wait-thread = off 2491.74 mbps 93495 > compress-wait-thread-adaptive 1982.94 mbps 162529 > = on I think you mean compress-wait-thread = adaptive here. Can you sketch use cases for all three settings? The resulting guidance should then be worked into documentation.
Patchew URL: https://patchew.org/QEMU/20190111063732.10484-1-xiaoguangrong@tencent.com/ Hi, This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally. === TEST SCRIPT BEGIN === #!/bin/bash time make docker-test-quick@centos7 SHOW_ENV=1 J=8 === TEST SCRIPT END === The full log is available at http://patchew.org/logs/20190111063732.10484-1-xiaoguangrong@tencent.com/testing.docker-quick@centos7/?type=message. --- Email generated automatically by Patchew [http://patchew.org/]. Please send your feedback to patchew-devel@redhat.com
Patchew URL: https://patchew.org/QEMU/20190111063732.10484-1-xiaoguangrong@tencent.com/
Hi,
This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-mingw@fedora SHOW_ENV=1 J=14
=== TEST SCRIPT END ===
CC aarch64-softmmu/trace/generated-helpers.o
CC aarch64-softmmu/target/arm/translate-sve.o
/tmp/qemu-test/src/migration/ram.c: In function 'compress_page_with_multi_thread':
/tmp/qemu-test/src/migration/ram.c:2064:26: error: comparison of constant '2' with boolean expression is always false [-Werror=bool-compare]
adaptive = (adaptive == COMPRESS_WAIT_THREAD_ADAPTIVE);
^~
/tmp/qemu-test/src/migration/ram.c:2060:9: error: unused variable 'compress_wait_thread' [-Werror=unused-variable]
int compress_wait_thread = migrate_compress_wait_thread();
^~~~~~~~~~~~~~~~~~~~
/tmp/qemu-test/src/migration/ram.c:2063:10: error: 'adaptive' is used uninitialized in this function [-Werror=uninitialized]
wait = (adaptive == COMPRESS_WAIT_THREAD_ON);
~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
make[1]: *** [/tmp/qemu-test/src/rules.mak:69: migration/ram.o] Error 1
make[1]: *** Waiting for unfinished jobs....
/tmp/qemu-test/src/migration/ram.c: In function 'compress_page_with_multi_thread':
/tmp/qemu-test/src/migration/ram.c:2064:26: error: comparison of constant '2' with boolean expression is always false [-Werror=bool-compare]
adaptive = (adaptive == COMPRESS_WAIT_THREAD_ADAPTIVE);
^~
/tmp/qemu-test/src/migration/ram.c:2060:9: error: unused variable 'compress_wait_thread' [-Werror=unused-variable]
int compress_wait_thread = migrate_compress_wait_thread();
^~~~~~~~~~~~~~~~~~~~
/tmp/qemu-test/src/migration/ram.c:2063:10: error: 'adaptive' is used uninitialized in this function [-Werror=uninitialized]
wait = (adaptive == COMPRESS_WAIT_THREAD_ON);
~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
The full log is available at
http://patchew.org/logs/20190111063732.10484-1-xiaoguangrong@tencent.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
From: Xiao Guangrong <xiaoguangrong@tencent.com> Changelog in v2: squash 'compress-wait-thread-adaptive' into 'compress-wait-thread' based on peter's suggestion Currently we have two behaviors if all threads are busy to do compression, the main thread mush wait one of them becoming free if @compress-wait-thread set to on or the main thread can directly return without wait and post the page out as normal one Both of them have its profits and short-comes, however, if the bandwidth is not limited extremely so that compression can not use out all of it bandwidth, at the same time, the migration process is easily throttled if we posted too may pages as normal pages. None of them can work properly under this case In order to use the bandwidth more effectively, we introduce the third behavior, adaptive, which make the main thread wait if there is no bandwidth left or let the page go out as normal page if there has enough bandwidth to make sure the migration process will not be throttled Another patch introduces a new statistic, pages-per-second, as bandwidth or mbps is not enough to measure the performance of posting pages out as we have compression, xbzrle, which can significantly reduce the amount of the data size, instead, pages-per-second is the one we want Performance data ================ We have limited the bandwidth to 300 Used Bandwidth Pages-per-Second compress-wait-thread = on 951.66 mbps 131784 compress-wait-thread = off 2491.74 mbps 93495 compress-wait-thread-adaptive 1982.94 mbps 162529 = on Xiao Guangrong (3): migration: introduce pages-per-second migration: fix memory leak when updating tls-creds and tls-hostname migration: introduce adaptive model for waiting thread hmp.c | 8 ++- migration/migration.c | 137 +++++++++++++++++++++++++++++++++++++++++--------- migration/migration.h | 21 ++++++++ migration/ram.c | 122 ++++++++++++++++++++++++++++++++++++++++---- qapi/migration.json | 44 +++++++++++----- 5 files changed, 281 insertions(+), 51 deletions(-)