From patchwork Tue Oct 18 12:10:04 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhanghailiang X-Patchwork-Id: 9382089 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 98C94607D0 for ; Tue, 18 Oct 2016 12:36:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 88E3A295C6 for ; Tue, 18 Oct 2016 12:36:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7D888295D0; Tue, 18 Oct 2016 12:36:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 43DDB295C6 for ; Tue, 18 Oct 2016 12:36:02 +0000 (UTC) Received: from localhost ([::1]:41188 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bwTcf-00027T-JR for patchwork-qemu-devel@patchwork.kernel.org; Tue, 18 Oct 2016 08:36:01 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37142) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bwTHa-0000vx-CX for qemu-devel@nongnu.org; Tue, 18 Oct 2016 08:14:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bwTHW-0006HV-8W for qemu-devel@nongnu.org; Tue, 18 Oct 2016 08:14:14 -0400 Received: from szxga01-in.huawei.com ([58.251.152.64]:64610) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1bwTHT-00066b-4a for qemu-devel@nongnu.org; Tue, 18 Oct 2016 08:14:10 -0400 Received: from 172.24.1.47 (EHLO szxeml431-hub.china.huawei.com) ([172.24.1.47]) by szxrg01-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id DTC35922; Tue, 18 Oct 2016 20:10:56 +0800 (CST) Received: from localhost (10.177.24.212) by szxeml431-hub.china.huawei.com (10.82.67.208) with Microsoft SMTP Server id 14.3.235.1; Tue, 18 Oct 2016 20:10:50 +0800 From: zhanghailiang To: , Date: Tue, 18 Oct 2016 20:10:04 +0800 Message-ID: <1476792613-11712-9-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1476792613-11712-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1476792613-11712-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] X-Received-From: 58.251.152.64 Subject: [Qemu-devel] [PATCH COLO-Frame (Base) v21 08/17] COLO: Send PVM state to secondary side when do checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: xiecl.fnst@cn.fujitsu.com, zhanghailiang , lizhijian@cn.fujitsu.com, qemu-devel@nongnu.org, dgilbert@redhat.com, Gonglei Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP VM checkpointing is to synchronize the state of PVM to SVM, just like migration does, we re-use save helpers to achieve migrating PVM's state to Secondary side. COLO need to cache the data of VM's state in the secondary side before synchronize it to SVM. COLO need the size of the data to determine how much data should be read in the secondary side. So here, we can get the size of the data by saving it into I/O channel before send it to the secondary side. Signed-off-by: zhanghailiang Signed-off-by: Gonglei Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert Cc: Dr. David Alan Gilbert Reviewed-by: Amit Shah --- v19: - fix title and comment. v17: - Rebase to master, use the new channel-buffer API v16: - Rename colo_put_cmd_value() to colo_send_message_value() v13: - Refactor colo_put_cmd_value() to use 'Error **errp' to indicate success or failure. v12: - Replace the old colo_ctl_get() with the new helper function colo_put_cmd_value() v11: - Add Reviewed-by tag --- migration/colo.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++----- migration/ram.c | 37 +++++++++++++++++------- 2 files changed, 105 insertions(+), 17 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index bf32d63..9736ccd 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -13,10 +13,13 @@ #include "qemu/osdep.h" #include "sysemu/sysemu.h" #include "migration/colo.h" +#include "io/channel-buffer.h" #include "trace.h" #include "qemu/error-report.h" #include "qapi/error.h" +#define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024) + bool colo_supported(void) { return false; @@ -55,6 +58,27 @@ static void colo_send_message(QEMUFile *f, COLOMessage msg, trace_colo_send_message(COLOMessage_lookup[msg]); } +static void colo_send_message_value(QEMUFile *f, COLOMessage msg, + uint64_t value, Error **errp) +{ + Error *local_err = NULL; + int ret; + + colo_send_message(f, msg, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + qemu_put_be64(f, value); + qemu_fflush(f); + + ret = qemu_file_get_error(f); + if (ret < 0) { + error_setg_errno(errp, -ret, "Failed to send value for message:%s", + COLOMessage_lookup[msg]); + } +} + static COLOMessage colo_receive_message(QEMUFile *f, Error **errp) { COLOMessage msg; @@ -91,9 +115,12 @@ static void colo_receive_check_message(QEMUFile *f, COLOMessage expect_msg, } } -static int colo_do_checkpoint_transaction(MigrationState *s) +static int colo_do_checkpoint_transaction(MigrationState *s, + QIOChannelBuffer *bioc, + QEMUFile *fb) { Error *local_err = NULL; + int ret = -1; colo_send_message(s->to_dst_file, COLO_MESSAGE_CHECKPOINT_REQUEST, &local_err); @@ -106,15 +133,46 @@ static int colo_do_checkpoint_transaction(MigrationState *s) if (local_err) { goto out; } + /* Reset channel-buffer directly */ + qio_channel_io_seek(QIO_CHANNEL(bioc), 0, 0, NULL); + bioc->usage = 0; - /* TODO: suspend and save vm state to colo buffer */ + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + qemu_mutex_unlock_iothread(); + trace_colo_vm_state_change("run", "stop"); + + /* Disable block migration */ + s->params.blk = 0; + s->params.shared = 0; + qemu_savevm_state_header(fb); + qemu_savevm_state_begin(fb, &s->params); + qemu_mutex_lock_iothread(); + qemu_savevm_state_complete_precopy(fb, false); + qemu_mutex_unlock_iothread(); + + qemu_fflush(fb); colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_err); if (local_err) { goto out; } + /* + * We need the size of the VMstate data in Secondary side, + * With which we can decide how much data should be read. + */ + colo_send_message_value(s->to_dst_file, COLO_MESSAGE_VMSTATE_SIZE, + bioc->usage, &local_err); + if (local_err) { + goto out; + } - /* TODO: send vmstate to Secondary */ + qemu_put_buffer(s->to_dst_file, bioc->data, bioc->usage); + qemu_fflush(s->to_dst_file); + ret = qemu_file_get_error(s->to_dst_file); + if (ret < 0) { + goto out; + } colo_receive_check_message(s->rp_state.from_dst_file, COLO_MESSAGE_VMSTATE_RECEIVED, &local_err); @@ -128,18 +186,24 @@ static int colo_do_checkpoint_transaction(MigrationState *s) goto out; } - /* TODO: resume Primary */ + ret = 0; + + qemu_mutex_lock_iothread(); + vm_start(); + qemu_mutex_unlock_iothread(); + trace_colo_vm_state_change("stop", "run"); - return 0; out: if (local_err) { error_report_err(local_err); } - return -EINVAL; + return ret; } static void colo_process_checkpoint(MigrationState *s) { + QIOChannelBuffer *bioc; + QEMUFile *fb = NULL; Error *local_err = NULL; int ret; @@ -158,6 +222,9 @@ static void colo_process_checkpoint(MigrationState *s) if (local_err) { goto out; } + bioc = qio_channel_buffer_new(COLO_BUFFER_BASE_SIZE); + fb = qemu_fopen_channel_output(QIO_CHANNEL(bioc)); + object_unref(OBJECT(bioc)); qemu_mutex_lock_iothread(); vm_start(); @@ -165,7 +232,7 @@ static void colo_process_checkpoint(MigrationState *s) trace_colo_vm_state_change("stop", "run"); while (s->state == MIGRATION_STATUS_COLO) { - ret = colo_do_checkpoint_transaction(s); + ret = colo_do_checkpoint_transaction(s, bioc, fb); if (ret < 0) { goto out; } @@ -179,6 +246,10 @@ out: migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + if (fb) { + qemu_fclose(fb); + } + if (s->rp_state.from_dst_file) { qemu_fclose(s->rp_state.from_dst_file); } diff --git a/migration/ram.c b/migration/ram.c index bc6154f..d632305 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -43,6 +43,7 @@ #include "trace.h" #include "exec/ram_addr.h" #include "qemu/rcu_queue.h" +#include "migration/colo.h" #ifdef DEBUG_MIGRATION_RAM #define DPRINTF(fmt, ...) \ @@ -1870,16 +1871,8 @@ err: return ret; } - -/* Each of ram_save_setup, ram_save_iterate and ram_save_complete has - * long-running RCU critical section. When rcu-reclaims in the code - * start to become numerous it will be necessary to reduce the - * granularity of these critical sections. - */ - -static int ram_save_setup(QEMUFile *f, void *opaque) +static int ram_save_init_globals(void) { - RAMBlock *block; int64_t ram_bitmap_pages; /* Size of bitmap in pages, including gaps */ dirty_rate_high_cnt = 0; @@ -1945,6 +1938,29 @@ static int ram_save_setup(QEMUFile *f, void *opaque) migration_bitmap_sync(); qemu_mutex_unlock_ramlist(); qemu_mutex_unlock_iothread(); + rcu_read_unlock(); + + return 0; +} + +/* Each of ram_save_setup, ram_save_iterate and ram_save_complete has + * long-running RCU critical section. When rcu-reclaims in the code + * start to become numerous it will be necessary to reduce the + * granularity of these critical sections. + */ + +static int ram_save_setup(QEMUFile *f, void *opaque) +{ + RAMBlock *block; + + /* migration has already setup the bitmap, reuse it. */ + if (!migration_in_colo_state()) { + if (ram_save_init_globals() < 0) { + return -1; + } + } + + rcu_read_lock(); qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE); @@ -2046,7 +2062,8 @@ static int ram_save_complete(QEMUFile *f, void *opaque) while (true) { int pages; - pages = ram_find_and_save_block(f, true, &bytes_transferred); + pages = ram_find_and_save_block(f, !migration_in_colo_state(), + &bytes_transferred); /* no more blocks to sent */ if (pages == 0) { break;