From patchwork Mon Feb 22 02:40:23 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhanghailiang X-Patchwork-Id: 8370591 Return-Path: X-Original-To: patchwork-qemu-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 2398D9F314 for ; Mon, 22 Feb 2016 02:51:47 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 6161820382 for ; Mon, 22 Feb 2016 02:51:46 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6FF9920220 for ; Mon, 22 Feb 2016 02:51:45 +0000 (UTC) Received: from localhost ([::1]:46149 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aXgbA-0006n8-N1 for patchwork-qemu-devel@patchwork.kernel.org; Sun, 21 Feb 2016 21:51:44 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59865) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aXgRf-0001mq-Gu for qemu-devel@nongnu.org; Sun, 21 Feb 2016 21:41:56 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aXgRd-0007Xu-R5 for qemu-devel@nongnu.org; Sun, 21 Feb 2016 21:41:55 -0500 Received: from szxga01-in.huawei.com ([58.251.152.64]:64429) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aXgRd-0007XA-3c for qemu-devel@nongnu.org; Sun, 21 Feb 2016 21:41:53 -0500 Received: from 172.24.1.49 (EHLO SZXEML423-HUB.china.huawei.com) ([172.24.1.49]) by szxrg01-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id DEY15423; Mon, 22 Feb 2016 10:41:39 +0800 (CST) Received: from localhost (10.177.24.212) by SZXEML423-HUB.china.huawei.com (10.82.67.154) with Microsoft SMTP Server id 14.3.235.1; Mon, 22 Feb 2016 10:41:32 +0800 From: zhanghailiang To: Date: Mon, 22 Feb 2016 10:40:23 +0800 Message-ID: <1456108832-24212-30-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 1.9.0.msysgit.0 In-Reply-To: <1456108832-24212-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1456108832-24212-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020205.56CA7563.0052, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2013-06-18 04:22:30, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 80e96b816fd840d32a8f4968888db975 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] X-Received-From: 58.251.152.64 Cc: xiecl.fnst@cn.fujitsu.com, lizhijian@cn.fujitsu.com, quintela@redhat.com, armbru@redhat.com, yunhong.jiang@intel.com, eddie.dong@intel.com, peter.huangpeng@huawei.com, dgilbert@redhat.com, zhanghailiang , arei.gonglei@huawei.com, stefanha@redhat.com, amit.shah@redhat.com, zhangchen.fnst@cn.fujitsu.com, hongyang.yang@easystack.cn Subject: [Qemu-devel] [PATCH COLO-Frame v15 29/38] COLO: Separate the process of saving/loading ram and device state X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We separate the process of saving/loading ram and device state when do checkpoint, we add new helpers for save/load ram/device. With this change, we can directly transfer ram from master to slave without using QEMUSizeBufferas as assistant, which also reduce the size of extra memory been used during checkpoint. Besides, we move the colo_flush_ram_cache to the proper position after the above change. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- v14: - split two new patches from this patch - Some minor fixes from Dave v13: - Re-use some existed helper functions to realize saving/loading ram and device. v11: - Remove load configuration section in qemu_loadvm_state_begin() --- migration/colo.c | 48 ++++++++++++++++++++++++++++++++++++++---------- migration/ram.c | 5 ----- migration/savevm.c | 5 +++++ 3 files changed, 43 insertions(+), 15 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 16bada6..300fa54 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -288,21 +288,37 @@ static int colo_do_checkpoint_transaction(MigrationState *s, goto out; } + colo_put_cmd(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_err); + if (local_err) { + goto out; + } + /* Disable block migration */ s->params.blk = 0; s->params.shared = 0; - qemu_savevm_state_header(trans); - qemu_savevm_state_begin(trans, &s->params); + qemu_savevm_state_begin(s->to_dst_file, &s->params); + ret = qemu_file_get_error(s->to_dst_file); + if (ret < 0) { + error_report("Save vm state begin error"); + goto out; + } + qemu_mutex_lock_iothread(); - qemu_savevm_state_complete_precopy(trans, false); + /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + /* Note: device state is saved into buffer */ + ret = qemu_save_device_state(trans); qemu_mutex_unlock_iothread(); - - qemu_fflush(trans); - - colo_put_cmd(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_err); - if (local_err) { + if (ret < 0) { + error_report("Save device state error"); goto out; } + qemu_fflush(trans); + /* we send the total size of the vmstate first */ size = qsb_get_length(buffer); colo_put_cmd_value(s->to_dst_file, COLO_MESSAGE_VMSTATE_SIZE, @@ -573,6 +589,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + ret = qemu_loadvm_state_begin(mis->from_src_file); + if (ret < 0) { + error_report("Load vm state begin error, ret=%d", ret); + goto out; + } + ret = qemu_loadvm_state_main(mis->from_src_file, mis); + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } /* read the VM state total size first */ value = colo_get_cmd_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); @@ -605,8 +631,10 @@ void *colo_process_incoming_thread(void *opaque) qemu_mutex_lock_iothread(); qemu_system_reset(VMRESET_SILENT); vmstate_loading = true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + colo_flush_ram_cache(); + ret = qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } diff --git a/migration/ram.c b/migration/ram.c index 891f3b2..8f416d5 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2465,7 +2465,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) * be atomic */ bool postcopy_running = postcopy_state_get() >= POSTCOPY_INCOMING_LISTENING; - bool need_flush = false; seq_iter++; @@ -2500,7 +2499,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) /* After going into COLO, we should load the Page into colo_cache */ if (ram_cache_enable) { host = colo_cache_from_block_offset(block, addr); - need_flush = true; } else { host = host_from_ram_block_offset(block, addr); } @@ -2594,9 +2592,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) rcu_read_unlock(); - if (!ret && ram_cache_enable && need_flush) { - colo_flush_ram_cache(); - } DPRINTF("Completed load of VM with exit code %d seq iteration " "%" PRIu64 "\n", ret, seq_iter); return ret; diff --git a/migration/savevm.c b/migration/savevm.c index 60c7b57..1551fbb 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -50,6 +50,7 @@ #include "qemu/iov.h" #include "block/snapshot.h" #include "block/qapi.h" +#include "migration/colo.h" #ifndef ETH_P_RARP @@ -923,6 +924,10 @@ void qemu_savevm_state_begin(QEMUFile *f, break; } } + if (migration_in_colo_state()) { + qemu_put_byte(f, QEMU_VM_EOF); + qemu_fflush(f); + } } /*