From patchwork Sat Feb 6 09:28:22 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhanghailiang X-Patchwork-Id: 8242091 Return-Path: X-Original-To: patchwork-qemu-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 5D7119FC36 for ; Sat, 6 Feb 2016 09:39:12 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 2759920204 for ; Sat, 6 Feb 2016 09:39:10 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1300D200F2 for ; Sat, 6 Feb 2016 09:39:09 +0000 (UTC) Received: from localhost ([::1]:52636 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aRzKe-0002yd-EY for patchwork-qemu-devel@patchwork.kernel.org; Sat, 06 Feb 2016 04:39:08 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38040) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aRzBU-0002Ft-Jl for qemu-devel@nongnu.org; Sat, 06 Feb 2016 04:29:42 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aRzBP-0001NA-Au for qemu-devel@nongnu.org; Sat, 06 Feb 2016 04:29:40 -0500 Received: from szxga02-in.huawei.com ([119.145.14.65]:19433) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aRzBO-0001M3-27 for qemu-devel@nongnu.org; Sat, 06 Feb 2016 04:29:35 -0500 Received: from 172.24.1.49 (EHLO szxeml428-hub.china.huawei.com) ([172.24.1.49]) by szxrg02-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id DBA11985; Sat, 06 Feb 2016 17:29:22 +0800 (CST) Received: from localhost (10.177.24.212) by szxeml428-hub.china.huawei.com (10.82.67.183) with Microsoft SMTP Server id 14.3.235.1; Sat, 6 Feb 2016 17:29:12 +0800 From: zhanghailiang To: Date: Sat, 6 Feb 2016 17:28:22 +0800 Message-ID: <1454750932-7556-11-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 1.9.0.msysgit.0 In-Reply-To: <1454750932-7556-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1454750932-7556-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020202.56B5BCF2.004D, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2013-06-18 04:22:30, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 06522c13067f1e34f732270a80a41f7f X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] X-Received-From: 119.145.14.65 Cc: xiecl.fnst@cn.fujitsu.com, lizhijian@cn.fujitsu.com, quintela@redhat.com, armbru@redhat.com, yunhong.jiang@intel.com, eddie.dong@intel.com, peter.huangpeng@huawei.com, dgilbert@redhat.com, zhanghailiang , arei.gonglei@huawei.com, stefanha@redhat.com, amit.shah@redhat.com, zhangchen.fnst@cn.fujitsu.com, hongyang.yang@easystack.cn Subject: [Qemu-devel] [PATCH COLO-Frame v14 10/40] COLO: Save PVM state to secondary side when do checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The main process of checkpoint is to synchronize SVM with PVM. VM's state includes ram and device state. So we will migrate PVM's state to SVM when do checkpoint, just like migration does. We will cache PVM's state in slave, we use QEMUSizedBuffer to store the data, we need to know the size of VM state, so in master, we use qsb to store VM state temporarily, get the data size by call qsb_get_length() and then migrate the data to the qsb in the secondary side. Signed-off-by: zhanghailiang Signed-off-by: Gonglei Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert Cc: Dr. David Alan Gilbert --- v13: - Refactor colo_put_cmd_value() to use 'Error **errp' to indicate success or failure. v12: - Replace the old colo_ctl_get() with the new helper function colo_put_cmd_value() v11: - Add Reviewed-by tag --- migration/colo.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++----- migration/ram.c | 39 ++++++++++++++++++------ 2 files changed, 114 insertions(+), 17 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index c0ff088..7e4692c 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -16,6 +16,9 @@ #include "trace.h" #include "qemu/error-report.h" +/* colo buffer */ +#define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024) + bool colo_supported(void) { return true; @@ -54,6 +57,27 @@ static void colo_put_cmd(QEMUFile *f, COLOMessage cmd, trace_colo_put_cmd(COLOMessage_lookup[cmd]); } +static void colo_put_cmd_value(QEMUFile *f, COLOMessage cmd, + uint64_t value, Error **errp) +{ + Error *local_err = NULL; + int ret; + + colo_put_cmd(f, cmd, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + qemu_put_be64(f, value); + qemu_fflush(f); + + ret = qemu_file_get_error(f); + if (ret < 0) { + error_setg_errno(errp, -ret, "Failed to send value for command:%s", + COLOMessage_lookup[cmd]); + } +} + static COLOMessage colo_get_cmd(QEMUFile *f, Error **errp) { COLOMessage cmd; @@ -90,9 +114,13 @@ static void colo_get_check_cmd(QEMUFile *f, COLOMessage expect_cmd, } } -static int colo_do_checkpoint_transaction(MigrationState *s) +static int colo_do_checkpoint_transaction(MigrationState *s, + QEMUSizedBuffer *buffer) { + QEMUFile *trans = NULL; + size_t size; Error *local_err = NULL; + int ret = -1; colo_put_cmd(s->to_dst_file, COLO_MESSAGE_CHECKPOINT_REQUEST, &local_err); @@ -105,15 +133,48 @@ static int colo_do_checkpoint_transaction(MigrationState *s) if (local_err) { goto out; } + /* Reset colo buffer and open it for write */ + qsb_set_length(buffer, 0); + trans = qemu_bufopen("w", buffer); + if (!trans) { + error_report("Open colo buffer for write failed"); + goto out; + } - /* TODO: suspend and save vm state to colo buffer */ + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + qemu_mutex_unlock_iothread(); + trace_colo_vm_state_change("run", "stop"); + + /* Disable block migration */ + s->params.blk = 0; + s->params.shared = 0; + qemu_savevm_state_header(trans); + qemu_savevm_state_begin(trans, &s->params); + qemu_mutex_lock_iothread(); + qemu_savevm_state_complete_precopy(trans, false); + qemu_mutex_unlock_iothread(); + + qemu_fflush(trans); colo_put_cmd(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_err); if (local_err) { goto out; } + /* we send the total size of the vmstate first */ + size = qsb_get_length(buffer); + colo_put_cmd_value(s->to_dst_file, COLO_MESSAGE_VMSTATE_SIZE, + size, &local_err); + if (local_err) { + goto out; + } - /* TODO: send vmstate to Secondary */ + qsb_put_buffer(s->to_dst_file, buffer, size); + qemu_fflush(s->to_dst_file); + ret = qemu_file_get_error(s->to_dst_file); + if (ret < 0) { + goto out; + } colo_get_check_cmd(s->rp_state.from_dst_file, COLO_MESSAGE_VMSTATE_RECEIVED, &local_err); @@ -127,18 +188,26 @@ static int colo_do_checkpoint_transaction(MigrationState *s) goto out; } - /* TODO: resume Primary */ + ret = 0; + /* Resume primary guest */ + qemu_mutex_lock_iothread(); + vm_start(); + qemu_mutex_unlock_iothread(); + trace_colo_vm_state_change("stop", "run"); - return 0; out: if (local_err) { error_report_err(local_err); } - return -EINVAL; + if (trans) { + qemu_fclose(trans); + } + return ret; } static void colo_process_checkpoint(MigrationState *s) { + QEMUSizedBuffer *buffer = NULL; Error *local_err = NULL; int ret; @@ -158,6 +227,12 @@ static void colo_process_checkpoint(MigrationState *s) goto out; } + buffer = qsb_create(NULL, COLO_BUFFER_BASE_SIZE); + if (buffer == NULL) { + error_report("Failed to allocate colo buffer!"); + goto out; + } + qemu_mutex_lock_iothread(); vm_start(); qemu_mutex_unlock_iothread(); @@ -165,7 +240,7 @@ static void colo_process_checkpoint(MigrationState *s) while (s->state == MIGRATION_STATUS_COLO) { /* start a colo checkpoint */ - ret = colo_do_checkpoint_transaction(s); + ret = colo_do_checkpoint_transaction(s, buffer); if (ret < 0) { goto out; } @@ -179,6 +254,9 @@ out: migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + qsb_free(buffer); + buffer = NULL; + if (s->rp_state.from_dst_file) { qemu_fclose(s->rp_state.from_dst_file); } diff --git a/migration/ram.c b/migration/ram.c index 3cdfea4..7b3e1db 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -40,6 +40,7 @@ #include "trace.h" #include "exec/ram_addr.h" #include "qemu/rcu_queue.h" +#include "migration/colo.h" #ifdef DEBUG_MIGRATION_RAM #define DPRINTF(fmt, ...) \ @@ -1874,16 +1875,8 @@ err: return ret; } - -/* Each of ram_save_setup, ram_save_iterate and ram_save_complete has - * long-running RCU critical section. When rcu-reclaims in the code - * start to become numerous it will be necessary to reduce the - * granularity of these critical sections. - */ - -static int ram_save_setup(QEMUFile *f, void *opaque) +static int ram_save_init_globals(void) { - RAMBlock *block; int64_t ram_bitmap_pages; /* Size of bitmap in pages, including gaps */ dirty_rate_high_cnt = 0; @@ -1948,6 +1941,31 @@ static int ram_save_setup(QEMUFile *f, void *opaque) migration_bitmap_sync(); qemu_mutex_unlock_ramlist(); qemu_mutex_unlock_iothread(); + rcu_read_unlock(); + + return 0; +} + +/* Each of ram_save_setup, ram_save_iterate and ram_save_complete has + * long-running RCU critical section. When rcu-reclaims in the code + * start to become numerous it will be necessary to reduce the + * granularity of these critical sections. + */ + +static int ram_save_setup(QEMUFile *f, void *opaque) +{ + RAMBlock *block; + + /* + * migration has already setup the bitmap, reuse it. + */ + if (!migration_in_colo_state()) { + if (ram_save_init_globals() < 0) { + return -1; + } + } + + rcu_read_lock(); qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE); @@ -2049,7 +2067,8 @@ static int ram_save_complete(QEMUFile *f, void *opaque) while (true) { int pages; - pages = ram_find_and_save_block(f, true, &bytes_transferred); + pages = ram_find_and_save_block(f, !migration_in_colo_state(), + &bytes_transferred); /* no more blocks to sent */ if (pages == 0) { break;