From patchwork Mon May 14 03:23:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10396793 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 585D66038F for ; Mon, 14 May 2018 06:31:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 457202909C for ; Mon, 14 May 2018 06:31:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3846B290B1; Mon, 14 May 2018 06:31:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6B01B2909C for ; Mon, 14 May 2018 06:31:02 +0000 (UTC) Received: from localhost ([::1]:42540 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI70e-0004hW-SF for patchwork-qemu-devel@patchwork.kernel.org; Mon, 14 May 2018 02:31:00 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47370) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6g5-0006dK-7A for qemu-devel@nongnu.org; Mon, 14 May 2018 02:09:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI46o-0000aS-7l for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:14 -0400 Received: from mail-pl0-x236.google.com ([2607:f8b0:400e:c01::236]:34424) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI46o-0000Zr-04 for qemu-devel@nongnu.org; Sun, 13 May 2018 23:25:10 -0400 Received: by mail-pl0-x236.google.com with SMTP id ay10-v6so6509678plb.1 for ; Sun, 13 May 2018 20:25:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=yWFrcXEFvFX0MP9vrdrR6VQWCgO9lRgtlgnt3l4KLCs=; b=dlexpN5YeXldTTFHvkTSE2YOTQ/0NoPwsS7q/02qzgG0OLxQl34xQltKCO1OSN96xr QwgnebvupASEJyBhmaEXd4J8NLb11BRP/AaE/w3ZXtv5Icidkmlua9V/fdh1L4kVgIf5 HhJlPMN73SPdzLTtZfSwaWetXqkH1nRuILXQJseATUBr5xmfcf+qoMK8e7Uh5FcmyUKE HEoCu1h/FUgmkmTaTUO97kksSsXQFQ01w4UKH68gOoLM35nCrUrQJk8K3cyJV5kSNuTK aOBuxH7q3DjcQNE7OTlotm1A5aSGteK3hpczL4pqw95YhKPLeoP2TT8a8jgISkShOha4 Om4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=yWFrcXEFvFX0MP9vrdrR6VQWCgO9lRgtlgnt3l4KLCs=; b=THLwzANcdNCZGYZLilIBO4Xr7ek80v82eY6bbwDxOVR85D3ifpqO2DzwkREjJjHSNN CTNjxPPIHw+qfsY5Dvw8hY71E4X8X2kPrl971EpmllpFnZGbwMIBEAKTxKyT5mAIr52I MlAe6zRywMF61I97TyJIJi0KvcZZrgSydY4MUf/NXFvlppkElwPi1jvV2KkcOYzW7Umf W3nKG+YeYOTgAyBtxk/Ue0Bff+I2hAzr93ZO72odbz81oCYOEcOXgXfUl409foNnKFOj 6t2fHmWw7nhSmTPHZ+fmqo+CSAJ4VbuXIEaQ+WuatjGTWQdxaSteDixo/b+a4NoC90fV lkaQ== X-Gm-Message-State: ALKqPwcz+6btZuPSKGiptu8kCuv/ufurfJb3LosI3PQ+WVNjjya0nibX RgLKnYDVW6wPA9XyWIu9W0F5vQ== X-Google-Smtp-Source: AB8JxZoAdEyt16cM+RkVes9eWYci0eJXFPgRK9fAwUhrfSozpI7KHhuTM6RCY0it8+iPbGrcBiNpeA== X-Received: by 2002:a17:902:6041:: with SMTP id a1-v6mr7944117plt.59.1526268308701; Sun, 13 May 2018 20:25:08 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.25.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:25:07 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:43 +0800 Message-Id: <1526268228-27951-13-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::236 Subject: [Qemu-devel] [PATCH V7 12/17] savevm: split the process of different stages for loadvm/savevm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang There are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections. We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state. So we add three new helper functions: qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration. Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 36 ++++++++++++++++++++++++++++-------- migration/savevm.c | 35 ++++++++++++++++++++++++++++------- migration/savevm.h | 4 ++++ 3 files changed, 60 insertions(+), 15 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 314344c..7869d55 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -30,6 +30,7 @@ #include "block/block.h" #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" +#include "sysemu/cpus.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -414,24 +415,31 @@ static int colo_do_checkpoint_transaction(MigrationState *s, /* Disable block migration */ migrate_set_block_enabled(false, &local_err); - qemu_savevm_state_header(fb); - qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); replication_do_checkpoint_all(&local_err); if (local_err) { qemu_mutex_unlock_iothread(); goto out; } - qemu_savevm_state_complete_precopy(fb, false, false); - qemu_mutex_unlock_iothread(); - - qemu_fflush(fb); colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_err); if (local_err) { goto out; } /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + /* Note: device state is saved into buffer */ + ret = qemu_save_device_state(fb); + + qemu_mutex_unlock_iothread(); + + qemu_fflush(fb); + + /* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read. */ @@ -643,6 +651,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t total_size; uint64_t value; Error *local_err = NULL; + int ret; qemu_sem_init(&mis->colo_incoming_sem, 0); @@ -715,6 +724,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + qemu_mutex_lock_iothread(); + cpu_synchronize_all_pre_loadvm(); + ret = qemu_loadvm_state_main(mis->from_src_file, mis); + qemu_mutex_unlock_iothread(); + + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } + value = colo_receive_message_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); if (local_err) { @@ -748,8 +767,9 @@ void *colo_process_incoming_thread(void *opaque) qemu_mutex_lock_iothread(); qemu_system_reset(SHUTDOWN_CAUSE_NONE); vmstate_loading = true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + ret = qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } diff --git a/migration/savevm.c b/migration/savevm.c index ec0bff0..0f61239 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1332,13 +1332,20 @@ done: return ret; } -static int qemu_save_device_state(QEMUFile *f) +void qemu_savevm_live_state(QEMUFile *f) { - SaveStateEntry *se; + /* save QEMU_VM_SECTION_END section */ + qemu_savevm_state_complete_precopy(f, true, false); + qemu_put_byte(f, QEMU_VM_EOF); +} - qemu_put_be32(f, QEMU_VM_FILE_MAGIC); - qemu_put_be32(f, QEMU_VM_FILE_VERSION); +int qemu_save_device_state(QEMUFile *f) +{ + SaveStateEntry *se; + if (!migration_in_colo_state()) { + qemu_savevm_state_header(f); + } cpu_synchronize_all_states(); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1394,8 +1401,6 @@ enum LoadVMExitCodes { LOADVM_QUIT = 1, }; -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); - /* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything @@ -2075,7 +2080,7 @@ void qemu_loadvm_state_cleanup(void) } } -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) { uint8_t section_type; int ret = 0; @@ -2229,6 +2234,22 @@ int qemu_loadvm_state(QEMUFile *f) return ret; } +int qemu_load_device_state(QEMUFile *f) +{ + MigrationIncomingState *mis = migration_incoming_get_current(); + int ret; + + /* Load QEMU_VM_SECTION_FULL section */ + ret = qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to load device state: %d", ret); + return ret; + } + + cpu_synchronize_all_post_init(); + return 0; +} + int save_snapshot(const char *name, Error **errp) { BlockDriverState *bs, *bs1; diff --git a/migration/savevm.h b/migration/savevm.h index c6d46b3..cf7935d 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -53,8 +53,12 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint64_t *start_list, uint64_t *length_list); void qemu_savevm_send_colo_enable(QEMUFile *f); +void qemu_savevm_live_state(QEMUFile *f); +int qemu_save_device_state(QEMUFile *f); int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); +int qemu_load_device_state(QEMUFile *f); #endif