From patchwork Sun Mar 11 07:45:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10274551 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 82E8A60211 for ; Sun, 11 Mar 2018 07:57:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 75FF728DE5 for ; Sun, 11 Mar 2018 07:57:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6891928DEE; Sun, 11 Mar 2018 07:57:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A3D7D28DE5 for ; Sun, 11 Mar 2018 07:57:10 +0000 (UTC) Received: from localhost ([::1]:53399 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvqv-0006eM-T5 for patchwork-qemu-devel@patchwork.kernel.org; Sun, 11 Mar 2018 03:57:09 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59325) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvgW-0006ay-FS for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvgV-0000vP-5y for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:24 -0400 Received: from mail-pl0-x244.google.com ([2607:f8b0:400e:c01::244]:37327) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvgU-0000uy-TE for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:23 -0400 Received: by mail-pl0-x244.google.com with SMTP id w12-v6so7606761plp.4 for ; Sat, 10 Mar 2018 23:46:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3ulfaou2re/01f94Yo2+jBdgKNZg8UcfXzE4VLleK8s=; b=FPwrXeM5Q7TdqjsZUH7rslfHC+alj13ZaP6uzI7VvHXN96zOKnYEK9RxbmpgFhhy11 v9ZSQb1ROn5D1OfpK7EPdWJa2gJr7gh4P0KhKQ7Ig13xuRByLcfFXFQi7YkeuC7n3/RE LC9QwKJIH5+8hfFEy8D5rmhpdryVDssHoij4wQ2+gUKRjoS53lK3ZPbhoKdWp52K4Prq 3ebQk02H47r33VFTWCKXeQdeB4+d/CdlfS5JVZi8ZyLz9brfn8uZW16ersdG6Rs1pC3g 0UBaE1Juw8dTf40xuiUFy4esHCVvuK+Pmbc/0DXxYxWDVvTl1pibjnEt/y5Yc+zoTRqN TPWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3ulfaou2re/01f94Yo2+jBdgKNZg8UcfXzE4VLleK8s=; b=sGwBWOIGhqccBxNvq3V+xgC2uogJi31wPsmOZ8/Ss7Gh2rJGuzJhUrGzsg4DsvOx8s X7d0IgJSjoPf9WSteu1O1R9QdI0mOwBWjQX2HBLEA2iTaa/ANkmBB53FItvyE7FADlI8 boXPYdtst10gAV5kA/xt1D/Z1Iy3gayVLqDKRnTbjYeWCzrJJIbARkGGyo3BvamjWnHC Rj4xC7OJBtmco2UnOhwyeBaHIHKeWumcgzL2MvrYyd3M3XoZYZnTLNChybqh6J2uvtSe b3KcNz9Eo4SS47c/rw1F1UZkVBwa9+GstFYjVrhfz+Of3rTlRw0R/LWJ15WAIZuRLe4O qBLg== X-Gm-Message-State: AElRT7FoQBKv3hUnLHTfUExETSNpY+3OKoWu61EqrmB18eMAqYnv6PTA wtOcz8TfQsmmGraZDfz6xHF5pEMW X-Google-Smtp-Source: AG47ELu9pq3rT+ZIwEiMAsrFFutDu1i6hHx6Cyy2U9HlmUIDy/x7SmoZDTZxTC96eu06i3p9BZ3huA== X-Received: by 2002:a17:902:8487:: with SMTP id c7-v6mr4349853plo.143.1520754381525; Sat, 10 Mar 2018 23:46:21 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.46.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:46:20 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:09 +0800 Message-Id: <1520754314-5969-14-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::244 Subject: [Qemu-devel] [PATCH V6 12/17] savevm: split the process of different stages for loadvm/savevm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang There are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections. We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state. So we add three new helper functions: qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration. Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 36 ++++++++++++++++++++++++++++-------- migration/savevm.c | 35 ++++++++++++++++++++++++++++------- migration/savevm.h | 4 ++++ 3 files changed, 60 insertions(+), 15 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 127db3c..f514492 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -29,6 +29,7 @@ #include "net/colo.h" #include "block/block.h" #include "qapi/qapi-events-migration.h" +#include "sysemu/cpus.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -411,24 +412,31 @@ static int colo_do_checkpoint_transaction(MigrationState *s, /* Disable block migration */ migrate_set_block_enabled(false, &local_err); - qemu_savevm_state_header(fb); - qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); replication_do_checkpoint_all(&local_err); if (local_err) { qemu_mutex_unlock_iothread(); goto out; } - qemu_savevm_state_complete_precopy(fb, false, false); - qemu_mutex_unlock_iothread(); - - qemu_fflush(fb); colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_err); if (local_err) { goto out; } /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + /* Note: device state is saved into buffer */ + ret = qemu_save_device_state(fb); + + qemu_mutex_unlock_iothread(); + + qemu_fflush(fb); + + /* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read. */ @@ -640,6 +648,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t total_size; uint64_t value; Error *local_err = NULL; + int ret; qemu_sem_init(&mis->colo_incoming_sem, 0); @@ -712,6 +721,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + qemu_mutex_lock_iothread(); + cpu_synchronize_all_pre_loadvm(); + ret = qemu_loadvm_state_main(mis->from_src_file, mis); + qemu_mutex_unlock_iothread(); + + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } + value = colo_receive_message_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); if (local_err) { @@ -745,8 +764,9 @@ void *colo_process_incoming_thread(void *opaque) qemu_mutex_lock_iothread(); qemu_system_reset(SHUTDOWN_CAUSE_NONE); vmstate_loading = true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + ret = qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } diff --git a/migration/savevm.c b/migration/savevm.c index 0d1f593..e347f73 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1322,13 +1322,20 @@ done: return ret; } -static int qemu_save_device_state(QEMUFile *f) +void qemu_savevm_live_state(QEMUFile *f) { - SaveStateEntry *se; + /* save QEMU_VM_SECTION_END section */ + qemu_savevm_state_complete_precopy(f, true, false); + qemu_put_byte(f, QEMU_VM_EOF); +} - qemu_put_be32(f, QEMU_VM_FILE_MAGIC); - qemu_put_be32(f, QEMU_VM_FILE_VERSION); +int qemu_save_device_state(QEMUFile *f) +{ + SaveStateEntry *se; + if (!migration_in_colo_state()) { + qemu_savevm_state_header(f); + } cpu_synchronize_all_states(); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1384,8 +1391,6 @@ enum LoadVMExitCodes { LOADVM_QUIT = 1, }; -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); - /* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything @@ -2050,7 +2055,7 @@ void qemu_loadvm_state_cleanup(void) } } -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) { uint8_t section_type; int ret = 0; @@ -2204,6 +2209,22 @@ int qemu_loadvm_state(QEMUFile *f) return ret; } +int qemu_load_device_state(QEMUFile *f) +{ + MigrationIncomingState *mis = migration_incoming_get_current(); + int ret; + + /* Load QEMU_VM_SECTION_FULL section */ + ret = qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to load device state: %d", ret); + return ret; + } + + cpu_synchronize_all_post_init(); + return 0; +} + int save_snapshot(const char *name, Error **errp) { BlockDriverState *bs, *bs1; diff --git a/migration/savevm.h b/migration/savevm.h index 041d23c..8d463fd 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -52,8 +52,12 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint64_t *start_list, uint64_t *length_list); void qemu_savevm_send_colo_enable(QEMUFile *f); +void qemu_savevm_live_state(QEMUFile *f); +int qemu_save_device_state(QEMUFile *f); int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); +int qemu_load_device_state(QEMUFile *f); #endif