From patchwork Sat Aug 11 20:59:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563559 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CBA6B13B4 for ; Sat, 11 Aug 2018 21:08:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BE6AD298D9 for ; Sat, 11 Aug 2018 21:08:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B1DBF29915; Sat, 11 Aug 2018 21:08:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 07453298D9 for ; Sat, 11 Aug 2018 21:08:24 +0000 (UTC) Received: from localhost ([::1]:33212 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob7X-0003H5-7k for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:08:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48250) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob0E-0005Ih-18 for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fob0C-0001VH-Pf for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:50 -0400 Received: from mail-pf1-x42e.google.com ([2607:f8b0:4864:20::42e]:46993) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fob0C-0001Uo-HP for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:48 -0400 Received: by mail-pf1-x42e.google.com with SMTP id u24-v6so5988319pfn.13 for ; Sat, 11 Aug 2018 14:00:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9X9qXjiTTLJdVwYXtKH6lkWk/cJysqmZIj5m9WkoURo=; b=Z8hyvupWIP0tvr+KzfFlQ61E3MdhE+jpBXkYKya2NCALY80WQfTlYbE3qeUei3wIa7 nf5NjjA0PLtX50bmSOAGSE7eLP8Y8MQKEnl7tCn62Scq9UMURfeCqdl/F9GW1kiMxMik 3LZo3kdulsxVTpIyw39Dcz+kmoEMyi8xCPrOFItruYd6kIdVmfRXvRE8ET2+XsEo97Q7 8ObnmpMWRmfLM5Lzflfmg1zpu33NKZhPAxoCUsojChGDvYMPpYoJNuOE9XfWzh9C1BUb dR1Qr9a8/PO0f3dL+c0K0PqTW+cpl+Z5ImYDKAqCupLRBxUQU3wp53Nz+JWi8rp82qqv vMbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9X9qXjiTTLJdVwYXtKH6lkWk/cJysqmZIj5m9WkoURo=; b=YQcl0lfrcF/NKjjgUvLcmXDacqQ3jw3SdzpSs8roYxNQIyebCbGLg6qn8LvWfLAd9Q IpS3gmNr+z75mJLSh/Q4I7VkzVlFXXXf4oRhh7Q93wIxmitXHQ5bDdG5HVR8Ui3EqpOW 7V/uYoc3aYgXbgkSlEKz/yECknY+GDsG1fXafr2RYGio0fsynrYw77IQVBeFVKQnyZ7W O9ITB3C+zsBvhTD0XLrBYaqnoe0svzvc1eKL9IKKmNkzuFnWWl9cAp0gJy7xHvxx2vxp bXvgJtrYmVoFU/ClODAlNCGm1o/vkwCXhUv6brqR6Iz7lKpdFPHqLSfKeLpLO7nFkysB wWIA== X-Gm-Message-State: AOUpUlGTYQYI3B6pXxAgPJ/CBDQVQ8DGc978HWNezibu8QBwXJgC0DSL PeORrbtSyoj1gsggNfQCqNdjwnNw X-Google-Smtp-Source: AA+uWPzyv9SDimHhL6paQg63ftmprAFz8yl0WU/d0NcPWGf5HXF8V6NZmK0F7RKpVKRWq6iwy++CwQ== X-Received: by 2002:a63:704f:: with SMTP id a15-v6mr11342789pgn.443.1534021247258; Sat, 11 Aug 2018 14:00:47 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.00.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:00:46 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:17 +0800 Message-Id: <20180811205924.4113-14-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42e Subject: [Qemu-devel] [PATCH V11 13/20] savevm: split the process of different stages for loadvm/savevm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP There are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections. We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state. So we add three new helper functions: qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration. Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 41 ++++++++++++++++++++++++++++++++--------- migration/savevm.c | 36 +++++++++++++++++++++++++++++------- migration/savevm.h | 4 ++++ 3 files changed, 65 insertions(+), 16 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index b355de3f01..688d6f40b2 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -30,6 +30,7 @@ #include "block/block.h" #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" +#include "sysemu/cpus.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -401,23 +402,34 @@ static int colo_do_checkpoint_transaction(MigrationState *s, /* Disable block migration */ migrate_set_block_enabled(false, &local_err); - qemu_savevm_state_header(fb); - qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); replication_do_checkpoint_all(&local_err); if (local_err) { qemu_mutex_unlock_iothread(); goto out; } - qemu_savevm_state_complete_precopy(fb, false, false); - qemu_mutex_unlock_iothread(); - - qemu_fflush(fb); colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_err); if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* Note: device state is saved into buffer */ + ret = qemu_save_device_state(fb); + + qemu_mutex_unlock_iothread(); + if (ret < 0) { goto out; } + /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + + qemu_fflush(fb); + /* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read. @@ -635,6 +647,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t total_size; uint64_t value; Error *local_err = NULL; + int ret; qemu_sem_init(&mis->colo_incoming_sem, 0); @@ -707,6 +720,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + qemu_mutex_lock_iothread(); + cpu_synchronize_all_pre_loadvm(); + ret = qemu_loadvm_state_main(mis->from_src_file, mis); + qemu_mutex_unlock_iothread(); + + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } + value = colo_receive_message_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); if (local_err) { @@ -738,10 +761,10 @@ void *colo_process_incoming_thread(void *opaque) } qemu_mutex_lock_iothread(); - qemu_system_reset(SHUTDOWN_CAUSE_NONE); vmstate_loading = true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + ret = qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } diff --git a/migration/savevm.c b/migration/savevm.c index 33e9e7cda0..3a6aa747f9 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1378,13 +1378,21 @@ done: return ret; } -static int qemu_save_device_state(QEMUFile *f) +void qemu_savevm_live_state(QEMUFile *f) { - SaveStateEntry *se; + /* save QEMU_VM_SECTION_END section */ + qemu_savevm_state_complete_precopy(f, true, false); + qemu_put_byte(f, QEMU_VM_EOF); +} - qemu_put_be32(f, QEMU_VM_FILE_MAGIC); - qemu_put_be32(f, QEMU_VM_FILE_VERSION); +int qemu_save_device_state(QEMUFile *f) +{ + SaveStateEntry *se; + if (!migration_in_colo_state()) { + qemu_put_be32(f, QEMU_VM_FILE_MAGIC); + qemu_put_be32(f, QEMU_VM_FILE_VERSION); + } cpu_synchronize_all_states(); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1440,8 +1448,6 @@ enum LoadVMExitCodes { LOADVM_QUIT = 1, }; -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); - /* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything @@ -2241,7 +2247,7 @@ static bool postcopy_pause_incoming(MigrationIncomingState *mis) return true; } -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) { uint8_t section_type; int ret = 0; @@ -2410,6 +2416,22 @@ int qemu_loadvm_state(QEMUFile *f) return ret; } +int qemu_load_device_state(QEMUFile *f) +{ + MigrationIncomingState *mis = migration_incoming_get_current(); + int ret; + + /* Load QEMU_VM_SECTION_FULL section */ + ret = qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to load device state: %d", ret); + return ret; + } + + cpu_synchronize_all_post_init(); + return 0; +} + int save_snapshot(const char *name, Error **errp) { BlockDriverState *bs, *bs1; diff --git a/migration/savevm.h b/migration/savevm.h index 8373c2f6bd..51a4b9caa8 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -56,8 +56,12 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint64_t *start_list, uint64_t *length_list); void qemu_savevm_send_colo_enable(QEMUFile *f); +void qemu_savevm_live_state(QEMUFile *f); +int qemu_save_device_state(QEMUFile *f); int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); +int qemu_load_device_state(QEMUFile *f); #endif