From patchwork Thu Jan 4 06:01:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10143899 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8B8B86034B for ; Thu, 4 Jan 2018 06:09:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D95028417 for ; Thu, 4 Jan 2018 06:09:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7141828419; Thu, 4 Jan 2018 06:09:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id DA8EC28417 for ; Thu, 4 Jan 2018 06:09:34 +0000 (UTC) Received: from localhost ([::1]:47486 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWyib-0002IF-Nk for patchwork-qemu-devel@patchwork.kernel.org; Thu, 04 Jan 2018 01:09:33 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36949) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybW-0003nC-Kz for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:15 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybV-0004GV-E7 for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:14 -0500 Received: from mail-pf0-x230.google.com ([2607:f8b0:400e:c00::230]:45414) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybV-0004G1-8x for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:13 -0500 Received: by mail-pf0-x230.google.com with SMTP id u19so365592pfa.12 for ; Wed, 03 Jan 2018 22:02:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=o9YWNPx3CTT5V9kJXFl72B413c+NQAN7NhavooxbAVM=; b=ZVTidTRF/i6WHxmN/S2+LAlzpUMljHTipM97xSQqF/FJRO5i7i8Ii7udTI5lLnuaXs 0sgUhK4mdlpPbQsMjVWIC1zwu3xDFSJNmdJO1gS6lpJ2xHMKm9GwBuJn9ZS9PcURN6JQ 9DCBU95yY8P76+h0yOrgiBms6UgJRv7zg5siMQFWnLttvn2IOMM2LYctVUT9mjwbwN5I F8wsCMmpIt5Q1xUk1PBKALOP1N5kJneBr0aD3r8ivUW7RMVheEvzC6UBWBrnNYF895Yz XWaz4qewSYNOO3sgMJvOU+clVn7b118K1GW3mOrEveVSM3bItJLB31iemRFfOFcIZhVj Ql6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=o9YWNPx3CTT5V9kJXFl72B413c+NQAN7NhavooxbAVM=; b=cGFZHGAzrsPV90ruPlJEQMQ/iGzQHqywukxuhrlMIzZXmoTwuh1YyynlzotiTeuLEJ MCIDq5Jsc1b9T8GAXkRm8ZivcXR13PMUqM6+U5FZsTLq2H0GveHGLpLJV7CEo0doNiEa IcNm+r6Q48sqVTmlDwZ96qU2GPoruuaEggcUXIYLrEJNgBNW+6C/2tEix697kRAoVfvz 3uygGj8ZqSRQQM6fGtSNY2uAVojTi/sK37KHTZ7AeuBrh/T51UkKqgtC3Y1frMWsotv/ QZsJygESB7pyBPFbFk9hmPmiOsiH32p3DvNCvY2L1+5vVDTzJyEZ9R/ue+lOOJgVl33A tr1w== X-Gm-Message-State: AKGB3mJ8xv223KRU2eYdMpcbYvVAG6NfYyjLfjKoWBBMKhvU2hebuwd0 UnUR2bkJbiQ9W5qK+m3NWKNqtcoQ X-Google-Smtp-Source: ACJfBovPwaVW556LZoEQaeD14hc8mtwXczhANc3qy8WQqnJDwzCQPbdGnoM2Pu00ouw8hioQpDEg0g== X-Received: by 10.101.101.65 with SMTP id a1mr3155682pgw.289.1515045731915; Wed, 03 Jan 2018 22:02:11 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.02.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:02:11 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:08 +0800 Message-Id: <1515045675-6993-10-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::230 Subject: [Qemu-devel] [PATCH RESEND V3 09/16] COLO: Flush memory data from ram cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint time. So, the content of SVM's RAM cache will always be same with PVM's memory after checkpoint. Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way: Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's. Besides, we must ensure flush RAM cache before load device state. Cc: Juan Quintela Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 39 +++++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 2 files changed, 41 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 388333d..23c67e0 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2908,6 +2908,40 @@ static bool postcopy_is_running(void) return ps >= POSTCOPY_INCOMING_LISTENING && ps < POSTCOPY_INCOMING_END; } +/* + * Flush content of RAM cache into SVM's memory. + * Only flush the pages that be dirtied by PVM or SVM or both. + */ +static void colo_flush_ram_cache(void) +{ + RAMBlock *block = NULL; + void *dst_host; + void *src_host; + unsigned long offset = 0; + + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); + rcu_read_lock(); + block = QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset = migration_bitmap_find_dirty(ram_state, block, offset); + migration_bitmap_clear_dirty(ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >= block->used_length) { + offset = 0; + block = QLIST_NEXT_RCU(block, next); + } else { + dst_host = block->host + (offset << TARGET_PAGE_BITS); + src_host = block->colo_cache + (offset << TARGET_PAGE_BITS); + memcpy(dst_host, src_host, TARGET_PAGE_SIZE); + } + } + + rcu_read_unlock(); + trace_colo_flush_ram_cache_end(); + assert(ram_state->migration_dirty_pages == 0); +} + static int ram_load(QEMUFile *f, void *opaque, int version_id) { int flags = 0, ret = 0, invalid_flags = 0; @@ -2920,6 +2954,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) bool postcopy_running = postcopy_is_running(); /* ADVISE is earlier, it shows the source has the postcopy capability on */ bool postcopy_advised = postcopy_is_advised(); + bool need_flush = false; seq_iter++; @@ -3095,6 +3130,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) wait_for_decompress_done(); rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); + + if (!ret && migration_incoming_in_colo_state() && need_flush) { + colo_flush_ram_cache(); + } return ret; } diff --git a/migration/trace-events b/migration/trace-events index d4738c8..69261a0 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -78,6 +78,8 @@ ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x" ram_postcopy_send_discard_bitmap(void) "" ram_save_page(const char *rbname, uint64_t offset, void *host) "%s: offset: 0x%" PRIx64 " host: %p" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: start: 0x%zx len: 0x%zx" +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" # migration/migration.c await_return_path_close_on_source_close(void) ""