From patchwork Mon May 14 03:23:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10396751 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 5B37D60536 for ; Mon, 14 May 2018 05:54:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4C0DC290A8 for ; Mon, 14 May 2018 05:54:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 40A56290AA; Mon, 14 May 2018 05:54:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9169C290A8 for ; Mon, 14 May 2018 05:54:43 +0000 (UTC) Received: from localhost ([::1]:35678 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6RW-0007op-Hm for patchwork-qemu-devel@patchwork.kernel.org; Mon, 14 May 2018 01:54:42 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43336) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fI6Nr-00005o-7V for qemu-devel@nongnu.org; Mon, 14 May 2018 01:50:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fI46Y-0000GE-UI for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:56 -0400 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:45466) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fI46Y-0000EU-NH for qemu-devel@nongnu.org; Sun, 13 May 2018 23:24:54 -0400 Received: by mail-pf0-x241.google.com with SMTP id c10-v6so5305257pfi.12 for ; Sun, 13 May 2018 20:24:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=g819tnk+8Qd28uIgGno/qnvKAxrfmnG7hWA5eEBBsqQ=; b=IWd/XJobKRbRQS8cc3Ocl93cUeHRLdQligV0vdDX8agDaPL90GwCochTGCXaK5NC/d hMnWhHHKye9Y0h46pGDViwDkh8Bu2Bf8uzua4MJttLumpt8bJeEnCKi/Ks6qZl2MH7S1 D/0qT05SanrChgpzmnW7UkaLMMyBv25ZgPYuwp3ldsrmDSyu+ZPRxc6duzmmHNR8GeiO GFXoXw/0wpJjUZcgzEYePrJn8nUl7YZHCFYvfrA+87ZgFmryyrH+jHtB/D9ZXHD7yXBh 8GqhD61msGLEoJyuyJaSUfOOpmq4rqKmHA9dC7VPtm8GA0EtGMNss494cLZ1EJZkH89o wblw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=g819tnk+8Qd28uIgGno/qnvKAxrfmnG7hWA5eEBBsqQ=; b=nxm4/v1Z1xABOmCKkfs4yxoThQrdh1K7Mc68PU+U1uB8QLdFml25gvnLsQMQyrM6W8 sU91ZMIPsiQPf3xDDaWzgnUdha3cq0HaJ/fLtlcaSJ4oZLdEtdOrvAFuwmtMp55wve6d fQrz3W5RvWN3rNJlER6XSuMtGZp6YeuTU0amor1Y2ApiRVuNywjl8nWOE2s6HeLHAL8R gwJVksnyhoTgLK7oGebJa/sJDrvUlys7NrEBiE8/htJILzcQHkyV4iKTFCoB/uD+O30K rtAWBe4gh6atRCyO72RC8e0TDHiIz6HUybJujE2lChK4AmFxDxx7TnPbZOJOvczIAvXq pe7A== X-Gm-Message-State: ALKqPwcittSxjbkl99ToupbktQgKLqh/Rxe/StcUw5aKZ0E+LRX0mAWS ebuiHXxjADz9m7J5PJzVE/PGIw== X-Google-Smtp-Source: AB8JxZp2Tl9B4xljrm8rZkS07kWpKVEFjioBIYbS9Jkht/nQYH1ClKsdMFg3CZxLqAIiIMUVRckNQQ== X-Received: by 2002:a65:424d:: with SMTP id d13-v6mr5780818pgq.234.1526268293489; Sun, 13 May 2018 20:24:53 -0700 (PDT) Received: from localhost.localdomain (109.227.199.104.bc.googleusercontent.com. [104.199.227.109]) by smtp.gmail.com with ESMTPSA id p6-v6sm17183532pfg.157.2018.05.13.20.24.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 13 May 2018 20:24:52 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org Date: Mon, 14 May 2018 11:23:40 +0800 Message-Id: <1526268228-27951-10-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> References: <1526268228-27951-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH V7 09/17] COLO: Flush memory data from ram cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Jason Wang , Markus Armbruster , "Dr . David Alan Gilbert" , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint time. So, the content of SVM's RAM cache will always be same with PVM's memory after checkpoint. Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way: Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's. Besides, we must ensure flush RAM cache before load device state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 39 +++++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 2 files changed, 41 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index e35dfee..4235a8f 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3031,6 +3031,40 @@ static bool postcopy_is_running(void) return ps >= POSTCOPY_INCOMING_LISTENING && ps < POSTCOPY_INCOMING_END; } +/* + * Flush content of RAM cache into SVM's memory. + * Only flush the pages that be dirtied by PVM or SVM or both. + */ +static void colo_flush_ram_cache(void) +{ + RAMBlock *block = NULL; + void *dst_host; + void *src_host; + unsigned long offset = 0; + + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); + rcu_read_lock(); + block = QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset = migration_bitmap_find_dirty(ram_state, block, offset); + migration_bitmap_clear_dirty(ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >= block->used_length) { + offset = 0; + block = QLIST_NEXT_RCU(block, next); + } else { + dst_host = block->host + (offset << TARGET_PAGE_BITS); + src_host = block->colo_cache + (offset << TARGET_PAGE_BITS); + memcpy(dst_host, src_host, TARGET_PAGE_SIZE); + } + } + + rcu_read_unlock(); + trace_colo_flush_ram_cache_end(); + assert(ram_state->migration_dirty_pages == 0); +} + static int ram_load(QEMUFile *f, void *opaque, int version_id) { int flags = 0, ret = 0, invalid_flags = 0; @@ -3043,6 +3077,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) bool postcopy_running = postcopy_is_running(); /* ADVISE is earlier, it shows the source has the postcopy capability on */ bool postcopy_advised = postcopy_is_advised(); + bool need_flush = false; seq_iter++; @@ -3218,6 +3253,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) ret |= wait_for_decompress_done(); rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); + + if (!ret && migration_incoming_in_colo_state() && need_flush) { + colo_flush_ram_cache(); + } return ret; } diff --git a/migration/trace-events b/migration/trace-events index 9295b4c..8e2f974 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -78,6 +78,8 @@ ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x" ram_postcopy_send_discard_bitmap(void) "" ram_save_page(const char *rbname, uint64_t offset, void *host) "%s: offset: 0x%" PRIx64 " host: %p" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: start: 0x%zx len: 0x%zx" +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" # migration/migration.c await_return_path_close_on_source_close(void) ""