From patchwork Mon Feb 22 02:52:25 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Congyang X-Patchwork-Id: 8370951 Return-Path: X-Original-To: patchwork-xen-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id AEE2CC0553 for ; Mon, 22 Feb 2016 02:54:56 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id B938120450 for ; Mon, 22 Feb 2016 02:54:53 +0000 (UTC) Received: from lists.xen.org (lists.xenproject.org [50.57.142.19]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8540020412 for ; Mon, 22 Feb 2016 02:54:52 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1aXgbr-0001Wv-Ic; Mon, 22 Feb 2016 02:52:27 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1aXgbk-0001HW-SK for xen-devel@lists.xen.org; Mon, 22 Feb 2016 02:52:21 +0000 Received: from [85.158.139.211] by server-17.bemta-5.messagelabs.com id 97/32-03800-4E77AC65; Mon, 22 Feb 2016 02:52:20 +0000 X-Env-Sender: wency@cn.fujitsu.com X-Msg-Ref: server-12.tower-206.messagelabs.com!1456109535!23931305!2 X-Originating-IP: [59.151.112.132] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 7.35.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 8437 invoked from network); 22 Feb 2016 02:52:19 -0000 Received: from cn.fujitsu.com (HELO heian.cn.fujitsu.com) (59.151.112.132) by server-12.tower-206.messagelabs.com with SMTP; 22 Feb 2016 02:52:19 -0000 X-IronPort-AV: E=Sophos;i="5.20,346,1444665600"; d="scan'208";a="3766071" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 22 Feb 2016 10:52:14 +0800 Received: from G08CNEXCHPEKD01.g08.fujitsu.local (unknown [10.167.33.80]) by cn.fujitsu.com (Postfix) with ESMTP id 91AC34219196; Mon, 22 Feb 2016 10:50:17 +0800 (CST) Received: from G08FNSTD140052.g08.fujitsu.local (10.167.226.52) by G08CNEXCHPEKD01.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP Server (TLS) id 14.3.181.6; Mon, 22 Feb 2016 10:52:08 +0800 From: Wen Congyang To: xen devel , Konrad Rzeszutek Wilk , Andrew Cooper , Ian Campbell , Ian Jackson , Wei Liu Date: Mon, 22 Feb 2016 10:52:25 +0800 Message-ID: <1456109555-28299-22-git-send-email-wency@cn.fujitsu.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1456109555-28299-1-git-send-email-wency@cn.fujitsu.com> References: <1456109555-28299-1-git-send-email-wency@cn.fujitsu.com> MIME-Version: 1.0 X-Originating-IP: [10.167.226.52] X-yoursite-MailScanner-ID: 91AC34219196.A6194 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: wency@cn.fujitsu.com X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Cc: Lars Kurth , Changlong Xie , Wen Congyang , Gui Jianfeng , Jiang Yunhong , Dong Eddie , Shriram Rajagopalan , Yang Hongyang Subject: [Xen-devel] [PATCH v10 21/31] libxc/save: support COLO save X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP After suspend primary vm, get dirty bitmap on secondary vm, and send pages both dirty on primary/secondary to secondary. Signed-off-by: Yang Hongyang Signed-off-by: Wen Congyang CC: Andrew Cooper --- tools/libxc/xc_sr_common.h | 2 + tools/libxc/xc_sr_save.c | 104 ++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 101 insertions(+), 5 deletions(-) diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h index 29ab4eb..5301746 100644 --- a/tools/libxc/xc_sr_common.h +++ b/tools/libxc/xc_sr_common.h @@ -196,6 +196,8 @@ struct xc_sr_context { struct /* Save data. */ { + int recv_fd; + struct xc_sr_save_ops ops; struct save_callbacks *callbacks; diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c index 7393355..89a5eaf 100644 --- a/tools/libxc/xc_sr_save.c +++ b/tools/libxc/xc_sr_save.c @@ -517,6 +517,58 @@ static int send_memory_live(struct xc_sr_context *ctx) return rc; } +static int merge_secondary_dirty_bitmap(struct xc_sr_context *ctx) +{ + xc_interface *xch = ctx->xch; + struct xc_sr_record rec; + uint64_t *pfns = NULL; + uint64_t pfn; + unsigned count, i; + int rc; + DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap, + &ctx->save.dirty_bitmap_hbuf); + + rc = read_record(ctx, ctx->save.recv_fd, &rec); + if ( rc ) + goto err; + + if ( rec.type != REC_TYPE_CHECKPOINT_DIRTY_PFN_LIST ) + { + PERROR("Expect dirty bitmap record, but received %u", rec.type ); + rc = -1; + goto err; + } + + if ( rec.length % sizeof(*pfns) ) + { + PERROR("Invalid dirty pfn list record length %u", rec.length ); + rc = -1; + goto err; + } + + count = rec.length / sizeof(*pfns); + pfns = rec.data; + + for ( i = 0; i < count; i++ ) + { + pfn = pfns[i]; + if (pfn > ctx->save.p2m_size) + { + PERROR("Invalid pfn %#lx", pfn ); + rc = -1; + goto err; + } + + set_bit(pfn, dirty_bitmap); + } + + rc = 0; + + err: + free(rec.data); + return rc; +} + /* * Suspend the domain and send dirty memory. * This is the last iteration of the live migration and the @@ -558,6 +610,16 @@ static int suspend_and_send_dirty(struct xc_sr_context *ctx) bitmap_or(dirty_bitmap, ctx->save.deferred_pages, ctx->save.p2m_size); + if ( !ctx->save.live && ctx->save.checkpointed == MIG_STREAM_COLO ) + { + rc = merge_secondary_dirty_bitmap(ctx); + if ( rc ) + { + PERROR("Failed to get secondary vm's dirty pages"); + goto out; + } + } + rc = send_dirty_pages(ctx, stats.dirty_count + ctx->save.nr_deferred_pages); if ( rc ) goto out; @@ -791,13 +853,42 @@ static int save(struct xc_sr_context *ctx, uint16_t guest_type) if ( rc ) goto err; - rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data); - if ( rc <= 0 ) - goto err; + if ( ctx->save.checkpointed == MIG_STREAM_COLO ) + { + rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data); + if ( !rc ) + { + rc = -1; + goto err; + } + } - rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data); - if ( rc <= 0 ) + rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data); + if ( !rc ) + { + rc = -1; goto err; + } + + if ( ctx->save.checkpointed == MIG_STREAM_COLO ) + { + rc = ctx->save.callbacks->wait_checkpoint( + ctx->save.callbacks->data); + if ( rc <= 0 ) + goto err; + } + else if ( ctx->save.checkpointed == MIG_STREAM_REMUS ) + { + rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data); + if ( rc <= 0 ) + goto err; + } + else + { + ERROR("Unknown checkpointed stream"); + rc = -1; + goto err; + } } } while ( ctx->save.checkpointed != MIG_STREAM_NONE ); @@ -843,6 +934,7 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, ctx.save.live = !!(flags & XCFLAGS_LIVE); ctx.save.debug = !!(flags & XCFLAGS_DEBUG); ctx.save.checkpointed = checkpointed_stream; + ctx.save.recv_fd = recv_fd; /* If altering migration_stream update this assert too. */ assert(checkpointed_stream == MIG_STREAM_NONE || @@ -863,6 +955,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, assert(callbacks->switch_qemu_logdirty); if ( ctx.save.checkpointed ) assert(callbacks->checkpoint && callbacks->postcopy); + if ( ctx.save.checkpointed == MIG_STREAM_COLO ) + assert(callbacks->wait_checkpoint); DPRINTF("fd %d, dom %u, max_iters %u, max_factor %u, flags %u, hvm %d", io_fd, dom, max_iters, max_factor, flags, hvm);