From patchwork Mon Feb 22 02:52:09 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Congyang X-Patchwork-Id: 8370901 Return-Path: X-Original-To: patchwork-xen-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id AA30EC0555 for ; Mon, 22 Feb 2016 02:54:51 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 4DFC52039E for ; Mon, 22 Feb 2016 02:54:50 +0000 (UTC) Received: from lists.xen.org (lists.xenproject.org [50.57.142.19]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F178C203A9 for ; Mon, 22 Feb 2016 02:54:48 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1aXgbX-000137-2e; Mon, 22 Feb 2016 02:52:07 +0000 Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1aXgbU-00011X-Vv for xen-devel@lists.xen.org; Mon, 22 Feb 2016 02:52:05 +0000 Received: from [193.109.254.147] by server-16.bemta-14.messagelabs.com id AC/26-15991-4D77AC65; Mon, 22 Feb 2016 02:52:04 +0000 X-Env-Sender: wency@cn.fujitsu.com X-Msg-Ref: server-5.tower-27.messagelabs.com!1456109516!25574131!3 X-Originating-IP: [59.151.112.132] X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG X-StarScan-Received: X-StarScan-Version: 7.35.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 62919 invoked from network); 22 Feb 2016 02:52:02 -0000 Received: from cn.fujitsu.com (HELO heian.cn.fujitsu.com) (59.151.112.132) by server-5.tower-27.messagelabs.com with SMTP; 22 Feb 2016 02:52:02 -0000 X-IronPort-AV: E=Sophos;i="5.20,346,1444665600"; d="scan'208";a="3766005" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 22 Feb 2016 10:51:50 +0800 Received: from G08CNEXCHPEKD01.g08.fujitsu.local (unknown [10.167.33.80]) by cn.fujitsu.com (Postfix) with ESMTP id 7331942AC845; Mon, 22 Feb 2016 10:49:58 +0800 (CST) Received: from G08FNSTD140052.g08.fujitsu.local (10.167.226.52) by G08CNEXCHPEKD01.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP Server (TLS) id 14.3.181.6; Mon, 22 Feb 2016 10:51:49 +0800 From: Wen Congyang To: xen devel , Konrad Rzeszutek Wilk , Andrew Cooper , Ian Campbell , Ian Jackson , Wei Liu Date: Mon, 22 Feb 2016 10:52:09 +0800 Message-ID: <1456109555-28299-6-git-send-email-wency@cn.fujitsu.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1456109555-28299-1-git-send-email-wency@cn.fujitsu.com> References: <1456109555-28299-1-git-send-email-wency@cn.fujitsu.com> MIME-Version: 1.0 X-Originating-IP: [10.167.226.52] X-yoursite-MailScanner-ID: 7331942AC845.AA732 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: wency@cn.fujitsu.com X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Cc: Lars Kurth , Changlong Xie , Ian Campbell , Wen Congyang , Gui Jianfeng , Jiang Yunhong , Dong Eddie , Shriram Rajagopalan , Ian Jackson , Yang Hongyang Subject: [Xen-devel] [PATCH v10 05/31] tools/libx{l, c}: add back channel to libxc X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In COLO mode, both VMs are running, and are considered in sync if the visible network traffic is identical. After some time, they fall out of sync. At this point, the two VMs have definitely diverged. Lets call the primary dirty bitmap set A, while the secondary dirty bitmap set B. Sets A and B are different. Under normal migration, the page data for set A will be sent from the primary to the secondary. However, the set difference B - A (the one in B but not in A, lets call this C) is out-of-date on the secondary (with respect to the primary) and will not be sent by the primary (to secondary), as it was not memory dirtied by the primary. The secondary needs C page data to reconstruct an exact copy of the primary at the checkpoint. The secondary cannot calculate C as it doesn't know A. Instead, the secondary must send B to the primary, at which point the primary calculates the union of A and B (lets call this D) which is all the pages dirtied by both the primary and the secondary, and sends all page data covered by D. In the general case, D is a superset of both A and B. Without the backchannel dirty bitmap, a COLO checkpoint can't reconstruct a valid copy of the primary. We transfer the dirty bitmap on libxc side, so we need to introduce back channel to libxc. Note: it is different from the paper. We change the original design to the current one, according to our following concerns: 1. The original design needs extra memory on Secondary host. When there's multiple backups on one host, the memory cost is high. 2. The memory cache code will be another 1k+, it will make the review more time consuming. Note: the back channel will be used in the patch libxc/restore: send dirty pfn list to primary when checkpoint under COLO to send dirty pfn list from secondary to primary. The patch is posted in another series. Signed-off-by: Yang Hongyang Signed-off-by: Andrew Cooper CC: Ian Campbell CC: Ian Jackson CC: Wei Liu Reviewed-by: Konrad Rzeszutek Wilk --- tools/libxc/include/xenguest.h | 4 ++-- tools/libxc/xc_nomigrate.c | 4 ++-- tools/libxc/xc_sr_restore.c | 2 +- tools/libxc/xc_sr_save.c | 2 +- tools/libxl/libxl_save_callout.c | 21 +++++++++++++++------ tools/libxl/libxl_save_helper.c | 8 ++++++-- 6 files changed, 27 insertions(+), 14 deletions(-) diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h index affc42b..8040ac7 100644 --- a/tools/libxc/include/xenguest.h +++ b/tools/libxc/include/xenguest.h @@ -88,7 +88,7 @@ struct save_callbacks { int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iters, uint32_t max_factor, uint32_t flags /* XCFLAGS_xxx */, struct save_callbacks* callbacks, int hvm, - int checkpointed_stream); + int checkpointed_stream, int recv_fd); /* callbacks provided by xc_domain_restore */ struct restore_callbacks { @@ -127,7 +127,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, unsigned long *console_mfn, domid_t console_domid, unsigned int hvm, unsigned int pae, int superpages, int checkpointed_stream, - struct restore_callbacks *callbacks); + struct restore_callbacks *callbacks, int send_back_fd); /** * This function will create a domain for a paravirtualized Linux diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c index c9124df..3634743 100644 --- a/tools/libxc/xc_nomigrate.c +++ b/tools/libxc/xc_nomigrate.c @@ -23,7 +23,7 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iters, uint32_t max_factor, uint32_t flags, struct save_callbacks* callbacks, int hvm, - int checkpointed_stream) + int checkpointed_stream, int recv_fd) { errno = ENOSYS; return -1; @@ -35,7 +35,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, unsigned long *console_mfn, domid_t console_domid, unsigned int hvm, unsigned int pae, int superpages, int checkpointed_stream, - struct restore_callbacks *callbacks) + struct restore_callbacks *callbacks, int send_back_fd) { errno = ENOSYS; return -1; diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c index d4d33fd..e24560e 100644 --- a/tools/libxc/xc_sr_restore.c +++ b/tools/libxc/xc_sr_restore.c @@ -726,7 +726,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, unsigned long *console_gfn, domid_t console_domid, unsigned int hvm, unsigned int pae, int superpages, int checkpointed_stream, - struct restore_callbacks *callbacks) + struct restore_callbacks *callbacks, int send_back_fd) { struct xc_sr_context ctx = { diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c index e258b7c..fe210cc 100644 --- a/tools/libxc/xc_sr_save.c +++ b/tools/libxc/xc_sr_save.c @@ -830,7 +830,7 @@ static int save(struct xc_sr_context *ctx, uint16_t guest_type) int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iters, uint32_t max_factor, uint32_t flags, struct save_callbacks* callbacks, int hvm, - int checkpointed_stream) + int checkpointed_stream, int recv_fd) { struct xc_sr_context ctx = { diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c index a885504..0d6949a 100644 --- a/tools/libxl/libxl_save_callout.c +++ b/tools/libxl/libxl_save_callout.c @@ -27,7 +27,7 @@ */ static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs, const char *mode_arg, - int stream_fd, + int stream_fd, int back_channel_fd, const int *preserve_fds, int num_preserve_fds, const unsigned long *argnums, int num_argnums); @@ -50,6 +50,7 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs, /* Convenience aliases */ const uint32_t domid = dcs->guest_domid; const int restore_fd = dcs->libxc_fd; + const int send_back_fd = dcs->send_back_fd; libxl__domain_build_state *const state = &dcs->build_state; unsigned cbflags = @@ -71,7 +72,7 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs, shs->caller_state = dcs; shs->need_results = 1; - run_helper(egc, shs, "--restore-domain", restore_fd, 0, 0, + run_helper(egc, shs, "--restore-domain", restore_fd, send_back_fd, 0, 0, argnums, ARRAY_SIZE(argnums)); } @@ -95,7 +96,7 @@ void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_save_state *dss, shs->caller_state = dss; shs->need_results = 0; - run_helper(egc, shs, "--save-domain", dss->fd, + run_helper(egc, shs, "--save-domain", dss->fd, dss->recv_fd, NULL, 0, argnums, ARRAY_SIZE(argnums)); return; @@ -139,12 +140,14 @@ static int dup_fd_helper(libxl__gc *gc, int fd, const char *what) * 1) Path to libxl-save-helper. * 2) --[restore|save]-domain. * 3) stream file descriptor. + * 4) back channel file descriptor. * n) save/restore specific parameters. - * 4) A \0 at the end. + * 5) A \0 at the end. */ -#define HELPER_NR_ARGS 4 +#define HELPER_NR_ARGS 5 static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs, - const char *mode_arg, int stream_fd, + const char *mode_arg, + int stream_fd, int back_channel_fd, const int *preserve_fds, int num_preserve_fds, const unsigned long *argnums, int num_argnums) { @@ -177,6 +180,7 @@ static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs, *arg++ = getenv("LIBXL_SAVE_HELPER") ?: LIBEXEC_BIN "/" "libxl-save-helper"; *arg++ = mode_arg; const char **stream_fd_arg = arg++; + const char **back_channel_fd_arg = arg++; for (i=0; i= 0) + back_channel_fd = dup_fd_helper(gc, back_channel_fd, + "migration back channel fd"); + *back_channel_fd_arg = GCSPRINTF("%d", back_channel_fd); + for (i=0; i= 0) { assert(preserve_fds[i] > 2); diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c index 6bdcf13..2ff4c6e 100644 --- a/tools/libxl/libxl_save_helper.c +++ b/tools/libxl/libxl_save_helper.c @@ -238,6 +238,7 @@ static struct restore_callbacks helper_restore_callbacks; int main(int argc, char **argv) { int r; + int send_back_fd, recv_fd; #define NEXTARG (++argv, assert(*argv), *argv) @@ -247,6 +248,7 @@ int main(int argc, char **argv) if (!strcmp(mode,"--save-domain")) { io_fd = atoi(NEXTARG); + recv_fd = atoi(NEXTARG); uint32_t dom = strtoul(NEXTARG,0,10); uint32_t max_iters = strtoul(NEXTARG,0,10); uint32_t max_factor = strtoul(NEXTARG,0,10); @@ -262,12 +264,14 @@ int main(int argc, char **argv) setup_signals(save_signal_handler); r = xc_domain_save(xch, io_fd, dom, max_iters, max_factor, flags, - &helper_save_callbacks, hvm, checkpointed_stream); + &helper_save_callbacks, hvm, checkpointed_stream, + recv_fd); complete(r); } else if (!strcmp(mode,"--restore-domain")) { io_fd = atoi(NEXTARG); + send_back_fd = atoi(NEXTARG); uint32_t dom = strtoul(NEXTARG,0,10); unsigned store_evtchn = strtoul(NEXTARG,0,10); domid_t store_domid = strtoul(NEXTARG,0,10); @@ -292,7 +296,7 @@ int main(int argc, char **argv) store_domid, console_evtchn, &console_mfn, console_domid, hvm, pae, superpages, checkpointed, - &helper_restore_callbacks); + &helper_restore_callbacks, send_back_fd); helper_stub_restore_results(store_mfn,console_mfn,0); complete(r);