From patchwork Mon Sep 3 04:38:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585403 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 30D0316B1 for ; Mon, 3 Sep 2018 04:41:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1ECD3296C2 for ; Mon, 3 Sep 2018 04:41:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 128C1296C7; Mon, 3 Sep 2018 04:41:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4DD26296C2 for ; Mon, 3 Sep 2018 04:41:01 +0000 (UTC) Received: from localhost ([::1]:43359 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgfc-0008GA-IO for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:41:00 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55890) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgeH-0006Yk-5z for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:39:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwgeF-0004AX-Uu for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:39:37 -0400 Received: from mail-pf1-x442.google.com ([2607:f8b0:4864:20::442]:40492) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwgeF-0004A3-Lz for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:39:35 -0400 Received: by mail-pf1-x442.google.com with SMTP id s13-v6so8107277pfi.7 for ; Sun, 02 Sep 2018 21:39:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=2Ar+g9n4fDvTxeogf86aS/At5lIDSUtJ1XnU+jBR9vU=; b=ID1fZmH1qAc+ql4Oh/uwDeaKW0xqDoxkUA5Kn6jy8R6kkuCTrg+vOovmKyzdq/Y6zG Tm6WTqeVZmWWPZAsIUWbpXe2srA/+WKtPzhLBPwuJOCdoNRYB5gF2SJZZeFmHfsOf+JI yRbPT1cb1ZxjH9DKVXhP69p7vxcq3fTJ3QxCSuKQCUddB51hgCxix2ivxCDXMTCXlKYT 05IyNX6x7dq7e2v0497zN3nfnCytLQHD7ftByK2CKL0DzJ0p4RSImb4HXsP6C/5Q8rH5 HgVwDMVhya5bqoyDkS7FLfV2hNfebD24iS95CCgmpTl4Fh70qoyQMO8OGAr8UbscbCgg dy4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=2Ar+g9n4fDvTxeogf86aS/At5lIDSUtJ1XnU+jBR9vU=; b=iBdvvmFmvlRHzhuzT6Fmaccyn774X51eLA3uLehSYW1RmQP1eIdaXdIS3C5MIgOZLw MHeuZQMYDw7MuTC3Q68nL43l2HYvYy0QyYVHJqWJeyXQZuA5wdA/ns59k5Zss4ZmqM2/ MmECJgSB7PxorRBIZNHKvPHjfR35+njq81JuSUbTB+ziyhjSSWdfyrPZvC6bLpW39UaI mxHeaX5t/dk1F8DKM8my41CMSeX4AzR6KEXBCXciN2TPCimy3d2WuX6US51z4KAtOaZi J60CF8yv+5SAzBCDekVE7Zy9/ic+zSd9QVMdAZi3M5bteX5ZWZWHoHDMe78MH6qKU6cf y3fQ== X-Gm-Message-State: APzg51Cjz1DpB01WMx2uGO8Dk/uSmzSV4VTgjkLqFvgCGHtUURSUzXSD 05ujU82mfrGllZaucohelxJLh8re X-Google-Smtp-Source: ANB0VdbvZSssgqfnZhQkw0gqM2YAy4tWBtIDkfDugtf/FY0t0SihD2EnBsG/tCiI59UeKSEIDPV58Q== X-Received: by 2002:a63:fc07:: with SMTP id j7-v6mr22380689pgi.1.1535949573967; Sun, 02 Sep 2018 21:39:33 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.39.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:39:33 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:42 +0800 Message-Id: <20180903043900.28592-2-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::442 Subject: [Qemu-devel] [PATCH V12 01/19] filter-rewriter: Add TCP state machine and fix memory leak in connection_track_table X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP We add almost full TCP state machine in filter-rewriter, except TCPS_LISTEN and some simplify in VM active close FIN states. After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak. Let't track the state of net connection, if it is closed, its related resources will be cleared up. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen --- net/colo.c | 2 +- net/colo.h | 9 ++-- net/filter-rewriter.c | 105 ++++++++++++++++++++++++++++++++++++++---- 3 files changed, 100 insertions(+), 16 deletions(-) diff --git a/net/colo.c b/net/colo.c index 6dda4ed66e..97c8fc928f 100644 --- a/net/colo.c +++ b/net/colo.c @@ -137,7 +137,7 @@ Connection *connection_new(ConnectionKey *key) conn->ip_proto = key->ip_proto; conn->processing = false; conn->offset = 0; - conn->syn_flag = 0; + conn->tcp_state = TCPS_CLOSED; conn->pack = 0; conn->sack = 0; g_queue_init(&conn->primary_list); diff --git a/net/colo.h b/net/colo.h index da6c36dcf7..0277e0e9ba 100644 --- a/net/colo.h +++ b/net/colo.h @@ -18,6 +18,7 @@ #include "slirp/slirp.h" #include "qemu/jhash.h" #include "qemu/timer.h" +#include "slirp/tcp.h" #define HASHTABLE_MAX_SIZE 16384 @@ -81,11 +82,9 @@ typedef struct Connection { uint32_t sack; /* offset = secondary_seq - primary_seq */ tcp_seq offset; - /* - * we use this flag update offset func - * run once in independent tcp connection - */ - int syn_flag; + + int tcp_state; /* TCP FSM state */ + tcp_seq fin_ack_seq; /* the seq of 'fin=1,ack=1' */ } Connection; uint32_t connection_key_hash(const void *opaque); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index f584e4eba4..f18a71bf2e 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -59,9 +59,9 @@ static int is_tcp_packet(Packet *pkt) } /* handle tcp packet from primary guest */ -static int handle_primary_tcp_pkt(NetFilterState *nf, +static int handle_primary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; @@ -74,23 +74,28 @@ static int handle_primary_tcp_pkt(NetFilterState *nf, trace_colo_filter_rewriter_conn_offset(conn->offset); } + if (((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == (TH_ACK | TH_SYN)) && + conn->tcp_state == TCPS_SYN_SENT) { + conn->tcp_state = TCPS_ESTABLISHED; + } + if (((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == TH_SYN)) { /* * we use this flag update offset func * run once in independent tcp connection */ - conn->syn_flag = 1; + conn->tcp_state = TCPS_SYN_RECEIVED; } if (((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == TH_ACK)) { - if (conn->syn_flag) { + if (conn->tcp_state == TCPS_SYN_RECEIVED) { /* * offset = secondary_seq - primary seq * ack packet sent by guest from primary node, * so we use th_ack - 1 get primary_seq */ conn->offset -= (ntohl(tcp_pkt->th_ack) - 1); - conn->syn_flag = 0; + conn->tcp_state = TCPS_ESTABLISHED; } if (conn->offset) { /* handle packets to the secondary from the primary */ @@ -99,15 +104,63 @@ static int handle_primary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_len, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 1: + * Step 3: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'ack=1' packets from client side, it acks 'fin=1, ack=1' + * packet from server side. From this point, we can ensure that there + * will be no packets in the connection, except that, some errors + * happen between the path of 'filter object' and vNIC, if this rare + * case really happen, we can still create a new connection, + * So it is safe to remove the connection from connection_track_table. + * + */ + if ((conn->tcp_state == TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) == (conn->fin_ack_seq + 1))) { + conn->tcp_state = TCPS_CLOSED; + g_hash_table_remove(rf->connection_track_table, key); + } + } + + if ((tcp_pkt->th_flags & TH_FIN) == TH_FIN) { + /* + * Case 1: + * Step 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. We will into CLOSE_WAIT status. + */ + if (conn->tcp_state == TCPS_ESTABLISHED) { + conn->tcp_state = TCPS_CLOSE_WAIT; + } + + /* + * Case 2: + * Step 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. We will into CLOSE_WAIT status. + * We simplify the TCPS_FIN_WAIT_2, TCPS_TIME_WAIT and CLOSING status. + */ + if (conn->tcp_state == TCPS_FIN_WAIT_1) { + conn->tcp_state = TCPS_TIME_WAIT; + /* + * For simplify implementation, we needn't wait 2MSL time + * in filter rewriter. + */ + conn->tcp_state = TCPS_CLOSED; + g_hash_table_remove(rf->connection_track_table, key); + } } return 0; } /* handle tcp packet from secondary guest */ -static int handle_secondary_tcp_pkt(NetFilterState *nf, +static int handle_secondary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; @@ -121,7 +174,8 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, trace_colo_filter_rewriter_conn_offset(conn->offset); } - if (((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == (TH_ACK | TH_SYN))) { + if (conn->tcp_state == TCPS_SYN_RECEIVED && + ((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == (TH_ACK | TH_SYN))) { /* * save offset = secondary_seq and then * in handle_primary_tcp_pkt make offset @@ -130,6 +184,12 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, conn->offset = ntohl(tcp_pkt->th_seq); } + /* VM active connect */ + if (conn->tcp_state == TCPS_CLOSED && + ((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == TH_SYN)) { + conn->tcp_state = TCPS_SYN_SENT; + } + if ((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == TH_ACK) { /* Only need to adjust seq while offset is Non-zero */ if (conn->offset) { @@ -140,6 +200,31 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, pkt->size - pkt->vnet_hdr_len); } } + /* + * Case 1: + * Step 2: + * The *server* side of this connect is VM, *client* tries to close + * the connection. In this step we will into LAST_ACK status. + * + * We got 'fin=1, ack=1' packet from server side, we need to + * record the seq of 'fin=1, ack=1' packet. + */ + if (conn->tcp_state == TCPS_CLOSE_WAIT && + (tcp_pkt->th_flags & (TH_ACK | TH_FIN)) == (TH_ACK | TH_FIN)) { + conn->fin_ack_seq = ntohl(tcp_pkt->th_seq); + conn->tcp_state = TCPS_LAST_ACK; + } + + /* + * Case 2: + * Step 1: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + */ + if (conn->tcp_state == TCPS_ESTABLISHED && + (tcp_pkt->th_flags & (TH_ACK | TH_FIN)) == TH_FIN) { + conn->tcp_state = TCPS_FIN_WAIT_1; + } return 0; } @@ -190,7 +275,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, if (sender == nf->netdev) { /* NET_FILTER_DIRECTION_TX */ - if (!handle_primary_tcp_pkt(nf, conn, pkt)) { + if (!handle_primary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); @@ -203,7 +288,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, } } else { /* NET_FILTER_DIRECTION_RX */ - if (!handle_secondary_tcp_pkt(nf, conn, pkt)) { + if (!handle_secondary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); From patchwork Mon Sep 3 04:38:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585405 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 36D5E14E0 for ; Mon, 3 Sep 2018 04:41:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 251BA296C7 for ; Mon, 3 Sep 2018 04:41:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 19AC8296CF; Mon, 3 Sep 2018 04:41:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6D035296C7 for ; Mon, 3 Sep 2018 04:41:13 +0000 (UTC) Received: from localhost ([::1]:43349 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgfo-0006kw-Kf for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:41:12 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55930) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgeU-0006iz-9V for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:39:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwgeR-0004Fo-0p for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:39:50 -0400 Received: from mail-pg1-x52b.google.com ([2607:f8b0:4864:20::52b]:37921) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwgeQ-0004FC-Os for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:39:46 -0400 Received: by mail-pg1-x52b.google.com with SMTP id f4-v6so2725658pgq.5 for ; Sun, 02 Sep 2018 21:39:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TvJO8udGdRUsX1kOVbv+P3kHWq6wiXsaeR/C1Uk4ojc=; b=f34R/vS1XDg5x4U9d4q2T5YlWfCWomWqwchQe8Y5noK5tZoZ3B+kVACuRNUXJ+Cb9S XD3cm2/505pOAW0dTTnluCvh3bjcSyyNXiFuW0ojpCV4hZYQIYcpte9xFQmBOTAib2pe 26uy7hnFgNxzLFbVrrhgfl9bGmnSQDR3g51wckzpRXSGjWuzx3L9jrAj3i+DAyIlJE3J ByczSigxDEvFy/ZfLBwJvu2p+ET1iQtZq8NSPnmow9+oqMvuos8dbfAf00inR4jlAlUd 9oWASsEGvdwndGeV0J4ym/5r/JsDnveMjcGMgNV9FGgBss/eVxqYwg85SdbfMWLoX3NX 082w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=TvJO8udGdRUsX1kOVbv+P3kHWq6wiXsaeR/C1Uk4ojc=; b=sC7AgoXRvbm+nZAiVedePeN0kd1vT64W8OzpdfCYMHV0jC4Cnb1ujZgMV6B40uypAg IGb1wuAGLbRVG7HKXS08C5g4fPcvp+al6a13agFphjAOu3oHkVU55AuKLjWT5B1XsZQI csOna5tpu8m2PudPMO6RdpziWEW2joXwBS4NcR/9LZx2Hiik6itxwiVUUKBfLi7ezomn t+UAfKXfboEzJvAQgPV17ohfOaGSbx0Po4bfQE8GH4G4OfqcgZpjuA6sJy9HpYLLy9/q va5/+dN15FL4C4A3bFCip++/pyNz+pQr7zqZ6loRaPZoTFHLSqUx/ZhiaJuOuZGkj5Mb 6A9Q== X-Gm-Message-State: APzg51BYNzcdhFU9u+McqhJN+AhIgqUsPgQ/onliuShQ6ZWYQ9R60VwG jFj7r1kmHwgNFF8YYDsfrDS1+96i X-Google-Smtp-Source: ANB0VdY3DE7Sp8HLJeFeHi7G/RKVjlHeIP32SQuwM0fQUM1KE5lL39tn4fmFknEwdmIDgmk4uLLdGw== X-Received: by 2002:a63:3e8b:: with SMTP id l133-v6mr24140237pga.355.1535949585600; Sun, 02 Sep 2018 21:39:45 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.39.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:39:44 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:43 +0800 Message-Id: <20180903043900.28592-3-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::52b Subject: [Qemu-devel] [PATCH V12 02/19] colo-compare: implement the process of checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen --- include/migration/colo.h | 6 ++++ net/colo-compare.c | 78 ++++++++++++++++++++++++++++++++++++++++ net/colo-compare.h | 22 ++++++++++++ 3 files changed, 106 insertions(+) create mode 100644 net/colo-compare.h diff --git a/include/migration/colo.h b/include/migration/colo.h index 2fe48ad353..fefb2fcf4c 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -16,6 +16,12 @@ #include "qemu-common.h" #include "qapi/qapi-types-migration.h" +enum colo_event { + COLO_EVENT_NONE, + COLO_EVENT_CHECKPOINT, + COLO_EVENT_FAILOVER, +}; + void colo_info_init(void); void migrate_start_colo_process(MigrationState *s); diff --git a/net/colo-compare.c b/net/colo-compare.c index dd745a491b..80e6532e8b 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -27,11 +27,16 @@ #include "qemu/sockets.h" #include "colo.h" #include "sysemu/iothread.h" +#include "net/colo-compare.h" +#include "migration/colo.h" #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) +static QTAILQ_HEAD(, CompareState) net_compares = + QTAILQ_HEAD_INITIALIZER(net_compares); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 @@ -41,6 +46,10 @@ /* TODO: Should be configurable */ #define REGULAR_PACKET_CHECK_MS 3000 +static QemuMutex event_mtx; +static QemuCond event_complete_cond; +static int event_unhandled_count; + /* * + CompareState ++ * | | @@ -87,6 +96,11 @@ typedef struct CompareState { IOThread *iothread; GMainContext *worker_context; QEMUTimer *packet_check_timer; + + QEMUBH *event_bh; + enum colo_event event; + + QTAILQ_ENTRY(CompareState) next; } CompareState; typedef struct CompareClass { @@ -736,6 +750,25 @@ static void check_old_packet_regular(void *opaque) REGULAR_PACKET_CHECK_MS); } +/* Public API, Used for COLO frame to notify compare event */ +void colo_notify_compares_event(void *opaque, int event, Error **errp) +{ + CompareState *s; + + qemu_mutex_lock(&event_mtx); + QTAILQ_FOREACH(s, &net_compares, next) { + s->event = event; + qemu_bh_schedule(s->event_bh); + event_unhandled_count++; + } + /* Wait all compare threads to finish handling this event */ + while (event_unhandled_count > 0) { + qemu_cond_wait(&event_complete_cond, &event_mtx); + } + + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_timer_init(CompareState *s) { AioContext *ctx = iothread_get_aio_context(s->iothread); @@ -756,6 +789,30 @@ static void colo_compare_timer_del(CompareState *s) } } +static void colo_flush_packets(void *opaque, void *user_data); + +static void colo_compare_handle_event(void *opaque) +{ + CompareState *s = opaque; + + switch (s->event) { + case COLO_EVENT_CHECKPOINT: + g_queue_foreach(&s->conn_list, colo_flush_packets, s); + break; + case COLO_EVENT_FAILOVER: + break; + default: + break; + } + + assert(event_unhandled_count > 0); + + qemu_mutex_lock(&event_mtx); + event_unhandled_count--; + qemu_cond_broadcast(&event_complete_cond); + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_iothread(CompareState *s) { object_ref(OBJECT(s->iothread)); @@ -769,6 +826,7 @@ static void colo_compare_iothread(CompareState *s) s, s->worker_context, true); colo_compare_timer_init(s); + s->event_bh = qemu_bh_new(colo_compare_handle_event, s); } static char *compare_get_pri_indev(Object *obj, Error **errp) @@ -926,8 +984,13 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); + QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); + qemu_mutex_init(&event_mtx); + qemu_cond_init(&event_complete_cond); + s->connection_track_table = g_hash_table_new_full(connection_key_hash, connection_key_equal, g_free, @@ -990,6 +1053,7 @@ static void colo_compare_init(Object *obj) static void colo_compare_finalize(Object *obj) { CompareState *s = COLO_COMPARE(obj); + CompareState *tmp = NULL; qemu_chr_fe_deinit(&s->chr_pri_in, false); qemu_chr_fe_deinit(&s->chr_sec_in, false); @@ -997,6 +1061,16 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { colo_compare_timer_del(s); } + + qemu_bh_delete(s->event_bh); + + QTAILQ_FOREACH(tmp, &net_compares, next) { + if (tmp == s) { + QTAILQ_REMOVE(&net_compares, s, next); + break; + } + } + /* Release all unhandled packets after compare thead exited */ g_queue_foreach(&s->conn_list, colo_flush_packets, s); @@ -1009,6 +1083,10 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { object_unref(OBJECT(s->iothread)); } + + qemu_mutex_destroy(&event_mtx); + qemu_cond_destroy(&event_complete_cond); + g_free(s->pri_indev); g_free(s->sec_indev); g_free(s->outdev); diff --git a/net/colo-compare.h b/net/colo-compare.h new file mode 100644 index 0000000000..1b1ce76aea --- /dev/null +++ b/net/colo-compare.h @@ -0,0 +1,22 @@ +/* + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) + * (a.k.a. Fault Tolerance or Continuous Replication) + * + * Copyright (c) 2017 HUAWEI TECHNOLOGIES CO., LTD. + * Copyright (c) 2017 FUJITSU LIMITED + * Copyright (c) 2017 Intel Corporation + * + * Authors: + * zhanghailiang + * Zhang Chen + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * later. See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event(void *opaque, int event, Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */ From patchwork Mon Sep 3 04:38:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585407 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2418214E0 for ; Mon, 3 Sep 2018 04:43:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0FEFF296D0 for ; Mon, 3 Sep 2018 04:43:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F3B3A296D4; Mon, 3 Sep 2018 04:43:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 746D0296D0 for ; Mon, 3 Sep 2018 04:43:31 +0000 (UTC) Received: from localhost ([::1]:43369 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgi2-00044r-Fx for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:43:30 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55955) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgeY-0006nb-CF for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:39:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwgeX-0004KD-9r for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:39:54 -0400 Received: from mail-pg1-x532.google.com ([2607:f8b0:4864:20::532]:33337) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwgeX-0004Ia-30 for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:39:53 -0400 Received: by mail-pg1-x532.google.com with SMTP id s7-v6so996598pgc.0 for ; Sun, 02 Sep 2018 21:39:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bpx5nNk4c2i+Qel/d4lL7jT0w+UFVx5G4ipkEW0Rma0=; b=Egw7ZPTQ7DcugYmSyepHnxaMi8BbMiuCh29HVEQ5ETI63nTPMuJ6sTJiBxVcZ8TJbs Ea2gFElWv+SUdvW1VlNA794/M+XPY/mQUk/N7eXRO64m0yixxDtduaeXS/QM9QSf5R3s SO0kNlYWOlaXj/25DvGnpOFZWngLWpHd/FsvemD7TYKFvj/LK+XQCrrLRivJdccL2Wzl xwNr3Gph6s7s5yC31UHkkfgEJ33HyLvgDLLojNAEvr+hBX0sBlnIHdL5GL9et8zeB7rj Q3OAOCpu78YHjIibssGMMmsBwT0ocLiQVGmENdA0PUQqUglBcKi04FrEFp5l3NavHkG4 0qog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=bpx5nNk4c2i+Qel/d4lL7jT0w+UFVx5G4ipkEW0Rma0=; b=Zg8FmaAX+nITrORM4QqhTT5gi4LkawZBgUNtEgqQHwEY1yNPYfb45pDfFv2D3uthyA 8HJz+26ZcpAg9T0h1Pgeor5vaLRLZdBZk2IJlGqQitQAiD9wYACcrveXMQvxafiCbsty ltqHgkizndFY6TvsI7yhX9P47VFKRcYzHGDPkDWXo/K4QOTKu0i9YO1rIYmzSkkz4u9u YKKr0iMxqwX7t1R6/6Op49d5EWTQOVSp+LJN4mFxPd6TnJA/yJLOgXDRO5oi5DKc5cLe 7jEY/KFPqIaXf1QdhuxI5LHmTmqBJoj0m1nNLXJovABNLOk3OyggdZAd9nWTdYvm+xhE KN3A== X-Gm-Message-State: APzg51DmhzI9HgiyMi8jOAD0194vql8FUusjyaAJmCR9LBPu5WPpDhn2 /69qMUgf0gXM1sLgmYkz7WcoV/9i X-Google-Smtp-Source: ANB0VdZwH55iVyeqMe/RdrfFN/1azvr1PmPe0hxHXg4PTNEZ96tOmR/exPYE/tyh3aYuC25MRdWPDg== X-Received: by 2002:a62:6bc5:: with SMTP id g188-v6mr23603896pfc.91.1535949591927; Sun, 02 Sep 2018 21:39:51 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.39.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:39:51 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:44 +0800 Message-Id: <20180903043900.28592-4-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::532 Subject: [Qemu-devel] [PATCH V12 03/19] colo-compare: use notifier to notify packets comparing result X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 37 ++++++++++++++++++++++++++----------- net/colo-compare.h | 2 ++ 2 files changed, 28 insertions(+), 11 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index 80e6532e8b..426eab5973 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -29,6 +29,7 @@ #include "sysemu/iothread.h" #include "net/colo-compare.h" #include "migration/colo.h" +#include "migration/migration.h" #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ @@ -37,6 +38,9 @@ static QTAILQ_HEAD(, CompareState) net_compares = QTAILQ_HEAD_INITIALIZER(net_compares); +static NotifierList colo_compare_notifiers = + NOTIFIER_LIST_INITIALIZER(colo_compare_notifiers); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 @@ -427,10 +431,7 @@ sec: qemu_hexdump((char *)spkt->data, stderr, "colo-compare spkt", spkt->size); - /* - * colo_compare_inconsistent_notify(); - * TODO: notice to checkpoint(); - */ + colo_compare_inconsistency_notify(); } } @@ -561,8 +562,24 @@ static int colo_old_packet_check_one(Packet *pkt, int64_t *check_time) } } +static void colo_compare_inconsistency_notify(void) +{ + notifier_list_notify(&colo_compare_notifiers, + migrate_get_current()); +} + +void colo_compare_register_notifier(Notifier *notify) +{ + notifier_list_add(&colo_compare_notifiers, notify); +} + +void colo_compare_unregister_notifier(Notifier *notify) +{ + notifier_remove(notify); +} + static int colo_old_packet_check_one_conn(Connection *conn, - void *user_data) + void *user_data) { GList *result = NULL; int64_t check_time = REGULAR_PACKET_CHECK_MS; @@ -573,10 +590,7 @@ static int colo_old_packet_check_one_conn(Connection *conn, if (result) { /* Do checkpoint will flush old packet */ - /* - * TODO: Notify colo frame to do checkpoint. - * colo_compare_inconsistent_notify(); - */ + colo_compare_inconsistency_notify(); return 0; } @@ -620,11 +634,12 @@ static void colo_compare_packet(CompareState *s, Connection *conn, /* * If one packet arrive late, the secondary_list or * primary_list will be empty, so we can't compare it - * until next comparison. + * until next comparison. If the packets in the list are + * timeout, it will trigger a checkpoint request. */ trace_colo_compare_main("packet different"); g_queue_push_head(&conn->primary_list, pkt); - /* TODO: colo_notify_checkpoint();*/ + colo_compare_inconsistency_notify(); break; } } diff --git a/net/colo-compare.h b/net/colo-compare.h index 1b1ce76aea..22ddd512e2 100644 --- a/net/colo-compare.h +++ b/net/colo-compare.h @@ -18,5 +18,7 @@ #define QEMU_COLO_COMPARE_H void colo_notify_compares_event(void *opaque, int event, Error **errp); +void colo_compare_register_notifier(Notifier *notify); +void colo_compare_unregister_notifier(Notifier *notify); #endif /* QEMU_COLO_COMPARE_H */ From patchwork Mon Sep 3 04:38:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585411 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB60A14E0 for ; Mon, 3 Sep 2018 04:43:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DA08B296D3 for ; Mon, 3 Sep 2018 04:43:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CE2F1296D7; Mon, 3 Sep 2018 04:43:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4AC4E296D3 for ; Mon, 3 Sep 2018 04:43:53 +0000 (UTC) Received: from localhost ([::1]:43374 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgiO-0004NW-Jc for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:43:52 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56020) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgek-0007NX-67 for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:40:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwgeg-0004Pk-Ui for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:40:06 -0400 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]:38786) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwgeg-0004PO-Ni for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:40:02 -0400 Received: by mail-pl1-x635.google.com with SMTP id u11-v6so8046021plq.5 for ; Sun, 02 Sep 2018 21:40:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=QxSyb7bdhCZ2W8suOPykje3xhRgVGiuGq/waG6csnMI=; b=mjgMu/lItU8y8FRVVbUBH/lC3+58NSbbgIXTZZuwEY3CwTSbnbJlJPfiv5bw0XI3J1 0qHKvx6gg3O1aTvvFaepiOsJFVE1nRUI8KXYAiFz0J5SLrPPjhewzwyA0grKsLH7SpA7 qPk+VzzV40i9q1OrLTKh+qxvUuMrzkGTaoW71etNgvP4XXpbiFk6zSAx09KIPF9yFzCE 0BxDsfluGDTaEd5p4QMhKFoaoi57HrjQrKsmQA1FfElEdY+aRMN514J0aTk66GQ2q29P Bfi9CuwDYGhgaan6K8CguID1+KBXJYVcR1II/6Q3J0nJLmoGh+QBFY4fx+737lMKY5dQ bK6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=QxSyb7bdhCZ2W8suOPykje3xhRgVGiuGq/waG6csnMI=; b=Ec0emHa7cCE94wvnFNasU0bH+hplFEgs0oNMRa1CDQunLEBl2BbiEIrd2oEk1x9Xn3 4q8ETN929MX9t8Hus3SAaNQ6YNWJLdb8gsaRX9Sb95uC3oWy7htYeL4jRxO/Mh9W1SLI uZOrdr8G7ASv3UHqyZs3f5Cmff9A66+PG5+/dwJRVTwRdpVg7jX1/CLAFYibrPgLIqAS EACpCCUzmyFCxO1yDfTCO2OLrcfNR4Y22LqkHm9r3Nr8Z7a8xP/VG8AaxMVl77mbnXDx QapM7Ib7wbbgj/MyEEZGOfC3L/O774xAearzxCYDhjsmR/UjJkRGANVVi0zU7VV7iT3y 05Kw== X-Gm-Message-State: APzg51A1CflLZ1zvYzfFaQHHQa367eRXNdFCohkcPOzQXunX8CtPSuP6 M9f9A0z0GEQzU5CU86NF1HMERyHP X-Google-Smtp-Source: ANB0VdYumayLu9sIPpeNsvnF90/K0XurWhyy8+aosTbwp3dlk3l364dNqOdVc7FuGW8wAM2oKImoGA== X-Received: by 2002:a17:902:d706:: with SMTP id w6-v6mr26783025ply.158.1535949601628; Sun, 02 Sep 2018 21:40:01 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.39.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:40:00 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:45 +0800 Message-Id: <20180903043900.28592-5-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::635 Subject: [Qemu-devel] [PATCH V12 04/19] COLO: integrate colo compare with colo frame X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP For COLO FT, both the PVM and SVM run at the same time, only sync the state while it needs. So here, let SVM runs while not doing checkpoint, change DEFAULT_MIGRATE_X_CHECKPOINT_DELAY to 200*100. Besides, we forgot to release colo_checkpoint_semd and colo_delay_timer, fix them here. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 ++++++++++++++++++++++++++++++++++++++++-- migration/migration.c | 6 ++---- 2 files changed, 42 insertions(+), 6 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 88936f5962..f4bdfde170 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -25,8 +25,11 @@ #include "qemu/error-report.h" #include "migration/failover.h" #include "replication.h" +#include "net/colo-compare.h" +#include "net/colo.h" static bool vmstate_loading; +static Notifier packets_compare_notifier; #define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024) @@ -343,6 +346,11 @@ static int colo_do_checkpoint_transaction(MigrationState *s, goto out; } + colo_notify_compares_event(NULL, COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + goto out; + } + /* Disable block migration */ migrate_set_block_enabled(false, &local_err); qemu_savevm_state_header(fb); @@ -400,6 +408,11 @@ out: return ret; } +static void colo_compare_notify_checkpoint(Notifier *notifier, void *data) +{ + colo_checkpoint_notify(data); +} + static void colo_process_checkpoint(MigrationState *s) { QIOChannelBuffer *bioc; @@ -416,6 +429,9 @@ static void colo_process_checkpoint(MigrationState *s) goto out; } + packets_compare_notifier.notify = colo_compare_notify_checkpoint; + colo_compare_register_notifier(&packets_compare_notifier); + /* * Wait for Secondary finish loading VM states and enter COLO * restore. @@ -461,11 +477,21 @@ out: qemu_fclose(fb); } - timer_del(s->colo_delay_timer); - /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); + + /* + * It is safe to unregister notifier after failover finished. + * Besides, colo_delay_timer and colo_checkpoint_sem can't be + * released befor unregister notifier, or there will be use-after-free + * error. + */ + colo_compare_unregister_notifier(&packets_compare_notifier); + timer_del(s->colo_delay_timer); + timer_free(s->colo_delay_timer); + qemu_sem_destroy(&s->colo_checkpoint_sem); + /* * Must be called after failover BH is completed, * Or the failover BH may shutdown the wrong fd that @@ -559,6 +585,11 @@ void *colo_process_incoming_thread(void *opaque) fb = qemu_fopen_channel_input(QIO_CHANNEL(bioc)); object_unref(OBJECT(bioc)); + qemu_mutex_lock_iothread(); + vm_start(); + trace_colo_vm_state_change("stop", "run"); + qemu_mutex_unlock_iothread(); + colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_READY, &local_err); if (local_err) { @@ -578,6 +609,11 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + trace_colo_vm_state_change("run", "stop"); + qemu_mutex_unlock_iothread(); + /* FIXME: This is unnecessary for periodic checkpoint mode */ colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_REPLY, &local_err); @@ -631,6 +667,8 @@ void *colo_process_incoming_thread(void *opaque) } vmstate_loading = false; + vm_start(); + trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); if (failover_get_state() == FAILOVER_STATUS_RELAUNCH) { diff --git a/migration/migration.c b/migration/migration.c index 4b316ec343..2f3e9d40d1 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -76,10 +76,8 @@ /* Migration XBZRLE default cache size */ #define DEFAULT_MIGRATE_XBZRLE_CACHE_SIZE (64 * 1024 * 1024) -/* The delay time (in ms) between two COLO checkpoints - * Note: Please change this default value to 10000 when we support hybrid mode. - */ -#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY 200 +/* The delay time (in ms) between two COLO checkpoints */ +#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100) #define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2 #define DEFAULT_MIGRATE_MULTIFD_PAGE_COUNT 16 From patchwork Mon Sep 3 04:38:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585415 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E1FC4112B for ; Mon, 3 Sep 2018 04:46:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CD5C0296D0 for ; Mon, 3 Sep 2018 04:46:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BF19C296D4; Mon, 3 Sep 2018 04:46:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2B573296D0 for ; Mon, 3 Sep 2018 04:46:21 +0000 (UTC) Received: from localhost ([::1]:43390 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgkm-0006Pl-E6 for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:46:20 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56114) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgf3-0007lB-BF for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:40:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwgf0-0004Yc-3Z for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:40:25 -0400 Received: from mail-pg1-x543.google.com ([2607:f8b0:4864:20::543]:40617) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwgez-0004YI-TW for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:40:22 -0400 Received: by mail-pg1-x543.google.com with SMTP id l63-v6so3521970pga.7 for ; Sun, 02 Sep 2018 21:40:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ccg+qGTJrnYh1JFBtXgKPrCUY4ftrmk1pV2d7nhir7s=; b=rj8gZIVr/ziWpYcEQTgz12YBNGDo9rktuOrTry4ip+IZDrBnZcWcc2GmTofrNAx5cX 1krQa1BgTwcQYOp5xEAhJOVf2MtGn/lF4X8ARK4fd7i6FV9KHSDLapbhGguURpiikKGW 4BEPGhonuCmYmoDl0Iljqu4q6rl6R0Y79cAKqer4HQjIbRzIRftHZTvP4RUvW4HMlcVO ojtSsadVPn4G8oEDNx02nKgvs+FJcmxramgaxwK6xjba7UM2m3B22KrEGPnj12yjYcfs rH7AtJwFu+FDdC8dGjpqt4bW/gHi5+E/FZ6RfZNCwtn2PVKYv7ZkABoc8mqYZNwp49zr CZ4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ccg+qGTJrnYh1JFBtXgKPrCUY4ftrmk1pV2d7nhir7s=; b=MRxV1FN1+bCoQfhJFDBbPT/bbwAZOpSO5TWSD5b4CNJZCWwnQpoGiIWG1f9jRZTgZy hSUuSMx7I1Hpl/oz40MMOgacEVdozXwmKtCC3riSxfdoP8e4tUwss9fyZl0J1YuOT6Ty paxq2SR3jl7VC6HQFSOMgaYFTIYcu7FHrPg1oU2V9EJLNf+Ka5TKwt2Jo1Yuxw5NI6Ss H9d1qo6/BvHSL/rb/9t4noD4ZlgEPshhyC/+aEpe3qSI9vrLeO1gVCBoL+8XXdYfo5io IxmVyamaPLv8gD8IxyuYZDaEsroOdWKsUbR7SQ6hQRHf8vDTLyUQ2CMvBySzF9rXFXnA qUjw== X-Gm-Message-State: APzg51B7e+AItlr/6gI39sPr1JSi3pdOu271MjavDSQrQuvAJMaCIcdQ l3lJvnBdX8f33vBiQtOnjp8/fKJ6 X-Google-Smtp-Source: ANB0VdYkLxQOFN/jfhlB6LYVmxkB0MY0wTjExXPoO9DJ40rwpzgx1msW4gzwZJOiWeIHT48PTvqVYA== X-Received: by 2002:a63:4860:: with SMTP id x32-v6mr20610165pgk.375.1535949620753; Sun, 02 Sep 2018 21:40:20 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.40.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:40:19 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:46 +0800 Message-Id: <20180903043900.28592-6-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::543 Subject: [Qemu-devel] [PATCH V12 05/19] COLO: Add block replication into colo process X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 10 ++++++++++ 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index f4bdfde170..af04010061 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -27,6 +27,7 @@ #include "replication.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "block/block.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -56,6 +57,7 @@ static void secondary_vm_do_failover(void) { int old_state; MigrationIncomingState *mis = migration_incoming_get_current(); + Error *local_err = NULL; /* Can not do failover during the process of VM's loading VMstate, Or * it will break the secondary VM. @@ -73,6 +75,11 @@ static void secondary_vm_do_failover(void) migrate_set_state(&mis->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + } + if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side"); /* recover runstate to normal migration finish state */ @@ -110,6 +117,7 @@ static void primary_vm_do_failover(void) { MigrationState *s = migrate_get_current(); int old_state; + Error *local_err = NULL; migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); @@ -133,6 +141,13 @@ static void primary_vm_do_failover(void) FailoverStatus_str(old_state)); return; } + + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + local_err = NULL; + } + /* Notify COLO thread that failover work is finished */ qemu_sem_post(&s->colo_exit_sem); } @@ -356,6 +371,11 @@ static int colo_do_checkpoint_transaction(MigrationState *s, qemu_savevm_state_header(fb); qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } qemu_savevm_state_complete_precopy(fb, false, false); qemu_mutex_unlock_iothread(); @@ -446,6 +466,12 @@ static void colo_process_checkpoint(MigrationState *s) object_unref(OBJECT(bioc)); qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_PRIMARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vm_start(); qemu_mutex_unlock_iothread(); trace_colo_vm_state_change("stop", "run"); @@ -586,6 +612,11 @@ void *colo_process_incoming_thread(void *opaque) object_unref(OBJECT(bioc)); qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_SECONDARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } vm_start(); trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); @@ -666,6 +697,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + replication_get_error_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* discard colo disk buffer */ + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading = false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/migration/migration.c b/migration/migration.c index 2f3e9d40d1..f4ab7b2d01 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -386,6 +386,7 @@ static void process_incoming_migration_co(void *opaque) MigrationIncomingState *mis = migration_incoming_get_current(); PostcopyState ps; int ret; + Error *local_err = NULL; assert(mis->from_src_file); mis->migration_incoming_co = qemu_coroutine_self(); @@ -418,6 +419,15 @@ static void process_incoming_migration_co(void *opaque) /* we get COLO info, and know if we are in COLO mode */ if (!ret && migration_incoming_enable_colo()) { + /* Make sure all file formats flush their mutable metadata */ + bdrv_invalidate_cache_all(&local_err); + if (local_err) { + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + error_report_err(local_err); + exit(EXIT_FAILURE); + } + qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); mis->have_colo_incoming_thread = true; From patchwork Mon Sep 3 04:38:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585419 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 336F3180E for ; Mon, 3 Sep 2018 04:46:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 22590296D0 for ; Mon, 3 Sep 2018 04:46:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 15F1B296D4; Mon, 3 Sep 2018 04:46:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4BC75296D0 for ; Mon, 3 Sep 2018 04:46:48 +0000 (UTC) Received: from localhost ([::1]:43392 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwglD-0006jw-Jd for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:46:47 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56171) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgfC-0007tG-VZ for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:40:36 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwgf9-0004cL-My for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:40:34 -0400 Received: from mail-pf1-x432.google.com ([2607:f8b0:4864:20::432]:42076) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwgf9-0004bx-Eu for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:40:31 -0400 Received: by mail-pf1-x432.google.com with SMTP id l9-v6so8101680pff.9 for ; Sun, 02 Sep 2018 21:40:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mL+Q+ivB361kS0dC7HePzbkeO15Jc+ZPNhKleSoW/38=; b=Q36Ibf2WLpQj5+/6jFMlUs+Dxa861Gs88DHnuNma7WLcjfqa2g3WmB49BbThGJpAvg w7VDzb9nRCrTHUu07oEPM766pGeo2JGw2JRnbOQT/Gp3dvEJsxmCBpgUfh/HnbuqXez/ tmntSSMaz17gsZ5PrZ+4arFvYqdz9A33lIs2sS9zxXuUHDbt6rilgr9oDiFuSx3gw9qY SRavYYbf6S22J2K41J7Rir7IFo+nFuCVEoXut3Kww5Msi4FXqduMXJ4WAQ4rooVtMPFf jAiQlAIoMSrc/GZJhEpbL6V5/s1JSmOidgoF79EB9h52y7IZqOj5WH7Ng8hldCBloTIn Gg5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mL+Q+ivB361kS0dC7HePzbkeO15Jc+ZPNhKleSoW/38=; b=W1mePZgS60Ty5B0QSx0rJ3BoSoholwM97RxLGdmykvkJ6Lz3r9CbqP3i4TgIVGeREd EiNToGTiIjyA8wmdgKjmxYCDvft960sUL/T5IsfQtKxMHlp2cquwzk1wS2WktN0T/M+I hqI5HPfnA65Te3Ni8BJK5oEylMwHYA+YjjAJfdOvv0Sk5Eon/VCnWLT+3Ru3nkhN0PJ3 iwQ1cUbhTFsGbhlBqZf8tw1d7Y2OcNT0tXGDgUSWJ8R5uNNWdvXAUHYRuhniWDb/AQBK ALD4AgZ+wxiIHswJg700OPqeW3im5dtToybVJSDOwxIBuyRHhhB0ZHk1OHMt9l/IRfjk LmdA== X-Gm-Message-State: APzg51Bsn5IeaGAH+2TPXIATGI8Cfqgg42oDIFs0g8xmKZkDXxcukWf6 Gs2wcmQLl0RCMjzvsF4egg63ehBr X-Google-Smtp-Source: ANB0VdY9VjyQCtIupo7qqO8IrJqH/ls/KKKJ9lYCzHfWHGZK4L5P13N7LhcAzI3qQLs8bbK2djPiOw== X-Received: by 2002:aa7:83cd:: with SMTP id j13-v6mr27384105pfn.236.1535949630156; Sun, 02 Sep 2018 21:40:30 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.40.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:40:29 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:47 +0800 Message-Id: <20180903043900.28592-7-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::432 Subject: [Qemu-devel] [PATCH V12 06/19] COLO: Remove colo_state migration struct X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP We need to know if migration is going into COLO state for incoming side before start normal migration. Instead by using the VMStateDescription to send colo_state from source side to destination side, we use MIG_CMD_ENABLE_COLO to indicate whether COLO is enabled or not. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- include/migration/colo.h | 5 +-- migration/Makefile.objs | 2 +- migration/colo-comm.c | 76 ---------------------------------------- migration/colo.c | 13 ++++++- migration/migration.c | 23 +++++++++++- migration/savevm.c | 17 +++++++++ migration/savevm.h | 1 + migration/trace-events | 1 + vl.c | 2 -- 9 files changed, 57 insertions(+), 83 deletions(-) delete mode 100644 migration/colo-comm.c diff --git a/include/migration/colo.h b/include/migration/colo.h index fefb2fcf4c..99ce17aca7 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -28,8 +28,9 @@ void migrate_start_colo_process(MigrationState *s); bool migration_in_colo_state(void); /* loadvm */ -bool migration_incoming_enable_colo(void); -void migration_incoming_exit_colo(void); +void migration_incoming_enable_colo(void); +void migration_incoming_disable_colo(void); +bool migration_incoming_colo_enabled(void); void *colo_process_incoming_thread(void *opaque); bool migration_incoming_in_colo_state(void); diff --git a/migration/Makefile.objs b/migration/Makefile.objs index c83ec47ba8..a4f3bafd86 100644 --- a/migration/Makefile.objs +++ b/migration/Makefile.objs @@ -1,6 +1,6 @@ common-obj-y += migration.o socket.o fd.o exec.o common-obj-y += tls.o channel.o savevm.o -common-obj-y += colo-comm.o colo.o colo-failover.o +common-obj-y += colo.o colo-failover.o common-obj-y += vmstate.o vmstate-types.o page_cache.o common-obj-y += qemu-file.o global_state.o common-obj-y += qemu-file-channel.o diff --git a/migration/colo-comm.c b/migration/colo-comm.c deleted file mode 100644 index df26e4dfe7..0000000000 --- a/migration/colo-comm.c +++ /dev/null @@ -1,76 +0,0 @@ -/* - * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) - * (a.k.a. Fault Tolerance or Continuous Replication) - * - * Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. - * Copyright (c) 2016 FUJITSU LIMITED - * Copyright (c) 2016 Intel Corporation - * - * This work is licensed under the terms of the GNU GPL, version 2 or - * later. See the COPYING file in the top-level directory. - * - */ - -#include "qemu/osdep.h" -#include "migration.h" -#include "migration/colo.h" -#include "migration/vmstate.h" -#include "trace.h" - -typedef struct { - bool colo_requested; -} COLOInfo; - -static COLOInfo colo_info; - -COLOMode get_colo_mode(void) -{ - if (migration_in_colo_state()) { - return COLO_MODE_PRIMARY; - } else if (migration_incoming_in_colo_state()) { - return COLO_MODE_SECONDARY; - } else { - return COLO_MODE_UNKNOWN; - } -} - -static int colo_info_pre_save(void *opaque) -{ - COLOInfo *s = opaque; - - s->colo_requested = migrate_colo_enabled(); - - return 0; -} - -static bool colo_info_need(void *opaque) -{ - return migrate_colo_enabled(); -} - -static const VMStateDescription colo_state = { - .name = "COLOState", - .version_id = 1, - .minimum_version_id = 1, - .pre_save = colo_info_pre_save, - .needed = colo_info_need, - .fields = (VMStateField[]) { - VMSTATE_BOOL(colo_requested, COLOInfo), - VMSTATE_END_OF_LIST() - }, -}; - -void colo_info_init(void) -{ - vmstate_register(NULL, 0, &colo_state, &colo_info); -} - -bool migration_incoming_enable_colo(void) -{ - return colo_info.colo_requested; -} - -void migration_incoming_exit_colo(void) -{ - colo_info.colo_requested = false; -} diff --git a/migration/colo.c b/migration/colo.c index af04010061..d3163b51c8 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -152,6 +152,17 @@ static void primary_vm_do_failover(void) qemu_sem_post(&s->colo_exit_sem); } +COLOMode get_colo_mode(void) +{ + if (migration_in_colo_state()) { + return COLO_MODE_PRIMARY; + } else if (migration_incoming_in_colo_state()) { + return COLO_MODE_SECONDARY; + } else { + return COLO_MODE_UNKNOWN; + } +} + void colo_do_failover(MigrationState *s) { /* Make sure VM stopped while failover happened. */ @@ -746,7 +757,7 @@ out: if (mis->to_src_file) { qemu_fclose(mis->to_src_file); } - migration_incoming_exit_colo(); + migration_incoming_disable_colo(); rcu_unregister_thread(); return NULL; diff --git a/migration/migration.c b/migration/migration.c index f4ab7b2d01..950b22df63 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -296,6 +296,22 @@ int migrate_send_rp_req_pages(MigrationIncomingState *mis, const char *rbname, return migrate_send_rp_message(mis, msg_type, msglen, bufc); } +static bool migration_colo_enabled; +bool migration_incoming_colo_enabled(void) +{ + return migration_colo_enabled; +} + +void migration_incoming_disable_colo(void) +{ + migration_colo_enabled = false; +} + +void migration_incoming_enable_colo(void) +{ + migration_colo_enabled = true; +} + void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p; @@ -418,7 +434,7 @@ static void process_incoming_migration_co(void *opaque) } /* we get COLO info, and know if we are in COLO mode */ - if (!ret && migration_incoming_enable_colo()) { + if (!ret && migration_incoming_colo_enabled()) { /* Make sure all file formats flush their mutable metadata */ bdrv_invalidate_cache_all(&local_err); if (local_err) { @@ -3010,6 +3026,11 @@ static void *migration_thread(void *opaque) qemu_savevm_send_postcopy_advise(s->to_dst_file); } + if (migrate_colo_enabled()) { + /* Notify migration destination that we enable COLO */ + qemu_savevm_send_colo_enable(s->to_dst_file); + } + qemu_savevm_state_setup(s->to_dst_file); s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start; diff --git a/migration/savevm.c b/migration/savevm.c index 13e51f0e34..0376e723b6 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -56,6 +56,7 @@ #include "io/channel-file.h" #include "sysemu/replay.h" #include "qjson.h" +#include "migration/colo.h" #ifndef ETH_P_RARP #define ETH_P_RARP 0x8035 @@ -82,6 +83,7 @@ enum qemu_vm_cmd { were previously sent during precopy but are dirty. */ MIG_CMD_PACKAGED, /* Send a wrapped stream within this stream */ + MIG_CMD_ENABLE_COLO, /* Enable COLO */ MIG_CMD_POSTCOPY_RESUME, /* resume postcopy on dest */ MIG_CMD_RECV_BITMAP, /* Request for recved bitmap on dst */ MIG_CMD_MAX @@ -841,6 +843,12 @@ static void qemu_savevm_command_send(QEMUFile *f, qemu_fflush(f); } +void qemu_savevm_send_colo_enable(QEMUFile *f) +{ + trace_savevm_send_colo_enable(); + qemu_savevm_command_send(f, MIG_CMD_ENABLE_COLO, 0, NULL); +} + void qemu_savevm_send_ping(QEMUFile *f, uint32_t value) { uint32_t buf; @@ -1921,6 +1929,12 @@ static int loadvm_handle_recv_bitmap(MigrationIncomingState *mis, return 0; } +static int loadvm_process_enable_colo(MigrationIncomingState *mis) +{ + migration_incoming_enable_colo(); + return 0; +} + /* * Process an incoming 'QEMU_VM_COMMAND' * 0 just a normal return @@ -2000,6 +2014,9 @@ static int loadvm_process_command(QEMUFile *f) case MIG_CMD_RECV_BITMAP: return loadvm_handle_recv_bitmap(mis, len); + + case MIG_CMD_ENABLE_COLO: + return loadvm_process_enable_colo(mis); } return 0; diff --git a/migration/savevm.h b/migration/savevm.h index a5e65b8ae3..8373c2f6bd 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -55,6 +55,7 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint16_t len, uint64_t *start_list, uint64_t *length_list); +void qemu_savevm_send_colo_enable(QEMUFile *f); int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); diff --git a/migration/trace-events b/migration/trace-events index 9430f3cbe0..fa0ff3f3bf 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -37,6 +37,7 @@ savevm_send_ping(uint32_t val) "0x%x" savevm_send_postcopy_listen(void) "" savevm_send_postcopy_run(void) "" savevm_send_postcopy_resume(void) "" +savevm_send_colo_enable(void) "" savevm_send_recv_bitmap(char *name) "%s" savevm_state_setup(void) "" savevm_state_resume_prepare(void) "" diff --git a/vl.c b/vl.c index 5ba06adf78..539e90e133 100644 --- a/vl.c +++ b/vl.c @@ -4396,8 +4396,6 @@ int main(int argc, char **argv, char **envp) #endif } - colo_info_init(); - if (net_init_clients(&err) < 0) { error_report_err(err); exit(1); From patchwork Mon Sep 3 04:38:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585409 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4AC8814E0 for ; Mon, 3 Sep 2018 04:43:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 386F8296D0 for ; Mon, 3 Sep 2018 04:43:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2BC82296D4; Mon, 3 Sep 2018 04:43:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7FAA3296D0 for ; Mon, 3 Sep 2018 04:43:45 +0000 (UTC) Received: from localhost ([::1]:43366 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgiG-0001oE-RF for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:43:44 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56239) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgfW-0008AV-2e for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:40:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwgfS-0004if-PJ for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:40:54 -0400 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:42563) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwgfS-0004i7-HV for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:40:50 -0400 Received: by mail-pl1-x644.google.com with SMTP id g23-v6so8033478plq.9 for ; Sun, 02 Sep 2018 21:40:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GkclWeVrO8jS1MIahu2gWatY+lpfTpxiBg+TTWPbieo=; b=uuPhIQnOEq9KxaZ0mPc4vMHvli/gmx9elFf3C8ObQBi4k+oaqoQTeePfWR0MCvU1PU SwovSh8H9XNrs7SpP+pZXNm5zAutcLdmA2JCOwJbl6KHP35q92QtQYgs5ZQyrtcy8WqQ iY8b33QW3VZd19RwW4BY7PPUw+LmtD0nLXBNFqgLUUevYeELNkozr/90JCLPEW4e851j pBg5oONjXJ32gRcXV6inQJLvRA+HnIgQV7aN6Pney78c9fbalSsW0HIpXdTeLEUFcUzv k67aAASQfleEb+f2Id8Kvwcq5NPTNwgPiAMi/29eJ7+UXSHYQh6J1SECtLYiMI4Kfog6 nz7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GkclWeVrO8jS1MIahu2gWatY+lpfTpxiBg+TTWPbieo=; b=jmylPxJNHhObe+bEsKAmd3/De9AC6EVeDRK9rT/c+MZPi/+M5jSBM5ArhfcjDc73ZP p0/362NXbUiLiWihCFPRjxpn7B4MkPF8T/ZvD3djs6zREfl3CbmClASD9lrAZYeOMVPx NKWA9WBPpyHRUoD7fD6lhCWBsOfdE5UvXXduJsj/6i+oK9yV6pd+/vZ6Igd2qh3ns/DG QVLfKZ+29CjP9zimElbVmqUNPho+DFpuWN9gXmLbQoyp3rdzSe1S8FzNPopo2fjN/6vk vn2ZqHnnSVQTdjjqHvQUubXkoFs1E6pvBuKnHzDNA1i8fL2gd+2rmqN2iQmcZaVCBF8N 4TxQ== X-Gm-Message-State: APzg51ArSISRN95XbTtNM9BVe6L+CFSqfZR97DxkTI2EzunFSbRM7BeA +1vIREbaRAEU4cX8hg5ERA/zzd0B X-Google-Smtp-Source: ANB0Vda3D0N++FLv77DgSTqmXlY2ULys/dos18NAyeVURDRTK3RDtuXm3k8sOypWqziu+fa4pS4VtA== X-Received: by 2002:a17:902:7845:: with SMTP id e5-v6mr26510789pln.197.1535949649359; Sun, 02 Sep 2018 21:40:49 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.40.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:40:48 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:48 +0800 Message-Id: <20180903043900.28592-8-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::644 Subject: [Qemu-devel] [PATCH V12 07/19] COLO: Load dirty pages into SVM's RAM cache firstly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP We should not load PVM's state directly into SVM, because there maybe some errors happen when SVM is receving data, which will break SVM. We need to ensure receving all data before load the state into SVM. We use an extra memory to cache these data (PVM's ram). The ram cache in secondary side is initially the same as SVM/PVM's memory. And in the process of checkpoint, we cache the dirty pages of PVM into this ram cache firstly, so this ram cache always the same as PVM's memory at every checkpoint, then we flush this cached ram to SVM after we receive all PVM's state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- include/exec/ram_addr.h | 1 + migration/migration.c | 7 ++++ migration/ram.c | 83 ++++++++++++++++++++++++++++++++++++++++- migration/ram.h | 4 ++ migration/savevm.c | 2 +- 5 files changed, 94 insertions(+), 3 deletions(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index 3abb639056..9ecd911c3e 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -27,6 +27,7 @@ struct RAMBlock { struct rcu_head rcu; struct MemoryRegion *mr; uint8_t *host; + uint8_t *colo_cache; /* For colo, VM's ram cache */ ram_addr_t offset; ram_addr_t used_length; ram_addr_t max_length; diff --git a/migration/migration.c b/migration/migration.c index 950b22df63..d67fcff359 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -444,6 +444,11 @@ static void process_incoming_migration_co(void *opaque) exit(EXIT_FAILURE); } + if (colo_init_ram_cache() < 0) { + error_report("Init ram cache failed"); + exit(EXIT_FAILURE); + } + qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); mis->have_colo_incoming_thread = true; @@ -451,6 +456,8 @@ static void process_incoming_migration_co(void *opaque) /* Wait checkpoint incoming thread exit before free resource */ qemu_thread_join(&mis->colo_incoming_thread); + /* We hold the global iothread lock, so it is safe here */ + colo_release_ram_cache(); } if (ret < 0) { diff --git a/migration/ram.c b/migration/ram.c index 79c89425a3..a63315533e 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3398,6 +3398,20 @@ static inline void *host_from_ram_block_offset(RAMBlock *block, return block->host + offset; } +static inline void *colo_cache_from_block_offset(RAMBlock *block, + ram_addr_t offset) +{ + if (!offset_in_ramblock(block, offset)) { + return NULL; + } + if (!block->colo_cache) { + error_report("%s: colo_cache is NULL in block :%s", + __func__, block->idstr); + return NULL; + } + return block->colo_cache + offset; +} + /** * ram_handle_compressed: handle the zero page case * @@ -3602,6 +3616,58 @@ static void decompress_data_with_multi_threads(QEMUFile *f, qemu_mutex_unlock(&decomp_done_lock); } +/* + * colo cache: this is for secondary VM, we cache the whole + * memory of the secondary VM, it is need to hold the global lock + * to call this helper. + */ +int colo_init_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + block->colo_cache = qemu_anon_ram_alloc(block->used_length, + NULL, + false); + if (!block->colo_cache) { + error_report("%s: Can't alloc memory for COLO cache of block %s," + "size 0x" RAM_ADDR_FMT, __func__, block->idstr, + block->used_length); + goto out_locked; + } + memcpy(block->colo_cache, block->host, block->used_length); + } + rcu_read_unlock(); + return 0; + +out_locked: + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache = NULL; + } + } + + rcu_read_unlock(); + return -errno; +} + +/* It is need to hold the global lock to call this helper */ +void colo_release_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache = NULL; + } + } + rcu_read_unlock(); +} + /** * ram_load_setup: Setup RAM for migration incoming side * @@ -3618,6 +3684,7 @@ static int ram_load_setup(QEMUFile *f, void *opaque) xbzrle_load_setup(); ramblock_recv_map_init(); + return 0; } @@ -3638,6 +3705,7 @@ static int ram_load_cleanup(void *opaque) g_free(rb->receivedmap); rb->receivedmap = NULL; } + return 0; } @@ -3875,13 +3943,24 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { RAMBlock *block = ram_block_from_stream(f, flags); - host = host_from_ram_block_offset(block, addr); + /* + * After going into COLO, we should load the Page into colo_cache. + */ + if (migration_incoming_in_colo_state()) { + host = colo_cache_from_block_offset(block, addr); + } else { + host = host_from_ram_block_offset(block, addr); + } if (!host) { error_report("Illegal RAM offset " RAM_ADDR_FMT, addr); ret = -EINVAL; break; } - ramblock_recv_bitmap_set(block, host); + + if (!migration_incoming_in_colo_state()) { + ramblock_recv_bitmap_set(block, host); + } + trace_ram_load_loop(block->idstr, (uint64_t)addr, flags, host); } diff --git a/migration/ram.h b/migration/ram.h index 457bf54b8c..d009480494 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -70,4 +70,8 @@ int64_t ramblock_recv_bitmap_send(QEMUFile *file, const char *block_name); int ram_dirty_bitmap_reload(MigrationState *s, RAMBlock *rb); +/* ram cache */ +int colo_init_ram_cache(void); +void colo_release_ram_cache(void); + #endif diff --git a/migration/savevm.c b/migration/savevm.c index 0376e723b6..96db539064 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1932,7 +1932,7 @@ static int loadvm_handle_recv_bitmap(MigrationIncomingState *mis, static int loadvm_process_enable_colo(MigrationIncomingState *mis) { migration_incoming_enable_colo(); - return 0; + return colo_init_ram_cache(); } /* From patchwork Mon Sep 3 04:38:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585413 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 43522174C for ; Mon, 3 Sep 2018 04:46:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 29B24296D0 for ; Mon, 3 Sep 2018 04:46:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1AF20296D4; Mon, 3 Sep 2018 04:46:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A4A05296D0 for ; Mon, 3 Sep 2018 04:46:02 +0000 (UTC) Received: from localhost ([::1]:43389 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgkT-0006B7-Ly for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:46:01 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56256) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgfb-0008Fo-5g for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwgfa-0004n8-C4 for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:40:59 -0400 Received: from mail-pl1-x62d.google.com ([2607:f8b0:4864:20::62d]:35272) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwgfa-0004mq-4Z for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:40:58 -0400 Received: by mail-pl1-x62d.google.com with SMTP id d9-v6so8052145plr.2 for ; Sun, 02 Sep 2018 21:40:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Exd1IDdhmKszy63Ebko1M5F8PB1LeR9RlGujcgIUx5o=; b=rVaCUcdMV7iEY0a5qPvuDogt5RiDAJef1ulzJBcpNb0Qdv5usXs7RXyw0R6DyghcgD tyGNIb91d9r6mLL7PWoZBCrz+J/JP97ByvPa2p4lq4TFaCuBCs3CcL8h5TN8eLa+6hf6 X18QBEsK0SvBe+K6yyyMc6tLuHfDs/jxYxlgV0GA8OkRohE8NJgKCWLwXE6bZds/F4ne pCZoFxIlWuse09ysvj75pURRB3to3OlSM0WoSEGSQroldEvm6eVlwTSpmcw+VHFxWJUj JZGEFDC6d5us73sUdD6WCWxsHTBrwhRL5NyLRqfuAnenxdZV6PvgCWJ7aNJRs6VKl2FV Av8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Exd1IDdhmKszy63Ebko1M5F8PB1LeR9RlGujcgIUx5o=; b=bGaB2I0x7YvYs+y2LLEWIvwmODHQTijmy/M4I+2JmXRGojPabBdci/3A68yVxfgVNB 9Ox5jM15G6EAo48cT/qxtkFGD/HiyLJps6WOiKeZS5V9zPz7Lu2V7RHOIWPCt+dMO7pm RNfi/ETtPChGY+a9+vblMzboKv+aNYEmVEvGcZ+RX73J4n4U/J/KReZ4ZywD0Xao40dp DwAXGuNf4ouy/avADVH+Sc24GArp0RYas4/wu5A0Bave1NwpU/Nycatxwf49TcPFWSsI SDs5tIKZf6FzU1ImIBjkOuxEQ5GJyEeYACECrB5FBiuFTcIxF7iSZ/MGdtqYyGFALxi+ sAUg== X-Gm-Message-State: APzg51AOjmbB0sPCGxRupuGEErn6VoF6VipDLIAvRtc5qclaQOQKTc5T wozLyiS3cH1qb6PkNub41+vsMdgT X-Google-Smtp-Source: ANB0VdYIXFiLzuCueQyR7NXKs4elxM4rku24JGuA7/CwwzF4DXK9a7eqoubo3nu0/0obXGcb6Uf3KA== X-Received: by 2002:a17:902:bf46:: with SMTP id u6-v6mr26608943pls.85.1535949657035; Sun, 02 Sep 2018 21:40:57 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.40.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:40:56 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:49 +0800 Message-Id: <20180903043900.28592-9-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::62d Subject: [Qemu-devel] [PATCH V12 08/19] ram/COLO: Record the dirty pages that SVM received X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 43 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 40 insertions(+), 3 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index a63315533e..df8202216f 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3409,6 +3409,15 @@ static inline void *colo_cache_from_block_offset(RAMBlock *block, __func__, block->idstr); return NULL; } + + /* + * During colo checkpoint, we need bitmap of these migrated pages. + * It help us to decide which pages in ram cache should be flushed + * into VM's RAM later. + */ + if (!test_and_set_bit(offset >> TARGET_PAGE_BITS, block->bmap)) { + ram_state->migration_dirty_pages++; + } return block->colo_cache + offset; } @@ -3626,7 +3635,7 @@ int colo_init_ram_cache(void) RAMBlock *block; rcu_read_lock(); - QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + RAMBLOCK_FOREACH_MIGRATABLE(block) { block->colo_cache = qemu_anon_ram_alloc(block->used_length, NULL, false); @@ -3639,10 +3648,29 @@ int colo_init_ram_cache(void) memcpy(block->colo_cache, block->host, block->used_length); } rcu_read_unlock(); + /* + * Record the dirty pages that sent by PVM, we use this dirty bitmap together + * with to decide which page in cache should be flushed into SVM's RAM. Here + * we use the same name 'ram_bitmap' as for migration. + */ + if (ram_bytes_total()) { + RAMBlock *block; + + RAMBLOCK_FOREACH_MIGRATABLE(block) { + unsigned long pages = block->max_length >> TARGET_PAGE_BITS; + + block->bmap = bitmap_new(pages); + bitmap_set(block->bmap, 0, pages); + } + } + ram_state = g_new0(RAMState, 1); + ram_state->migration_dirty_pages = 0; + return 0; out_locked: - QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + + RAMBLOCK_FOREACH_MIGRATABLE(block) { if (block->colo_cache) { qemu_anon_ram_free(block->colo_cache, block->used_length); block->colo_cache = NULL; @@ -3658,14 +3686,23 @@ void colo_release_ram_cache(void) { RAMBlock *block; + RAMBLOCK_FOREACH_MIGRATABLE(block) { + g_free(block->bmap); + block->bmap = NULL; + } + rcu_read_lock(); - QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + + RAMBLOCK_FOREACH_MIGRATABLE(block) { if (block->colo_cache) { qemu_anon_ram_free(block->colo_cache, block->used_length); block->colo_cache = NULL; } } + rcu_read_unlock(); + g_free(ram_state); + ram_state = NULL; } /** From patchwork Mon Sep 3 04:38:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585417 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A70D0112B for ; Mon, 3 Sep 2018 04:46:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 96265296D0 for ; Mon, 3 Sep 2018 04:46:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8A485296D4; Mon, 3 Sep 2018 04:46:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 16B4F296D0 for ; Mon, 3 Sep 2018 04:46:39 +0000 (UTC) Received: from localhost ([::1]:43391 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgl4-0006dt-C8 for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:46:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56277) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgfi-0008LX-2v for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwgfh-0004pt-9o for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:06 -0400 Received: from mail-pf1-x42a.google.com ([2607:f8b0:4864:20::42a]:39030) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwgfh-0004pd-4b for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:05 -0400 Received: by mail-pf1-x42a.google.com with SMTP id j8-v6so8107633pff.6 for ; Sun, 02 Sep 2018 21:41:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=w+z2NMcbiosMUoJrtQY1sknLV6dwxqLnjEceWuQ/STA=; b=srJW4Tb4e/fQmu8TS0cct0LFJClLpZABZb5fdv8RgcSWiCh39UFm2zfTMb6M1OwfCL +rOvVH+8AXfnSyNOHuFQHZvQLBfFASDmedIUInkAAdlsUF1Nkug0m0HZCymQMrKMk1nD Lb+M2L1DN78r0ft2JM/b6gkLOkU8VeuJgllSoO/KTMtjVBuRROK+Q3Xy8QKo4oilSBPt kpMihQ4UXKfwXkGmWNsKvzgOLufOoktvmeJLMRwxMxy6AVu7Hjec2eqxKTfKZ++PEobp aGYpYFQfpnsKMvSaOphx2y5JF+7i3k344ezCEkAvLuWFEv+CWoiq0MFrNBdc+t9xJVnb ipIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=w+z2NMcbiosMUoJrtQY1sknLV6dwxqLnjEceWuQ/STA=; b=YR/WFFUlKjP6Ao6HS6RqvdwVQCIpQDAdwke/uzjnhZrBvVqM2MsK2ICqn20iH5kAIf OKA8bSBAQF8cnUQbweA7DWJ5h4F5yVqBmuh332J47yrDrhWXYywAoLrftZtraOPlEJHh f5w+VUMhLHnuCQRNjW6EeZu3u9IifFYfSG4kYNIK/pQpuR+SWykzOeqhn+ugL4s51tOx C/U2V0EtOAXfDaRYoYZzuXGsEVAiA1RiWz3IM0+9DCvEeU368hrS3p1ZvBWYFeffcYFM B+mc/0tYK4+p7QBy2RzEcecde63Q+rd6VtQl8fQSH9gqZr/lZSS97JOjHuGVxC8ZtdoT WJMg== X-Gm-Message-State: APzg51Dwr/k2ujVVwVyjxz/ibWqCHBy1cJb5zemneIRzyyk1wEef0iGv 1w75pBIAJmJJfxru3jRItbgi+vc1 X-Google-Smtp-Source: ANB0VdYIF48EIMZORAUnTfjRVDINfbEZK+xkMnwgovuW64UBR+y8ZM9phT02ZiAS06KUbDK5vBWczw== X-Received: by 2002:a63:2605:: with SMTP id m5-v6mr18397561pgm.225.1535949664030; Sun, 02 Sep 2018 21:41:04 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.40.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:41:03 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:50 +0800 Message-Id: <20180903043900.28592-10-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42a Subject: [Qemu-devel] [PATCH V12 09/19] COLO: Flush memory data from ram cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint time. So, the content of SVM's RAM cache will always be same with PVM's memory after checkpoint. Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way: Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's. Besides, we must ensure flush RAM cache before load device state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 37 +++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 2 files changed, 39 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index df8202216f..a478d85740 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3925,6 +3925,39 @@ static bool postcopy_is_running(void) return ps >= POSTCOPY_INCOMING_LISTENING && ps < POSTCOPY_INCOMING_END; } +/* + * Flush content of RAM cache into SVM's memory. + * Only flush the pages that be dirtied by PVM or SVM or both. + */ +static void colo_flush_ram_cache(void) +{ + RAMBlock *block = NULL; + void *dst_host; + void *src_host; + unsigned long offset = 0; + + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); + rcu_read_lock(); + block = QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset = migration_bitmap_find_dirty(ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >= block->used_length) { + offset = 0; + block = QLIST_NEXT_RCU(block, next); + } else { + migration_bitmap_clear_dirty(ram_state, block, offset); + dst_host = block->host + (offset << TARGET_PAGE_BITS); + src_host = block->colo_cache + (offset << TARGET_PAGE_BITS); + memcpy(dst_host, src_host, TARGET_PAGE_SIZE); + } + } + + rcu_read_unlock(); + trace_colo_flush_ram_cache_end(); +} + static int ram_load(QEMUFile *f, void *opaque, int version_id) { int flags = 0, ret = 0, invalid_flags = 0; @@ -4101,6 +4134,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) ret |= wait_for_decompress_done(); rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); + + if (!ret && migration_incoming_in_colo_state()) { + colo_flush_ram_cache(); + } return ret; } diff --git a/migration/trace-events b/migration/trace-events index fa0ff3f3bf..bd2d0cd25a 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -102,6 +102,8 @@ ram_dirty_bitmap_sync_start(void) "" ram_dirty_bitmap_sync_wait(void) "" ram_dirty_bitmap_sync_complete(void) "" ram_state_resume_prepare(uint64_t v) "%" PRId64 +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" # migration/migration.c await_return_path_close_on_source_close(void) "" From patchwork Mon Sep 3 04:38:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585421 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 154F413AC for ; Mon, 3 Sep 2018 04:49:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 02DB429529 for ; Mon, 3 Sep 2018 04:49:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E9E4B29533; Mon, 3 Sep 2018 04:49:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6368329529 for ; Mon, 3 Sep 2018 04:49:06 +0000 (UTC) Received: from localhost ([::1]:43403 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgnR-0000Nv-Dq for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:49:05 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56297) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgfr-0008U2-Ps for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwgfo-00052z-HA for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:15 -0400 Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]:46050) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwgfo-00051k-Ay for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:12 -0400 Received: by mail-pl1-x62a.google.com with SMTP id j8-v6so8022790pll.12 for ; Sun, 02 Sep 2018 21:41:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=7lzId/i/kSCDSFoSw243CnN2MYB6TqXdPDlb86gKDB8=; b=UO7wwPiaaeIloJ5RHLQzqdQ5GLTvNZ6hTf/eUI1POCUYLPSqUEehH29ypzc3PnNGgF AIVFKdG29Lnq4dXwkhZCy3QBwQkM+3feIg/yC+TEO45glEKuNv9uVr3FrOJW5pM/AjL8 QDVW8hcfa0GizsM84NBd0rVQ/2/rEUPbecQg5Gu/OlU9oek+qnJyKzCT8lZCZfh5Glde Bwq9TAPO1QRvyCi+W7yMbUQZoP9i/658bj5tKvpUUnpMa1KJyJ/ng7Ou2A1+g0P+XoYK fhU6ZtzFI+rPAiNJAy50RElabsNYeq4hJD12W2WKS3BX+TxcNLKTm3xaB0rUxPB7D+bt swhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=7lzId/i/kSCDSFoSw243CnN2MYB6TqXdPDlb86gKDB8=; b=MgyVrmgYzaiSt3BxrawGPEvxsv57NAnCqS7ExJKive7kWwNAH7PZUKkZRLwSniiwvY NqwXZX0/9+5TN8aGGnK1A3Wr2Xsyrl4ASykOD07YiKGSJ7bofqqBuM553xX5XzGq4xKu 6VXdybHhNJomopOdUpgQYHayevzIJjjB6SHgD28TrUzWCAM41q6iqEVSHE1NNd/EIwNZ 664rQn6pmcmt6jK8SRcd564vFUS3ddHkoCFCBHWaT7WKvxkKYr2n5d6My+TZ10YITjCi s1XCa9UWukZNLgLaaX2sJfmG72YCJFI0SXWp3HFJDz76t9jWzrSgajpdvgWjCvhWGD5U Tpxg== X-Gm-Message-State: APzg51BhwKx/jICQ68PoaUc7/FLMBfXtX65MZvmJKVitW6dV7A6xT0JS ZNrpgDhzrUo5WwVp1vV9Ee+UtxXe X-Google-Smtp-Source: ANB0VdYpa3/9c2HQfNSW9gmyz+i2GpuboTvXJkb8I2DLbjxeNz3xiOyJ+sFiPREXcm1oQFMNWovu9Q== X-Received: by 2002:a17:902:7b97:: with SMTP id w23-v6mr26599087pll.66.1535949671220; Sun, 02 Sep 2018 21:41:11 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.41.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:41:10 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:51 +0800 Message-Id: <20180903043900.28592-11-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::62a Subject: [Qemu-devel] [PATCH V12 10/19] qmp event: Add COLO_EXIT event to notify users while exited COLO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x-colo-lost-heartbeat', Users can intervene in COLO's failover work immediately. If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Eric Blake Reviewed-by: Markus Armbruster --- migration/colo.c | 31 +++++++++++++++++++++++++++++++ qapi/migration.json | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index d3163b51c8..761fbb9f78 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -28,6 +28,7 @@ #include "net/colo-compare.h" #include "net/colo.h" #include "block/block.h" +#include "qapi/qapi-events-migration.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -514,6 +515,23 @@ out: qemu_fclose(fb); } + /* + * There are only two reasons we can get here, some error happened + * or the user triggered failover. + */ + switch (failover_get_state()) { + case FAILOVER_STATUS_NONE: + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_ERROR, NULL); + break; + case FAILOVER_STATUS_REQUIRE: + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_REQUEST, NULL); + break; + default: + abort(); + } + /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); @@ -746,6 +764,19 @@ out: error_report_err(local_err); } + switch (failover_get_state()) { + case FAILOVER_STATUS_NONE: + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_ERROR, NULL); + break; + case FAILOVER_STATUS_REQUIRE: + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_REQUEST, NULL); + break; + default: + abort(); + } + if (fb) { qemu_fclose(fb); } diff --git a/qapi/migration.json b/qapi/migration.json index f62d3f9a4b..9b3b1c0c4e 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -932,6 +932,44 @@ { 'enum': 'FailoverStatus', 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } +## +# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +# +# @mode: report COLO mode when COLO exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 3.1 +# +# Example: +# +# <- { "timestamp": {"seconds": 2032141960, "microseconds": 417172}, +# "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "request" } } +# +## +{ 'event': 'COLO_EXIT', + 'data': {'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## +# @COLOExitReason: +# +# The reason for a COLO exit +# +# @none: no failover has ever happened. This can't occur in the +# COLO_EXIT event, only in the result of query-colo-status. +# +# @request: COLO exit is due to an external request +# +# @error: COLO exit is due to an internal error +# +# Since: 3.1 +## +{ 'enum': 'COLOExitReason', + 'data': [ 'none', 'request', 'error' ] } + ## # @x-colo-lost-heartbeat: # From patchwork Mon Sep 3 04:38:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585427 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8F9B313AC for ; Mon, 3 Sep 2018 04:49:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7CC8729529 for ; Mon, 3 Sep 2018 04:49:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7009D29533; Mon, 3 Sep 2018 04:49:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 1D7BE29529 for ; Mon, 3 Sep 2018 04:49:54 +0000 (UTC) Received: from localhost ([::1]:43407 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgoD-0001DN-Ag for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:49:53 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56313) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgfx-00007K-5t for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:24 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwgfw-00057I-Cq for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:21 -0400 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:38212) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwgfw-00056f-6n for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:20 -0400 Received: by mail-pg1-x541.google.com with SMTP id f4-v6so2727088pgq.5 for ; Sun, 02 Sep 2018 21:41:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=zt7NRTIN42jXMPUy8MOSlel/C2qlQ2cbwWEGVrgl+wU=; b=Crbd7woMrPr3ea8u/arW/Ed00blXXGveW2A9c30X9Rb++IAtOsI0I1KeF+eTrPCKxa xPXZtyTlgl5o+v1U3a3dH/mHFCmQ439jT4ACDKAR6DLI9oYIVdvAuywBBYDUufY30ig8 YbGd1wMN4EQwJHJUD/gjbgJ9GWaw5FzUXhgGgdTZwIdyWFfQ0/+oAsmu+pGZyaQJ72Av MEE16K9lC9/bdEUfBny44Euhy4gCmXxpHh/z6/VOjatEiBEZY67BbP0Zh9LabqutI6r1 tskb5DWsOFnTIF9L+fgpbebByoRhzpArvctBIoGOtgvV8cnG2uiay7mq0jCzsB1TTVrH 4Ehg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=zt7NRTIN42jXMPUy8MOSlel/C2qlQ2cbwWEGVrgl+wU=; b=YzYINsDg0sJvFegku2s0p1GEQwaE35A2JybgK0t4UPiRNrU4ZG2+72iguXa30ZkwOA q/gyJyUwTKpRniXjqGfKWWP9Yade/DmZdKXXgUh9LC5kfHuh37JaI7v7oS/STwRA86Hq coITFoDFjE3+yEcukvuB5REiXI1ZCjRJ5J4WS4JLEorkvLXdecHPFQKv5qowxpTi5wu7 k69/slDKmZ/FZKA2TAjmuFmB920PNr8cp765tTRgA4rQlWtDjytkc7iU1cKrU0YWMhb/ cjMDZ15gLXEpW6rHgGE0RVbRscVf4FD4e2KqIzBhxMNpWso9Hwt3NbDvk9FOFx3ApsXm 4mow== X-Gm-Message-State: APzg51AiSL2yWFZJkWlIEno37dSSZLU7rNWKMQcJUSbh9y2+E/oMYexF lctO8YVR5s39eDMZxf8X2ME8fMXz X-Google-Smtp-Source: ANB0VdY4/OY2tDfoh7KJ5hwTYY6V00YA9byrMVnmlQ3+iqJU2PZkRuygud+jrWejsav9HO4zuTHxyg== X-Received: by 2002:aa7:8713:: with SMTP id b19-v6mr27555106pfo.151.1535949679040; Sun, 02 Sep 2018 21:41:19 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.41.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:41:18 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:52 +0800 Message-Id: <20180903043900.28592-12-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::541 Subject: [Qemu-devel] [PATCH V12 11/19] qapi/migration.json: Rename COLO unknown mode to none mode. X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Zhang Chen Suggested by Markus Armbruster rename COLO unknown mode to none mode. Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Eric Blake Reviewed-by: Markus Armbruster --- migration/colo-failover.c | 2 +- migration/colo.c | 2 +- qapi/migration.json | 10 +++++----- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/migration/colo-failover.c b/migration/colo-failover.c index 0ae0c41221..4854a96c92 100644 --- a/migration/colo-failover.c +++ b/migration/colo-failover.c @@ -77,7 +77,7 @@ FailoverStatus failover_get_state(void) void qmp_x_colo_lost_heartbeat(Error **errp) { - if (get_colo_mode() == COLO_MODE_UNKNOWN) { + if (get_colo_mode() == COLO_MODE_NONE) { error_setg(errp, QERR_FEATURE_DISABLED, "colo"); return; } diff --git a/migration/colo.c b/migration/colo.c index 761fbb9f78..9107ac124a 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -160,7 +160,7 @@ COLOMode get_colo_mode(void) } else if (migration_incoming_in_colo_state()) { return COLO_MODE_SECONDARY; } else { - return COLO_MODE_UNKNOWN; + return COLO_MODE_NONE; } } diff --git a/qapi/migration.json b/qapi/migration.json index 9b3b1c0c4e..c0bc77fd8d 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -899,18 +899,18 @@ ## # @COLOMode: # -# The colo mode +# The COLO current mode. # -# @unknown: unknown mode +# @none: COLO is disabled. # -# @primary: master side +# @primary: COLO node in primary side. # -# @secondary: slave side +# @secondary: COLO node in slave side. # # Since: 2.8 ## { 'enum': 'COLOMode', - 'data': [ 'unknown', 'primary', 'secondary'] } + 'data': [ 'none', 'primary', 'secondary'] } ## # @FailoverStatus: From patchwork Mon Sep 3 04:38:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585425 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 19600174C for ; Mon, 3 Sep 2018 04:49:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 071C629529 for ; Mon, 3 Sep 2018 04:49:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EFBEE29533; Mon, 3 Sep 2018 04:49:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 96EC629529 for ; Mon, 3 Sep 2018 04:49:44 +0000 (UTC) Received: from localhost ([::1]:43405 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgo3-00013j-T8 for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:49:43 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56413) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwggL-0000VY-PK for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwggK-0005fH-Ut for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:45 -0400 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:34652) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwggK-0005eC-7N for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:44 -0400 Received: by mail-pl1-x643.google.com with SMTP id f6-v6so8059994plo.1 for ; Sun, 02 Sep 2018 21:41:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aZuHHS6luCm6QPodXWJwWtpEq349UrPUbTFjnVdJmp4=; b=R5SLWE1/FE26BoBsQW1WvowpuPEWQsq3j9uN0YhNq/dWdTtXa6nAW9yOesTJrW0B41 ZreBAAwZP4yr9YKBIBSCs1xa3+izxrH0M94+gBoVSlquLVJydzeELgxE3iZZMEzpsq4U 6pWXqIHx9AjlkA+Tn5G51EYj0Miscm/yweqXas/r0wtm7mG2GB2IHgwW/OvGAU8F2tdn lZLejRxFeuHMTG+i/fEV9ASZLxfCdSVl5qFk7CmF/6ZFfaHP8tlErTQoRJl/sPCxLxK7 g+iuvQcKiVuIG5VIbsNrgPCQdMJ/Op/27SUPgY1YeLBgAbfEMG/a8WiqHTu1m7qe5MIS iwLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aZuHHS6luCm6QPodXWJwWtpEq349UrPUbTFjnVdJmp4=; b=ZNS6GQJnibxLG/BgXVOq8dAtx6ZxihtayVe92lHQSQZg/CceADKVhc6z85y0uzq790 rsjWlfICva3nXv4zg3Dnwk2ESYibrqUO2HMxyhZogeLB52VLVgJ7K7ECCX8r2cMqjVb8 YSDLutSMEek9hfdUIaf9V1et7UheAZaRneNJX5rGc4uOqc80HPIXzEFm6xMDhrncQhai N7zDK6MogVH+LYgCXyhJA3Ve3OfS6ljrG8uZHl632KOAX2o9lpL1kBluENTjiBNAuqJe NBrVayLiuSdowDbqTsbF7r1TmkJ+YSEcKPFbDPVp4gin9ooCKf7tJQsGT5K4NB6dqcyu mFYA== X-Gm-Message-State: APzg51Cx5JPM6T/miIxNrvPUiXNXrthdLhnjAOkPIwrkDzV+oA2nIZ4c FE5leMV7n/tM34jGqVUrjCacvbB8 X-Google-Smtp-Source: ANB0VdZ2mXMws4oKczfF9Mo24E3aw2wxkCQLw9WhqFIu3oE7K7m5E3/XQBwOCJqBKudMaxV1A7bzVg== X-Received: by 2002:a17:902:7b97:: with SMTP id w23-v6mr26600425pll.66.1535949702630; Sun, 02 Sep 2018 21:41:42 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.41.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:41:41 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:53 +0800 Message-Id: <20180903043900.28592-13-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::643 Subject: [Qemu-devel] [PATCH V12 12/19] qapi: Add new command to query colo status X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Libvirt or other high level software can use this command query colo status. You can test this command like that: {'execute':'query-colo-status'} Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen --- migration/colo.c | 21 +++++++++++++++++++++ qapi/migration.json | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 9107ac124a..8c4ba373aa 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -29,6 +29,7 @@ #include "net/colo.h" #include "block/block.h" #include "qapi/qapi-events-migration.h" +#include "qapi/qmp/qerror.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -237,6 +238,26 @@ void qmp_xen_colo_do_checkpoint(Error **errp) #endif } +COLOStatus *qmp_query_colo_status(Error **errp) +{ + COLOStatus *s = g_new0(COLOStatus, 1); + + s->mode = get_colo_mode(); + + switch (failover_get_state()) { + case FAILOVER_STATUS_NONE: + s->reason = COLO_EXIT_REASON_NONE; + break; + case FAILOVER_STATUS_REQUIRE: + s->reason = COLO_EXIT_REASON_REQUEST; + break; + default: + s->reason = COLO_EXIT_REASON_ERROR; + } + + return s; +} + static void colo_send_message(QEMUFile *f, COLOMessage msg, Error **errp) { diff --git a/qapi/migration.json b/qapi/migration.json index c0bc77fd8d..840a118d11 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1283,6 +1283,38 @@ ## { 'command': 'xen-colo-do-checkpoint' } +## +# @COLOStatus: +# +# The result format for 'query-colo-status'. +# +# @mode: COLO running mode. If COLO is running, this field will return +# 'primary' or 'secondary'. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 3.0 +## +{ 'struct': 'COLOStatus', + 'data': { 'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## +# @query-colo-status: +# +# Query COLO status while the vm is running. +# +# Returns: A @COLOStatus object showing the status. +# +# Example: +# +# -> { "execute": "query-colo-status" } +# <- { "return": { "mode": "primary", "active": true, "reason": "request" } } +# +# Since: 3.0 +## +{ 'command': 'query-colo-status', + 'returns': 'COLOStatus' } + ## # @migrate-recover: # From patchwork Mon Sep 3 04:38:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585433 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5FC0413AC for ; Mon, 3 Sep 2018 04:52:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4E6FD29124 for ; Mon, 3 Sep 2018 04:52:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4155A29533; Mon, 3 Sep 2018 04:52:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9D55529124 for ; Mon, 3 Sep 2018 04:52:39 +0000 (UTC) Received: from localhost ([::1]:43423 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgqs-0003Wn-U1 for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:52:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56484) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwggW-0000gO-9s for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwggT-0005oy-RL for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:56 -0400 Received: from mail-pf1-x436.google.com ([2607:f8b0:4864:20::436]:42083) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwggT-0005oB-IY for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:53 -0400 Received: by mail-pf1-x436.google.com with SMTP id l9-v6so8103041pff.9 for ; Sun, 02 Sep 2018 21:41:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Z8FDDLV5+hp/7eXYQF3STKEdQsjIeTMTqbMlW6FViSs=; b=bD6XPUJhh1+QH1OBDrN4tStLrGJ08XHBjW6HVHnZhb/hrwRvYpTGqU2SYL2aBbDYoO 7CTQxPm9sLQygEkHs/8nWOMfQUOfROsJa2IMMxS2bOj/V1krFpjw8qsbXgGe+j8hrzOb qfl3bcseh61lqyNDlZ0fWWarvmyjhQAKgEVSQWXBMKw2T261QYF2j5f32WSanDQd+9MM oWPQ9/+BRj/akST3xaBXpq6jduXIDfKEL/nCQrUDkdJrtyZFKdc++QNHyvvGvJDyyC86 bR/4WFVQwPWzY8YIJoW+2w3wmdhaxRq6hHTZ63QQXsNANmqOHRViEcoro3KT1LqrNYK9 QaHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Z8FDDLV5+hp/7eXYQF3STKEdQsjIeTMTqbMlW6FViSs=; b=ksR4jJLhAglkBN7a165Fw5ggpKjAbVz9ThreIKS/WmIG61wJ6sghBYYdXMACqgg8eI ec4V0JQgeedXugHrqiXxPj2WMlXPV7088u1GUDzyw3GLwqAnW6aGobpxKv0wd7BSzDOa +igalALrr5RRu+gRnMeI8sMyLVMnsW4rSxKFAtU+JmuDehZpRIi/jvkO8E1RzVV5cIx8 EtLli22E7Kw9/MPgcRf3liSZUSkgaFSDahtyZLXOl2rAbpCWAnp5OneZZtwTNIC0RywI hGCsBLJuY6GMa97g6t7MWwngbqmshZOE+YafK20U8pzsAAEMIXlrzFkSI0DzC0X+YX/G 2FYA== X-Gm-Message-State: APzg51BBV5UlF8vT9n9cEJgctRQlzpjymRSdavawommDEelkyxuURda6 LiWot3wvOp2S4Uy1xGFetCHCecyf X-Google-Smtp-Source: ANB0Vdbse/gU+e1tLuXGW+Gzn9KpCT4OJ7cOdHnOSuDFh9i1Wlk4Fc3/S0VHOSeZYH9uXuqcL2IarA== X-Received: by 2002:a62:6bc5:: with SMTP id g188-v6mr23609153pfc.91.1535949712361; Sun, 02 Sep 2018 21:41:52 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.41.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:41:51 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:54 +0800 Message-Id: <20180903043900.28592-14-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::436 Subject: [Qemu-devel] [PATCH V12 13/19] savevm: split the process of different stages for loadvm/savevm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP There are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections. We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state. So we add three new helper functions: qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration. Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 41 ++++++++++++++++++++++++++++++++--------- migration/savevm.c | 36 +++++++++++++++++++++++++++++------- migration/savevm.h | 4 ++++ 3 files changed, 65 insertions(+), 16 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 8c4ba373aa..25d279decf 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -30,6 +30,7 @@ #include "block/block.h" #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" +#include "sysemu/cpus.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -401,23 +402,34 @@ static int colo_do_checkpoint_transaction(MigrationState *s, /* Disable block migration */ migrate_set_block_enabled(false, &local_err); - qemu_savevm_state_header(fb); - qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); replication_do_checkpoint_all(&local_err); if (local_err) { qemu_mutex_unlock_iothread(); goto out; } - qemu_savevm_state_complete_precopy(fb, false, false); - qemu_mutex_unlock_iothread(); - - qemu_fflush(fb); colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_err); if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* Note: device state is saved into buffer */ + ret = qemu_save_device_state(fb); + + qemu_mutex_unlock_iothread(); + if (ret < 0) { goto out; } + /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + + qemu_fflush(fb); + /* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read. @@ -635,6 +647,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t total_size; uint64_t value; Error *local_err = NULL; + int ret; rcu_register_thread(); qemu_sem_init(&mis->colo_incoming_sem, 0); @@ -708,6 +721,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + qemu_mutex_lock_iothread(); + cpu_synchronize_all_pre_loadvm(); + ret = qemu_loadvm_state_main(mis->from_src_file, mis); + qemu_mutex_unlock_iothread(); + + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } + value = colo_receive_message_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); if (local_err) { @@ -739,10 +762,10 @@ void *colo_process_incoming_thread(void *opaque) } qemu_mutex_lock_iothread(); - qemu_system_reset(SHUTDOWN_CAUSE_NONE); vmstate_loading = true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + ret = qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } diff --git a/migration/savevm.c b/migration/savevm.c index 96db539064..5636cbeae2 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1378,13 +1378,21 @@ done: return ret; } -static int qemu_save_device_state(QEMUFile *f) +void qemu_savevm_live_state(QEMUFile *f) { - SaveStateEntry *se; + /* save QEMU_VM_SECTION_END section */ + qemu_savevm_state_complete_precopy(f, true, false); + qemu_put_byte(f, QEMU_VM_EOF); +} - qemu_put_be32(f, QEMU_VM_FILE_MAGIC); - qemu_put_be32(f, QEMU_VM_FILE_VERSION); +int qemu_save_device_state(QEMUFile *f) +{ + SaveStateEntry *se; + if (!migration_in_colo_state()) { + qemu_put_be32(f, QEMU_VM_FILE_MAGIC); + qemu_put_be32(f, QEMU_VM_FILE_VERSION); + } cpu_synchronize_all_states(); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1440,8 +1448,6 @@ enum LoadVMExitCodes { LOADVM_QUIT = 1, }; -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); - /* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything @@ -2244,7 +2250,7 @@ static bool postcopy_pause_incoming(MigrationIncomingState *mis) return true; } -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) { uint8_t section_type; int ret = 0; @@ -2413,6 +2419,22 @@ int qemu_loadvm_state(QEMUFile *f) return ret; } +int qemu_load_device_state(QEMUFile *f) +{ + MigrationIncomingState *mis = migration_incoming_get_current(); + int ret; + + /* Load QEMU_VM_SECTION_FULL section */ + ret = qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to load device state: %d", ret); + return ret; + } + + cpu_synchronize_all_post_init(); + return 0; +} + int save_snapshot(const char *name, Error **errp) { BlockDriverState *bs, *bs1; diff --git a/migration/savevm.h b/migration/savevm.h index 8373c2f6bd..51a4b9caa8 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -56,8 +56,12 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint64_t *start_list, uint64_t *length_list); void qemu_savevm_send_colo_enable(QEMUFile *f); +void qemu_savevm_live_state(QEMUFile *f); +int qemu_save_device_state(QEMUFile *f); int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); +int qemu_load_device_state(QEMUFile *f); #endif From patchwork Mon Sep 3 04:38:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585429 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 23F13174C for ; Mon, 3 Sep 2018 04:52:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0D33129124 for ; Mon, 3 Sep 2018 04:52:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F372729533; Mon, 3 Sep 2018 04:52:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 8401829124 for ; Mon, 3 Sep 2018 04:52:04 +0000 (UTC) Received: from localhost ([::1]:43421 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgqJ-0002za-61 for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:52:03 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56510) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgga-0000kZ-OD for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwgga-0005v9-1Z for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:00 -0400 Received: from mail-pf1-x444.google.com ([2607:f8b0:4864:20::444]:39344) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwggZ-0005uF-R5 for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:41:59 -0400 Received: by mail-pf1-x444.google.com with SMTP id j8-v6so8108538pff.6 for ; Sun, 02 Sep 2018 21:41:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=peHDRWSd9yuYJ9BNLfmqLKO/zt2uBcH7yOGqyg6XRrc=; b=qfudZ6zJuoPH6V2fdwlNEzEKMYkpYQS4vC0aAmz2xr2UzgvACZ2qFwn7iEFwH4fneb blOceGbYlgqXedg3My/RVBKuG5KJPKQ6GBG/3C+9VwPxPJ6mfhuxcPmsvy7l/Y0IgbvL CA8RFQ7VVnV71oiDJKmk+q91BGGQBwlIpuBMKgNd2nB4d4WM1h7RdWA/lOz3bpQChDeH uT3zgzsERP7Ar8ONY6Ar8/jdhwfSU6a32ZCwyPyruu9u0dAba9YGZF+54R9jd9djKHkw F338nKb87MLySjtyb3fhj8pAM8urHNJGMkQGPz0clZzApWNq3P9n10Flva66VkMM7zAG KwDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=peHDRWSd9yuYJ9BNLfmqLKO/zt2uBcH7yOGqyg6XRrc=; b=sjJkPM3p818U5d8dBcUdVgcbfruv/+uKfNq0NrVRWErg3CgJZ+9hGKyNeSSClPWWUg bbLJlmA+s7U/ZOLrNCq0eC8zH8oRAALTG1yaH295bTIyEqKhdU407YOvtYx3ZfCg9asT v0tfh84CzxCrZ5EE16UN5ZBAJvrpwjn23oDaTaHyku53xWIZIiSASZLomKJBxQBVnB07 BUgoLPrz8Ze0+yF1ISEasELJpm3/bOyfTlnLeVw119TAbXLgGEvluWF9Vl3DSfWrfbyK xy8VirF0mDQpju6dKu8M3WrXq3GiRngkNuf0bNK9I/gjZSjCgFaCG+ycLg1mSwsyWx79 ZMWw== X-Gm-Message-State: APzg51BT/CvrkC9Mh2OHErg0AiGGKRBltdVhNgtamHXBplozKjU2kvjD eKqyDqU2v+Xftnt9YYy14kjtroMZ X-Google-Smtp-Source: ANB0VdaQep5H6irPAJ9PCD7K8+N1Dvly2gzy+i0LzfZrEqMnZrwyTjhiAVZz/wmwdqUyYNJQp2p2Lg== X-Received: by 2002:a62:d74e:: with SMTP id v14-v6mr24098957pfl.88.1535949718715; Sun, 02 Sep 2018 21:41:58 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.41.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:41:57 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:55 +0800 Message-Id: <20180903043900.28592-15-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::444 Subject: [Qemu-devel] [PATCH V12 14/19] COLO: flush host dirty ram from cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index a478d85740..739458be5b 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3665,6 +3665,7 @@ int colo_init_ram_cache(void) } ram_state = g_new0(RAMState, 1); ram_state->migration_dirty_pages = 0; + memory_global_dirty_log_start(); return 0; @@ -3686,6 +3687,7 @@ void colo_release_ram_cache(void) { RAMBlock *block; + memory_global_dirty_log_stop(); RAMBLOCK_FOREACH_MIGRATABLE(block) { g_free(block->bmap); block->bmap = NULL; @@ -3936,6 +3938,13 @@ static void colo_flush_ram_cache(void) void *src_host; unsigned long offset = 0; + memory_global_dirty_log_sync(); + rcu_read_lock(); + RAMBLOCK_FOREACH_MIGRATABLE(block) { + migration_bitmap_sync_range(ram_state, block, 0, block->used_length); + } + rcu_read_unlock(); + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); rcu_read_lock(); block = QLIST_FIRST_RCU(&ram_list.blocks); From patchwork Mon Sep 3 04:38:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585431 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A8C46174C for ; Mon, 3 Sep 2018 04:52:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 96F0529124 for ; Mon, 3 Sep 2018 04:52:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8AC2C29533; Mon, 3 Sep 2018 04:52:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 1C98029124 for ; Mon, 3 Sep 2018 04:52:30 +0000 (UTC) Received: from localhost ([::1]:43422 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgqk-0003R5-9v for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:52:30 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56545) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwggj-0000sA-1q for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwggh-000614-Cp for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:08 -0400 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]:44513) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwggh-00060l-6E for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:07 -0400 Received: by mail-pl1-x62f.google.com with SMTP id ba4-v6so8033478plb.11 for ; Sun, 02 Sep 2018 21:42:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=nOCtUKXmmcYITFocP0lsd6EtsrO318oN3y/72KNpQqw=; b=UFwvjbORyI7TaIojdfIuOZffkiEXvTZXvUt1gN1Zfa+oRS7pw8Unl74gX/8a1ogY4u 2Ab+ogbEnIwc2UnOz4LUvPHidOLI9xbptHcQm4UINfacDjm7mNqDuUvh+2VOPHYR2RCy HrdNP88uM60eHhF0VvM21m/iHmOePgB39m1bVXuf/zlUOCwo9ZtIHYPTL8zWvpMcZhPs FsDLU3wusva643mGGDiwA0DVyiq93w4akiICZ0Te1fV2l0OFnncD48UohAGWrLaZl19O w2W3Zs6OGzk8ECuw/Uu61GWtMwmjZteo/pZkagcvZQgAPdY7icuKP0aneiTwcCzrU//w UOqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=nOCtUKXmmcYITFocP0lsd6EtsrO318oN3y/72KNpQqw=; b=qPjEd+My/f0m4XjFdrS9zWb6YPJ1mpSnLoXWBrO6kYEkFOm7UVg7ajmUgBDy4tp6XK UxSyRth6tE6vIPvQo2abTGCtOpRRMkhGWEHs8IMuIYuEewes3xSNvdeP0PHHqExr3KMs 2SUD3jdXcolF9CCz6c97HTfvzqXUgEyuJ2l/bgJwzUzPNUgrD/M1KmMSPsIWhGiMip6m pIkmtfGf4jy9qr6i3jtkLFid4ct+3mmhu+umexSWlKtvIGpqV0naMVAnmL2K5QM8f3P9 rah+j72VUOhamPncMgh2mY7rlXNrFvAQ3rRdVEoTuvN19i/DitMm/FOBMa7kGAZC5kka T7MQ== X-Gm-Message-State: APzg51BET0oXC2reFIRC0O2iSZ8XajfjKGyFuJVEkrBqwGkrYCS5njK0 Dvsik/y+XrDif8PsnMBGXfFaRRTh X-Google-Smtp-Source: ANB0VdYsr+DA3nf3Jdg0xgwuR6bBERVvPykRiAj/8JDz2QQxe0NgShIlViD1QMrfSc8OmWH25LDJHA== X-Received: by 2002:a17:902:2904:: with SMTP id g4-v6mr25915331plb.70.1535949726079; Sun, 02 Sep 2018 21:42:06 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.41.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:42:05 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:56 +0800 Message-Id: <20180903043900.28592-16-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::62f Subject: [Qemu-devel] [PATCH V12 15/19] filter: Add handle_event method for NetFilterClass X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen --- include/net/filter.h | 5 +++++ net/filter.c | 17 +++++++++++++++++ net/net.c | 19 +++++++++++++++++++ 3 files changed, 41 insertions(+) diff --git a/include/net/filter.h b/include/net/filter.h index 435acd6f82..49da666ac0 100644 --- a/include/net/filter.h +++ b/include/net/filter.h @@ -38,6 +38,8 @@ typedef ssize_t (FilterReceiveIOV)(NetFilterState *nc, typedef void (FilterStatusChanged) (NetFilterState *nf, Error **errp); +typedef void (FilterHandleEvent) (NetFilterState *nf, int event, Error **errp); + typedef struct NetFilterClass { ObjectClass parent_class; @@ -45,6 +47,7 @@ typedef struct NetFilterClass { FilterSetup *setup; FilterCleanup *cleanup; FilterStatusChanged *status_changed; + FilterHandleEvent *handle_event; /* mandatory */ FilterReceiveIOV *receive_iov; } NetFilterClass; @@ -77,4 +80,6 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender, int iovcnt, void *opaque); +void colo_notify_filters_event(int event, Error **errp); + #endif /* QEMU_NET_FILTER_H */ diff --git a/net/filter.c b/net/filter.c index 2fd7d7d663..c9f9e5fa08 100644 --- a/net/filter.c +++ b/net/filter.c @@ -17,6 +17,8 @@ #include "net/vhost_net.h" #include "qom/object_interfaces.h" #include "qemu/iov.h" +#include "net/colo.h" +#include "migration/colo.h" static inline bool qemu_can_skip_netfilter(NetFilterState *nf) { @@ -245,11 +247,26 @@ static void netfilter_finalize(Object *obj) g_free(nf->netdev_id); } +static void default_handle_event(NetFilterState *nf, int event, Error **errp) +{ + switch (event) { + case COLO_EVENT_CHECKPOINT: + break; + case COLO_EVENT_FAILOVER: + object_property_set_str(OBJECT(nf), "off", "status", errp); + break; + default: + break; + } +} + static void netfilter_class_init(ObjectClass *oc, void *data) { UserCreatableClass *ucc = USER_CREATABLE_CLASS(oc); + NetFilterClass *nfc = NETFILTER_CLASS(oc); ucc->complete = netfilter_complete; + nfc->handle_event = default_handle_event; } static const TypeInfo netfilter_info = { diff --git a/net/net.c b/net/net.c index 2a3133990c..fd8efebfdb 100644 --- a/net/net.c +++ b/net/net.c @@ -1331,6 +1331,25 @@ void hmp_info_network(Monitor *mon, const QDict *qdict) } } +void colo_notify_filters_event(int event, Error **errp) +{ + NetClientState *nc; + NetFilterState *nf; + NetFilterClass *nfc = NULL; + Error *local_err = NULL; + + QTAILQ_FOREACH(nc, &net_clients, next) { + QTAILQ_FOREACH(nf, &nc->filters, next) { + nfc = NETFILTER_GET_CLASS(OBJECT(nf)); + nfc->handle_event(nf, event, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + } + } +} + void qmp_set_link(const char *name, bool up, Error **errp) { NetClientState *ncs[MAX_QUEUE_NUM]; From patchwork Mon Sep 3 04:38:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585437 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C4DDF13AC for ; Mon, 3 Sep 2018 04:54:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B105F29515 for ; Mon, 3 Sep 2018 04:54:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A332B29532; Mon, 3 Sep 2018 04:54:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 0932B29515 for ; Mon, 3 Sep 2018 04:54:48 +0000 (UTC) Received: from localhost ([::1]:43433 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgsx-0005Hp-B0 for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:54:47 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56602) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwggx-0001gF-8u for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:24 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwggw-0006AI-9q for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:23 -0400 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:34654) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwggw-0006A6-2t for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:22 -0400 Received: by mail-pl1-x643.google.com with SMTP id f6-v6so8060560plo.1 for ; Sun, 02 Sep 2018 21:42:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BDQEFWqRZlVlpIbknzXKVcTZfv0kyu1oe5H4msXUAzY=; b=jgNfwZc69p18i2arO2tU7HeQBaXn55S+UnPAqRkNAyT4BuT1uG05EzYWNAbrlHoxbL RDZmu6sEArTqPYrOpSuT52jCkzfU6GRFFn1aSqgzoxGy4UWL4I9thbgXs8T8JFAbE0zD naa3ggYl8iC9GspUnk18D8oxCqYilym7lkANYI1KN1GW5kpLrTKX5pYebtzLiARPglVS 6EF/ZMkN/3Sw5nbG1jv4XTuoKE5KnZ6fVzytCI/syCoPIabEmtUO41ikyAXTfiaO+E5h uMNtS3le2p2LxwKWJMVqCKKqjHLBkZZgu5FJH4NbE3ZMtOAZayXac8IXGDnLEpgGk4H8 Viqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BDQEFWqRZlVlpIbknzXKVcTZfv0kyu1oe5H4msXUAzY=; b=XqDSYq6T9usYEpUvx01+Ac/Xf0GpLpAc6fCxDhVZlDO0huJEioOd3WyiK6dvl9Gso2 aPPobw8a+2Np8kEcOpf3pVSZyXnGR/gDc2FpLgBF/Eumf75qfz7yfe73zhCBb7kjSxOQ owTDR7wbHZMfpxM6I5/WM9IoGppFZj4nz2HcN8qVRB+XtggbVzYhimFF2uAUPSCfx0xC 17Bj9tAZheT0tAVUvjEnPwtwQbQDNIP2CzWuT1osio+Tx8s3QCiE4GDM4sI7cxDi/wAW R2N4uGylA+Ep/OgUh8VOhS9lhHSzVBT/P2cYOseQUuLf8ziBiHxvSyQUOVYRv+Qy7bn8 zryA== X-Gm-Message-State: APzg51CSNQ9qghx6UQgxsifAevr1VWWDFxCpd4EHLkkll9REFCnchgyb rTgHA1f4/cmQK8f+OE5rPcqr+gUU X-Google-Smtp-Source: ANB0VdbUpO0qCJLOrRaSnlKchLX+EJYdae52lKa8zbBDvGZnwTqojNnl38/659lxiKvie1BFPKnSJA== X-Received: by 2002:a17:902:1a9:: with SMTP id b38-v6mr26584889plb.89.1535949741036; Sun, 02 Sep 2018 21:42:21 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.42.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:42:20 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:57 +0800 Message-Id: <20180903043900.28592-17-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::643 Subject: [Qemu-devel] [PATCH V12 16/19] filter-rewriter: handle checkpoint and failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP After one round of checkpoint, the states between PVM and SVM become consistent, so it is unnecessary to adjust the sequence of net packets for old connections, besides, while failover happens, filter-rewriter will into failover mode that needn't handle the new TCP connection. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen --- net/colo-compare.c | 12 ++++----- net/colo.c | 8 ++++++ net/colo.h | 2 ++ net/filter-rewriter.c | 57 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 73 insertions(+), 6 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index 426eab5973..a39191d522 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -116,6 +116,12 @@ enum { SECONDARY_IN, }; +static void colo_compare_inconsistency_notify(void) +{ + notifier_list_notify(&colo_compare_notifiers, + migrate_get_current()); +} + static int compare_chr_send(CompareState *s, const uint8_t *buf, uint32_t size, @@ -562,12 +568,6 @@ static int colo_old_packet_check_one(Packet *pkt, int64_t *check_time) } } -static void colo_compare_inconsistency_notify(void) -{ - notifier_list_notify(&colo_compare_notifiers, - migrate_get_current()); -} - void colo_compare_register_notifier(Notifier *notify) { notifier_list_add(&colo_compare_notifiers, notify); diff --git a/net/colo.c b/net/colo.c index 97c8fc928f..49176bf07b 100644 --- a/net/colo.c +++ b/net/colo.c @@ -221,3 +221,11 @@ Connection *connection_get(GHashTable *connection_track_table, return conn; } + +bool connection_has_tracked(GHashTable *connection_track_table, + ConnectionKey *key) +{ + Connection *conn = g_hash_table_lookup(connection_track_table, key); + + return conn ? true : false; +} diff --git a/net/colo.h b/net/colo.h index 0277e0e9ba..11c5226488 100644 --- a/net/colo.h +++ b/net/colo.h @@ -98,6 +98,8 @@ void connection_destroy(void *opaque); Connection *connection_get(GHashTable *connection_track_table, ConnectionKey *key, GQueue *conn_list); +bool connection_has_tracked(GHashTable *connection_track_table, + ConnectionKey *key); void connection_hashtable_reset(GHashTable *connection_track_table); Packet *packet_new(const void *data, int size, int vnet_hdr_len); void packet_destroy(void *opaque, void *user_data); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index f18a71bf2e..359494fc72 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -20,11 +20,15 @@ #include "qemu/main-loop.h" #include "qemu/iov.h" #include "net/checksum.h" +#include "net/colo.h" +#include "migration/colo.h" #define FILTER_COLO_REWRITER(obj) \ OBJECT_CHECK(RewriterState, (obj), TYPE_FILTER_REWRITER) #define TYPE_FILTER_REWRITER "filter-rewriter" +#define FAILOVER_MODE_ON true +#define FAILOVER_MODE_OFF false typedef struct RewriterState { NetFilterState parent_obj; @@ -32,8 +36,14 @@ typedef struct RewriterState { /* hashtable to save connection */ GHashTable *connection_track_table; bool vnet_hdr; + bool failover_mode; } RewriterState; +static void filter_rewriter_failover_mode(RewriterState *s) +{ + s->failover_mode = FAILOVER_MODE_ON; +} + static void filter_rewriter_flush(NetFilterState *nf) { RewriterState *s = FILTER_COLO_REWRITER(nf); @@ -269,6 +279,13 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, */ reverse_connection_key(&key); } + + /* After failover we needn't change new TCP packet */ + if (s->failover_mode && + !connection_has_tracked(s->connection_track_table, &key)) { + goto out; + } + conn = connection_get(s->connection_track_table, &key, NULL); @@ -302,11 +319,49 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, } } +out: packet_destroy(pkt, NULL); pkt = NULL; return 0; } +static void reset_seq_offset(gpointer key, gpointer value, gpointer user_data) +{ + Connection *conn = (Connection *)value; + + conn->offset = 0; +} + +static gboolean offset_is_nonzero(gpointer key, + gpointer value, + gpointer user_data) +{ + Connection *conn = (Connection *)value; + + return conn->offset ? true : false; +} + +static void colo_rewriter_handle_event(NetFilterState *nf, int event, + Error **errp) +{ + RewriterState *rs = FILTER_COLO_REWRITER(nf); + + switch (event) { + case COLO_EVENT_CHECKPOINT: + g_hash_table_foreach(rs->connection_track_table, + reset_seq_offset, NULL); + break; + case COLO_EVENT_FAILOVER: + if (!g_hash_table_find(rs->connection_track_table, + offset_is_nonzero, NULL)) { + filter_rewriter_failover_mode(rs); + } + break; + default: + break; + } +} + static void colo_rewriter_cleanup(NetFilterState *nf) { RewriterState *s = FILTER_COLO_REWRITER(nf); @@ -350,6 +405,7 @@ static void filter_rewriter_init(Object *obj) RewriterState *s = FILTER_COLO_REWRITER(obj); s->vnet_hdr = false; + s->failover_mode = FAILOVER_MODE_OFF; object_property_add_bool(obj, "vnet_hdr_support", filter_rewriter_get_vnet_hdr, filter_rewriter_set_vnet_hdr, NULL); @@ -362,6 +418,7 @@ static void colo_rewriter_class_init(ObjectClass *oc, void *data) nfc->setup = colo_rewriter_setup; nfc->cleanup = colo_rewriter_cleanup; nfc->receive_iov = colo_rewriter_receive_iov; + nfc->handle_event = colo_rewriter_handle_event; } static const TypeInfo colo_rewriter_info = { From patchwork Mon Sep 3 04:38:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585439 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 53E3913AC for ; Mon, 3 Sep 2018 04:54:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 41ADC29515 for ; Mon, 3 Sep 2018 04:54:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 350E329532; Mon, 3 Sep 2018 04:54:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id DC8FA29515 for ; Mon, 3 Sep 2018 04:54:54 +0000 (UTC) Received: from localhost ([::1]:43434 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgt4-0005O0-4d for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:54:54 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56615) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgh6-0002Lg-GY for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwgh2-0006Bo-Jl for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:32 -0400 Received: from mail-pg1-x543.google.com ([2607:f8b0:4864:20::543]:46322) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwgh2-0006Bc-DE for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:28 -0400 Received: by mail-pg1-x543.google.com with SMTP id b129-v6so8003135pga.13 for ; Sun, 02 Sep 2018 21:42:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=SPbpnEYkE02fEANS6dtUBnvhJ9fE2CIuRiZeuzr7BqY=; b=nBFziBr4qu6D+JR8PP1nexuyyGlZOxewiMenOORpB1Hmn8KWPC2iLTVIhzPYE2ushq dCizHKZ1ktaH+7BI9trBf8ELuHuBUl775hyjcJ4YfmWbA9yL6PBnpkWeUF8gmCsNFhkq wzOpspHVCTYbm13dexQMAmJKDqviExO65CRWdtvNHciqrhipnzQPG3ZxiOVZeog+o4Um xH3tQOgrRK/N7EtSDBJdlIyjL+2Vw1dB5g4TRlTQvRFtP01Dpm+OWwOXmS+0e5xrXKBI KoHU7uijQMJGxGXtp100fcpOp0ntZuNt+EogEJrM0hCDxsthYVz4Tv8GaQD/MRtKGPXv TsOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=SPbpnEYkE02fEANS6dtUBnvhJ9fE2CIuRiZeuzr7BqY=; b=MkWiv+1PuPirkLEFqvM6VfGQSEr5hQaPVsScvudMGXtYNFgjeVugQOveQzkaKbeBwy FzuLRvAw8TW6rwLY/21pyNzBqq1D10enf2uaDBemuUEo0cJ17ZShxYQZ0XaVb3S/0YmF DfB5WzmHVMag7BFgas1hEAfnbqoPjfPuWkVtzOON7YRWCKxPiNda2BISsHFAbwP77mmb yLuQXEaUIygObxbokofEIaul8Fo65hlDX799492x7Dxe+1bFFNXG5X4cgZN+EA1vI56u /EFRaySLMYJ2m42HdlkpPtqzg+aF/flKPUX3ulWStcDjYwUo4vBpRfNNcsiMpQZTPVYb 7gUw== X-Gm-Message-State: APzg51D+WXxgAtWloa5PXOT9jpTgtaVvgAD3Czj5pQI/+v88b6Iz2K2p 7mqYy6rBUU7U9+o+P5Y31TKqc883 X-Google-Smtp-Source: ANB0VdbmwB8jNPXrcxuRJsQBp3vAizBUY/HNlCm4jqNwJc5a5XL09zfNgzeJatjV2cA/cUNK62UzJw== X-Received: by 2002:a63:fc07:: with SMTP id j7-v6mr22387214pgi.1.1535949747346; Sun, 02 Sep 2018 21:42:27 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.42.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:42:26 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:58 +0800 Message-Id: <20180903043900.28592-18-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::543 Subject: [Qemu-devel] [PATCH V12 17/19] COLO: notify net filters about checkpoint/failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang Notify all net filters about the checkpoint and failover event. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 25d279decf..365e913e51 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -31,6 +31,7 @@ #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" #include "sysemu/cpus.h" +#include "net/filter.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -83,6 +84,12 @@ static void secondary_vm_do_failover(void) error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } + if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side"); /* recover runstate to normal migration finish state */ @@ -782,6 +789,14 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading = false; vm_start(); trace_colo_vm_state_change("stop", "run"); From patchwork Mon Sep 3 04:38:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585435 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B910F920 for ; Mon, 3 Sep 2018 04:54:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A58E829515 for ; Mon, 3 Sep 2018 04:54:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9861D29532; Mon, 3 Sep 2018 04:54:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 50AF929515 for ; Mon, 3 Sep 2018 04:54:18 +0000 (UTC) Received: from localhost ([::1]:43432 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgsT-0004vb-Ja for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:54:17 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56636) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwghC-0002kp-7F for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwghA-0006Dl-BE for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:37 -0400 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:37546) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwgh8-0006DB-WE for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:35 -0400 Received: by mail-pl1-x644.google.com with SMTP id f1-v6so1289490plt.4 for ; Sun, 02 Sep 2018 21:42:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ydWWVJm8+lt2One58Iz1g6Mmsyo7LHeIHX9sAAhSyw4=; b=fSUDmyFH/Z9gplw3RPsMVlIhNowdZpN72S83GtNR98bcjG368P9FlVnLr5WCbbIVj0 OFOZ0qW5R8lF6TYSRR+1XAX+VdDA9I7zBeAWaGoHsAOe/uLe7wW81xL0c5nWj3fvUYzD UvG/yqchy/i5OwDIMS1MOTj7+2fPd7GsSmooGyFPMq1Aj4v8oB7WTJgaLaxTkoVronJC AlstbwrFfhn6+MfS7a7u/aDEzgda/jFU4LToE0rwwUyN9yNIaVfxycNTDmELKwlrG2LK avWHhDGHI4V1CxygRb4QhVnGsgQWdsyb5k3srexTy5IMiEqiPm3/ajPyg7nBeN7ddIhj xpcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ydWWVJm8+lt2One58Iz1g6Mmsyo7LHeIHX9sAAhSyw4=; b=O+HA3TOHrDfq7ZOkecYyQjmNtC8bsapfFy1GmLSQPKwf0aZtHP4b3Rg2JmvLj0DwYG 1EzPaYbxoPmlWmcGIoGkcoicyu0rl91cme/uRkLHwqNqUTeDgMi0PTsMSzKhaLygnv6a W/yGwQlAp/FySVTLt6jELu4QIa/UoFH+qBMs9tfT336sqjqvxhl7F7arYd1Q7w20ywy1 jvaivkSEuG8NiywB86GvryhA6dbq3tKTPgFaoTnaC+yo4E+ME1ogRDXMgmwyVBfkdJkL TS80+OdCYKXa58P1Bn1oOb5pfoTn9Vppx0j2x+swAVe/0s0f1GyFp+tSdUDE18mWtX6I SjxA== X-Gm-Message-State: APzg51CDaoJiAx9p6Wv7c34+OpNFJ0yM7T0wgk4yZVRliiskfTovVdeM Cs5XTg8irhzKaJnqf585mLAQRdx5 X-Google-Smtp-Source: ANB0VdZZ5WJp7psgJNuxELJoLsGTlfffywwZN/ysOnwMj585yjM35oLGA6Sx0AV8LrKa8fJwyTa8pQ== X-Received: by 2002:a17:902:6946:: with SMTP id k6-v6mr26638524plt.268.1535949753340; Sun, 02 Sep 2018 21:42:33 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.42.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:42:32 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:38:59 +0800 Message-Id: <20180903043900.28592-19-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::644 Subject: [Qemu-devel] [PATCH V12 18/19] COLO: quick failover process by kick COLO thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 365e913e51..3a3efb916b 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -131,6 +131,11 @@ static void primary_vm_do_failover(void) migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + /* + * kick COLO thread which might wait at + * qemu_sem_wait(&s->colo_checkpoint_sem). + */ + colo_checkpoint_notify(migrate_get_current()); /* * Wake up COLO thread which may blocked in recv() or send(), @@ -539,6 +544,9 @@ static void colo_process_checkpoint(MigrationState *s) qemu_sem_wait(&s->colo_checkpoint_sem); + if (s->state != MIGRATION_STATUS_COLO) { + goto out; + } ret = colo_do_checkpoint_transaction(s, bioc, fb); if (ret < 0) { goto out; From patchwork Mon Sep 3 04:39:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10585423 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 91449174C for ; Mon, 3 Sep 2018 04:49:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D77D29529 for ; Mon, 3 Sep 2018 04:49:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6F4D829533; Mon, 3 Sep 2018 04:49:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 1F1AF29529 for ; Mon, 3 Sep 2018 04:49:28 +0000 (UTC) Received: from localhost ([::1]:43404 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwgnn-0000mB-BG for patchwork-qemu-devel@patchwork.kernel.org; Mon, 03 Sep 2018 00:49:27 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56667) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fwghI-0003EU-Tn for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fwghI-0006Fm-3U for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:44 -0400 Received: from mail-pg1-x52b.google.com ([2607:f8b0:4864:20::52b]:33336) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fwghG-0006Ev-4A for qemu-devel@nongnu.org; Mon, 03 Sep 2018 00:42:43 -0400 Received: by mail-pg1-x52b.google.com with SMTP id s7-v6so999219pgc.0 for ; Sun, 02 Sep 2018 21:42:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=06HycV6v4ZN/DheKI9cIvbvXxU2TRCnL8VF/KmZnq/U=; b=ompHGMntM8YLxhdvafMIstaTIDp6nhh56ZvARjy7LX/Nw6fVPZjHNfZb7t3zoRZsLL TqO/OA0FXp6t+WZrVru1FgBGYtN/fuLGsEe0pJNPK5YwCLLbbVuBJVCj9rcz1hEaQb/B LE6MQnTFuxPZWkbckB4oOzvtS5xWQ8aiitJvakTiHvmHjF5R21NXWn8z6tRfHPXaSpGb c8F1RgSTWa0E66USC1WD3F3RWaRD3/uS//tePWazSIypSnGNCiPAzBWkHDflXCy0V+6+ 7ZjEFp7GE4V9pUAqY+kY54UQYG/D//B6Lg/XD5+IRdrY4gG91H5KfhqckwFlU9RKP3Hi szkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=06HycV6v4ZN/DheKI9cIvbvXxU2TRCnL8VF/KmZnq/U=; b=qeLrqJWRKyjfTA39onf2b38XYONATP3wYp/wQQ8xi+bPkyodHO/o/zjCq3gysc24aV TfT5MNbYqnwbYdxT8NAqAaR5TGA/5MdAgXhmwgGK/nMnqqaTenkgQmh4fQUJYlbWVlaa azOlOVs/UVKRQ+aIv9iW1NuRp7cp8oi6N2AQiNLgDu1MLLQeaZYWEvF011Z/NAIxuLZj PgAVSc2TJGvzIrZr0RIoury0bjmN+EoAtsNUYl8x9GbKbw5QVfuJ+/FJO1mCGM8WRT+B XNadgOqXAQGe9Es9OPrg0OV6znpdPjoc8H+6Jm6vPUqGmHWhTUr/dMSqJg7SGaM5dc9F g0qw== X-Gm-Message-State: APzg51BPeb1Hfwks4mpuvZB7lHj8cqGrdF3GRSDiboL4OkoLIq+nyzSG 4F8uaA+OJ1gVury/xZJ3TeBk2mIg X-Google-Smtp-Source: ANB0VdYfFa0ZkWiIFAiPoYYiIAMT3p5rHcyOJd8fKxkW0tcF6egNLJzLx609XFVGdEQWVSAaBXALqA== X-Received: by 2002:a62:a216:: with SMTP id m22-v6mr27289918pff.163.1535949759794; Sun, 02 Sep 2018 21:42:39 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id g6-v6sm32738308pfb.11.2018.09.02.21.42.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 21:42:38 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 3 Sep 2018 12:39:00 +0800 Message-Id: <20180903043900.28592-20-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903043900.28592-1-zhangckid@gmail.com> References: <20180903043900.28592-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::52b Subject: [Qemu-devel] [PATCH V12 19/19] docs: Add COLO status diagram to COLO-FT.txt X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Zhang Chen This diagram make user better understand COLO. Suggested by Markus Armbruster. Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen --- docs/COLO-FT.txt | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/docs/COLO-FT.txt b/docs/COLO-FT.txt index d7c7dcda8f..d5007895d1 100644 --- a/docs/COLO-FT.txt +++ b/docs/COLO-FT.txt @@ -110,6 +110,40 @@ Note: HeartBeat has not been implemented yet, so you need to trigger failover process by using 'x-colo-lost-heartbeat' command. +== COLO operation status == + ++-----------------+ +| | +| Start COLO | +| | ++--------+--------+ + | + | Main qmp command: + | migrate-set-capabilities with x-colo + | migrate + | + v ++--------+--------+ +| | +| COLO running | +| | ++--------+--------+ + | + | Main qmp command: + | x-colo-lost-heartbeat + | or + | some error happened + v ++--------+--------+ +| | send qmp event: +| COLO failover | COLO_EXIT +| | ++-----------------+ + +COLO use the qmp command switching and report operation status. +The diagram just write the main qmp command, you can get the detail +in test procedure. + == Test procedure == 1. Startup qemu Primary: