From patchwork Sat Aug 11 20:59:05 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563553 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 67A581510 for ; Sat, 11 Aug 2018 21:05:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5863D2842E for ; Sat, 11 Aug 2018 21:05:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 48D3C297B6; Sat, 11 Aug 2018 21:05:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 931D72842E for ; Sat, 11 Aug 2018 21:05:45 +0000 (UTC) Received: from localhost ([::1]:33199 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob4y-0001Hv-Gz for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:05:44 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48023) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1foaz9-0003kA-2V for qemu-devel@nongnu.org; Sat, 11 Aug 2018 16:59:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1foaz7-0000vP-Ij for qemu-devel@nongnu.org; Sat, 11 Aug 2018 16:59:43 -0400 Received: from mail-pf1-x441.google.com ([2607:f8b0:4864:20::441]:35069) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1foaz7-0000uq-AL for qemu-devel@nongnu.org; Sat, 11 Aug 2018 16:59:41 -0400 Received: by mail-pf1-x441.google.com with SMTP id p12-v6so6016283pfh.2 for ; Sat, 11 Aug 2018 13:59:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=2Ar+g9n4fDvTxeogf86aS/At5lIDSUtJ1XnU+jBR9vU=; b=Ut0lwdpnBjL7Wy/QgLsINaZsla9LuKcE9zfnfsI4ohIJ/ymv45TN+2MrHxUyfPJ2iN PyK2RMJooYqFwMQctlhC8TUpRsYBY6Va04dVVijRzHcrINAayUZtceVIPtGSA8RltTd5 m2nV/yHlwWIfD+87aMo+5Bkp4jawIk1PwNoPueytKwTWLpmqZrnNOwoPkSv7u1si0vDN SRhl4I8qYMHaXyF8W5Qrsmw1ecUgBXKmgJlUByvk9ICE67siGDBdTZPSZIeXfF/q64fj axdrFnZAX1coxx1JLhkhGOWFH5L10dF/9QsoqJSQD6Z/uLZfBf6HMBnLnTRX0BjnP8db 2UZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=2Ar+g9n4fDvTxeogf86aS/At5lIDSUtJ1XnU+jBR9vU=; b=hysX1+aN8Q4PL/AqLFNL3hJIYaDa0RVrPo2AFeY60xn1JITFiJIFzO5ar/7yRd7wMq XLBgUobvreRJwIPh7gFqUV20dasFTW205Vl2HTxWBy6qX0BhB/AOI+dQy4+B2RERoca9 AD26zP2i0Qgh1KEOit029gDGBWYLZmXt8qH8IkOSMpLY2QdTTBao1E9b6mKEHWJ/0mFk q0J9D9n6ZxtRsDG/1Mf5UUuqtQtz8d4R2+/l4fmjfVBgN28Is0Opc3QdF5X6anjMvL9s S2q/LgdjUJ9Qm5m61zyP4xKSKJqN7Y3zKvpLNbRQjnmj0HaeU+QsL8BuCs2aNc+JkH2D Wy+Q== X-Gm-Message-State: AOUpUlFvxKffdgDXQXIsLFATjYzcwvpkB/8+G5ZakSoCGP3DudSqixDJ PSjN5OpcfJbttK3icfNzRBXBhz7x X-Google-Smtp-Source: AA+uWPwGsop+izCRk6pdPsy2Sl+KukeAMnE8NUd8xHHMfh/cwPdxiOV0ADBqz/w/JWfkgnel4ZKNbQ== X-Received: by 2002:a62:586:: with SMTP id 128-v6mr12684926pff.80.1534021179892; Sat, 11 Aug 2018 13:59:39 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.13.59.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 13:59:38 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:05 +0800 Message-Id: <20180811205924.4113-2-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::441 Subject: [Qemu-devel] [PATCH V11 01/20] filter-rewriter: Add TCP state machine and fix memory leak in connection_track_table X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP We add almost full TCP state machine in filter-rewriter, except TCPS_LISTEN and some simplify in VM active close FIN states. After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak. Let't track the state of net connection, if it is closed, its related resources will be cleared up. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen --- net/colo.c | 2 +- net/colo.h | 9 ++-- net/filter-rewriter.c | 105 ++++++++++++++++++++++++++++++++++++++---- 3 files changed, 100 insertions(+), 16 deletions(-) diff --git a/net/colo.c b/net/colo.c index 6dda4ed66e..97c8fc928f 100644 --- a/net/colo.c +++ b/net/colo.c @@ -137,7 +137,7 @@ Connection *connection_new(ConnectionKey *key) conn->ip_proto = key->ip_proto; conn->processing = false; conn->offset = 0; - conn->syn_flag = 0; + conn->tcp_state = TCPS_CLOSED; conn->pack = 0; conn->sack = 0; g_queue_init(&conn->primary_list); diff --git a/net/colo.h b/net/colo.h index da6c36dcf7..0277e0e9ba 100644 --- a/net/colo.h +++ b/net/colo.h @@ -18,6 +18,7 @@ #include "slirp/slirp.h" #include "qemu/jhash.h" #include "qemu/timer.h" +#include "slirp/tcp.h" #define HASHTABLE_MAX_SIZE 16384 @@ -81,11 +82,9 @@ typedef struct Connection { uint32_t sack; /* offset = secondary_seq - primary_seq */ tcp_seq offset; - /* - * we use this flag update offset func - * run once in independent tcp connection - */ - int syn_flag; + + int tcp_state; /* TCP FSM state */ + tcp_seq fin_ack_seq; /* the seq of 'fin=1,ack=1' */ } Connection; uint32_t connection_key_hash(const void *opaque); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index f584e4eba4..f18a71bf2e 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -59,9 +59,9 @@ static int is_tcp_packet(Packet *pkt) } /* handle tcp packet from primary guest */ -static int handle_primary_tcp_pkt(NetFilterState *nf, +static int handle_primary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; @@ -74,23 +74,28 @@ static int handle_primary_tcp_pkt(NetFilterState *nf, trace_colo_filter_rewriter_conn_offset(conn->offset); } + if (((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == (TH_ACK | TH_SYN)) && + conn->tcp_state == TCPS_SYN_SENT) { + conn->tcp_state = TCPS_ESTABLISHED; + } + if (((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == TH_SYN)) { /* * we use this flag update offset func * run once in independent tcp connection */ - conn->syn_flag = 1; + conn->tcp_state = TCPS_SYN_RECEIVED; } if (((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == TH_ACK)) { - if (conn->syn_flag) { + if (conn->tcp_state == TCPS_SYN_RECEIVED) { /* * offset = secondary_seq - primary seq * ack packet sent by guest from primary node, * so we use th_ack - 1 get primary_seq */ conn->offset -= (ntohl(tcp_pkt->th_ack) - 1); - conn->syn_flag = 0; + conn->tcp_state = TCPS_ESTABLISHED; } if (conn->offset) { /* handle packets to the secondary from the primary */ @@ -99,15 +104,63 @@ static int handle_primary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_len, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 1: + * Step 3: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'ack=1' packets from client side, it acks 'fin=1, ack=1' + * packet from server side. From this point, we can ensure that there + * will be no packets in the connection, except that, some errors + * happen between the path of 'filter object' and vNIC, if this rare + * case really happen, we can still create a new connection, + * So it is safe to remove the connection from connection_track_table. + * + */ + if ((conn->tcp_state == TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) == (conn->fin_ack_seq + 1))) { + conn->tcp_state = TCPS_CLOSED; + g_hash_table_remove(rf->connection_track_table, key); + } + } + + if ((tcp_pkt->th_flags & TH_FIN) == TH_FIN) { + /* + * Case 1: + * Step 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. We will into CLOSE_WAIT status. + */ + if (conn->tcp_state == TCPS_ESTABLISHED) { + conn->tcp_state = TCPS_CLOSE_WAIT; + } + + /* + * Case 2: + * Step 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. We will into CLOSE_WAIT status. + * We simplify the TCPS_FIN_WAIT_2, TCPS_TIME_WAIT and CLOSING status. + */ + if (conn->tcp_state == TCPS_FIN_WAIT_1) { + conn->tcp_state = TCPS_TIME_WAIT; + /* + * For simplify implementation, we needn't wait 2MSL time + * in filter rewriter. + */ + conn->tcp_state = TCPS_CLOSED; + g_hash_table_remove(rf->connection_track_table, key); + } } return 0; } /* handle tcp packet from secondary guest */ -static int handle_secondary_tcp_pkt(NetFilterState *nf, +static int handle_secondary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; @@ -121,7 +174,8 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, trace_colo_filter_rewriter_conn_offset(conn->offset); } - if (((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == (TH_ACK | TH_SYN))) { + if (conn->tcp_state == TCPS_SYN_RECEIVED && + ((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == (TH_ACK | TH_SYN))) { /* * save offset = secondary_seq and then * in handle_primary_tcp_pkt make offset @@ -130,6 +184,12 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, conn->offset = ntohl(tcp_pkt->th_seq); } + /* VM active connect */ + if (conn->tcp_state == TCPS_CLOSED && + ((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == TH_SYN)) { + conn->tcp_state = TCPS_SYN_SENT; + } + if ((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == TH_ACK) { /* Only need to adjust seq while offset is Non-zero */ if (conn->offset) { @@ -140,6 +200,31 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, pkt->size - pkt->vnet_hdr_len); } } + /* + * Case 1: + * Step 2: + * The *server* side of this connect is VM, *client* tries to close + * the connection. In this step we will into LAST_ACK status. + * + * We got 'fin=1, ack=1' packet from server side, we need to + * record the seq of 'fin=1, ack=1' packet. + */ + if (conn->tcp_state == TCPS_CLOSE_WAIT && + (tcp_pkt->th_flags & (TH_ACK | TH_FIN)) == (TH_ACK | TH_FIN)) { + conn->fin_ack_seq = ntohl(tcp_pkt->th_seq); + conn->tcp_state = TCPS_LAST_ACK; + } + + /* + * Case 2: + * Step 1: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + */ + if (conn->tcp_state == TCPS_ESTABLISHED && + (tcp_pkt->th_flags & (TH_ACK | TH_FIN)) == TH_FIN) { + conn->tcp_state = TCPS_FIN_WAIT_1; + } return 0; } @@ -190,7 +275,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, if (sender == nf->netdev) { /* NET_FILTER_DIRECTION_TX */ - if (!handle_primary_tcp_pkt(nf, conn, pkt)) { + if (!handle_primary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); @@ -203,7 +288,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, } } else { /* NET_FILTER_DIRECTION_RX */ - if (!handle_secondary_tcp_pkt(nf, conn, pkt)) { + if (!handle_secondary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); From patchwork Sat Aug 11 20:59:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563541 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 582631390 for ; Sat, 11 Aug 2018 21:01:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4AC5E29CCD for ; Sat, 11 Aug 2018 21:01:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3E1D029CD1; Sat, 11 Aug 2018 21:01:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 90A6629CCD for ; Sat, 11 Aug 2018 21:01:38 +0000 (UTC) Received: from localhost ([::1]:33177 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob0z-0005rI-PK for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:01:37 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48044) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1foazG-0004Dk-9Q for qemu-devel@nongnu.org; Sat, 11 Aug 2018 16:59:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1foazD-00012g-6x for qemu-devel@nongnu.org; Sat, 11 Aug 2018 16:59:50 -0400 Received: from mail-pf1-x430.google.com ([2607:f8b0:4864:20::430]:41484) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1foazC-00011S-UQ for qemu-devel@nongnu.org; Sat, 11 Aug 2018 16:59:47 -0400 Received: by mail-pf1-x430.google.com with SMTP id y10-v6so6000323pfn.8 for ; Sat, 11 Aug 2018 13:59:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TvJO8udGdRUsX1kOVbv+P3kHWq6wiXsaeR/C1Uk4ojc=; b=pcQpTn1tt92EbybzYFU/RycWh8uH2BPyGycVZRfaZub0bRmuYhCato5fvLU4iCpnaL emjl3f30NJbe36tvZIVpVq5HMv6uOOL0CCC9cqtnMLGRDWrrMwAipBi9lwPuHQd3RBgu ZMJcNHciUvMwYTUmrPvYEhrggHx5Nx7IN5BypUZBIUs931zbrdB8eO4AyBYSNowIYLGN rh6GPPw9LeTuLoykrp0hL62mExFkDGVfjYOLGUcCUmw+nxj8bMmGP8L3VZNK6/NQOQO0 49oClr+W5MtvhbZL6aw3WG3vwxASKt/19CyK2LcirJuRzfIR/mpde0SA6mFCNPAJESm8 etng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=TvJO8udGdRUsX1kOVbv+P3kHWq6wiXsaeR/C1Uk4ojc=; b=S4pk6+E99DN8+VVq88PdFS/1nZHduX5uCipV9WN2Re+kMxMGrNpD534R8wjmP9DiOd cxRjRH2mWSvpwZWu9qFuH0FA/vizYnyEAGfFG8yd8g73BEqppAKKi3KtyIL9OFNHtcwE eLmDuW4Zcud/SqaHLJAZdPJ/L8ewmkVNdz1WAm1ODh4vd7kme8IjrkzoGeSNWJcZpNIm ZTjJiYB09x/YLKDgnWNCvy274c9ItRznP6+ty01n8Qw3jaLy/Xfht2b126frsnysUCTj FGS9k7TC5VrZNMRFoBOhrGxtt7agGXfxPi9k1Yy1Lq3LmoTQ5JgV4VkPR4GDbtt0Cc2p ZvVQ== X-Gm-Message-State: AOUpUlE0F9pbesxizSRRMN55zH+TIhAl/l9Wm3wl7LFJ2JAi6SfvHoEF jbLP2aJOkHzGuq0uBViHff4SEiw4 X-Google-Smtp-Source: AA+uWPwJ4X0BeVRjdufIi9HfxZQbh5wqQzpnU008HQvmzb7ej2Q8iloK50vrh74ug7pSZM8K8JT0zQ== X-Received: by 2002:a62:9849:: with SMTP id q70-v6mr12526575pfd.178.1534021185482; Sat, 11 Aug 2018 13:59:45 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.13.59.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 13:59:44 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:06 +0800 Message-Id: <20180811205924.4113-3-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::430 Subject: [Qemu-devel] [PATCH V11 02/20] colo-compare: implement the process of checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen --- include/migration/colo.h | 6 ++++ net/colo-compare.c | 78 ++++++++++++++++++++++++++++++++++++++++ net/colo-compare.h | 22 ++++++++++++ 3 files changed, 106 insertions(+) create mode 100644 net/colo-compare.h diff --git a/include/migration/colo.h b/include/migration/colo.h index 2fe48ad353..fefb2fcf4c 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -16,6 +16,12 @@ #include "qemu-common.h" #include "qapi/qapi-types-migration.h" +enum colo_event { + COLO_EVENT_NONE, + COLO_EVENT_CHECKPOINT, + COLO_EVENT_FAILOVER, +}; + void colo_info_init(void); void migrate_start_colo_process(MigrationState *s); diff --git a/net/colo-compare.c b/net/colo-compare.c index dd745a491b..80e6532e8b 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -27,11 +27,16 @@ #include "qemu/sockets.h" #include "colo.h" #include "sysemu/iothread.h" +#include "net/colo-compare.h" +#include "migration/colo.h" #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) +static QTAILQ_HEAD(, CompareState) net_compares = + QTAILQ_HEAD_INITIALIZER(net_compares); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 @@ -41,6 +46,10 @@ /* TODO: Should be configurable */ #define REGULAR_PACKET_CHECK_MS 3000 +static QemuMutex event_mtx; +static QemuCond event_complete_cond; +static int event_unhandled_count; + /* * + CompareState ++ * | | @@ -87,6 +96,11 @@ typedef struct CompareState { IOThread *iothread; GMainContext *worker_context; QEMUTimer *packet_check_timer; + + QEMUBH *event_bh; + enum colo_event event; + + QTAILQ_ENTRY(CompareState) next; } CompareState; typedef struct CompareClass { @@ -736,6 +750,25 @@ static void check_old_packet_regular(void *opaque) REGULAR_PACKET_CHECK_MS); } +/* Public API, Used for COLO frame to notify compare event */ +void colo_notify_compares_event(void *opaque, int event, Error **errp) +{ + CompareState *s; + + qemu_mutex_lock(&event_mtx); + QTAILQ_FOREACH(s, &net_compares, next) { + s->event = event; + qemu_bh_schedule(s->event_bh); + event_unhandled_count++; + } + /* Wait all compare threads to finish handling this event */ + while (event_unhandled_count > 0) { + qemu_cond_wait(&event_complete_cond, &event_mtx); + } + + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_timer_init(CompareState *s) { AioContext *ctx = iothread_get_aio_context(s->iothread); @@ -756,6 +789,30 @@ static void colo_compare_timer_del(CompareState *s) } } +static void colo_flush_packets(void *opaque, void *user_data); + +static void colo_compare_handle_event(void *opaque) +{ + CompareState *s = opaque; + + switch (s->event) { + case COLO_EVENT_CHECKPOINT: + g_queue_foreach(&s->conn_list, colo_flush_packets, s); + break; + case COLO_EVENT_FAILOVER: + break; + default: + break; + } + + assert(event_unhandled_count > 0); + + qemu_mutex_lock(&event_mtx); + event_unhandled_count--; + qemu_cond_broadcast(&event_complete_cond); + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_iothread(CompareState *s) { object_ref(OBJECT(s->iothread)); @@ -769,6 +826,7 @@ static void colo_compare_iothread(CompareState *s) s, s->worker_context, true); colo_compare_timer_init(s); + s->event_bh = qemu_bh_new(colo_compare_handle_event, s); } static char *compare_get_pri_indev(Object *obj, Error **errp) @@ -926,8 +984,13 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); + QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); + qemu_mutex_init(&event_mtx); + qemu_cond_init(&event_complete_cond); + s->connection_track_table = g_hash_table_new_full(connection_key_hash, connection_key_equal, g_free, @@ -990,6 +1053,7 @@ static void colo_compare_init(Object *obj) static void colo_compare_finalize(Object *obj) { CompareState *s = COLO_COMPARE(obj); + CompareState *tmp = NULL; qemu_chr_fe_deinit(&s->chr_pri_in, false); qemu_chr_fe_deinit(&s->chr_sec_in, false); @@ -997,6 +1061,16 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { colo_compare_timer_del(s); } + + qemu_bh_delete(s->event_bh); + + QTAILQ_FOREACH(tmp, &net_compares, next) { + if (tmp == s) { + QTAILQ_REMOVE(&net_compares, s, next); + break; + } + } + /* Release all unhandled packets after compare thead exited */ g_queue_foreach(&s->conn_list, colo_flush_packets, s); @@ -1009,6 +1083,10 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { object_unref(OBJECT(s->iothread)); } + + qemu_mutex_destroy(&event_mtx); + qemu_cond_destroy(&event_complete_cond); + g_free(s->pri_indev); g_free(s->sec_indev); g_free(s->outdev); diff --git a/net/colo-compare.h b/net/colo-compare.h new file mode 100644 index 0000000000..1b1ce76aea --- /dev/null +++ b/net/colo-compare.h @@ -0,0 +1,22 @@ +/* + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) + * (a.k.a. Fault Tolerance or Continuous Replication) + * + * Copyright (c) 2017 HUAWEI TECHNOLOGIES CO., LTD. + * Copyright (c) 2017 FUJITSU LIMITED + * Copyright (c) 2017 Intel Corporation + * + * Authors: + * zhanghailiang + * Zhang Chen + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * later. See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event(void *opaque, int event, Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */ From patchwork Sat Aug 11 20:59:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563543 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E68831390 for ; Sat, 11 Aug 2018 21:01:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D819D29CCD for ; Sat, 11 Aug 2018 21:01:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CB28629CD1; Sat, 11 Aug 2018 21:01:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 5236129CCD for ; Sat, 11 Aug 2018 21:01:41 +0000 (UTC) Received: from localhost ([::1]:33178 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob12-0005tE-Fj for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:01:40 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48055) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1foazJ-0004NX-Pk for qemu-devel@nongnu.org; Sat, 11 Aug 2018 16:59:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1foazI-000155-T5 for qemu-devel@nongnu.org; Sat, 11 Aug 2018 16:59:53 -0400 Received: from mail-pg1-x52c.google.com ([2607:f8b0:4864:20::52c]:35824) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1foazI-00014j-Mp for qemu-devel@nongnu.org; Sat, 11 Aug 2018 16:59:52 -0400 Received: by mail-pg1-x52c.google.com with SMTP id w10-v6so5872207pgv.2 for ; Sat, 11 Aug 2018 13:59:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bpx5nNk4c2i+Qel/d4lL7jT0w+UFVx5G4ipkEW0Rma0=; b=ClZBExym7OBkgvKHO7xIrM0C+y2E0WJki4DPWcQfetTv3G/ETV+Yk+8gSkbBYl9YnH Kp0QMimkC5TkwSBx9Hhv5ymF5IGVNFSynxTdpWBSe8o/gZbosahGTn1p3AKsVUD8FqPI C4VJsG5p/rmdSUwKWu5HhgM/AYTroBtGkQ1RC1hi/AjKcMPrnkKgq1e1MjL5IycEPfU2 pcy9sscECuIolGOFWq+4gSjLZ/8sewbjb6S0o8eMM+EDukyWQq6qNTwn4JN6nH5PSbXg 5HMlT4JeLqZt5V7WviQswoKp4KbLIoxeiuDDqH8RA3VJwFYhyEDE9L+IwaUfapcvl8eU ZkUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=bpx5nNk4c2i+Qel/d4lL7jT0w+UFVx5G4ipkEW0Rma0=; b=EPYbowDj1DlJjkojqkhnFhCb04fqOhs+EoU9F+EwG8BXaTxJcDXngCHXdg71p7Qthv 9okYtdwMVKDKkGg6vwB00Gy9v2X93oOJCPGZ3Juimxp5lBRxRcX99om22A9viISw3wwH IfZeQwnuDESHiRLS1h0Ui4EEHZ4nugis/ax/3e+Glr8Imj5CZXYwmqC/f8ORSPwF5Jng GruWtsN1wKumLmVfOaFTlImGgUBeDBRjKr7mCayjGlmuHkrhMSxxXbAlVLWvHP4gla4o dEtLE35/eqFK1vao6ojqTzw/hL6vsEjTQgyMw8E7ztRIOqmxkpZF15QRIVh8s2L8Y1zk zyLA== X-Gm-Message-State: AOUpUlHNEslKKSdJUxtPPZ1pNB4xAvttmVGLTFZfZaLltCKOko6XkRAL cUWLASolfXTA+Wl1q3jCtNlehsRJ X-Google-Smtp-Source: AA+uWPxHqd4AICH8jcTb+tbnUpJY6rk6WOzOUhuO5q6G3K0355jrjvgvKFcJGWNHthftCkm9DAJYvw== X-Received: by 2002:a62:1d49:: with SMTP id d70-v6mr12760657pfd.101.1534021191258; Sat, 11 Aug 2018 13:59:51 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.13.59.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 13:59:50 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:07 +0800 Message-Id: <20180811205924.4113-4-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::52c Subject: [Qemu-devel] [PATCH V11 03/20] colo-compare: use notifier to notify packets comparing result X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 37 ++++++++++++++++++++++++++----------- net/colo-compare.h | 2 ++ 2 files changed, 28 insertions(+), 11 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index 80e6532e8b..426eab5973 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -29,6 +29,7 @@ #include "sysemu/iothread.h" #include "net/colo-compare.h" #include "migration/colo.h" +#include "migration/migration.h" #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ @@ -37,6 +38,9 @@ static QTAILQ_HEAD(, CompareState) net_compares = QTAILQ_HEAD_INITIALIZER(net_compares); +static NotifierList colo_compare_notifiers = + NOTIFIER_LIST_INITIALIZER(colo_compare_notifiers); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 @@ -427,10 +431,7 @@ sec: qemu_hexdump((char *)spkt->data, stderr, "colo-compare spkt", spkt->size); - /* - * colo_compare_inconsistent_notify(); - * TODO: notice to checkpoint(); - */ + colo_compare_inconsistency_notify(); } } @@ -561,8 +562,24 @@ static int colo_old_packet_check_one(Packet *pkt, int64_t *check_time) } } +static void colo_compare_inconsistency_notify(void) +{ + notifier_list_notify(&colo_compare_notifiers, + migrate_get_current()); +} + +void colo_compare_register_notifier(Notifier *notify) +{ + notifier_list_add(&colo_compare_notifiers, notify); +} + +void colo_compare_unregister_notifier(Notifier *notify) +{ + notifier_remove(notify); +} + static int colo_old_packet_check_one_conn(Connection *conn, - void *user_data) + void *user_data) { GList *result = NULL; int64_t check_time = REGULAR_PACKET_CHECK_MS; @@ -573,10 +590,7 @@ static int colo_old_packet_check_one_conn(Connection *conn, if (result) { /* Do checkpoint will flush old packet */ - /* - * TODO: Notify colo frame to do checkpoint. - * colo_compare_inconsistent_notify(); - */ + colo_compare_inconsistency_notify(); return 0; } @@ -620,11 +634,12 @@ static void colo_compare_packet(CompareState *s, Connection *conn, /* * If one packet arrive late, the secondary_list or * primary_list will be empty, so we can't compare it - * until next comparison. + * until next comparison. If the packets in the list are + * timeout, it will trigger a checkpoint request. */ trace_colo_compare_main("packet different"); g_queue_push_head(&conn->primary_list, pkt); - /* TODO: colo_notify_checkpoint();*/ + colo_compare_inconsistency_notify(); break; } } diff --git a/net/colo-compare.h b/net/colo-compare.h index 1b1ce76aea..22ddd512e2 100644 --- a/net/colo-compare.h +++ b/net/colo-compare.h @@ -18,5 +18,7 @@ #define QEMU_COLO_COMPARE_H void colo_notify_compares_event(void *opaque, int event, Error **errp); +void colo_compare_register_notifier(Notifier *notify); +void colo_compare_unregister_notifier(Notifier *notify); #endif /* QEMU_COLO_COMPARE_H */ From patchwork Sat Aug 11 20:59:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563547 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 244CC1510 for ; Sat, 11 Aug 2018 21:04:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0E0E629239 for ; Sat, 11 Aug 2018 21:04:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E038229758; Sat, 11 Aug 2018 21:04:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4950D29239 for ; Sat, 11 Aug 2018 21:04:53 +0000 (UTC) Received: from localhost ([::1]:33190 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob48-0008Gv-BZ for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:04:52 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48071) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1foazR-0004hb-NV for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1foazO-000175-Kg for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:01 -0400 Received: from mail-pl0-x22c.google.com ([2607:f8b0:400e:c01::22c]:38386) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1foazO-00016i-Ci for qemu-devel@nongnu.org; Sat, 11 Aug 2018 16:59:58 -0400 Received: by mail-pl0-x22c.google.com with SMTP id u11-v6so5387292plq.5 for ; Sat, 11 Aug 2018 13:59:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+xIPG0Ka1irzU3UL2shk3CuB2CKRPEKS0yY2Gtuow8U=; b=YZ0xrZA16aDNlFz3bJVrDTjRktt7jnQ8gWtAaUDxIKbKQe9U3dKx2TfC2V7NUjiuR0 JA7xMAGtzj++b0BYMYMV0JCCtWJiC/58WUvziXgkNXtYROrE55WKpUNQOkQVXn2MHU0G lUjO92z+ixecQdi99bggBHrtkzc9BWNOb1+e+mRVtzq48Vi9agGMtDGngYAkk/EKL0Jf Sh46u1n0gIN0YkUka3HkbJlLODJ9K3IDL/yhpHYCMHsLib8wXbs2IkjN7bLzRCd0o2Q4 0TES461Ie3PQ0m1d8qfTrp7CZWgrVCw46DiHnkrCuYKFZL8WrhWpWCajeQHDDKKdAHj+ zzWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+xIPG0Ka1irzU3UL2shk3CuB2CKRPEKS0yY2Gtuow8U=; b=U/MNZ9lvn4e2+0Be8BnRY6afoIfHvV05TfQsRUpPbFSxMD8DgrUM+1/Xk3EOmUoGnE PmT1a3pFMzbs/yQ6UmdCufS+BZjPFcmGFQ0cF9+NmGtFl70IhguihOjDk5p5CR0kFS3a C8O1/5pvyt0yaYvNYvnkxDripGTu0wt2t6/w7DvohFYLIoKHuzCuF5uaNTZaRLRoVZij 2cRpCeqChtNFOJqMA1vpA4c710ufE9Plt+HY68pNuKZ4Ga/qby4ii0GuKle5CSfr7gNe /mKzw7RVfQP4iEUP1+s3Et6pbzlDCQIm5vKl1tMUSJwJA4lzOsoHIHr2Zg2Xtjc2JLjt fRlQ== X-Gm-Message-State: AOUpUlFaFTXCYDP1JfOanZ7EY2O2L9jd3XXT/QpSc4kapP7iPCec3Stt XW/FTm9LNrgcv2IHNfcHSJLB6Jb6 X-Google-Smtp-Source: AA+uWPy4SajA0B3jydAKdmXxgD7XBCmqRzr6glXFi8/whoLSEVtKGgixwWzJqvNsEjFjag82mHjaBA== X-Received: by 2002:a17:902:9a83:: with SMTP id w3-v6mr11043776plp.75.1534021196867; Sat, 11 Aug 2018 13:59:56 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.13.59.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 13:59:55 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:08 +0800 Message-Id: <20180811205924.4113-5-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::22c Subject: [Qemu-devel] [PATCH V11 04/20] COLO: integrate colo compare with colo frame X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP For COLO FT, both the PVM and SVM run at the same time, only sync the state while it needs. So here, let SVM runs while not doing checkpoint, change DEFAULT_MIGRATE_X_CHECKPOINT_DELAY to 200*100. Besides, we forgot to release colo_checkpoint_semd and colo_delay_timer, fix them here. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 ++++++++++++++++++++++++++++++++++++++++-- migration/migration.c | 6 ++---- 2 files changed, 42 insertions(+), 6 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 4381067ed4..081df1835f 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -25,8 +25,11 @@ #include "qemu/error-report.h" #include "migration/failover.h" #include "replication.h" +#include "net/colo-compare.h" +#include "net/colo.h" static bool vmstate_loading; +static Notifier packets_compare_notifier; #define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024) @@ -343,6 +346,11 @@ static int colo_do_checkpoint_transaction(MigrationState *s, goto out; } + colo_notify_compares_event(NULL, COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + goto out; + } + /* Disable block migration */ migrate_set_block_enabled(false, &local_err); qemu_savevm_state_header(fb); @@ -400,6 +408,11 @@ out: return ret; } +static void colo_compare_notify_checkpoint(Notifier *notifier, void *data) +{ + colo_checkpoint_notify(data); +} + static void colo_process_checkpoint(MigrationState *s) { QIOChannelBuffer *bioc; @@ -416,6 +429,9 @@ static void colo_process_checkpoint(MigrationState *s) goto out; } + packets_compare_notifier.notify = colo_compare_notify_checkpoint; + colo_compare_register_notifier(&packets_compare_notifier); + /* * Wait for Secondary finish loading VM states and enter COLO * restore. @@ -461,11 +477,21 @@ out: qemu_fclose(fb); } - timer_del(s->colo_delay_timer); - /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); + + /* + * It is safe to unregister notifier after failover finished. + * Besides, colo_delay_timer and colo_checkpoint_sem can't be + * released befor unregister notifier, or there will be use-after-free + * error. + */ + colo_compare_unregister_notifier(&packets_compare_notifier); + timer_del(s->colo_delay_timer); + timer_free(s->colo_delay_timer); + qemu_sem_destroy(&s->colo_checkpoint_sem); + /* * Must be called after failover BH is completed, * Or the failover BH may shutdown the wrong fd that @@ -558,6 +584,11 @@ void *colo_process_incoming_thread(void *opaque) fb = qemu_fopen_channel_input(QIO_CHANNEL(bioc)); object_unref(OBJECT(bioc)); + qemu_mutex_lock_iothread(); + vm_start(); + trace_colo_vm_state_change("stop", "run"); + qemu_mutex_unlock_iothread(); + colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_READY, &local_err); if (local_err) { @@ -577,6 +608,11 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + trace_colo_vm_state_change("run", "stop"); + qemu_mutex_unlock_iothread(); + /* FIXME: This is unnecessary for periodic checkpoint mode */ colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_REPLY, &local_err); @@ -630,6 +666,8 @@ void *colo_process_incoming_thread(void *opaque) } vmstate_loading = false; + vm_start(); + trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); if (failover_get_state() == FAILOVER_STATUS_RELAUNCH) { diff --git a/migration/migration.c b/migration/migration.c index b7d9854bda..33a2fd74c6 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -75,10 +75,8 @@ /* Migration XBZRLE default cache size */ #define DEFAULT_MIGRATE_XBZRLE_CACHE_SIZE (64 * 1024 * 1024) -/* The delay time (in ms) between two COLO checkpoints - * Note: Please change this default value to 10000 when we support hybrid mode. - */ -#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY 200 +/* The delay time (in ms) between two COLO checkpoints */ +#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100) #define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2 #define DEFAULT_MIGRATE_MULTIFD_PAGE_COUNT 16 From patchwork Sat Aug 11 20:59:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563545 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E1F5B13B4 for ; Sat, 11 Aug 2018 21:01:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D46B429CCD for ; Sat, 11 Aug 2018 21:01:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C807129CD1; Sat, 11 Aug 2018 21:01:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3249F29CCD for ; Sat, 11 Aug 2018 21:01:57 +0000 (UTC) Received: from localhost ([::1]:33179 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob1I-000655-90 for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:01:56 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48082) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1foazV-0004jv-D6 for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1foazU-00019x-9j for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:05 -0400 Received: from mail-pf1-x42e.google.com ([2607:f8b0:4864:20::42e]:42569) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1foazU-00019d-1n for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:04 -0400 Received: by mail-pf1-x42e.google.com with SMTP id l9-v6so5998245pff.9 for ; Sat, 11 Aug 2018 14:00:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=zg3UpdwPhIVS+58+okcfZSbgGtXcG6c8nBCI1yBhYHM=; b=uVLcyjq4t4+SIooL22yKG72DQvpMEsy90vksLGAnoXhxRVq5svDr+p0eM+0zwYBWS0 yhD3tmOSWXC+wLNZiA+Pm50xEAhCaPCIES6UzjpndqT3WI6BVFBm6kexgLXtZado3Jwa xHOjP6CNwpYc3WhCLxmGw6tX/nlnUFuFoDy5E8SSkI8570qmsYRs9o4NIB1omn8NKTo+ 3JaHEgyz0MT/sfIJXngsBpEZnTbufN/4avwLKdoGzAyt9vTfzDjRbctFGNI+R0ECZdXK xSpqaz09yDjfh8vdkWCHuoghY0UpedNzSEInDEkEvOhEVB2lmvNOHZ7WonYvYwVegKau mqYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=zg3UpdwPhIVS+58+okcfZSbgGtXcG6c8nBCI1yBhYHM=; b=Bd6L56vhxwzfYPWvo390VdjHAvDgXCbC8MBN8JpxfBTyRvOf4D4g9QXpspGG3wnyvn MYx1sukZQDmYvLSDUshGBLU4mOkHbdCsJ0Q22WU1KQUSKHaQnMcwBRxmUoOKiiMRe2Og Iif0yku8qr+QUlH0T3jhWZ/xfPAL6qRNyYtdtJeQLOqeuiMkectnPxPTT9j1+Pgnlt17 hwRB44f7mkT2d5TzRtLl/DSqZu0WntQAXnoXhHMZZngxIPgE6c/hy3kuZDYN6LF8eroD GJ4sZnuyrKqyrzx5sKEujd880rvy4xyKVSds/sclY7VMDRUTx5mt289G4Keu5LpJfWBA a9HA== X-Gm-Message-State: AOUpUlFNLQGIEdByqcrDreNahBYMgrCNmkpJM6Wk4ZWvz3w1jR49cbsM kCnp/gfT4lDKhIT3slHlI6t0Nn5x X-Google-Smtp-Source: AA+uWPxwCdfPW56IrKZyMXW0Uon3JY7KpNuRf5FxS/pa/Ek75M2SJ6myPKIyM7ZFviutciZWfpLXkA== X-Received: by 2002:a62:954:: with SMTP id e81-v6mr12579539pfd.231.1534021202687; Sat, 11 Aug 2018 14:00:02 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.13.59.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:00:01 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:09 +0800 Message-Id: <20180811205924.4113-6-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42e Subject: [Qemu-devel] [PATCH V11 05/20] COLO: Add block replication into colo process X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 9 +++++++++ 2 files changed, 52 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 081df1835f..e06640c3d6 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -27,6 +27,7 @@ #include "replication.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "block/block.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -56,6 +57,7 @@ static void secondary_vm_do_failover(void) { int old_state; MigrationIncomingState *mis = migration_incoming_get_current(); + Error *local_err = NULL; /* Can not do failover during the process of VM's loading VMstate, Or * it will break the secondary VM. @@ -73,6 +75,11 @@ static void secondary_vm_do_failover(void) migrate_set_state(&mis->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + } + if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side"); /* recover runstate to normal migration finish state */ @@ -110,6 +117,7 @@ static void primary_vm_do_failover(void) { MigrationState *s = migrate_get_current(); int old_state; + Error *local_err = NULL; migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); @@ -133,6 +141,13 @@ static void primary_vm_do_failover(void) FailoverStatus_str(old_state)); return; } + + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + local_err = NULL; + } + /* Notify COLO thread that failover work is finished */ qemu_sem_post(&s->colo_exit_sem); } @@ -356,6 +371,11 @@ static int colo_do_checkpoint_transaction(MigrationState *s, qemu_savevm_state_header(fb); qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } qemu_savevm_state_complete_precopy(fb, false, false); qemu_mutex_unlock_iothread(); @@ -446,6 +466,12 @@ static void colo_process_checkpoint(MigrationState *s) object_unref(OBJECT(bioc)); qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_PRIMARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vm_start(); qemu_mutex_unlock_iothread(); trace_colo_vm_state_change("stop", "run"); @@ -585,6 +611,11 @@ void *colo_process_incoming_thread(void *opaque) object_unref(OBJECT(bioc)); qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_SECONDARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } vm_start(); trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); @@ -665,6 +696,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + replication_get_error_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* discard colo disk buffer */ + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading = false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/migration/migration.c b/migration/migration.c index 33a2fd74c6..71492c642d 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -385,6 +385,7 @@ static void process_incoming_migration_co(void *opaque) MigrationIncomingState *mis = migration_incoming_get_current(); PostcopyState ps; int ret; + Error *local_err = NULL; assert(mis->from_src_file); mis->largest_page_size = qemu_ram_pagesize_largest(); @@ -416,6 +417,14 @@ static void process_incoming_migration_co(void *opaque) /* we get COLO info, and know if we are in COLO mode */ if (!ret && migration_incoming_enable_colo()) { + /* Make sure all file formats flush their mutable metadata */ + bdrv_invalidate_cache_all(&local_err); + if (local_err) { + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + error_report_err(local_err); + exit(EXIT_FAILURE); + } mis->migration_incoming_co = qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); From patchwork Sat Aug 11 20:59:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563549 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 34D1F13B4 for ; Sat, 11 Aug 2018 21:04:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2675929239 for ; Sat, 11 Aug 2018 21:04:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1A49C29758; Sat, 11 Aug 2018 21:04:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 5674629239 for ; Sat, 11 Aug 2018 21:04:57 +0000 (UTC) Received: from localhost ([::1]:33191 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob4C-0008Tn-E2 for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:04:56 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48124) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1foazd-0004pO-Am for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1foaza-0001C1-7z for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:13 -0400 Received: from mail-pl0-x241.google.com ([2607:f8b0:400e:c01::241]:39671) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1foazZ-0001Bp-Vr for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:10 -0400 Received: by mail-pl0-x241.google.com with SMTP id w14-v6so5383615plp.6 for ; Sat, 11 Aug 2018 14:00:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uhk9gk3t3FaE7CeqiUKGUtHN6rf5khX2J2VT7ZGWKN0=; b=PJAtxF0jcy0BPhSwV1ebO0EJU8YgHQn2PPHXAESQCuG0X1KPErq91CXoSPCWDvl8o2 kWaXTl3Mc9X7QfoHGCoErvX+mcAB2BbqRftqOJidI843FmdGxcduN+V4GJiB+SAYqGC2 HcIxCc6k9eZ4Fv8P2uy+dBgnZXNA76JW6zGCyWXwcEOjKHEfseoahucNXEGP8UHmJmfy 6a4SMi0PvxOZxcuibfgk4IHw9e8tLL+WQ/DtVBUQgJxp7DsvlY5gnww2YGNfWVdsUxuL /ILFXxzooVS93rZ2kJeez0pfuptHqeB7YiSO0SMtK8rdANxzvJgCHgR/JDMZiha/+LTc +u8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uhk9gk3t3FaE7CeqiUKGUtHN6rf5khX2J2VT7ZGWKN0=; b=dgglMStpYzpAnG0qRGEy9Mo4ReKdvccbUuPnfiTgyR3pXOUV6K5gH57LvdrOzOBteI ei4xLEP94PBeM50L975OOknQ6jhJM5GR1Ah6YaNgQlcY0k+bcFFgOVODzrZYICJiLr2d gG7f1AaZrMJL9M+Zlcobd6ETI/pByEfIPtvVWSkz32NYOe9S2Rl2dnY9sWt2SWyVTpDS bXhXvyVxdncFlYnLLrB3hAjXxUmGIHGWyOVS8JIOT7AH/9MvsS1UOb3mNX0F4ljJvOm5 821+cRjVmy4YSekixm1tiBYDKEqmIcRSmt4I5ouotXR9XchmiLzv6/uKoZIjYlmH9WNg u2gQ== X-Gm-Message-State: AOUpUlE31wZj+jX8UW3QrNpNULrflLmU0NkmzU06hlY9ApK7zKrWCM7d 1wZWeuZQTWQqVsTac4cvv3iqfGLI X-Google-Smtp-Source: AA+uWPwe/2wGyMDfmXW0Ep7rQoIg3zL3a+evoT8+HFFM5qxE88rr3pq9h53jmfjavYsoY+hehlfrzA== X-Received: by 2002:a17:902:b60b:: with SMTP id b11-v6mr10950659pls.301.1534021208585; Sat, 11 Aug 2018 14:00:08 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.00.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:00:07 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:10 +0800 Message-Id: <20180811205924.4113-7-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::241 Subject: [Qemu-devel] [PATCH V11 06/20] COLO: Remove colo_state migration struct X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP We need to know if migration is going into COLO state for incoming side before start normal migration. Instead by using the VMStateDescription to send colo_state from source side to destination side, we use MIG_CMD_ENABLE_COLO to indicate whether COLO is enabled or not. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- include/migration/colo.h | 5 +-- migration/Makefile.objs | 2 +- migration/colo-comm.c | 76 ---------------------------------------- migration/colo.c | 13 ++++++- migration/migration.c | 23 +++++++++++- migration/savevm.c | 17 +++++++++ migration/savevm.h | 1 + migration/trace-events | 1 + vl.c | 2 -- 9 files changed, 57 insertions(+), 83 deletions(-) delete mode 100644 migration/colo-comm.c diff --git a/include/migration/colo.h b/include/migration/colo.h index fefb2fcf4c..99ce17aca7 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -28,8 +28,9 @@ void migrate_start_colo_process(MigrationState *s); bool migration_in_colo_state(void); /* loadvm */ -bool migration_incoming_enable_colo(void); -void migration_incoming_exit_colo(void); +void migration_incoming_enable_colo(void); +void migration_incoming_disable_colo(void); +bool migration_incoming_colo_enabled(void); void *colo_process_incoming_thread(void *opaque); bool migration_incoming_in_colo_state(void); diff --git a/migration/Makefile.objs b/migration/Makefile.objs index c83ec47ba8..a4f3bafd86 100644 --- a/migration/Makefile.objs +++ b/migration/Makefile.objs @@ -1,6 +1,6 @@ common-obj-y += migration.o socket.o fd.o exec.o common-obj-y += tls.o channel.o savevm.o -common-obj-y += colo-comm.o colo.o colo-failover.o +common-obj-y += colo.o colo-failover.o common-obj-y += vmstate.o vmstate-types.o page_cache.o common-obj-y += qemu-file.o global_state.o common-obj-y += qemu-file-channel.o diff --git a/migration/colo-comm.c b/migration/colo-comm.c deleted file mode 100644 index df26e4dfe7..0000000000 --- a/migration/colo-comm.c +++ /dev/null @@ -1,76 +0,0 @@ -/* - * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) - * (a.k.a. Fault Tolerance or Continuous Replication) - * - * Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. - * Copyright (c) 2016 FUJITSU LIMITED - * Copyright (c) 2016 Intel Corporation - * - * This work is licensed under the terms of the GNU GPL, version 2 or - * later. See the COPYING file in the top-level directory. - * - */ - -#include "qemu/osdep.h" -#include "migration.h" -#include "migration/colo.h" -#include "migration/vmstate.h" -#include "trace.h" - -typedef struct { - bool colo_requested; -} COLOInfo; - -static COLOInfo colo_info; - -COLOMode get_colo_mode(void) -{ - if (migration_in_colo_state()) { - return COLO_MODE_PRIMARY; - } else if (migration_incoming_in_colo_state()) { - return COLO_MODE_SECONDARY; - } else { - return COLO_MODE_UNKNOWN; - } -} - -static int colo_info_pre_save(void *opaque) -{ - COLOInfo *s = opaque; - - s->colo_requested = migrate_colo_enabled(); - - return 0; -} - -static bool colo_info_need(void *opaque) -{ - return migrate_colo_enabled(); -} - -static const VMStateDescription colo_state = { - .name = "COLOState", - .version_id = 1, - .minimum_version_id = 1, - .pre_save = colo_info_pre_save, - .needed = colo_info_need, - .fields = (VMStateField[]) { - VMSTATE_BOOL(colo_requested, COLOInfo), - VMSTATE_END_OF_LIST() - }, -}; - -void colo_info_init(void) -{ - vmstate_register(NULL, 0, &colo_state, &colo_info); -} - -bool migration_incoming_enable_colo(void) -{ - return colo_info.colo_requested; -} - -void migration_incoming_exit_colo(void) -{ - colo_info.colo_requested = false; -} diff --git a/migration/colo.c b/migration/colo.c index e06640c3d6..c083d3696f 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -152,6 +152,17 @@ static void primary_vm_do_failover(void) qemu_sem_post(&s->colo_exit_sem); } +COLOMode get_colo_mode(void) +{ + if (migration_in_colo_state()) { + return COLO_MODE_PRIMARY; + } else if (migration_incoming_in_colo_state()) { + return COLO_MODE_SECONDARY; + } else { + return COLO_MODE_UNKNOWN; + } +} + void colo_do_failover(MigrationState *s) { /* Make sure VM stopped while failover happened. */ @@ -745,7 +756,7 @@ out: if (mis->to_src_file) { qemu_fclose(mis->to_src_file); } - migration_incoming_exit_colo(); + migration_incoming_disable_colo(); return NULL; } diff --git a/migration/migration.c b/migration/migration.c index 71492c642d..e5330dbede 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -295,6 +295,22 @@ int migrate_send_rp_req_pages(MigrationIncomingState *mis, const char *rbname, return migrate_send_rp_message(mis, msg_type, msglen, bufc); } +static bool migration_colo_enabled; +bool migration_incoming_colo_enabled(void) +{ + return migration_colo_enabled; +} + +void migration_incoming_disable_colo(void) +{ + migration_colo_enabled = false; +} + +void migration_incoming_enable_colo(void) +{ + migration_colo_enabled = true; +} + void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p; @@ -416,7 +432,7 @@ static void process_incoming_migration_co(void *opaque) } /* we get COLO info, and know if we are in COLO mode */ - if (!ret && migration_incoming_enable_colo()) { + if (!ret && migration_incoming_colo_enabled()) { /* Make sure all file formats flush their mutable metadata */ bdrv_invalidate_cache_all(&local_err); if (local_err) { @@ -2970,6 +2986,11 @@ static void *migration_thread(void *opaque) qemu_savevm_send_postcopy_advise(s->to_dst_file); } + if (migrate_colo_enabled()) { + /* Notify migration destination that we enable COLO */ + qemu_savevm_send_colo_enable(s->to_dst_file); + } + qemu_savevm_state_setup(s->to_dst_file); s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start; diff --git a/migration/savevm.c b/migration/savevm.c index 7f92567a10..437308877a 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -56,6 +56,7 @@ #include "io/channel-file.h" #include "sysemu/replay.h" #include "qjson.h" +#include "migration/colo.h" #ifndef ETH_P_RARP #define ETH_P_RARP 0x8035 @@ -82,6 +83,7 @@ enum qemu_vm_cmd { were previously sent during precopy but are dirty. */ MIG_CMD_PACKAGED, /* Send a wrapped stream within this stream */ + MIG_CMD_ENABLE_COLO, /* Enable COLO */ MIG_CMD_POSTCOPY_RESUME, /* resume postcopy on dest */ MIG_CMD_RECV_BITMAP, /* Request for recved bitmap on dst */ MIG_CMD_MAX @@ -841,6 +843,12 @@ static void qemu_savevm_command_send(QEMUFile *f, qemu_fflush(f); } +void qemu_savevm_send_colo_enable(QEMUFile *f) +{ + trace_savevm_send_colo_enable(); + qemu_savevm_command_send(f, MIG_CMD_ENABLE_COLO, 0, NULL); +} + void qemu_savevm_send_ping(QEMUFile *f, uint32_t value) { uint32_t buf; @@ -1918,6 +1926,12 @@ static int loadvm_handle_recv_bitmap(MigrationIncomingState *mis, return 0; } +static int loadvm_process_enable_colo(MigrationIncomingState *mis) +{ + migration_incoming_enable_colo(); + return 0; +} + /* * Process an incoming 'QEMU_VM_COMMAND' * 0 just a normal return @@ -1997,6 +2011,9 @@ static int loadvm_process_command(QEMUFile *f) case MIG_CMD_RECV_BITMAP: return loadvm_handle_recv_bitmap(mis, len); + + case MIG_CMD_ENABLE_COLO: + return loadvm_process_enable_colo(mis); } return 0; diff --git a/migration/savevm.h b/migration/savevm.h index a5e65b8ae3..8373c2f6bd 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -55,6 +55,7 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint16_t len, uint64_t *start_list, uint64_t *length_list); +void qemu_savevm_send_colo_enable(QEMUFile *f); int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); diff --git a/migration/trace-events b/migration/trace-events index 9430f3cbe0..fa0ff3f3bf 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -37,6 +37,7 @@ savevm_send_ping(uint32_t val) "0x%x" savevm_send_postcopy_listen(void) "" savevm_send_postcopy_run(void) "" savevm_send_postcopy_resume(void) "" +savevm_send_colo_enable(void) "" savevm_send_recv_bitmap(char *name) "%s" savevm_state_setup(void) "" savevm_state_resume_prepare(void) "" diff --git a/vl.c b/vl.c index 16b913f9d5..a4e877e2ff 100644 --- a/vl.c +++ b/vl.c @@ -4392,8 +4392,6 @@ int main(int argc, char **argv, char **envp) #endif } - colo_info_init(); - if (net_init_clients(&err) < 0) { error_report_err(err); exit(1); From patchwork Sat Aug 11 20:59:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563557 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 829E213B4 for ; Sat, 11 Aug 2018 21:08:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 74AE529595 for ; Sat, 11 Aug 2018 21:08:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6641E2947B; Sat, 11 Aug 2018 21:08:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B6BAF2947B for ; Sat, 11 Aug 2018 21:08:11 +0000 (UTC) Received: from localhost ([::1]:33211 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob7K-000374-U9 for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:08:10 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48139) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1foazh-0004sw-CS for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:20 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1foazg-0001EJ-1R for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:17 -0400 Received: from mail-pf1-x442.google.com ([2607:f8b0:4864:20::442]:39867) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1foazf-0001Dz-NY for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:15 -0400 Received: by mail-pf1-x442.google.com with SMTP id j8-v6so6007846pff.6 for ; Sat, 11 Aug 2018 14:00:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=DzHL68LIqS1UCEBCc+7sb2AXjsOjx/iGla+q3l1kSls=; b=Px3f13siuv+hgKl+begj8vp/xiwbVkp9FWgX3Wzx3jmwHc/buG6gaeM3iy7T08fV+c t97rL+832MtFZH0pavFUHja94eNDjHVthy+DuWQ6Z+OggqbK6CBlD+uRqQvWOkGOCu5j iZuHuE8Pn5ib1siW47eepWcaG8HWp0A1cxH5TcAiu/oo4A7kHLaHniuPaqMKsHMiW2fw +nkcrt7FGYmI+djZkHNScJKdXzXbEjhNvxO6+nYzSnQ80EVQdFslba3w7eNXPHXJTTZZ JkqxlXoys9aZIfMPoujrSLQxb6aS+GEDlAPSnW2ZBFO8h3d89SVBgcmlq8xv9Ebn+I4v Y19Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=DzHL68LIqS1UCEBCc+7sb2AXjsOjx/iGla+q3l1kSls=; b=XdDcWhW+ekuMb0HXv7eDburC5fencZbRsiXp3v70NNDTmm90ZYSDRSxct4n89oXakw 7T4u9yKnX4jb761wd3GGGcpNR8NPhfT/uu9nwDBu/mgJfQVSB9AcpOfrf7gyF2fweMsv DgH9Ww7TB5ZzwT6S5dbinGkL6lfW70FK4k+7Hj1D0lAO++GHtDs8BCHtiphm/FXGTz5I g+LJAhuU71jK5lpWA/WcwKmuQhUuX56lS0fVo5+ANy460Ob6CIHKcgehG4+Y7gdmzD9m D2QJDcNRyivra8xeWNCR2J6v0QwzeSvakcWTlgBEC9m1zp7WRjjoDJPVhaxqIpwyEMeX CPrA== X-Gm-Message-State: AOUpUlHv1VWf+v3S1mmZLjaUKafqjBPgw21Bp34i4nNY8YtkW9yEExkp Q90oV+4zrq4WxLVatAbjWr3g8rm9 X-Google-Smtp-Source: AA+uWPxXn3VT6ommOoA46P/wy/Fv1M1m0aii+j/nTyZ4vEI2tFR2qqcmJNLsTrXgoxr2ZIyxkU7kcA== X-Received: by 2002:a65:6104:: with SMTP id z4-v6mr11382766pgu.361.1534021214332; Sat, 11 Aug 2018 14:00:14 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.00.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:00:13 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:11 +0800 Message-Id: <20180811205924.4113-8-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::442 Subject: [Qemu-devel] [PATCH V11 07/20] COLO: Load dirty pages into SVM's RAM cache firstly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP We should not load PVM's state directly into SVM, because there maybe some errors happen when SVM is receving data, which will break SVM. We need to ensure receving all data before load the state into SVM. We use an extra memory to cache these data (PVM's ram). The ram cache in secondary side is initially the same as SVM/PVM's memory. And in the process of checkpoint, we cache the dirty pages of PVM into this ram cache firstly, so this ram cache always the same as PVM's memory at every checkpoint, then we flush this cached ram to SVM after we receive all PVM's state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- include/exec/ram_addr.h | 1 + migration/migration.c | 6 +++ migration/ram.c | 83 ++++++++++++++++++++++++++++++++++++++++- migration/ram.h | 4 ++ migration/savevm.c | 2 +- 5 files changed, 93 insertions(+), 3 deletions(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index cf4ce06248..a78c1c99a7 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -27,6 +27,7 @@ struct RAMBlock { struct rcu_head rcu; struct MemoryRegion *mr; uint8_t *host; + uint8_t *colo_cache; /* For colo, VM's ram cache */ ram_addr_t offset; ram_addr_t used_length; ram_addr_t max_length; diff --git a/migration/migration.c b/migration/migration.c index e5330dbede..d702a847f2 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -441,6 +441,10 @@ static void process_incoming_migration_co(void *opaque) error_report_err(local_err); exit(EXIT_FAILURE); } + if (colo_init_ram_cache() < 0) { + error_report("Init ram cache failed"); + exit(EXIT_FAILURE); + } mis->migration_incoming_co = qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); @@ -449,6 +453,8 @@ static void process_incoming_migration_co(void *opaque) /* Wait checkpoint incoming thread exit before free resource */ qemu_thread_join(&mis->colo_incoming_thread); + /* We hold the global iothread lock, so it is safe here */ + colo_release_ram_cache(); } if (ret < 0) { diff --git a/migration/ram.c b/migration/ram.c index 24dea2730c..ce2cf99bf5 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3321,6 +3321,20 @@ static inline void *host_from_ram_block_offset(RAMBlock *block, return block->host + offset; } +static inline void *colo_cache_from_block_offset(RAMBlock *block, + ram_addr_t offset) +{ + if (!offset_in_ramblock(block, offset)) { + return NULL; + } + if (!block->colo_cache) { + error_report("%s: colo_cache is NULL in block :%s", + __func__, block->idstr); + return NULL; + } + return block->colo_cache + offset; +} + /** * ram_handle_compressed: handle the zero page case * @@ -3525,6 +3539,58 @@ static void decompress_data_with_multi_threads(QEMUFile *f, qemu_mutex_unlock(&decomp_done_lock); } +/* + * colo cache: this is for secondary VM, we cache the whole + * memory of the secondary VM, it is need to hold the global lock + * to call this helper. + */ +int colo_init_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + block->colo_cache = qemu_anon_ram_alloc(block->used_length, + NULL, + false); + if (!block->colo_cache) { + error_report("%s: Can't alloc memory for COLO cache of block %s," + "size 0x" RAM_ADDR_FMT, __func__, block->idstr, + block->used_length); + goto out_locked; + } + memcpy(block->colo_cache, block->host, block->used_length); + } + rcu_read_unlock(); + return 0; + +out_locked: + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache = NULL; + } + } + + rcu_read_unlock(); + return -errno; +} + +/* It is need to hold the global lock to call this helper */ +void colo_release_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache = NULL; + } + } + rcu_read_unlock(); +} + /** * ram_load_setup: Setup RAM for migration incoming side * @@ -3541,6 +3607,7 @@ static int ram_load_setup(QEMUFile *f, void *opaque) xbzrle_load_setup(); ramblock_recv_map_init(); + return 0; } @@ -3554,6 +3621,7 @@ static int ram_load_cleanup(void *opaque) g_free(rb->receivedmap); rb->receivedmap = NULL; } + return 0; } @@ -3791,13 +3859,24 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { RAMBlock *block = ram_block_from_stream(f, flags); - host = host_from_ram_block_offset(block, addr); + /* + * After going into COLO, we should load the Page into colo_cache. + */ + if (migration_incoming_in_colo_state()) { + host = colo_cache_from_block_offset(block, addr); + } else { + host = host_from_ram_block_offset(block, addr); + } if (!host) { error_report("Illegal RAM offset " RAM_ADDR_FMT, addr); ret = -EINVAL; break; } - ramblock_recv_bitmap_set(block, host); + + if (!migration_incoming_in_colo_state()) { + ramblock_recv_bitmap_set(block, host); + } + trace_ram_load_loop(block->idstr, (uint64_t)addr, flags, host); } diff --git a/migration/ram.h b/migration/ram.h index 457bf54b8c..d009480494 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -70,4 +70,8 @@ int64_t ramblock_recv_bitmap_send(QEMUFile *file, const char *block_name); int ram_dirty_bitmap_reload(MigrationState *s, RAMBlock *rb); +/* ram cache */ +int colo_init_ram_cache(void); +void colo_release_ram_cache(void); + #endif diff --git a/migration/savevm.c b/migration/savevm.c index 437308877a..33e9e7cda0 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1929,7 +1929,7 @@ static int loadvm_handle_recv_bitmap(MigrationIncomingState *mis, static int loadvm_process_enable_colo(MigrationIncomingState *mis) { migration_incoming_enable_colo(); - return 0; + return colo_init_ram_cache(); } /* From patchwork Sat Aug 11 20:59:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563565 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB2E313B4 for ; Sat, 11 Aug 2018 21:11:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DE680298D9 for ; Sat, 11 Aug 2018 21:11:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D2BA029915; Sat, 11 Aug 2018 21:11:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6EF71298D9 for ; Sat, 11 Aug 2018 21:11:07 +0000 (UTC) Received: from localhost ([::1]:33229 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fobAA-0006QF-KS for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:11:06 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48157) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1foazp-000500-7H for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1foazl-0001I3-W9 for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:25 -0400 Received: from mail-pl0-x242.google.com ([2607:f8b0:400e:c01::242]:36149) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1foazl-0001HR-Ln for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:21 -0400 Received: by mail-pl0-x242.google.com with SMTP id e11-v6so5386116plb.3 for ; Sat, 11 Aug 2018 14:00:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=t0ew/6qCUx4k4vyc0bty50OVKj22t3leUPx2lVu+1E0=; b=E69gCNRTKUeXvpVrfRVIbMaEBsj+QQJtQZcKorfg52yF/Lir4UW5amkob+zmIdIoW2 qVFkMO2H/0aqcgMLNRBJGntsp23uXmHa00k4TpcOUQQLBH0gTUkOBjm0dlhKIRpcAi2d ohVOM5/pFngnYSMJdT58foGqwlvt0VRwYMt7+n2g3c+YUTWlBWSFvrLOP2tzpQ8BH5xZ 9vf/rrvR38toKgErXYYhI+kE8gSH8F5Pd1HoTouc324xsooZlIMFXr97a3sWUbekUUVS LRwide2PqzNtVsQFoF5id1W9c0WyuLO3BwckxoWFYrokVpNeke7nJY9Srb89fqefxDBC J4qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=t0ew/6qCUx4k4vyc0bty50OVKj22t3leUPx2lVu+1E0=; b=D+QGfMBIC852ATSp+H32XF+lR0IjdUeEMbyWYWN4TacYVm60SeV1AbwSF4SbjRl2kl 0tUkXvwNph/QSHe2rfHP5+gCzGg6f2qBGzfk5ucGkR9DRmdIxClc1IuV450ujF5w4WIk L5QFYHwkkbZk11ebAr3JOvwYiNEOl9UVaknCANfbUxNrmMm+OCaMjCdZdYoyI64zo4h5 fOEWqhqUnocnvGsHWrL9rmasRHyeJfw0kxAw8SrBg+973h2C8U0wwJovSptUIU/ZC8WX i5di0tBwTYKmc2bUngV8v/R9ux0w9B5tzFrZJjNc2GHd3jZRbuNw0hNoog13KHsfI9lf heRQ== X-Gm-Message-State: AOUpUlFK8khhV9lJqZmEjTC9vSYyEhyBgy+xzPod0OGn4HjFvyCVWOuh SY/HHIhg5GRDjpq+EGBKgmgpg6zW X-Google-Smtp-Source: AA+uWPzJCiHINFJNrrZ4FbAGpOdunR/BBrM2QNyvMz+o4435VnbWnJtMYfY25tnG7CvNCCOT3bMz9Q== X-Received: by 2002:a17:902:a40b:: with SMTP id p11-v6mr11167271plq.228.1534021220008; Sat, 11 Aug 2018 14:00:20 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.00.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:00:18 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:12 +0800 Message-Id: <20180811205924.4113-9-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::242 Subject: [Qemu-devel] [PATCH V11 08/20] ram/COLO: Record the dirty pages that SVM received X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 43 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 40 insertions(+), 3 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index ce2cf99bf5..3dc403adb5 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3332,6 +3332,15 @@ static inline void *colo_cache_from_block_offset(RAMBlock *block, __func__, block->idstr); return NULL; } + + /* + * During colo checkpoint, we need bitmap of these migrated pages. + * It help us to decide which pages in ram cache should be flushed + * into VM's RAM later. + */ + if (!test_and_set_bit(offset >> TARGET_PAGE_BITS, block->bmap)) { + ram_state->migration_dirty_pages++; + } return block->colo_cache + offset; } @@ -3549,7 +3558,7 @@ int colo_init_ram_cache(void) RAMBlock *block; rcu_read_lock(); - QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + RAMBLOCK_FOREACH_MIGRATABLE(block) { block->colo_cache = qemu_anon_ram_alloc(block->used_length, NULL, false); @@ -3562,10 +3571,29 @@ int colo_init_ram_cache(void) memcpy(block->colo_cache, block->host, block->used_length); } rcu_read_unlock(); + /* + * Record the dirty pages that sent by PVM, we use this dirty bitmap together + * with to decide which page in cache should be flushed into SVM's RAM. Here + * we use the same name 'ram_bitmap' as for migration. + */ + if (ram_bytes_total()) { + RAMBlock *block; + + RAMBLOCK_FOREACH_MIGRATABLE(block) { + unsigned long pages = block->max_length >> TARGET_PAGE_BITS; + + block->bmap = bitmap_new(pages); + bitmap_set(block->bmap, 0, pages); + } + } + ram_state = g_new0(RAMState, 1); + ram_state->migration_dirty_pages = 0; + return 0; out_locked: - QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + + RAMBLOCK_FOREACH_MIGRATABLE(block) { if (block->colo_cache) { qemu_anon_ram_free(block->colo_cache, block->used_length); block->colo_cache = NULL; @@ -3581,14 +3609,23 @@ void colo_release_ram_cache(void) { RAMBlock *block; + RAMBLOCK_FOREACH_MIGRATABLE(block) { + g_free(block->bmap); + block->bmap = NULL; + } + rcu_read_lock(); - QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + + RAMBLOCK_FOREACH_MIGRATABLE(block) { if (block->colo_cache) { qemu_anon_ram_free(block->colo_cache, block->used_length); block->colo_cache = NULL; } } + rcu_read_unlock(); + g_free(ram_state); + ram_state = NULL; } /** From patchwork Sat Aug 11 20:59:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563551 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7868913B4 for ; Sat, 11 Aug 2018 21:05:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 697A229239 for ; Sat, 11 Aug 2018 21:05:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5A81629758; Sat, 11 Aug 2018 21:05:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id EB37729239 for ; Sat, 11 Aug 2018 21:05:15 +0000 (UTC) Received: from localhost ([::1]:33198 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob4V-0000tx-7b for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:05:15 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48172) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1foazr-000533-Vf for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1foazq-0001NH-Tk for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:27 -0400 Received: from mail-pl0-x236.google.com ([2607:f8b0:400e:c01::236]:45121) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1foazq-0001MV-GZ for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:26 -0400 Received: by mail-pl0-x236.google.com with SMTP id j8-v6so5375679pll.12 for ; Sat, 11 Aug 2018 14:00:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=hl4GecGoDAwGK9lm9frRd4U00FYZiPtvthNssXbfQco=; b=Dul44DDBRnDFrf8kPM2Od8PDVChLotH4obsw4tpcTrjTjxljOzcDps11sff+bv8Dte af5FP1fZk84tlmStlrsg3knsmE6qULsi6UfE5JBC1JX4hAtTMafWU5c6jVZnZ5SxOlxq xX34vt36mZuxMW/pncrwWaYyGMddM6CFXBXkX8l/kFqi5jdDG3yCUIvyWvG8WiKHq3hp Qo7666Hqt4mRw7HbHcNo4E9BmcwLx8bLFdtb188FfKNWMYCTmYjqe2aoUi99s5gYEwYc BrFKOfdep4wzbnMBmJHE9CDD9BxQpa+HDElnLUIh7wjQuW973epECiysMSESys9X2nX4 FiDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=hl4GecGoDAwGK9lm9frRd4U00FYZiPtvthNssXbfQco=; b=Zou+UGrmZQ17VucSDgWyzI7BBzWfkN3aDysoRPvWZ2gblfNifQFn7l7xtekBPB4qXr jYZPwMdw9h61wAbIp9qP0DDQbtBuPLANqFflcqfWMoR5K2nio/StMu8jpH6qLuL4ANFd RImqa0SBZOnwR27x3139SmoQguyk8DxMeteK7WQRsae+AsDczs72c9krhEjjO+y7RMaC 2pwAf3d45o9dzowGoJmTqLQovx1MqjpBKmPo4hehz4GOII/8iJLNBz+ExUb9InFhCQfG QHDsQrXEPhhWzy5VD6TZbTNECyGttRgCnKOgf3ukGnR+GYucsJc8n6RceSiCkZHU3TEt qX6A== X-Gm-Message-State: AOUpUlFUVClr/WXEq+NzvNleMv9Rrs6OtLABTP0+4l16MhF8gzdkvZ7g iV78pR9c9rVDDyYFmZxOuHHgBSk7 X-Google-Smtp-Source: AA+uWPyISmh4AoYFhX33KZzrdtQ03XyRshGsvqvYax0drWfC2DIMELCKSupe87FbA1pV5kVcHrLQiw== X-Received: by 2002:a17:902:8d8c:: with SMTP id v12-v6mr10915205plo.94.1534021225198; Sat, 11 Aug 2018 14:00:25 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.00.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:00:24 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:13 +0800 Message-Id: <20180811205924.4113-10-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::236 Subject: [Qemu-devel] [PATCH V11 09/20] COLO: Flush memory data from ram cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint time. So, the content of SVM's RAM cache will always be same with PVM's memory after checkpoint. Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way: Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's. Besides, we must ensure flush RAM cache before load device state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 37 +++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 2 files changed, 39 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 3dc403adb5..b4cbcf783f 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3841,6 +3841,39 @@ static bool postcopy_is_running(void) return ps >= POSTCOPY_INCOMING_LISTENING && ps < POSTCOPY_INCOMING_END; } +/* + * Flush content of RAM cache into SVM's memory. + * Only flush the pages that be dirtied by PVM or SVM or both. + */ +static void colo_flush_ram_cache(void) +{ + RAMBlock *block = NULL; + void *dst_host; + void *src_host; + unsigned long offset = 0; + + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); + rcu_read_lock(); + block = QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset = migration_bitmap_find_dirty(ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >= block->used_length) { + offset = 0; + block = QLIST_NEXT_RCU(block, next); + } else { + migration_bitmap_clear_dirty(ram_state, block, offset); + dst_host = block->host + (offset << TARGET_PAGE_BITS); + src_host = block->colo_cache + (offset << TARGET_PAGE_BITS); + memcpy(dst_host, src_host, TARGET_PAGE_SIZE); + } + } + + rcu_read_unlock(); + trace_colo_flush_ram_cache_end(); +} + static int ram_load(QEMUFile *f, void *opaque, int version_id) { int flags = 0, ret = 0, invalid_flags = 0; @@ -4017,6 +4050,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) ret |= wait_for_decompress_done(); rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); + + if (!ret && migration_incoming_in_colo_state()) { + colo_flush_ram_cache(); + } return ret; } diff --git a/migration/trace-events b/migration/trace-events index fa0ff3f3bf..bd2d0cd25a 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -102,6 +102,8 @@ ram_dirty_bitmap_sync_start(void) "" ram_dirty_bitmap_sync_wait(void) "" ram_dirty_bitmap_sync_complete(void) "" ram_state_resume_prepare(uint64_t v) "%" PRId64 +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" # migration/migration.c await_return_path_close_on_source_close(void) "" From patchwork Sat Aug 11 20:59:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563555 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B5E3213B4 for ; Sat, 11 Aug 2018 21:08:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 307C82947B for ; Sat, 11 Aug 2018 21:08:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0CCD0294B9; Sat, 11 Aug 2018 21:08:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 733222947B for ; Sat, 11 Aug 2018 21:08:05 +0000 (UTC) Received: from localhost ([::1]:33210 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob7E-00033D-29 for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:08:04 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48183) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1foazz-00059S-CX for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:36 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1foazw-0001Oz-AR for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:35 -0400 Received: from mail-pg1-x52b.google.com ([2607:f8b0:4864:20::52b]:46821) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1foazw-0001Oh-3t for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:32 -0400 Received: by mail-pg1-x52b.google.com with SMTP id f14-v6so5850536pgv.13 for ; Sat, 11 Aug 2018 14:00:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RS0WVorMcxBw+4/+ip7fUKa54Kq6NT6yPbDyMpZ4Vz8=; b=oXTwXleIoTXYpcYggsh9tK8YbzLoA/y+SLEd9uYo5BIAXWNfM8WT4FWu4U7BZtvN7r acstqziBxfrErP2xIychrbOmbmvi1OcRZ6os7bSAnk+fRZBvCPdRpSl2P9FMDxL8y4S6 vPUEkjBw5M+T275BzyVMfT6vm61kZyW2V7TKwpUIXeYCZWITBzMBoQj5scPtnUdNRrGP 6B9LeTYMbQC7m8CKTeTmfaORQPETl285rlswrJrOOnv24qJFwzJbbvEMLpBkMW1HlCrr SgSysC1pmthI8mZIzJIOfKCH0SGN3N2KiN+EllJoZvkC0XuBISEr2GX8eaB+Klr3Eq6H U/8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RS0WVorMcxBw+4/+ip7fUKa54Kq6NT6yPbDyMpZ4Vz8=; b=ow/kgkOHgcKcCPIymFk+M+A865hJHQtFSIo1WJKPxKou/qmUOXf9EosUKJOA/Y3BT4 KLjYmUQHuM9Qbf7RaZPzcTEVyOczJjeMgB5QYi4n0rFLTvecLwSymykgNVskcUUXU/4u 92ea2+RnNM8pFpFntePTBFweLYvS01YgBPvE8/0rAtptM4TBqyiK+hDfOc3FOHlkUapx F6ZhxyQHNPGmGUcGTCnVTAuYtUdi3j6vXox0C9v5uAddhAkjonR/q3HUN+kz8/CYm7/A rV3+eSE3+3pSrgD1GEA6c4I/gFkBwsioAT2uK/sH61JsUmx8x37m8MJV5nelXD2M1B01 uHrw== X-Gm-Message-State: AOUpUlFxaqH9FpGQ2/RLYE3k27sT71UuxrGGoTjVP56DtGZaetfEmEdS x16UiZmkO/27Bb0XNENVLfOKXq+E X-Google-Smtp-Source: AA+uWPx1LlIKZMOCLTyO23u5nohDYlUC8Yjc1Fec79/x60svZ6IUcLfYjv6NiW3SSWU7eI6LFjOscA== X-Received: by 2002:a63:d20e:: with SMTP id a14-v6mr11386382pgg.226.1534021230782; Sat, 11 Aug 2018 14:00:30 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.00.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:00:29 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:14 +0800 Message-Id: <20180811205924.4113-11-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::52b Subject: [Qemu-devel] [PATCH V11 10/20] qmp event: Add COLO_EXIT event to notify users while exited COLO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x-colo-lost-heartbeat', Users can intervene in COLO's failover work immediately. If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Eric Blake Reviewed-by: Markus Armbruster --- migration/colo.c | 31 +++++++++++++++++++++++++++++++ qapi/migration.json | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index c083d3696f..ab484ad754 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -28,6 +28,7 @@ #include "net/colo-compare.h" #include "net/colo.h" #include "block/block.h" +#include "qapi/qapi-events-migration.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -514,6 +515,23 @@ out: qemu_fclose(fb); } + /* + * There are only two reasons we can get here, some error happened + * or the user triggered failover. + */ + switch (failover_get_state()) { + case FAILOVER_STATUS_NONE: + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_ERROR, NULL); + break; + case FAILOVER_STATUS_REQUIRE: + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_REQUEST, NULL); + break; + default: + abort(); + } + /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); @@ -745,6 +763,19 @@ out: error_report_err(local_err); } + switch (failover_get_state()) { + case FAILOVER_STATUS_NONE: + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_ERROR, NULL); + break; + case FAILOVER_STATUS_REQUIRE: + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_REQUEST, NULL); + break; + default: + abort(); + } + if (fb) { qemu_fclose(fb); } diff --git a/qapi/migration.json b/qapi/migration.json index 186e8a7303..36faea59c1 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -900,6 +900,44 @@ { 'enum': 'FailoverStatus', 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } +## +# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +# +# @mode: report COLO mode when COLO exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 3.1 +# +# Example: +# +# <- { "timestamp": {"seconds": 2032141960, "microseconds": 417172}, +# "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "request" } } +# +## +{ 'event': 'COLO_EXIT', + 'data': {'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## +# @COLOExitReason: +# +# The reason for a COLO exit +# +# @none: no failover has ever happened. This can't occur in the +# COLO_EXIT event, only in the result of query-colo-status. +# +# @request: COLO exit is due to an external request +# +# @error: COLO exit is due to an internal error +# +# Since: 3.1 +## +{ 'enum': 'COLOExitReason', + 'data': [ 'none', 'request', 'error' ] } + ## # @x-colo-lost-heartbeat: # From patchwork Sat Aug 11 20:59:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563561 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B8FEA1510 for ; Sat, 11 Aug 2018 21:08:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AAB54298D9 for ; Sat, 11 Aug 2018 21:08:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9B0B629915; Sat, 11 Aug 2018 21:08:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 404C4298D9 for ; Sat, 11 Aug 2018 21:08:56 +0000 (UTC) Received: from localhost ([::1]:33213 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob83-0003f8-Fi for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:08:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48200) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob02-0005DA-HJ for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fob01-0001Qr-P4 for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:38 -0400 Received: from mail-pl0-x22f.google.com ([2607:f8b0:400e:c01::22f]:33031) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fob01-0001QY-JV for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:37 -0400 Received: by mail-pl0-x22f.google.com with SMTP id b90-v6so5396307plb.0 for ; Sat, 11 Aug 2018 14:00:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=r8TzZ9DQANYfrVKfulYSTmRjRXN9eA0MPtIvg6/ga8Y=; b=je4M50hST6wCYYxczr+raDAanLodTQdf+DJbhUQPsdGXrhpxGdEN2AGzjgY9DIs5H9 4upmeQXHrSmeCb05FjkjdbuC3r8XuOgtM3mviT4dLaKY2wPakYGDm3wZ6K1j/RoO+mZh Q79w9MIsnbbwvtFxCCi2WfxtNNl4gdZsk6ffwkJeBezxBsYeGcQk2H3EmZko0NTp9yTY +kmqTf56ZwOZoqtnx//5RmfsAYmenbHhNsOmGvKVjKnkmaOaSahq8W2BDzbbmB0UxmrB OnFJhvMQvjXkdsbSYWwfqewAhRwD6t0JT+TrNN70jesaH/fNs507x7ExrSsRybvb2dwM hYKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=r8TzZ9DQANYfrVKfulYSTmRjRXN9eA0MPtIvg6/ga8Y=; b=gIEB3Xe+G/Hft4L8GB67XJzzlP+RgR1Ny2gS/7ZBaaJQm2L7chlaxAJ/8pqtVX9E6a 8Rbdt2u+aFG8n7c9+Q3k5lT7Nih11veakryeFqs0nj1G6e07XUVRdzsgB6Io6WYDECOG H9SJR22ikpYL3dEkcz8odMLtupUz5Jkj0OXW7eWwq735w8kUqDdR9UIlwa10znG+TNF3 nZq8WHcSoaIm+r4JfN89WTbZ8C1mPBPlaX33DliuVw+xyXkBTD9eh6SaonL6qbAfSLJy ObwziDCXlDGxiP5mBrjDd8GBQgx2u7T9kCLgCVO7z8ypCab3CL4N8Spl4GcHSEALLiIF qnnA== X-Gm-Message-State: AOUpUlEnPHznRKxImBroYfLX8TSFNj9B6kli1CGKPjook/7u0Js6H4oQ GvdgUXYC/4/trpNb5bTZOIxJumRr X-Google-Smtp-Source: AA+uWPzjq6PCMeW9E7psh8tERiA5uTzcfap2akYjg8VOe3S7HbOqrUXLHzGwFsmub+vnZdcI6jFArw== X-Received: by 2002:a17:902:900a:: with SMTP id a10-v6mr11067234plp.143.1534021236278; Sat, 11 Aug 2018 14:00:36 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.00.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:00:35 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:15 +0800 Message-Id: <20180811205924.4113-12-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::22f Subject: [Qemu-devel] [PATCH V11 11/20] qapi/migration.json: Rename COLO unknown mode to none mode. X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Zhang Chen Suggested by Markus Armbruster rename COLO unknown mode to none mode. Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Eric Blake Reviewed-by: Markus Armbruster --- migration/colo-failover.c | 2 +- migration/colo.c | 2 +- qapi/migration.json | 10 +++++----- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/migration/colo-failover.c b/migration/colo-failover.c index 0ae0c41221..4854a96c92 100644 --- a/migration/colo-failover.c +++ b/migration/colo-failover.c @@ -77,7 +77,7 @@ FailoverStatus failover_get_state(void) void qmp_x_colo_lost_heartbeat(Error **errp) { - if (get_colo_mode() == COLO_MODE_UNKNOWN) { + if (get_colo_mode() == COLO_MODE_NONE) { error_setg(errp, QERR_FEATURE_DISABLED, "colo"); return; } diff --git a/migration/colo.c b/migration/colo.c index ab484ad754..8fdb79ac73 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -160,7 +160,7 @@ COLOMode get_colo_mode(void) } else if (migration_incoming_in_colo_state()) { return COLO_MODE_SECONDARY; } else { - return COLO_MODE_UNKNOWN; + return COLO_MODE_NONE; } } diff --git a/qapi/migration.json b/qapi/migration.json index 36faea59c1..78932206dd 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -867,18 +867,18 @@ ## # @COLOMode: # -# The colo mode +# The COLO current mode. # -# @unknown: unknown mode +# @none: COLO is disabled. # -# @primary: master side +# @primary: COLO node in primary side. # -# @secondary: slave side +# @secondary: COLO node in slave side. # # Since: 2.8 ## { 'enum': 'COLOMode', - 'data': [ 'unknown', 'primary', 'secondary'] } + 'data': [ 'none', 'primary', 'secondary'] } ## # @FailoverStatus: From patchwork Sat Aug 11 20:59:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563573 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 096B11510 for ; Sat, 11 Aug 2018 21:13:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F20A32990B for ; Sat, 11 Aug 2018 21:13:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E3EF029946; Sat, 11 Aug 2018 21:13:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 881312990B for ; Sat, 11 Aug 2018 21:13:56 +0000 (UTC) Received: from localhost ([::1]:33242 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fobCt-0000Ce-Lc for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:13:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48224) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob0A-0005Hp-8G for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fob07-0001So-2W for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:46 -0400 Received: from mail-pl0-x241.google.com ([2607:f8b0:400e:c01::241]:42183) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fob06-0001SP-TJ for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:43 -0400 Received: by mail-pl0-x241.google.com with SMTP id g6-v6so5380157plq.9 for ; Sat, 11 Aug 2018 14:00:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PWweyNM/R6oxWdlr2CZr7URUz0go+xHI7GiQr8rdHHk=; b=sGGBweLNF0faVRJPjaRo4eISW6TIyMAYsv5TINLABJG6QdlrJITGifxoDK6AwfVnC9 ysN0XqohTTGJcvzz2H59kZ1cGf+7+1784dyPMafk1T+BcSahi4ebxLNulWAufb+2QjEs X/kfWWvoFaAuXIrd6fynZH8sEtmuNRsYINR7wRvoivWnqrrd7cprXjj5lbbob7VCPHDQ 3JdGbhJ2pMzCTkg6onaGmJS8YmT+LySlkXyBf0zXRz/RcLbjkzODnlP3XYeGos7E/wlZ UHyvmYLJPPRHwi7EpiipgnPjUzpUZR0+I9v6dDcuK+GBndpGrZ/5/iEaKZa2cd0Ompus 96jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PWweyNM/R6oxWdlr2CZr7URUz0go+xHI7GiQr8rdHHk=; b=Mib0GX6epC4koSjiUhX9g+jp0KypQA1mwraEcFYwRMBqx3/QhZGcUXbWqG0Jwx/Leu 4n/NvSf8RRz+vu+wl0ZngslmgetWhdo1J0P4Zfg1fmgovJYyPhpoWahxvm1GhfT0VoLn s1PsFSnm6ALvOfsBYUr4HHkxXeewiarKXRnlVA1OXMRYWKlFqGmAv6cDHNLRPsIbFYDH XROn/7GKkHh6EzILCXJurWMUy0fP2sdRGD9Vwj/hIP6z2GEJJ63B00VcE5M+nQKOcjGs wy0A9HFcAfnW+hsu4nqoL1Fno62ZYfcuhmO6NvRMociR1LxnADsirD7UlVReFru1JcgN 5ncg== X-Gm-Message-State: AOUpUlE0rvIRVutuCn31MuRgAYWQNACXmZNyzA48pXUbBketRTCxLQqK fcwNAnj0+tLazrjMYCBTHe3KN3Uj X-Google-Smtp-Source: AA+uWPxwtpMgucgh9CnXAIw/RDPBvAm82WXyWMX+aHJ+z++XtNQ09updsz98pKi+NDKccSO1XkHL1w== X-Received: by 2002:a17:902:4381:: with SMTP id j1-v6mr11098234pld.104.1534021241675; Sat, 11 Aug 2018 14:00:41 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.00.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:00:40 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:16 +0800 Message-Id: <20180811205924.4113-13-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::241 Subject: [Qemu-devel] [PATCH V11 12/20] qapi: Add new command to query colo status X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Libvirt or other high level software can use this command query colo status. You can test this command like that: {'execute':'query-colo-status'} Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen --- migration/colo.c | 21 +++++++++++++++++++++ qapi/migration.json | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 8fdb79ac73..b355de3f01 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -29,6 +29,7 @@ #include "net/colo.h" #include "block/block.h" #include "qapi/qapi-events-migration.h" +#include "qapi/qmp/qerror.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -237,6 +238,26 @@ void qmp_xen_colo_do_checkpoint(Error **errp) #endif } +COLOStatus *qmp_query_colo_status(Error **errp) +{ + COLOStatus *s = g_new0(COLOStatus, 1); + + s->mode = get_colo_mode(); + + switch (failover_get_state()) { + case FAILOVER_STATUS_NONE: + s->reason = COLO_EXIT_REASON_NONE; + break; + case FAILOVER_STATUS_REQUIRE: + s->reason = COLO_EXIT_REASON_REQUEST; + break; + default: + s->reason = COLO_EXIT_REASON_ERROR; + } + + return s; +} + static void colo_send_message(QEMUFile *f, COLOMessage msg, Error **errp) { diff --git a/qapi/migration.json b/qapi/migration.json index 78932206dd..87312dd093 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1251,6 +1251,38 @@ ## { 'command': 'xen-colo-do-checkpoint' } +## +# @COLOStatus: +# +# The result format for 'query-colo-status'. +# +# @mode: COLO running mode. If COLO is running, this field will return +# 'primary' or 'secondary'. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 3.0 +## +{ 'struct': 'COLOStatus', + 'data': { 'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## +# @query-colo-status: +# +# Query COLO status while the vm is running. +# +# Returns: A @COLOStatus object showing the status. +# +# Example: +# +# -> { "execute": "query-colo-status" } +# <- { "return": { "mode": "primary", "active": true, "reason": "request" } } +# +# Since: 3.0 +## +{ 'command': 'query-colo-status', + 'returns': 'COLOStatus' } + ## # @migrate-recover: # From patchwork Sat Aug 11 20:59:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563559 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CBA6B13B4 for ; Sat, 11 Aug 2018 21:08:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BE6AD298D9 for ; Sat, 11 Aug 2018 21:08:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B1DBF29915; Sat, 11 Aug 2018 21:08:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 07453298D9 for ; Sat, 11 Aug 2018 21:08:24 +0000 (UTC) Received: from localhost ([::1]:33212 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob7X-0003H5-7k for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:08:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48250) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob0E-0005Ih-18 for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fob0C-0001VH-Pf for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:50 -0400 Received: from mail-pf1-x42e.google.com ([2607:f8b0:4864:20::42e]:46993) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fob0C-0001Uo-HP for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:48 -0400 Received: by mail-pf1-x42e.google.com with SMTP id u24-v6so5988319pfn.13 for ; Sat, 11 Aug 2018 14:00:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9X9qXjiTTLJdVwYXtKH6lkWk/cJysqmZIj5m9WkoURo=; b=Z8hyvupWIP0tvr+KzfFlQ61E3MdhE+jpBXkYKya2NCALY80WQfTlYbE3qeUei3wIa7 nf5NjjA0PLtX50bmSOAGSE7eLP8Y8MQKEnl7tCn62Scq9UMURfeCqdl/F9GW1kiMxMik 3LZo3kdulsxVTpIyw39Dcz+kmoEMyi8xCPrOFItruYd6kIdVmfRXvRE8ET2+XsEo97Q7 8ObnmpMWRmfLM5Lzflfmg1zpu33NKZhPAxoCUsojChGDvYMPpYoJNuOE9XfWzh9C1BUb dR1Qr9a8/PO0f3dL+c0K0PqTW+cpl+Z5ImYDKAqCupLRBxUQU3wp53Nz+JWi8rp82qqv vMbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9X9qXjiTTLJdVwYXtKH6lkWk/cJysqmZIj5m9WkoURo=; b=YQcl0lfrcF/NKjjgUvLcmXDacqQ3jw3SdzpSs8roYxNQIyebCbGLg6qn8LvWfLAd9Q IpS3gmNr+z75mJLSh/Q4I7VkzVlFXXXf4oRhh7Q93wIxmitXHQ5bDdG5HVR8Ui3EqpOW 7V/uYoc3aYgXbgkSlEKz/yECknY+GDsG1fXafr2RYGio0fsynrYw77IQVBeFVKQnyZ7W O9ITB3C+zsBvhTD0XLrBYaqnoe0svzvc1eKL9IKKmNkzuFnWWl9cAp0gJy7xHvxx2vxp bXvgJtrYmVoFU/ClODAlNCGm1o/vkwCXhUv6brqR6Iz7lKpdFPHqLSfKeLpLO7nFkysB wWIA== X-Gm-Message-State: AOUpUlGTYQYI3B6pXxAgPJ/CBDQVQ8DGc978HWNezibu8QBwXJgC0DSL PeORrbtSyoj1gsggNfQCqNdjwnNw X-Google-Smtp-Source: AA+uWPzyv9SDimHhL6paQg63ftmprAFz8yl0WU/d0NcPWGf5HXF8V6NZmK0F7RKpVKRWq6iwy++CwQ== X-Received: by 2002:a63:704f:: with SMTP id a15-v6mr11342789pgn.443.1534021247258; Sat, 11 Aug 2018 14:00:47 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.00.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:00:46 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:17 +0800 Message-Id: <20180811205924.4113-14-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42e Subject: [Qemu-devel] [PATCH V11 13/20] savevm: split the process of different stages for loadvm/savevm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP There are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections. We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state. So we add three new helper functions: qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration. Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 41 ++++++++++++++++++++++++++++++++--------- migration/savevm.c | 36 +++++++++++++++++++++++++++++------- migration/savevm.h | 4 ++++ 3 files changed, 65 insertions(+), 16 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index b355de3f01..688d6f40b2 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -30,6 +30,7 @@ #include "block/block.h" #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" +#include "sysemu/cpus.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -401,23 +402,34 @@ static int colo_do_checkpoint_transaction(MigrationState *s, /* Disable block migration */ migrate_set_block_enabled(false, &local_err); - qemu_savevm_state_header(fb); - qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); replication_do_checkpoint_all(&local_err); if (local_err) { qemu_mutex_unlock_iothread(); goto out; } - qemu_savevm_state_complete_precopy(fb, false, false); - qemu_mutex_unlock_iothread(); - - qemu_fflush(fb); colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_err); if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* Note: device state is saved into buffer */ + ret = qemu_save_device_state(fb); + + qemu_mutex_unlock_iothread(); + if (ret < 0) { goto out; } + /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + + qemu_fflush(fb); + /* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read. @@ -635,6 +647,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t total_size; uint64_t value; Error *local_err = NULL; + int ret; qemu_sem_init(&mis->colo_incoming_sem, 0); @@ -707,6 +720,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + qemu_mutex_lock_iothread(); + cpu_synchronize_all_pre_loadvm(); + ret = qemu_loadvm_state_main(mis->from_src_file, mis); + qemu_mutex_unlock_iothread(); + + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } + value = colo_receive_message_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); if (local_err) { @@ -738,10 +761,10 @@ void *colo_process_incoming_thread(void *opaque) } qemu_mutex_lock_iothread(); - qemu_system_reset(SHUTDOWN_CAUSE_NONE); vmstate_loading = true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + ret = qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } diff --git a/migration/savevm.c b/migration/savevm.c index 33e9e7cda0..3a6aa747f9 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1378,13 +1378,21 @@ done: return ret; } -static int qemu_save_device_state(QEMUFile *f) +void qemu_savevm_live_state(QEMUFile *f) { - SaveStateEntry *se; + /* save QEMU_VM_SECTION_END section */ + qemu_savevm_state_complete_precopy(f, true, false); + qemu_put_byte(f, QEMU_VM_EOF); +} - qemu_put_be32(f, QEMU_VM_FILE_MAGIC); - qemu_put_be32(f, QEMU_VM_FILE_VERSION); +int qemu_save_device_state(QEMUFile *f) +{ + SaveStateEntry *se; + if (!migration_in_colo_state()) { + qemu_put_be32(f, QEMU_VM_FILE_MAGIC); + qemu_put_be32(f, QEMU_VM_FILE_VERSION); + } cpu_synchronize_all_states(); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1440,8 +1448,6 @@ enum LoadVMExitCodes { LOADVM_QUIT = 1, }; -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); - /* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything @@ -2241,7 +2247,7 @@ static bool postcopy_pause_incoming(MigrationIncomingState *mis) return true; } -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) { uint8_t section_type; int ret = 0; @@ -2410,6 +2416,22 @@ int qemu_loadvm_state(QEMUFile *f) return ret; } +int qemu_load_device_state(QEMUFile *f) +{ + MigrationIncomingState *mis = migration_incoming_get_current(); + int ret; + + /* Load QEMU_VM_SECTION_FULL section */ + ret = qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to load device state: %d", ret); + return ret; + } + + cpu_synchronize_all_post_init(); + return 0; +} + int save_snapshot(const char *name, Error **errp) { BlockDriverState *bs, *bs1; diff --git a/migration/savevm.h b/migration/savevm.h index 8373c2f6bd..51a4b9caa8 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -56,8 +56,12 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint64_t *start_list, uint64_t *length_list); void qemu_savevm_send_colo_enable(QEMUFile *f); +void qemu_savevm_live_state(QEMUFile *f); +int qemu_save_device_state(QEMUFile *f); int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); +int qemu_load_device_state(QEMUFile *f); #endif From patchwork Sat Aug 11 20:59:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563567 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 663351510 for ; Sat, 11 Aug 2018 21:11:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 58A10298D9 for ; Sat, 11 Aug 2018 21:11:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4BC4629915; Sat, 11 Aug 2018 21:11:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id BE2AE298D9 for ; Sat, 11 Aug 2018 21:11:47 +0000 (UTC) Received: from localhost ([::1]:33230 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fobAo-0006x1-VE for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:11:46 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48265) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob0L-0005MI-6o for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fob0I-0001XI-5w for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:57 -0400 Received: from mail-pg1-x543.google.com ([2607:f8b0:4864:20::543]:45671) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fob0H-0001Wy-W1 for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:54 -0400 Received: by mail-pg1-x543.google.com with SMTP id f1-v6so5849141pgq.12 for ; Sat, 11 Aug 2018 14:00:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FQkHD0iF4ZCPUwLAsj1ShX86lwX7Htt016788pFu0iI=; b=PFxyjeqfK9uYnfvUVQuBhAshiTyQwFanwnQ5KtDX6VJYiaIEVUOhLuntYKBIzm9qzB Uh5KOdRwHv74j9OXmvobZNfnRlQ2pmaUzJOlgNs9Y1P27hz2nTaVWgNS24Xo4tHqanXf UDPOVqIYZFy4ldwrABj2kF6+bKgDlGj6tlhRBS4C9hx2ijdEonJ6y4AZ8qVKgRGABOob awjLF8EY5T8FrC9iB0QzxpQ1+y8WAlLUfBhDYlwhP2Gf1ZNRX3W07+JFCD+yqDhHbJUx 97Lw6d2cLm0zTRBfMQ5/jwrngC/jmwR6dPw7IqSM8lgL4zDI3IBxF47QTmsPwYjE0Nu5 5moA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FQkHD0iF4ZCPUwLAsj1ShX86lwX7Htt016788pFu0iI=; b=Bh8CRWsGyveVO60SmhmyY5Gl6yvafv3iuU6YoL2Y2RVvgUWzGk7uKD3+lIlpvCECOS 8sTZM/PbuEX5XN7XNZ19/86YijCZBa4hT1MNtG9gg9m2apvB4GFdZvP750cb6Yx+SJKQ XTIrlc4Ti3H0bBL+XenMaXmyh9DHPkOw+SVDuwraloys5NJMK/yj9Or2kBlBEFUMGLOK 4Lw546jOk2GiHg3YnFW873kFn987yId8FtSky9aI3s2CSeradCX1XvrN+ns9NJaRE9IR ooYpbVOiou7CR8ICHy38fU6ebP9Bl3+aJmNo9P00fPWqoIcbHkZYS1Tj+4dc8HkaR+UO 6bcQ== X-Gm-Message-State: AOUpUlFoXbOuP4h6czPzfiYR7qyAM8oKc5L4uzJfAe8azJvYjXHhqinL tSaX8OfdX3b0keirQ33gwEjtR3sQ X-Google-Smtp-Source: AA+uWPwnUBdTBFvnVbDeJfKgUagCDS8rMdgTs0bJJzSeQjtXpy9owfLgoh0W/uHtoXhW/xl3qsucDg== X-Received: by 2002:a62:8d84:: with SMTP id p4-v6mr12796371pfk.251.1534021252661; Sat, 11 Aug 2018 14:00:52 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.00.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:00:51 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:18 +0800 Message-Id: <20180811205924.4113-15-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::543 Subject: [Qemu-devel] [PATCH V11 14/20] COLO: flush host dirty ram from cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index b4cbcf783f..734e944a21 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3588,6 +3588,7 @@ int colo_init_ram_cache(void) } ram_state = g_new0(RAMState, 1); ram_state->migration_dirty_pages = 0; + memory_global_dirty_log_start(); return 0; @@ -3609,6 +3610,7 @@ void colo_release_ram_cache(void) { RAMBlock *block; + memory_global_dirty_log_stop(); RAMBLOCK_FOREACH_MIGRATABLE(block) { g_free(block->bmap); block->bmap = NULL; @@ -3852,6 +3854,13 @@ static void colo_flush_ram_cache(void) void *src_host; unsigned long offset = 0; + memory_global_dirty_log_sync(); + rcu_read_lock(); + RAMBLOCK_FOREACH_MIGRATABLE(block) { + migration_bitmap_sync_range(ram_state, block, 0, block->used_length); + } + rcu_read_unlock(); + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); rcu_read_lock(); block = QLIST_FIRST_RCU(&ram_list.blocks); From patchwork Sat Aug 11 20:59:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563563 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AB92F1510 for ; Sat, 11 Aug 2018 21:11:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9A35C29906 for ; Sat, 11 Aug 2018 21:11:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8BDDC298D9; Sat, 11 Aug 2018 21:11:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 10D0E298D9 for ; Sat, 11 Aug 2018 21:11:01 +0000 (UTC) Received: from localhost ([::1]:33228 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fobA5-0006Lt-38 for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:11:01 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48276) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob0O-0005Ov-AK for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fob0N-0001Z2-Ih for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:00 -0400 Received: from mail-pg1-x536.google.com ([2607:f8b0:4864:20::536]:46832) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fob0N-0001Ym-CV for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:00:59 -0400 Received: by mail-pg1-x536.google.com with SMTP id f14-v6so5850797pgv.13 for ; Sat, 11 Aug 2018 14:00:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=kUQIPs69dPjU+OLFvSIP7I8hJ4jrJ7hci6aj9qXare4=; b=VM20Qo+zYM0YoOuNIYjt2ImNkEuT4xnfWRfFYSDowjVDSee9sK/vwMj4p7nyjeC71B tNX+WFOKj1fnfhgSAaPLJFYULLiLZfpVg9F4POWzwTH5n09nSwjLjpKPZreGbQv1aSgE airkJ9w5vCKXKrUs2xi8xMTrkl82Wy+LTPNp2ufUObrPjXlADQreR05FLMcWpaxwiNNI a5yCUL/MhFcfori/pjSzg0QKSMrn8yrKci/sES1SMraidU/jvc/YueZt75KE/pAPzsoN v9KYlrt+wE10ixeWuxIytpeI4oPfk1whZIXistieREgUL5fLLuw2NcD8Oq9VdoAAgniE CZvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=kUQIPs69dPjU+OLFvSIP7I8hJ4jrJ7hci6aj9qXare4=; b=V1lzgs4dvNj2iHcjfoIrafGM0Tj1x9nwhi3qzYEfPLJXX12BpQbLLk/zrfm1WKAn9M roofg9ots6t7LAXFORzqZQZbgqLMa8UkK6VRRppdyL85gOs9cQokE8u32AKnLWu8LLeQ 40w03PT8WjHTMHHxrp2N1DAAD73yJL//V9zEbvTtN72zfCHffgrHG00WmcvuM6wNEZaq Y4R518TEiX9AMNbkM/q0EN8DenEy2p2pONt4adabuBXAOgh/+NB3aOfk5TV2ltkETcmv FVUKqWR0kSdTHyNTE6ihrnGibnjKw3oEcSi+R0l9mn06o4k2C/zCy2PMzVMJAhZGSnD3 YgnQ== X-Gm-Message-State: AOUpUlGL8yCZZuQcrT5yen3hYVOaQJ2lp2+CvTZ7zXI/J8rm8WajMPAa kFSvXJB/s8oPmpYLUYs4CpS9dXl0 X-Google-Smtp-Source: AA+uWPxmA/Ni+S3D8QeszkMxXYpX3QHVMyJhmXlvRbucl3BGGTLzOT8liFmlR9aD5ZDEELF/IZRcVA== X-Received: by 2002:a63:7c5c:: with SMTP id l28-v6mr11562876pgn.352.1534021258091; Sat, 11 Aug 2018 14:00:58 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.00.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:00:57 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:19 +0800 Message-Id: <20180811205924.4113-16-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::536 Subject: [Qemu-devel] [PATCH V11 15/20] net/net.c: Add net client type check function for COLO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Zhang Chen We add is_colo_support_client_type() to check the net client type for COLO-compare. Currently we just support TAP. Suggested by Jason. Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen --- include/net/net.h | 1 + net/colo-compare.c | 5 +++++ net/net.c | 14 ++++++++++++++ 3 files changed, 20 insertions(+) diff --git a/include/net/net.h b/include/net/net.h index 1425960f76..dcbc7ba9c0 100644 --- a/include/net/net.h +++ b/include/net/net.h @@ -212,6 +212,7 @@ void hmp_host_net_add(Monitor *mon, const QDict *qdict); void hmp_host_net_remove(Monitor *mon, const QDict *qdict); void netdev_add(QemuOpts *opts, Error **errp); void qmp_netdev_add(QDict *qdict, QObject **ret, Error **errp); +bool is_colo_support_client_type(void); int net_hub_id_for_client(NetClientState *nc, int *id); NetClientState *net_hub_port_find(int hub_id); diff --git a/net/colo-compare.c b/net/colo-compare.c index 426eab5973..b8c0240725 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -996,6 +996,11 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) return; } + if (!is_colo_support_client_type()) { + error_setg(errp, "COLO-compare: Net client type is not supported"); + return; + } + net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); diff --git a/net/net.c b/net/net.c index 2a3133990c..a77ea88fff 100644 --- a/net/net.c +++ b/net/net.c @@ -1733,3 +1733,17 @@ int net_fill_rstate(SocketReadState *rs, const uint8_t *buf, int size) assert(size == 0); return 0; } + +/* Currently, COLO just support TAP */ +bool is_colo_support_client_type(void) +{ + NetClientState *nc; + + QTAILQ_FOREACH(nc, &net_clients, next) { + if (nc->info->type != NET_CLIENT_DRIVER_TAP) { + return false; + } + } + + return true; +} From patchwork Sat Aug 11 20:59:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563575 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 57B8F13B4 for ; Sat, 11 Aug 2018 21:14:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4904B2990B for ; Sat, 11 Aug 2018 21:14:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3AFF029982; Sat, 11 Aug 2018 21:14:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B9A422990B for ; Sat, 11 Aug 2018 21:14:27 +0000 (UTC) Received: from localhost ([::1]:33244 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fobDO-0000b3-Te for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:14:26 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48288) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob0V-0005Rw-NW for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fob0S-0001ap-Ma for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:07 -0400 Received: from mail-pl0-x234.google.com ([2607:f8b0:400e:c01::234]:40187) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fob0S-0001aY-GL for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:04 -0400 Received: by mail-pl0-x234.google.com with SMTP id s17-v6so5377864plp.7 for ; Sat, 11 Aug 2018 14:01:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ktX0J9bUdPM6g6Qi3dOomt5iQ9JyknqMDbDqhS9ZkG4=; b=BCuSWrZH4GrtKKthHzzINm3gx3/PCHHKl3g8ekq3v6guqyOuX6zJl7tLrXBJ+gJUeC BKl+4jC4Sou226Yi/buQxSE4+FECdbThElk8KNJnTkc/wr3uUIzQfkrFwJndRO0i31iI v2duZQQN/8vj8yWVUY8vE4qQCeF8l3esXTTRnOw1rFsWjr2FS06ELMT6DbGnvn0H8hu7 ZJVyJaxcsVofEjxMjdnA+pJByCT9gf/CTK9P2223ICLpiMM/BqGiokeaJ5lI/dcPbLRM nezwynqb4e1MSvJHsN8UOXwWmZNVcsQ48VtRSnVV1OcYwjbhuP2Cgf/3Ucs8tsInihoT ndgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ktX0J9bUdPM6g6Qi3dOomt5iQ9JyknqMDbDqhS9ZkG4=; b=fpbCuDk2tC9n3r1MXVRDtIj5ETYbZ2OqukEXu6LgoH3beqI5X6hkYnPyURPgFl1TSJ wY0uNIR5rriHmFMXA9D4509nk4oukxWlq8uwDtdGKkj+1Qf3AOvwTEcGWUNKk9lUXoM3 usHcxDz3xjxAAohXAa3sYcqmjVqzSOwqycyoufzQ9a2x4l8pHxCdDLWkkWo0AhqBWbDi JP2EDzoiNBcFsagA+lhBAdyQReOacPMWhqGK1KjRO+Ude7Fv1sY7h77ZY1shzL/dRRQM 5VbBBlLpKjbc6okFT84d0mC36EDG37B1BYVwYZC/3jdzTfQKMbPro89U8Xm3AKXmVUxV vPjw== X-Gm-Message-State: AOUpUlEgIrDlKiaFJGL6THyVZOSi/3eRmnq2VyIFLb1NCvdMBTs56rLm kOg+sdQkpFuQkIz5cek6qRDTpZS3 X-Google-Smtp-Source: AA+uWPyVFQhrdqaxEPSAf0No1UjsbPVFhyh6ib0ctqZPkBJmuEAX0SV1q0C/vieDfBcyPAKwHtjuXw== X-Received: by 2002:a17:902:b60b:: with SMTP id b11-v6mr10953400pls.301.1534021263210; Sat, 11 Aug 2018 14:01:03 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.00.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:01:02 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:20 +0800 Message-Id: <20180811205924.4113-17-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::234 Subject: [Qemu-devel] [PATCH V11 16/20] filter: Add handle_event method for NetFilterClass X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Signed-off-by: zhanghailiang --- include/net/filter.h | 5 +++++ net/filter.c | 17 +++++++++++++++++ net/net.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+) diff --git a/include/net/filter.h b/include/net/filter.h index 435acd6f82..49da666ac0 100644 --- a/include/net/filter.h +++ b/include/net/filter.h @@ -38,6 +38,8 @@ typedef ssize_t (FilterReceiveIOV)(NetFilterState *nc, typedef void (FilterStatusChanged) (NetFilterState *nf, Error **errp); +typedef void (FilterHandleEvent) (NetFilterState *nf, int event, Error **errp); + typedef struct NetFilterClass { ObjectClass parent_class; @@ -45,6 +47,7 @@ typedef struct NetFilterClass { FilterSetup *setup; FilterCleanup *cleanup; FilterStatusChanged *status_changed; + FilterHandleEvent *handle_event; /* mandatory */ FilterReceiveIOV *receive_iov; } NetFilterClass; @@ -77,4 +80,6 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender, int iovcnt, void *opaque); +void colo_notify_filters_event(int event, Error **errp); + #endif /* QEMU_NET_FILTER_H */ diff --git a/net/filter.c b/net/filter.c index 2fd7d7d663..0f17eba143 100644 --- a/net/filter.c +++ b/net/filter.c @@ -17,6 +17,8 @@ #include "net/vhost_net.h" #include "qom/object_interfaces.h" #include "qemu/iov.h" +#include "net/colo.h" +#include "migration/colo.h" static inline bool qemu_can_skip_netfilter(NetFilterState *nf) { @@ -245,11 +247,26 @@ static void netfilter_finalize(Object *obj) g_free(nf->netdev_id); } +static void dummy_handle_event(NetFilterState *nf, int event, Error **errp) +{ + switch (event) { + case COLO_EVENT_CHECKPOINT: + break; + case COLO_EVENT_FAILOVER: + object_property_set_str(OBJECT(nf), "off", "status", errp); + break; + default: + break; + } +} + static void netfilter_class_init(ObjectClass *oc, void *data) { UserCreatableClass *ucc = USER_CREATABLE_CLASS(oc); + NetFilterClass *nfc = NETFILTER_CLASS(oc); ucc->complete = netfilter_complete; + nfc->handle_event = dummy_handle_event; } static const TypeInfo netfilter_info = { diff --git a/net/net.c b/net/net.c index a77ea88fff..b4f6a2efb2 100644 --- a/net/net.c +++ b/net/net.c @@ -1331,6 +1331,34 @@ void hmp_info_network(Monitor *mon, const QDict *qdict) } } +void colo_notify_filters_event(int event, Error **errp) +{ + NetClientState *nc, *peer; + NetClientDriver type; + NetFilterState *nf; + NetFilterClass *nfc = NULL; + Error *local_err = NULL; + + QTAILQ_FOREACH(nc, &net_clients, next) { + peer = nc->peer; + type = nc->info->type; + if (!peer || type != NET_CLIENT_DRIVER_TAP) { + continue; + } + QTAILQ_FOREACH(nf, &nc->filters, next) { + nfc = NETFILTER_GET_CLASS(OBJECT(nf)); + if (!nfc->handle_event) { + continue; + } + nfc->handle_event(nf, event, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + } + } +} + void qmp_set_link(const char *name, bool up, Error **errp) { NetClientState *ncs[MAX_QUEUE_NUM]; From patchwork Sat Aug 11 20:59:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563571 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3DD481510 for ; Sat, 11 Aug 2018 21:13:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 246B52990B for ; Sat, 11 Aug 2018 21:13:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 13B5629946; Sat, 11 Aug 2018 21:13:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 68A222990B for ; Sat, 11 Aug 2018 21:13:52 +0000 (UTC) Received: from localhost ([::1]:33241 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fobCp-00008y-78 for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:13:51 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48310) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob0Z-0005VX-Jk for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fob0Y-0001dG-Eg for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:11 -0400 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:36002) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fob0Y-0001cv-5z for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:10 -0400 Received: by mail-pg1-x541.google.com with SMTP id h12-v6so5871269pgs.3 for ; Sat, 11 Aug 2018 14:01:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=F9zSpxYMysq3FvGzJajTPFG8q/vQadcxc2+IrJzc1JQ=; b=PT5GgEmDSnZFjvMUpwAO3gXRtjMa/g2zYVROFYnG2TEPpDbUtk3Aw5neMz8Km7RJRe f3JvFt87m+Ipco3hppERCPCi1VdXAN9qjutGskbSfstr23fsXuFpYOlc1r9RhaugqS76 /SljnGRkgzYEEtwkLMMHXXY3/BgkZL3HJ0Xt/fTxKk0jusROwsvAz+UFm1Sk0z53VReJ x96ILHuzORI2mNtcsEmaAcRDi6H5aS9C5Obt9jv+kkT/kDuHo3p78MpEjeMvDkNmR3yN wKEAnWmUNDQGmIB/JeA9jyBa8nByphSv+KkSmkQXYiB2oHYc/RfVjEsMtvvaXyl3NzQJ rPlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=F9zSpxYMysq3FvGzJajTPFG8q/vQadcxc2+IrJzc1JQ=; b=FPVulrHzwWRkogeg3Tsa9H0q/ucW3p1FlpDtXGvqwGy8mBnHdESAvc8cqKstymPHQa Ct63UPg6p5R5nopqkMlQ8LvzuMW72hO9g3/njk0/uEeDr1vSdRJLlXOIp8aHZ7sAU3cz 2FwnITN487XOdg+kvi7YtfHihGj5jm8e2LgKAyyLtsSC2cbYTsGLZiEcicAS4PgxplQQ prmTIDhR3qecLPEoz+CJb1GsyzL+7FDW5WwQtObfFaKGpY4wSKF6IFsrX+D/RX+n7t5p w2HmZfPKnOZeJg2K3j2PgM4NDRvBisPN7CJVK51VRR+0zNMW3PVIGkjW70bzN+8QNW4w dcXA== X-Gm-Message-State: AOUpUlHVMhPVMoe9hFfUNxj3HSHFqE+UZaBVJhkYSFARHY7oUg2vmwY2 DxBMlMuZHtx/QTviSOOQ1ZgCg2PS X-Google-Smtp-Source: AA+uWPx68LHfezuomym9iAl78K4mfoRfhg8q9iegdWNEM6KxRVVDHAJ3LheiGuBioCIEPh2w/oRIUw== X-Received: by 2002:a63:1403:: with SMTP id u3-v6mr11417411pgl.13.1534021268862; Sat, 11 Aug 2018 14:01:08 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.01.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:01:07 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:21 +0800 Message-Id: <20180811205924.4113-18-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::541 Subject: [Qemu-devel] [PATCH V11 17/20] filter-rewriter: handle checkpoint and failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP After one round of checkpoint, the states between PVM and SVM become consistent, so it is unnecessary to adjust the sequence of net packets for old connections, besides, while failover happens, filter-rewriter will into failover mode that needn't handle the new TCP connection. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen --- net/colo-compare.c | 12 ++++----- net/colo.c | 8 ++++++ net/colo.h | 2 ++ net/filter-rewriter.c | 57 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 73 insertions(+), 6 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index b8c0240725..462e822be6 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -116,6 +116,12 @@ enum { SECONDARY_IN, }; +static void colo_compare_inconsistency_notify(void) +{ + notifier_list_notify(&colo_compare_notifiers, + migrate_get_current()); +} + static int compare_chr_send(CompareState *s, const uint8_t *buf, uint32_t size, @@ -562,12 +568,6 @@ static int colo_old_packet_check_one(Packet *pkt, int64_t *check_time) } } -static void colo_compare_inconsistency_notify(void) -{ - notifier_list_notify(&colo_compare_notifiers, - migrate_get_current()); -} - void colo_compare_register_notifier(Notifier *notify) { notifier_list_add(&colo_compare_notifiers, notify); diff --git a/net/colo.c b/net/colo.c index 97c8fc928f..49176bf07b 100644 --- a/net/colo.c +++ b/net/colo.c @@ -221,3 +221,11 @@ Connection *connection_get(GHashTable *connection_track_table, return conn; } + +bool connection_has_tracked(GHashTable *connection_track_table, + ConnectionKey *key) +{ + Connection *conn = g_hash_table_lookup(connection_track_table, key); + + return conn ? true : false; +} diff --git a/net/colo.h b/net/colo.h index 0277e0e9ba..11c5226488 100644 --- a/net/colo.h +++ b/net/colo.h @@ -98,6 +98,8 @@ void connection_destroy(void *opaque); Connection *connection_get(GHashTable *connection_track_table, ConnectionKey *key, GQueue *conn_list); +bool connection_has_tracked(GHashTable *connection_track_table, + ConnectionKey *key); void connection_hashtable_reset(GHashTable *connection_track_table); Packet *packet_new(const void *data, int size, int vnet_hdr_len); void packet_destroy(void *opaque, void *user_data); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index f18a71bf2e..c463a0c1d0 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -20,11 +20,15 @@ #include "qemu/main-loop.h" #include "qemu/iov.h" #include "net/checksum.h" +#include "net/colo.h" +#include "migration/colo.h" #define FILTER_COLO_REWRITER(obj) \ OBJECT_CHECK(RewriterState, (obj), TYPE_FILTER_REWRITER) #define TYPE_FILTER_REWRITER "filter-rewriter" +#define FAILOVER_MODE_ON true +#define FAILOVER_MODE_OFF false typedef struct RewriterState { NetFilterState parent_obj; @@ -32,8 +36,14 @@ typedef struct RewriterState { /* hashtable to save connection */ GHashTable *connection_track_table; bool vnet_hdr; + bool failover_mode; } RewriterState; +static void filter_rewriter_failover_mode(RewriterState *s) +{ + s->failover_mode = FAILOVER_MODE_ON; +} + static void filter_rewriter_flush(NetFilterState *nf) { RewriterState *s = FILTER_COLO_REWRITER(nf); @@ -269,6 +279,13 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, */ reverse_connection_key(&key); } + + /* After failover we needn't change new TCP packet */ + if (s->failover_mode && + connection_has_tracked(s->connection_track_table, &key)) { + goto out; + } + conn = connection_get(s->connection_track_table, &key, NULL); @@ -302,11 +319,49 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, } } +out: packet_destroy(pkt, NULL); pkt = NULL; return 0; } +static void reset_seq_offset(gpointer key, gpointer value, gpointer user_data) +{ + Connection *conn = (Connection *)value; + + conn->offset = 0; +} + +static gboolean offset_is_nonzero(gpointer key, + gpointer value, + gpointer user_data) +{ + Connection *conn = (Connection *)value; + + return conn->offset ? true : false; +} + +static void colo_rewriter_handle_event(NetFilterState *nf, int event, + Error **errp) +{ + RewriterState *rs = FILTER_COLO_REWRITER(nf); + + switch (event) { + case COLO_EVENT_CHECKPOINT: + g_hash_table_foreach(rs->connection_track_table, + reset_seq_offset, NULL); + break; + case COLO_EVENT_FAILOVER: + if (!g_hash_table_find(rs->connection_track_table, + offset_is_nonzero, NULL)) { + filter_rewriter_failover_mode(rs); + } + break; + default: + break; + } +} + static void colo_rewriter_cleanup(NetFilterState *nf) { RewriterState *s = FILTER_COLO_REWRITER(nf); @@ -350,6 +405,7 @@ static void filter_rewriter_init(Object *obj) RewriterState *s = FILTER_COLO_REWRITER(obj); s->vnet_hdr = false; + s->failover_mode = FAILOVER_MODE_OFF; object_property_add_bool(obj, "vnet_hdr_support", filter_rewriter_get_vnet_hdr, filter_rewriter_set_vnet_hdr, NULL); @@ -362,6 +418,7 @@ static void colo_rewriter_class_init(ObjectClass *oc, void *data) nfc->setup = colo_rewriter_setup; nfc->cleanup = colo_rewriter_cleanup; nfc->receive_iov = colo_rewriter_receive_iov; + nfc->handle_event = colo_rewriter_handle_event; } static const TypeInfo colo_rewriter_info = { From patchwork Sat Aug 11 20:59:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563579 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F355C13BB for ; Sat, 11 Aug 2018 21:16:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E69A52990B for ; Sat, 11 Aug 2018 21:16:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D8F1529982; Sat, 11 Aug 2018 21:16:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 859382990B for ; Sat, 11 Aug 2018 21:16:51 +0000 (UTC) Received: from localhost ([::1]:33260 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fobFi-0003GS-O2 for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:16:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48344) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob0g-0005ai-JQ for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:20 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fob0d-0001fV-Im for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:18 -0400 Received: from mail-pf1-x444.google.com ([2607:f8b0:4864:20::444]:35076) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fob0d-0001f8-CS for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:15 -0400 Received: by mail-pf1-x444.google.com with SMTP id p12-v6so6017204pfh.2 for ; Sat, 11 Aug 2018 14:01:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=icdrUsJnfbDed6Ff8GShXGHEA9pGkgHwJ593EjVc+iw=; b=ciVrvTeg3v4mRHWU7WVYlitqmT0FAPVz1KJ1lntVJ7B9+HQYa3et+tMbFjvte8bJ3M ybWXGDyBIdTi4IWOZDzOHCduZxDvi3dSIMlp73VISpvhJXchRnYlEOcJhnqOf6mMqx5G xCPWVAdVX4nM2+nVUH0eY7zED5cNxUqh5MuTcFmgBC/3X5x3unCQ9+Yw/3JJ6QtFL3cY Fp/0vSWKKkNeMeqVBw+hq/dsxt6Lkd1rkR23Cspr2zqAXLTUlwJ11uIFXywe2IaYIKpm /o19PJ2CIa5NgJ1Xy9fyB04HX4OKNxOLMJiR5liBfTMCIvCjOP/8vnygcUwyQinwnudv eUqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=icdrUsJnfbDed6Ff8GShXGHEA9pGkgHwJ593EjVc+iw=; b=kJH0+6gEl63g7G0Rb0Icz8udhbAH1hn7QEKjynHNdWQWgtg6FYQcAVv54H/kFOm9g9 bCixzjc+uOQVqXG6nfdbnBxsocDC2gUZv3ehwtXES/4CfXZ7ZFeN3SrzCFZbfteqpMy5 Zxik94s8/FAtdnjfsiiZH/2x4AQI59t58Y8Uia/HZIofiKQSCZO2xGNElTCXFuYrONK/ f3yJ4DLXSeJSn4MM6hg06ISAm9yxSVJ6TzdJDY/PEbrRuYOqZm5wJ+l/qK7rpTd88h35 xIzYxVu3B80vRVlMXVzwtnHcf+Szdv12ep8OMbBd1O2y7kGPFLSEtcIqXWb/1koeUr6b r6oA== X-Gm-Message-State: AOUpUlHDO7aHhcG6S2liKteyFkc+KQdDXkj1upM0Qd2aItGWfJy2VLb+ +inQ46IAHu0x5NJ1MHEUTjeig0rz X-Google-Smtp-Source: AA+uWPzHBqLS+m+M99wPa78Ut9YKSPjPZ2v+IVP6j/AuUAZNxvcFnclzNYRePJcrvyAPqYSQNE/YUA== X-Received: by 2002:a62:d544:: with SMTP id d65-v6mr12442596pfg.107.1534021273991; Sat, 11 Aug 2018 14:01:13 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.01.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:01:13 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:22 +0800 Message-Id: <20180811205924.4113-19-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::444 Subject: [Qemu-devel] [PATCH V11 18/20] COLO: notify net filters about checkpoint/failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang Notify all net filters about the checkpoint and failover event. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 688d6f40b2..8221a18add 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -31,6 +31,7 @@ #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" #include "sysemu/cpus.h" +#include "net/filter.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -83,6 +84,12 @@ static void secondary_vm_do_failover(void) error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } + if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side"); /* recover runstate to normal migration finish state */ @@ -781,6 +788,14 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading = false; vm_start(); trace_colo_vm_state_change("stop", "run"); From patchwork Sat Aug 11 20:59:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563581 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 75D5913B4 for ; Sat, 11 Aug 2018 21:18:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6648C29C87 for ; Sat, 11 Aug 2018 21:18:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 56BD529C9C; Sat, 11 Aug 2018 21:18:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 0752029C87 for ; Sat, 11 Aug 2018 21:18:31 +0000 (UTC) Received: from localhost ([::1]:33268 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fobHK-00044n-Aa for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:18:30 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48356) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob0j-0005bi-8d for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fob0i-0001kX-HQ for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:21 -0400 Received: from mail-pf1-x442.google.com ([2607:f8b0:4864:20::442]:43715) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fob0i-0001jx-AJ for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:20 -0400 Received: by mail-pf1-x442.google.com with SMTP id j26-v6so5995949pfi.10 for ; Sat, 11 Aug 2018 14:01:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=HJs1IyP47L5VWSWXnfNk3rt/hlTP0bhjrgYjnFnswxY=; b=F+PxArHkFXwasL4icOxC+VlvGsD+1YYj3EeDvs9eiVS7Tnt6wOJrXUYGzX+0iJx11d jR/0YgULQIw5lXqfE+65/R5R5uKsZ+gEvHhE8XqoSeC2W+xWTjK07JIZqAs5b/KnP7a6 4eHMquGaWlCY0exhhuv+L5E4tGG8r/DLpNbusz3drHV7za91ljtEdQUKrjBRoMoyIVyf tBS743q6KsAlCJNAfNT2AsFskDOxAaTY7ml1vGyx24nA7m2DSwAfrhx0pM/G/ygDLD34 59+7VGfUJn9fVaUkh1S3AydVt9Lzhe9/E8eXSaNwp7/HsqDhBdx7Rnhqf5ZcqclRAe3X Ne4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=HJs1IyP47L5VWSWXnfNk3rt/hlTP0bhjrgYjnFnswxY=; b=cCUdbbh8mcCnYM1Drtojai3vOPhU7VQbnoaPt4HZI6I05uTVWFdYww5gaeExmeI7nl Gn5IQK17sY2rgnsU9c46uNh/0XUHp11Skfm2kclf9+ZwxJVYKhsPVh6I/rX/GkcvBS9F +f+lhMYAqAbc94tX8bGOWC9SnrFEHSA1gP7dnQP3N7XKyuP5vKPhP+W0EB6YskkRbz7m jaIZWXnnKp6bM+m7FqzUgP/5kDdmhA5uyQnEQd1RHN44N9BkVL/jTuCi/iVVb5h5RC/C c4Xpdclrb+jWVRN6igsUZsmdSWDl61c9QNXiThio4JVl3bz2UH+ZafuFPLcURoeLMrig zl4w== X-Gm-Message-State: AOUpUlGORGkRNOhMwPwRRS9laOKBEZebva9WRtZE795kswBUxSxFHGJe xQjQajdb9vEftESJ3MsdsgqI4Ve7 X-Google-Smtp-Source: AA+uWPzB5Cc+0Wy2422R/K+4rK+aj7sAPVmV9c7u5QPZ9QFgqz6gyLkoXut0oXB3d8oPCLCOF3KzQQ== X-Received: by 2002:a62:f208:: with SMTP id m8-v6mr12504888pfh.171.1534021279079; Sat, 11 Aug 2018 14:01:19 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.01.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:01:18 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:23 +0800 Message-Id: <20180811205924.4113-20-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::442 Subject: [Qemu-devel] [PATCH V11 19/20] COLO: quick failover process by kick COLO thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 8221a18add..e951159071 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -131,6 +131,11 @@ static void primary_vm_do_failover(void) migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + /* + * kick COLO thread which might wait at + * qemu_sem_wait(&s->colo_checkpoint_sem). + */ + colo_checkpoint_notify(migrate_get_current()); /* * Wake up COLO thread which may blocked in recv() or send(), @@ -539,6 +544,9 @@ static void colo_process_checkpoint(MigrationState *s) qemu_sem_wait(&s->colo_checkpoint_sem); + if (s->state != MIGRATION_STATUS_COLO) { + goto out; + } ret = colo_do_checkpoint_transaction(s, bioc, fb); if (ret < 0) { goto out; From patchwork Sat Aug 11 20:59:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10563577 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 97BC713B4 for ; Sat, 11 Aug 2018 21:16:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 748482990B for ; Sat, 11 Aug 2018 21:16:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 65E8D29982; Sat, 11 Aug 2018 21:16:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 153FB2990B for ; Sat, 11 Aug 2018 21:16:36 +0000 (UTC) Received: from localhost ([::1]:33259 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fobFU-00037j-3U for patchwork-qemu-devel@patchwork.kernel.org; Sat, 11 Aug 2018 17:16:36 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48395) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fob0r-0005jr-1q for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fob0n-0001nE-WC for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:29 -0400 Received: from mail-pl0-x231.google.com ([2607:f8b0:400e:c01::231]:39393) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fob0n-0001mq-PZ for qemu-devel@nongnu.org; Sat, 11 Aug 2018 17:01:25 -0400 Received: by mail-pl0-x231.google.com with SMTP id w14-v6so5384385plp.6 for ; Sat, 11 Aug 2018 14:01:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=06HycV6v4ZN/DheKI9cIvbvXxU2TRCnL8VF/KmZnq/U=; b=hf0a4N5Cf8olGAhVFUKN3vPm/m8QKzMRn3oQfad2kLbsmR25ozYx1AYEZDH/9c50p0 JtaANuVVBh+A1C7o/X5kwuJg74FMCGT4eNbdsnjAm7dQwliTV5wNFM5ZkVn6zPY+ZCIo QIYrX11sK+wweF2xhOV23yvvdMbkdrf3HQN4yUM6a/1QOtZglibFS35wLK5sJiueN7E6 YUPyaFeZqUgTu1+avwKe2VdIDpMcM/mAlU2iOhsvFRETubtovcXjV3Jcyjn03U/g1EOF kKKocnwsMSKLm33lZhoS4nBu3bBOtx8O4n21fgPnbvYrilioM042L+KBPw8i4rDl9ngT LC5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=06HycV6v4ZN/DheKI9cIvbvXxU2TRCnL8VF/KmZnq/U=; b=ZLUxfbXjC6MTVAINwj5pmxnQsCAT2OLASyKvkB1Rg4YfwwSIieuXS/6efQjq9Qrsh3 6Oev2gWiFw/ATqnL8gK3yxEJ/E8jD/+QUX0veNlxSFlHOJrbyDuoYJNdMD6Cg8ZKJcaP imCtJ4kWX/OLSj+CIx1Ihqkc3zJstWotTLZaqRvPHVEa3k1449KPL0Hw0a2GEIY03zE1 21F5AXfORebBhHWNM9SYLPJhJUzHVVwNuwgJCLjkGN/ZbSxk1xu7Wbdi7cwFbeQCb2oF ij7agqOASdAhxEaNgbHiMct8DhhrHW+3E3NhaomvcMIht4FykV25ChnPEwvjGjJO65lC u8TQ== X-Gm-Message-State: AOUpUlG3NAvBN+8f9BgO6SJvk9WcXizdAWiTcyuUmeKBrxXjHZkCpoWV QsRSAQ2wggzTng897rBcjnXzpUMn X-Google-Smtp-Source: AA+uWPyGyzZmppAYcDMez12y6YmLIA5Jm7eiVZ6+sAIsVKrqGGatuz5XoXKpUY5jAVLJWktnMH+xqw== X-Received: by 2002:a17:902:a40b:: with SMTP id p11-v6mr11170708plq.228.1534021284541; Sat, 11 Aug 2018 14:01:24 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id h24-v6sm23502641pfk.113.2018.08.11.14.01.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 11 Aug 2018 14:01:23 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 12 Aug 2018 04:59:24 +0800 Message-Id: <20180811205924.4113-21-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180811205924.4113-1-zhangckid@gmail.com> References: <20180811205924.4113-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::231 Subject: [Qemu-devel] [PATCH V11 20/20] docs: Add COLO status diagram to COLO-FT.txt X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Zhang Chen This diagram make user better understand COLO. Suggested by Markus Armbruster. Signed-off-by: Zhang Chen Signed-off-by: Zhang Chen --- docs/COLO-FT.txt | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/docs/COLO-FT.txt b/docs/COLO-FT.txt index d7c7dcda8f..d5007895d1 100644 --- a/docs/COLO-FT.txt +++ b/docs/COLO-FT.txt @@ -110,6 +110,40 @@ Note: HeartBeat has not been implemented yet, so you need to trigger failover process by using 'x-colo-lost-heartbeat' command. +== COLO operation status == + ++-----------------+ +| | +| Start COLO | +| | ++--------+--------+ + | + | Main qmp command: + | migrate-set-capabilities with x-colo + | migrate + | + v ++--------+--------+ +| | +| COLO running | +| | ++--------+--------+ + | + | Main qmp command: + | x-colo-lost-heartbeat + | or + | some error happened + v ++--------+--------+ +| | send qmp event: +| COLO failover | COLO_EXIT +| | ++-----------------+ + +COLO use the qmp command switching and report operation status. +The diagram just write the main qmp command, you can get the detail +in test procedure. + == Test procedure == 1. Startup qemu Primary: