From patchwork Sun Jul 22 19:33:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539367 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0ABB71823 for ; Sun, 22 Jul 2018 19:36:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EF7E22841D for ; Sun, 22 Jul 2018 19:36:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E3D522842A; Sun, 22 Jul 2018 19:36:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 37FA12841D for ; Sun, 22 Jul 2018 19:36:01 +0000 (UTC) Received: from localhost ([::1]:57211 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK9A-0003X9-7N for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:36:00 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37621) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK7W-0001tX-Br for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK7V-0000QY-1l for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:18 -0400 Received: from mail-pg1-x542.google.com ([2607:f8b0:4864:20::542]:33896) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK7U-0000QQ-Q7 for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:16 -0400 Received: by mail-pg1-x542.google.com with SMTP id y5-v6so10670946pgv.1 for ; Sun, 22 Jul 2018 12:34:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=z4ov75nrbEBWmaDinxPw/JqZz8lqE0MTX0TZ4nBA8T4=; b=APQERZfjn5PcLZoon14GkEG0IHDqwi2LJ5aUPkTRPnZq2iA4S4hBPGy3shGspMFzZ6 DSTHIjCYXqLMElfkxMy8Cta9nZIi7QfCRBpIJVziCyKt1g/Veq+2JVr6Ou9JccYIV5tp +hLFoHZt+8QA9gS90evBtEi594bARRTXyhSO1vtnmXMphn0XLLm57HMgZ3iKXa3m8eG8 toH/TTbDj+aGckmmOnq2rnf/KVbErKxGIMdwqZx2MisYkd71yxFtGFys+sh4gkEgjqOd +jhpDqYS6ZTehEGUri32dlABNoL+DrkCPCipPERUo/V/ILZe/lzZlo0ZhSS6C3qolR6L OSRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=z4ov75nrbEBWmaDinxPw/JqZz8lqE0MTX0TZ4nBA8T4=; b=geiI/MxOEYs7WShSJluRxtMAoSHICkXw/7YBFiClSE3GQc1G4W1QpMyU6apWMjycsW cVBvYNuOvW+Z6HkhLYK+AktXbYUNQ8B0ASPLcUgGGW/CUaIwWx81Ge9gyVvdhBeNaw7u D9ZaT9rHEWRg0FkcQD92Q2JiPpNTbT1MrluMX0BbK93CU3sspdslkA1bMjtxSw7BVzr4 4RNZ+/nh5cETRahZqCPnmevxQcnnD34nCbqeLZZtsTwJ4XClCzSMNh4ZRaxLE1iPkHtE TpdbZGW+oEakD9gtVovTXXMw6mmUQr78vfcZQCRsk2Mmmc2bUVd+fLO8DPjDkrkfHJ4x kKjQ== X-Gm-Message-State: AOUpUlFvjJO+RUshdmf6l1u1NlvcCgd8ac5KhZJjCMdYhAyHjS4GPsrG qpdpOqn+gQtPkljoGtyFJ7pka0iS X-Google-Smtp-Source: AAOMgpdmPtG0oMbPtyzxEoL6czGNnSxSXDbU0lgnUHx3SZbydEFlRWprkyNf+7w0HV6/AHb1tfSAjA== X-Received: by 2002:a62:4add:: with SMTP id c90-v6mr10444285pfj.23.1532288055606; Sun, 22 Jul 2018 12:34:15 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.34.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:34:14 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:31 +0800 Message-Id: <20180722193350.6028-2-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::542 Subject: [Qemu-devel] [PATCH V10 01/20] filter-rewriter: Add TCP state machine and fix memory leak in connection_track_table X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP We add almost full TCP state machine in filter-rewriter, except TCPS_LISTEN and some simplify in VM active close FIN states. After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak. Let't track the state of net connection, if it is closed, its related resources will be cleared up. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- net/colo.c | 2 +- net/colo.h | 9 ++-- net/filter-rewriter.c | 105 ++++++++++++++++++++++++++++++++++++++---- 3 files changed, 100 insertions(+), 16 deletions(-) diff --git a/net/colo.c b/net/colo.c index 6dda4ed66e..97c8fc928f 100644 --- a/net/colo.c +++ b/net/colo.c @@ -137,7 +137,7 @@ Connection *connection_new(ConnectionKey *key) conn->ip_proto = key->ip_proto; conn->processing = false; conn->offset = 0; - conn->syn_flag = 0; + conn->tcp_state = TCPS_CLOSED; conn->pack = 0; conn->sack = 0; g_queue_init(&conn->primary_list); diff --git a/net/colo.h b/net/colo.h index da6c36dcf7..0277e0e9ba 100644 --- a/net/colo.h +++ b/net/colo.h @@ -18,6 +18,7 @@ #include "slirp/slirp.h" #include "qemu/jhash.h" #include "qemu/timer.h" +#include "slirp/tcp.h" #define HASHTABLE_MAX_SIZE 16384 @@ -81,11 +82,9 @@ typedef struct Connection { uint32_t sack; /* offset = secondary_seq - primary_seq */ tcp_seq offset; - /* - * we use this flag update offset func - * run once in independent tcp connection - */ - int syn_flag; + + int tcp_state; /* TCP FSM state */ + tcp_seq fin_ack_seq; /* the seq of 'fin=1,ack=1' */ } Connection; uint32_t connection_key_hash(const void *opaque); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index f584e4eba4..f18a71bf2e 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -59,9 +59,9 @@ static int is_tcp_packet(Packet *pkt) } /* handle tcp packet from primary guest */ -static int handle_primary_tcp_pkt(NetFilterState *nf, +static int handle_primary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; @@ -74,23 +74,28 @@ static int handle_primary_tcp_pkt(NetFilterState *nf, trace_colo_filter_rewriter_conn_offset(conn->offset); } + if (((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == (TH_ACK | TH_SYN)) && + conn->tcp_state == TCPS_SYN_SENT) { + conn->tcp_state = TCPS_ESTABLISHED; + } + if (((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == TH_SYN)) { /* * we use this flag update offset func * run once in independent tcp connection */ - conn->syn_flag = 1; + conn->tcp_state = TCPS_SYN_RECEIVED; } if (((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == TH_ACK)) { - if (conn->syn_flag) { + if (conn->tcp_state == TCPS_SYN_RECEIVED) { /* * offset = secondary_seq - primary seq * ack packet sent by guest from primary node, * so we use th_ack - 1 get primary_seq */ conn->offset -= (ntohl(tcp_pkt->th_ack) - 1); - conn->syn_flag = 0; + conn->tcp_state = TCPS_ESTABLISHED; } if (conn->offset) { /* handle packets to the secondary from the primary */ @@ -99,15 +104,63 @@ static int handle_primary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_len, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 1: + * Step 3: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'ack=1' packets from client side, it acks 'fin=1, ack=1' + * packet from server side. From this point, we can ensure that there + * will be no packets in the connection, except that, some errors + * happen between the path of 'filter object' and vNIC, if this rare + * case really happen, we can still create a new connection, + * So it is safe to remove the connection from connection_track_table. + * + */ + if ((conn->tcp_state == TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) == (conn->fin_ack_seq + 1))) { + conn->tcp_state = TCPS_CLOSED; + g_hash_table_remove(rf->connection_track_table, key); + } + } + + if ((tcp_pkt->th_flags & TH_FIN) == TH_FIN) { + /* + * Case 1: + * Step 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. We will into CLOSE_WAIT status. + */ + if (conn->tcp_state == TCPS_ESTABLISHED) { + conn->tcp_state = TCPS_CLOSE_WAIT; + } + + /* + * Case 2: + * Step 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. We will into CLOSE_WAIT status. + * We simplify the TCPS_FIN_WAIT_2, TCPS_TIME_WAIT and CLOSING status. + */ + if (conn->tcp_state == TCPS_FIN_WAIT_1) { + conn->tcp_state = TCPS_TIME_WAIT; + /* + * For simplify implementation, we needn't wait 2MSL time + * in filter rewriter. + */ + conn->tcp_state = TCPS_CLOSED; + g_hash_table_remove(rf->connection_track_table, key); + } } return 0; } /* handle tcp packet from secondary guest */ -static int handle_secondary_tcp_pkt(NetFilterState *nf, +static int handle_secondary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; @@ -121,7 +174,8 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, trace_colo_filter_rewriter_conn_offset(conn->offset); } - if (((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == (TH_ACK | TH_SYN))) { + if (conn->tcp_state == TCPS_SYN_RECEIVED && + ((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == (TH_ACK | TH_SYN))) { /* * save offset = secondary_seq and then * in handle_primary_tcp_pkt make offset @@ -130,6 +184,12 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, conn->offset = ntohl(tcp_pkt->th_seq); } + /* VM active connect */ + if (conn->tcp_state == TCPS_CLOSED && + ((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == TH_SYN)) { + conn->tcp_state = TCPS_SYN_SENT; + } + if ((tcp_pkt->th_flags & (TH_ACK | TH_SYN)) == TH_ACK) { /* Only need to adjust seq while offset is Non-zero */ if (conn->offset) { @@ -140,6 +200,31 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, pkt->size - pkt->vnet_hdr_len); } } + /* + * Case 1: + * Step 2: + * The *server* side of this connect is VM, *client* tries to close + * the connection. In this step we will into LAST_ACK status. + * + * We got 'fin=1, ack=1' packet from server side, we need to + * record the seq of 'fin=1, ack=1' packet. + */ + if (conn->tcp_state == TCPS_CLOSE_WAIT && + (tcp_pkt->th_flags & (TH_ACK | TH_FIN)) == (TH_ACK | TH_FIN)) { + conn->fin_ack_seq = ntohl(tcp_pkt->th_seq); + conn->tcp_state = TCPS_LAST_ACK; + } + + /* + * Case 2: + * Step 1: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + */ + if (conn->tcp_state == TCPS_ESTABLISHED && + (tcp_pkt->th_flags & (TH_ACK | TH_FIN)) == TH_FIN) { + conn->tcp_state = TCPS_FIN_WAIT_1; + } return 0; } @@ -190,7 +275,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, if (sender == nf->netdev) { /* NET_FILTER_DIRECTION_TX */ - if (!handle_primary_tcp_pkt(nf, conn, pkt)) { + if (!handle_primary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); @@ -203,7 +288,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, } } else { /* NET_FILTER_DIRECTION_RX */ - if (!handle_secondary_tcp_pkt(nf, conn, pkt)) { + if (!handle_secondary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); From patchwork Sun Jul 22 19:33:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539373 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 09A3E1823 for ; Sun, 22 Jul 2018 19:38:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E59222842E for ; Sun, 22 Jul 2018 19:38:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D64D728434; Sun, 22 Jul 2018 19:38:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 383062842E for ; Sun, 22 Jul 2018 19:38:51 +0000 (UTC) Received: from localhost ([::1]:57225 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKBu-0005nD-D1 for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:38:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37636) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK7e-0001uv-2K for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK7a-0000SY-TZ for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:26 -0400 Received: from mail-pl0-x235.google.com ([2607:f8b0:400e:c01::235]:44491) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK7a-0000Ro-Jk for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:22 -0400 Received: by mail-pl0-x235.google.com with SMTP id m16-v6so7295734pls.11 for ; Sun, 22 Jul 2018 12:34:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=N7q5nKAwQ8wTjSQ2XfmKSsRuHefxeQQ4JpcGN03xRrY=; b=oJ4oDP6ourMvHwFYGi3Ke3eZa6cgN+dAsTDO6jVDuK//rjVTDNr6Xgm/RdXYZW3mBc IIjbaYPJP9Zm51vlRv78Q11Mw0wqWU78yxvdlcPCX2TvqTSxWqWNV2yLIRevIb4lb97q 7GZkRT0gD5mcJT9+AjbuCnDNKU/VPotE6WgtL70l1RPN35+l6h07gxXq/ESGM0CvjsJV aGjCSxCJXtjWd7iOdEDF+Qb8zN3/Kr7Rz+17JCP1WQpOZekHXVTxIImrH3elW/hCFMp4 nVcMkQ5MfXWovUpQzt30JCUIJ20Nx7Npt40ifTe8X+EfyMC1EF1KNf8EQoOh+YIPkr4Q uNvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=N7q5nKAwQ8wTjSQ2XfmKSsRuHefxeQQ4JpcGN03xRrY=; b=WtJZtAZazXQ+hMGarg47u87kQ/hvntKbZURNoibNQcMhnVzGJro1iuq9zomHnABwAg Me4nGmPnYDp011Iv4BDhAFWP/4jecV/33M5EC6DLU2esckGx6EcGl5zIDsNAsVjbnDCQ Cp8viNry3o334yj0w7mJyYlONebqATOyJiLnKyQ75/nsLVqQC8Tvzog87r6z/WrdpV5U GuG0/eUHVO378dl1ap/zC/pfjiXf0Ls+TOyAUyNFd4Jdpm0tTFWX0QjlLX5OGHL4BY98 S3IrpkhUe1tgNnmOUDCTSJZsMe/Z19tEJY2yTIafc7xi5Mhe9sat6nmyTClHofQwWF5G hY7g== X-Gm-Message-State: AOUpUlEl5LBFhaY/Wu6KT0QhBHzzklyOhuEPSDSuEmX0KwZUI49+dwKS xLO9VAySIfxl269RO1k2iDhXiK9/ X-Google-Smtp-Source: AAOMgpcQbiSUGf2KPgKF2aWXAazT5TS320ACVehN05ldQvSG2Hz5XHkAhA2yeLQeYiJa8PACneNXwQ== X-Received: by 2002:a17:902:ba88:: with SMTP id k8-v6mr6425492pls.259.1532288061204; Sun, 22 Jul 2018 12:34:21 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.34.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:34:20 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:32 +0800 Message-Id: <20180722193350.6028-3-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::235 Subject: [Qemu-devel] [PATCH V10 02/20] colo-compare: implement the process of checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 6 ++++ net/colo-compare.c | 78 ++++++++++++++++++++++++++++++++++++++++ net/colo-compare.h | 22 ++++++++++++ 3 files changed, 106 insertions(+) create mode 100644 net/colo-compare.h diff --git a/include/migration/colo.h b/include/migration/colo.h index 2fe48ad353..fefb2fcf4c 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -16,6 +16,12 @@ #include "qemu-common.h" #include "qapi/qapi-types-migration.h" +enum colo_event { + COLO_EVENT_NONE, + COLO_EVENT_CHECKPOINT, + COLO_EVENT_FAILOVER, +}; + void colo_info_init(void); void migrate_start_colo_process(MigrationState *s); diff --git a/net/colo-compare.c b/net/colo-compare.c index dd745a491b..80e6532e8b 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -27,11 +27,16 @@ #include "qemu/sockets.h" #include "colo.h" #include "sysemu/iothread.h" +#include "net/colo-compare.h" +#include "migration/colo.h" #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) +static QTAILQ_HEAD(, CompareState) net_compares = + QTAILQ_HEAD_INITIALIZER(net_compares); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 @@ -41,6 +46,10 @@ /* TODO: Should be configurable */ #define REGULAR_PACKET_CHECK_MS 3000 +static QemuMutex event_mtx; +static QemuCond event_complete_cond; +static int event_unhandled_count; + /* * + CompareState ++ * | | @@ -87,6 +96,11 @@ typedef struct CompareState { IOThread *iothread; GMainContext *worker_context; QEMUTimer *packet_check_timer; + + QEMUBH *event_bh; + enum colo_event event; + + QTAILQ_ENTRY(CompareState) next; } CompareState; typedef struct CompareClass { @@ -736,6 +750,25 @@ static void check_old_packet_regular(void *opaque) REGULAR_PACKET_CHECK_MS); } +/* Public API, Used for COLO frame to notify compare event */ +void colo_notify_compares_event(void *opaque, int event, Error **errp) +{ + CompareState *s; + + qemu_mutex_lock(&event_mtx); + QTAILQ_FOREACH(s, &net_compares, next) { + s->event = event; + qemu_bh_schedule(s->event_bh); + event_unhandled_count++; + } + /* Wait all compare threads to finish handling this event */ + while (event_unhandled_count > 0) { + qemu_cond_wait(&event_complete_cond, &event_mtx); + } + + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_timer_init(CompareState *s) { AioContext *ctx = iothread_get_aio_context(s->iothread); @@ -756,6 +789,30 @@ static void colo_compare_timer_del(CompareState *s) } } +static void colo_flush_packets(void *opaque, void *user_data); + +static void colo_compare_handle_event(void *opaque) +{ + CompareState *s = opaque; + + switch (s->event) { + case COLO_EVENT_CHECKPOINT: + g_queue_foreach(&s->conn_list, colo_flush_packets, s); + break; + case COLO_EVENT_FAILOVER: + break; + default: + break; + } + + assert(event_unhandled_count > 0); + + qemu_mutex_lock(&event_mtx); + event_unhandled_count--; + qemu_cond_broadcast(&event_complete_cond); + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_iothread(CompareState *s) { object_ref(OBJECT(s->iothread)); @@ -769,6 +826,7 @@ static void colo_compare_iothread(CompareState *s) s, s->worker_context, true); colo_compare_timer_init(s); + s->event_bh = qemu_bh_new(colo_compare_handle_event, s); } static char *compare_get_pri_indev(Object *obj, Error **errp) @@ -926,8 +984,13 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); + QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); + qemu_mutex_init(&event_mtx); + qemu_cond_init(&event_complete_cond); + s->connection_track_table = g_hash_table_new_full(connection_key_hash, connection_key_equal, g_free, @@ -990,6 +1053,7 @@ static void colo_compare_init(Object *obj) static void colo_compare_finalize(Object *obj) { CompareState *s = COLO_COMPARE(obj); + CompareState *tmp = NULL; qemu_chr_fe_deinit(&s->chr_pri_in, false); qemu_chr_fe_deinit(&s->chr_sec_in, false); @@ -997,6 +1061,16 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { colo_compare_timer_del(s); } + + qemu_bh_delete(s->event_bh); + + QTAILQ_FOREACH(tmp, &net_compares, next) { + if (tmp == s) { + QTAILQ_REMOVE(&net_compares, s, next); + break; + } + } + /* Release all unhandled packets after compare thead exited */ g_queue_foreach(&s->conn_list, colo_flush_packets, s); @@ -1009,6 +1083,10 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { object_unref(OBJECT(s->iothread)); } + + qemu_mutex_destroy(&event_mtx); + qemu_cond_destroy(&event_complete_cond); + g_free(s->pri_indev); g_free(s->sec_indev); g_free(s->outdev); diff --git a/net/colo-compare.h b/net/colo-compare.h new file mode 100644 index 0000000000..1b1ce76aea --- /dev/null +++ b/net/colo-compare.h @@ -0,0 +1,22 @@ +/* + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) + * (a.k.a. Fault Tolerance or Continuous Replication) + * + * Copyright (c) 2017 HUAWEI TECHNOLOGIES CO., LTD. + * Copyright (c) 2017 FUJITSU LIMITED + * Copyright (c) 2017 Intel Corporation + * + * Authors: + * zhanghailiang + * Zhang Chen + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * later. See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event(void *opaque, int event, Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */ From patchwork Sun Jul 22 19:33:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539369 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2F3B1174A for ; Sun, 22 Jul 2018 19:36:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 207A72841D for ; Sun, 22 Jul 2018 19:36:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 14CC12842A; Sun, 22 Jul 2018 19:36:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 94F402841D for ; Sun, 22 Jul 2018 19:36:07 +0000 (UTC) Received: from localhost ([::1]:57212 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK9G-0003di-NN for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:36:06 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37660) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK7g-0001wn-UW for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK7f-0000UE-Uh for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:28 -0400 Received: from mail-pg1-x52b.google.com ([2607:f8b0:4864:20::52b]:38102) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK7f-0000Tn-OW for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:27 -0400 Received: by mail-pg1-x52b.google.com with SMTP id k3-v6so10653962pgq.5 for ; Sun, 22 Jul 2018 12:34:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Wjh9k6AIGGdX5TYjeH54iduofbef0LaIRxc8FGJ9TAw=; b=AGGby0U6avEeNVFoEuno2BLz+3j824q74LQLrc0GviowKxrDvtCh64WC0Q/qnqv9wH ISiMrN8Wq/lI5B7zQuSG3hadxOfnfRvICqW3l+pmyJ1F6cU0UZxQRF0Zdpi6G4ODjbXw STU+S9Shq5vYpk0oOHR0W5ZfGecamwZPX/bckh9c1riq2Um6L/IS5oSt/UBO/Ob1wAF6 6Qbx8NxXfQq3AFzSdP/6/fvchKIvD9WndwIvHOQPCSQlihJN+iAcB8rGirkg3wfIXepq 7MIz8chQpH0qApLIgeoU9AbuqNalMvakMm2mBZv7JV2fHTSa68lBYV5NSKUyBRWOFJLy monw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Wjh9k6AIGGdX5TYjeH54iduofbef0LaIRxc8FGJ9TAw=; b=QIweKo4BWehsDF62RB/KoSQyL86E5g+1KT3TFCCyujIGpCGJwYYXtqAW32LmDHmJDL kjSgHUaej6WCtHJ/VNrzAafN+vFSX6Ce/oPCIcKMT+vQF87lq8+K2oyDJ4iZKuZZ6Gsj 9wCwR2aqWzj6u8QqnFeWMg42zHNdvumjceRtEAWpAvtKr1C+Xj6YUxjgHgWZKQklybzI MArR0sEBUDzGCWcrxggZ6K9VZj2oTxyNZQgBjmC3JcQ2FbdhLrtSB+uTp9A8AENyvkDT 1+kxFwAsr6Rjqh1LVSlXw4BpI/88NgpQ5jXOW55YBUkqdzzvIAWaU7/vTaeTz2Ck+ScA xbAQ== X-Gm-Message-State: AOUpUlHjwPDgoa4S/Ec8gAEG8yt8s3GfQ4W5AfdEk7XSUYHkIVZU91a1 GzMHJghWT/4hrggxuOsl7BITKVZU X-Google-Smtp-Source: AAOMgpfhX0DW8LaMYMuKWRpyj09jJwUPaPSjljdkFtbEPrs08S7UGzOH3ioJ2fXYZMSOzj2RTOI5yQ== X-Received: by 2002:a62:4494:: with SMTP id m20-v6mr10299400pfi.205.1532288066526; Sun, 22 Jul 2018 12:34:26 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.34.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:34:25 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:33 +0800 Message-Id: <20180722193350.6028-4-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::52b Subject: [Qemu-devel] [PATCH V10 03/20] colo-compare: use notifier to notify packets comparing result X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 37 ++++++++++++++++++++++++++----------- net/colo-compare.h | 2 ++ 2 files changed, 28 insertions(+), 11 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index 80e6532e8b..426eab5973 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -29,6 +29,7 @@ #include "sysemu/iothread.h" #include "net/colo-compare.h" #include "migration/colo.h" +#include "migration/migration.h" #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ @@ -37,6 +38,9 @@ static QTAILQ_HEAD(, CompareState) net_compares = QTAILQ_HEAD_INITIALIZER(net_compares); +static NotifierList colo_compare_notifiers = + NOTIFIER_LIST_INITIALIZER(colo_compare_notifiers); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 @@ -427,10 +431,7 @@ sec: qemu_hexdump((char *)spkt->data, stderr, "colo-compare spkt", spkt->size); - /* - * colo_compare_inconsistent_notify(); - * TODO: notice to checkpoint(); - */ + colo_compare_inconsistency_notify(); } } @@ -561,8 +562,24 @@ static int colo_old_packet_check_one(Packet *pkt, int64_t *check_time) } } +static void colo_compare_inconsistency_notify(void) +{ + notifier_list_notify(&colo_compare_notifiers, + migrate_get_current()); +} + +void colo_compare_register_notifier(Notifier *notify) +{ + notifier_list_add(&colo_compare_notifiers, notify); +} + +void colo_compare_unregister_notifier(Notifier *notify) +{ + notifier_remove(notify); +} + static int colo_old_packet_check_one_conn(Connection *conn, - void *user_data) + void *user_data) { GList *result = NULL; int64_t check_time = REGULAR_PACKET_CHECK_MS; @@ -573,10 +590,7 @@ static int colo_old_packet_check_one_conn(Connection *conn, if (result) { /* Do checkpoint will flush old packet */ - /* - * TODO: Notify colo frame to do checkpoint. - * colo_compare_inconsistent_notify(); - */ + colo_compare_inconsistency_notify(); return 0; } @@ -620,11 +634,12 @@ static void colo_compare_packet(CompareState *s, Connection *conn, /* * If one packet arrive late, the secondary_list or * primary_list will be empty, so we can't compare it - * until next comparison. + * until next comparison. If the packets in the list are + * timeout, it will trigger a checkpoint request. */ trace_colo_compare_main("packet different"); g_queue_push_head(&conn->primary_list, pkt); - /* TODO: colo_notify_checkpoint();*/ + colo_compare_inconsistency_notify(); break; } } diff --git a/net/colo-compare.h b/net/colo-compare.h index 1b1ce76aea..22ddd512e2 100644 --- a/net/colo-compare.h +++ b/net/colo-compare.h @@ -18,5 +18,7 @@ #define QEMU_COLO_COMPARE_H void colo_notify_compares_event(void *opaque, int event, Error **errp); +void colo_compare_register_notifier(Notifier *notify); +void colo_compare_unregister_notifier(Notifier *notify); #endif /* QEMU_COLO_COMPARE_H */ From patchwork Sun Jul 22 19:33:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539371 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 64406174A for ; Sun, 22 Jul 2018 19:36:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 53FEE2841D for ; Sun, 22 Jul 2018 19:36:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4580D2842A; Sun, 22 Jul 2018 19:36:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id AA2C12841D for ; Sun, 22 Jul 2018 19:36:13 +0000 (UTC) Received: from localhost ([::1]:57213 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK9M-0003kp-QR for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:36:12 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37673) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK7o-00029y-Rq for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK7l-0000XZ-N1 for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:36 -0400 Received: from mail-pg1-x535.google.com ([2607:f8b0:4864:20::535]:44970) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK7l-0000WQ-Ev for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:33 -0400 Received: by mail-pg1-x535.google.com with SMTP id r1-v6so10655093pgp.11 for ; Sun, 22 Jul 2018 12:34:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Wo1XxCi/VLFQhNhvp3VamahkuEggN5wilMfPdAw4/oY=; b=HelKHZy+vgAvFp2B0XarU35R89mRGz1n4vOMoUvmO9mR4GFvpgGr7YKb1vEAUhw79r dS3NeIzQebTguYPfrL0YsgTPYH5S7K0vAE2V3LR3RTegIT6j/ePY8/3Juv+tt/SDinrw LVcFMZCbRVH8HwkqQLmaoFvburM5Mv2POEqIAshJb97OGZMHFTGU5bQVwJ0jcsumUhzP cSy8z+w2ISARHgwm/U7azrlyefc4DhPhbmp4aaz2Q2ehaDIWWUZJpr8VA/mxm/sMHNgf 41lv7359kE021KKtI21/5Fvr9TeieT/n7QcW7b9oO/TZrZKFPlgb7YSXqD7m6KpZ5/uB lHCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Wo1XxCi/VLFQhNhvp3VamahkuEggN5wilMfPdAw4/oY=; b=bsGlUccpHXPFWtBef5+xfyhlDGp30Wuod2l+RVq4f1DtY+tFGMy6YLnTdXbmupRAXB axiyvR7XbkLGhdiu1hN5a/g/T0T3HUv8XaVfbWMMH/cKV0aeIoquOFB+8ae6Z7pA+gjd mnYBXBkrGR85+NUEq0iF/Cijn9wSXqhKtN/knIEEGY6jrXz8i5FWnE4cOT/EKDVW4y5k 1IPss/7ZKdnQ6ltbNvcpt3dz7Ix7SiEFKMe85Y1dLIPytQqCxDhc7KyxPialcDOOuIR7 g07uU1NcMIeBfd7pEl5eDaVajy4tCZCWgZ30H5h0xYqbYr5jN5WQtQ7N04r6zJYfOD0s TVgA== X-Gm-Message-State: AOUpUlEjoJjkQifYlFRqxasc31Ke62eLZU0bL0UXpr/Bd0xR4cqiI9xS pI2Pv29pUZORU6k1a2J4qjq0/Sij X-Google-Smtp-Source: AAOMgpfyZMNtMnyE8WMfuMW4vXIsATc/dp0IX43ujf27ziGIQOitQ1tif9zUU1zmi1nXVeAw5RIdsg== X-Received: by 2002:a63:1a20:: with SMTP id a32-v6mr9524712pga.446.1532288072243; Sun, 22 Jul 2018 12:34:32 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.34.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:34:31 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:34 +0800 Message-Id: <20180722193350.6028-5-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::535 Subject: [Qemu-devel] [PATCH V10 04/20] COLO: integrate colo compare with colo frame X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP For COLO FT, both the PVM and SVM run at the same time, only sync the state while it needs. So here, let SVM runs while not doing checkpoint, change DEFAULT_MIGRATE_X_CHECKPOINT_DELAY to 200*100. Besides, we forgot to release colo_checkpoint_semd and colo_delay_timer, fix them here. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 ++++++++++++++++++++++++++++++++++++++++-- migration/migration.c | 6 ++---- 2 files changed, 42 insertions(+), 6 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 4381067ed4..081df1835f 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -25,8 +25,11 @@ #include "qemu/error-report.h" #include "migration/failover.h" #include "replication.h" +#include "net/colo-compare.h" +#include "net/colo.h" static bool vmstate_loading; +static Notifier packets_compare_notifier; #define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024) @@ -343,6 +346,11 @@ static int colo_do_checkpoint_transaction(MigrationState *s, goto out; } + colo_notify_compares_event(NULL, COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + goto out; + } + /* Disable block migration */ migrate_set_block_enabled(false, &local_err); qemu_savevm_state_header(fb); @@ -400,6 +408,11 @@ out: return ret; } +static void colo_compare_notify_checkpoint(Notifier *notifier, void *data) +{ + colo_checkpoint_notify(data); +} + static void colo_process_checkpoint(MigrationState *s) { QIOChannelBuffer *bioc; @@ -416,6 +429,9 @@ static void colo_process_checkpoint(MigrationState *s) goto out; } + packets_compare_notifier.notify = colo_compare_notify_checkpoint; + colo_compare_register_notifier(&packets_compare_notifier); + /* * Wait for Secondary finish loading VM states and enter COLO * restore. @@ -461,11 +477,21 @@ out: qemu_fclose(fb); } - timer_del(s->colo_delay_timer); - /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); + + /* + * It is safe to unregister notifier after failover finished. + * Besides, colo_delay_timer and colo_checkpoint_sem can't be + * released befor unregister notifier, or there will be use-after-free + * error. + */ + colo_compare_unregister_notifier(&packets_compare_notifier); + timer_del(s->colo_delay_timer); + timer_free(s->colo_delay_timer); + qemu_sem_destroy(&s->colo_checkpoint_sem); + /* * Must be called after failover BH is completed, * Or the failover BH may shutdown the wrong fd that @@ -558,6 +584,11 @@ void *colo_process_incoming_thread(void *opaque) fb = qemu_fopen_channel_input(QIO_CHANNEL(bioc)); object_unref(OBJECT(bioc)); + qemu_mutex_lock_iothread(); + vm_start(); + trace_colo_vm_state_change("stop", "run"); + qemu_mutex_unlock_iothread(); + colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_READY, &local_err); if (local_err) { @@ -577,6 +608,11 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + trace_colo_vm_state_change("run", "stop"); + qemu_mutex_unlock_iothread(); + /* FIXME: This is unnecessary for periodic checkpoint mode */ colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_REPLY, &local_err); @@ -630,6 +666,8 @@ void *colo_process_incoming_thread(void *opaque) } vmstate_loading = false; + vm_start(); + trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); if (failover_get_state() == FAILOVER_STATUS_RELAUNCH) { diff --git a/migration/migration.c b/migration/migration.c index 8d56d56930..ce06941706 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -75,10 +75,8 @@ /* Migration XBZRLE default cache size */ #define DEFAULT_MIGRATE_XBZRLE_CACHE_SIZE (64 * 1024 * 1024) -/* The delay time (in ms) between two COLO checkpoints - * Note: Please change this default value to 10000 when we support hybrid mode. - */ -#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY 200 +/* The delay time (in ms) between two COLO checkpoints */ +#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100) #define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2 #define DEFAULT_MIGRATE_MULTIFD_PAGE_COUNT 16 From patchwork Sun Jul 22 19:33:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539377 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DCBF3174A for ; Sun, 22 Jul 2018 19:39:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CD2092842E for ; Sun, 22 Jul 2018 19:39:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C166D28434; Sun, 22 Jul 2018 19:39:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3C4CA2842E for ; Sun, 22 Jul 2018 19:39:08 +0000 (UTC) Received: from localhost ([::1]:57227 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKCB-00060Q-Cl for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:39:07 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37684) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK7s-0002Ho-Bz for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK7r-0000Z2-6C for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:40 -0400 Received: from mail-pg1-x543.google.com ([2607:f8b0:4864:20::543]:43393) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK7q-0000Yq-Tt for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:39 -0400 Received: by mail-pg1-x543.google.com with SMTP id v13-v6so10654853pgr.10 for ; Sun, 22 Jul 2018 12:34:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IFmo4cYw6R9A3e4c6WaA+TA/FlBWfHCi3Ur4TjwQ24g=; b=kw1e6VKoBN7DDDdWy6ZQ7xnGX/be7qc8yAWz8VSFfhl0GBwp1a0FLBE82Sas4l+pZU aCqnMXSDAPKf0GkBILotZN7k522MxSjwIuH16B2Lh8YGdPCKOTxz/xpPvR+d5xRA556f 83y1XUQoaXK+tlrEBpNfBgrXzz2w46VVw5hoWSTILwbHVCYrYfqW7q4XFvqM07viwmpN XS0Y+2kUNR1PIZtdNLIBr0NIvzjcH/psZRV/iMgE/9gx/GkkYnIYH5bR7NHubQ6f+cc6 n7J/fTtx33DRY37PNuSDwNR4fj/xQ0KKOYJVUWcfolSy3Adog/vkemuL+ye4uCdX0an4 2+vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IFmo4cYw6R9A3e4c6WaA+TA/FlBWfHCi3Ur4TjwQ24g=; b=XgXFBzATTSuJ4nlBMqhTWTryDi7RNCrHVfWPDivRTn3T9iiadejplAheuLG2e7swZs 5R9FKDy+vg29BM5KMO6/lWUN6UZZXRs/Z7svIP20TrHBbh0zC0Xbsu3Q0h6+izXMO6It NJ9BLJLjFXIh5Wt7JwK7ooFPwXyau1lrxj7m2xdmy0qcNfPrrBRQ6FQiO2BbSwEyoYlw BmbYL0j+HUQrAYmuF6b/h3AuUd0r0H2jg3WM5rTSTu19cYAKvcFHHImuBVtdB3kw2i6v 2WUOJcjDCBYvxwrD603K8CJSyn69u5iQbig6+8a9oRFfrVsCj4BpMprLRmTdCcRwglVB HpxA== X-Gm-Message-State: AOUpUlHxeKY7MZxdJJrxZe5fa52bFO/+uBT7ITcIhPdEAiXtv/gieNHa e+URwEqyZhnlVqW4p+Hi9TY/zmDC X-Google-Smtp-Source: AAOMgpfbxJtjbUVSDaM0azomhnbu584anOFBImF/AXGycCrmRL8zDcpA69g37eeGVCJVmrwmuazerA== X-Received: by 2002:a63:8449:: with SMTP id k70-v6mr9584766pgd.309.1532288077783; Sun, 22 Jul 2018 12:34:37 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.34.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:34:36 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:35 +0800 Message-Id: <20180722193350.6028-6-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::543 Subject: [Qemu-devel] [PATCH V10 05/20] COLO: Add block replication into colo process X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 9 +++++++++ 2 files changed, 52 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 081df1835f..e06640c3d6 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -27,6 +27,7 @@ #include "replication.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "block/block.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -56,6 +57,7 @@ static void secondary_vm_do_failover(void) { int old_state; MigrationIncomingState *mis = migration_incoming_get_current(); + Error *local_err = NULL; /* Can not do failover during the process of VM's loading VMstate, Or * it will break the secondary VM. @@ -73,6 +75,11 @@ static void secondary_vm_do_failover(void) migrate_set_state(&mis->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + } + if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side"); /* recover runstate to normal migration finish state */ @@ -110,6 +117,7 @@ static void primary_vm_do_failover(void) { MigrationState *s = migrate_get_current(); int old_state; + Error *local_err = NULL; migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); @@ -133,6 +141,13 @@ static void primary_vm_do_failover(void) FailoverStatus_str(old_state)); return; } + + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + local_err = NULL; + } + /* Notify COLO thread that failover work is finished */ qemu_sem_post(&s->colo_exit_sem); } @@ -356,6 +371,11 @@ static int colo_do_checkpoint_transaction(MigrationState *s, qemu_savevm_state_header(fb); qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } qemu_savevm_state_complete_precopy(fb, false, false); qemu_mutex_unlock_iothread(); @@ -446,6 +466,12 @@ static void colo_process_checkpoint(MigrationState *s) object_unref(OBJECT(bioc)); qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_PRIMARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vm_start(); qemu_mutex_unlock_iothread(); trace_colo_vm_state_change("stop", "run"); @@ -585,6 +611,11 @@ void *colo_process_incoming_thread(void *opaque) object_unref(OBJECT(bioc)); qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_SECONDARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } vm_start(); trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); @@ -665,6 +696,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + replication_get_error_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* discard colo disk buffer */ + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading = false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/migration/migration.c b/migration/migration.c index ce06941706..c97b7660af 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -385,6 +385,7 @@ static void process_incoming_migration_co(void *opaque) MigrationIncomingState *mis = migration_incoming_get_current(); PostcopyState ps; int ret; + Error *local_err = NULL; assert(mis->from_src_file); mis->largest_page_size = qemu_ram_pagesize_largest(); @@ -416,6 +417,14 @@ static void process_incoming_migration_co(void *opaque) /* we get COLO info, and know if we are in COLO mode */ if (!ret && migration_incoming_enable_colo()) { + /* Make sure all file formats flush their mutable metadata */ + bdrv_invalidate_cache_all(&local_err); + if (local_err) { + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + error_report_err(local_err); + exit(EXIT_FAILURE); + } mis->migration_incoming_co = qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); From patchwork Sun Jul 22 19:33:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539381 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8F7AB174A for ; Sun, 22 Jul 2018 19:41:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 767F42842E for ; Sun, 22 Jul 2018 19:41:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6A98128434; Sun, 22 Jul 2018 19:41:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A43BB2842E for ; Sun, 22 Jul 2018 19:41:48 +0000 (UTC) Received: from localhost ([::1]:57246 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKEl-00005K-Do for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:41:47 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37704) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK80-0002Zj-8A for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK7x-0000af-2L for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:48 -0400 Received: from mail-pl0-x241.google.com ([2607:f8b0:400e:c01::241]:37825) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK7w-0000aQ-PI for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:44 -0400 Received: by mail-pl0-x241.google.com with SMTP id 31-v6so7312511plc.4 for ; Sun, 22 Jul 2018 12:34:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3mZKuMnVABMqqnSxdiA2rkwZ665XedeqIvBhE8IIZhs=; b=JYFlxwom/ti7ES3AZgH2gUQXhneg/GrvABXoPkOoigXwh7cPJraYpfTr0tW/tmXRpK d6xUTPVQFjpqrBxUBgoUsokBoOa06B4rQ3kDqZRcedTChPGk5IfrM4OfXb6kOp6ke/FH VY22IuZ9CXuR84Emt+4umveEY7vuQ1XG6VhE4w/LjhgPyBiWrfzJJwm8rVrD+UWn5JV4 Sjrqpkjt1m7ztJN3E4FSG4i0i82Q33IFL/bbDdy2AjfDlQspj19qjZGXeY3z+GwR/c8l Ema6iP9oikvFI82I2DvylWZxSvKtrBJPOtzcQvL/83EvXDnrdArGdk3wH4/qJ+kzSP0v X5uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3mZKuMnVABMqqnSxdiA2rkwZ665XedeqIvBhE8IIZhs=; b=fF63RKoizmGIfDB58wxwRScxxULL/HbOG6TNfuo2wo08c2AWUJBYJSUtPLGfRYT67O nGmyii3u7q2y3TnJgQYQxp2lMDmVQcFjkQHDGz4JwLmwz2RWivRox7vYKprN7uuALOQJ MrtkjgrdIWaiHLM+jIs3PtiTIgtA6CRw3TdX89GtZxd4bgHtwuJh1XN7HCWIzQDemU+3 8cSwZnFwnnySgCpTSj70KpC9cYOHAk0wankT66yXp3TZyY3nz/8VF7fxI3k5u3+9g7Dq mkXmdwCKv6FgV9z2k501PhkCMu/MKktc5K8mRdBtl/FHAx2rP5eo4Gwv41QEte/XpC0e d6pQ== X-Gm-Message-State: AOUpUlGiu+SGsstxUy3SiKNtIWfBW5ZID8T1Fxk44TbCvZ8GvYL+Fp3D Rndwugw7cjiEcZ0aoSsP0cu7KXs+ X-Google-Smtp-Source: AAOMgpebacvopbVQx4xj986bQjR/hwX0z1Xv9sHCy059KJMSPdX8kF6/2clAzCarvJ7X6DRjMAIWvg== X-Received: by 2002:a17:902:561:: with SMTP id 88-v6mr10080437plf.320.1532288083468; Sun, 22 Jul 2018 12:34:43 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.34.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:34:42 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:36 +0800 Message-Id: <20180722193350.6028-7-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::241 Subject: [Qemu-devel] [PATCH V10 06/20] COLO: Remove colo_state migration struct X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP We need to know if migration is going into COLO state for incoming side before start normal migration. Instead by using the VMStateDescription to send colo_state from source side to destination side, we use MIG_CMD_ENABLE_COLO to indicate whether COLO is enabled or not. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- include/migration/colo.h | 5 +-- migration/Makefile.objs | 2 +- migration/colo-comm.c | 76 ---------------------------------------- migration/colo.c | 13 ++++++- migration/migration.c | 23 +++++++++++- migration/savevm.c | 17 +++++++++ migration/savevm.h | 1 + migration/trace-events | 1 + vl.c | 2 -- 9 files changed, 57 insertions(+), 83 deletions(-) delete mode 100644 migration/colo-comm.c diff --git a/include/migration/colo.h b/include/migration/colo.h index fefb2fcf4c..99ce17aca7 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -28,8 +28,9 @@ void migrate_start_colo_process(MigrationState *s); bool migration_in_colo_state(void); /* loadvm */ -bool migration_incoming_enable_colo(void); -void migration_incoming_exit_colo(void); +void migration_incoming_enable_colo(void); +void migration_incoming_disable_colo(void); +bool migration_incoming_colo_enabled(void); void *colo_process_incoming_thread(void *opaque); bool migration_incoming_in_colo_state(void); diff --git a/migration/Makefile.objs b/migration/Makefile.objs index c83ec47ba8..a4f3bafd86 100644 --- a/migration/Makefile.objs +++ b/migration/Makefile.objs @@ -1,6 +1,6 @@ common-obj-y += migration.o socket.o fd.o exec.o common-obj-y += tls.o channel.o savevm.o -common-obj-y += colo-comm.o colo.o colo-failover.o +common-obj-y += colo.o colo-failover.o common-obj-y += vmstate.o vmstate-types.o page_cache.o common-obj-y += qemu-file.o global_state.o common-obj-y += qemu-file-channel.o diff --git a/migration/colo-comm.c b/migration/colo-comm.c deleted file mode 100644 index df26e4dfe7..0000000000 --- a/migration/colo-comm.c +++ /dev/null @@ -1,76 +0,0 @@ -/* - * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) - * (a.k.a. Fault Tolerance or Continuous Replication) - * - * Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. - * Copyright (c) 2016 FUJITSU LIMITED - * Copyright (c) 2016 Intel Corporation - * - * This work is licensed under the terms of the GNU GPL, version 2 or - * later. See the COPYING file in the top-level directory. - * - */ - -#include "qemu/osdep.h" -#include "migration.h" -#include "migration/colo.h" -#include "migration/vmstate.h" -#include "trace.h" - -typedef struct { - bool colo_requested; -} COLOInfo; - -static COLOInfo colo_info; - -COLOMode get_colo_mode(void) -{ - if (migration_in_colo_state()) { - return COLO_MODE_PRIMARY; - } else if (migration_incoming_in_colo_state()) { - return COLO_MODE_SECONDARY; - } else { - return COLO_MODE_UNKNOWN; - } -} - -static int colo_info_pre_save(void *opaque) -{ - COLOInfo *s = opaque; - - s->colo_requested = migrate_colo_enabled(); - - return 0; -} - -static bool colo_info_need(void *opaque) -{ - return migrate_colo_enabled(); -} - -static const VMStateDescription colo_state = { - .name = "COLOState", - .version_id = 1, - .minimum_version_id = 1, - .pre_save = colo_info_pre_save, - .needed = colo_info_need, - .fields = (VMStateField[]) { - VMSTATE_BOOL(colo_requested, COLOInfo), - VMSTATE_END_OF_LIST() - }, -}; - -void colo_info_init(void) -{ - vmstate_register(NULL, 0, &colo_state, &colo_info); -} - -bool migration_incoming_enable_colo(void) -{ - return colo_info.colo_requested; -} - -void migration_incoming_exit_colo(void) -{ - colo_info.colo_requested = false; -} diff --git a/migration/colo.c b/migration/colo.c index e06640c3d6..c083d3696f 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -152,6 +152,17 @@ static void primary_vm_do_failover(void) qemu_sem_post(&s->colo_exit_sem); } +COLOMode get_colo_mode(void) +{ + if (migration_in_colo_state()) { + return COLO_MODE_PRIMARY; + } else if (migration_incoming_in_colo_state()) { + return COLO_MODE_SECONDARY; + } else { + return COLO_MODE_UNKNOWN; + } +} + void colo_do_failover(MigrationState *s) { /* Make sure VM stopped while failover happened. */ @@ -745,7 +756,7 @@ out: if (mis->to_src_file) { qemu_fclose(mis->to_src_file); } - migration_incoming_exit_colo(); + migration_incoming_disable_colo(); return NULL; } diff --git a/migration/migration.c b/migration/migration.c index c97b7660af..c645f66f4e 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -295,6 +295,22 @@ int migrate_send_rp_req_pages(MigrationIncomingState *mis, const char *rbname, return migrate_send_rp_message(mis, msg_type, msglen, bufc); } +static bool migration_colo_enabled; +bool migration_incoming_colo_enabled(void) +{ + return migration_colo_enabled; +} + +void migration_incoming_disable_colo(void) +{ + migration_colo_enabled = false; +} + +void migration_incoming_enable_colo(void) +{ + migration_colo_enabled = true; +} + void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p; @@ -416,7 +432,7 @@ static void process_incoming_migration_co(void *opaque) } /* we get COLO info, and know if we are in COLO mode */ - if (!ret && migration_incoming_enable_colo()) { + if (!ret && migration_incoming_colo_enabled()) { /* Make sure all file formats flush their mutable metadata */ bdrv_invalidate_cache_all(&local_err); if (local_err) { @@ -2950,6 +2966,11 @@ static void *migration_thread(void *opaque) qemu_savevm_send_postcopy_advise(s->to_dst_file); } + if (migrate_colo_enabled()) { + /* Notify migration destination that we enable COLO */ + qemu_savevm_send_colo_enable(s->to_dst_file); + } + qemu_savevm_state_setup(s->to_dst_file); s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start; diff --git a/migration/savevm.c b/migration/savevm.c index 7f92567a10..437308877a 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -56,6 +56,7 @@ #include "io/channel-file.h" #include "sysemu/replay.h" #include "qjson.h" +#include "migration/colo.h" #ifndef ETH_P_RARP #define ETH_P_RARP 0x8035 @@ -82,6 +83,7 @@ enum qemu_vm_cmd { were previously sent during precopy but are dirty. */ MIG_CMD_PACKAGED, /* Send a wrapped stream within this stream */ + MIG_CMD_ENABLE_COLO, /* Enable COLO */ MIG_CMD_POSTCOPY_RESUME, /* resume postcopy on dest */ MIG_CMD_RECV_BITMAP, /* Request for recved bitmap on dst */ MIG_CMD_MAX @@ -841,6 +843,12 @@ static void qemu_savevm_command_send(QEMUFile *f, qemu_fflush(f); } +void qemu_savevm_send_colo_enable(QEMUFile *f) +{ + trace_savevm_send_colo_enable(); + qemu_savevm_command_send(f, MIG_CMD_ENABLE_COLO, 0, NULL); +} + void qemu_savevm_send_ping(QEMUFile *f, uint32_t value) { uint32_t buf; @@ -1918,6 +1926,12 @@ static int loadvm_handle_recv_bitmap(MigrationIncomingState *mis, return 0; } +static int loadvm_process_enable_colo(MigrationIncomingState *mis) +{ + migration_incoming_enable_colo(); + return 0; +} + /* * Process an incoming 'QEMU_VM_COMMAND' * 0 just a normal return @@ -1997,6 +2011,9 @@ static int loadvm_process_command(QEMUFile *f) case MIG_CMD_RECV_BITMAP: return loadvm_handle_recv_bitmap(mis, len); + + case MIG_CMD_ENABLE_COLO: + return loadvm_process_enable_colo(mis); } return 0; diff --git a/migration/savevm.h b/migration/savevm.h index a5e65b8ae3..8373c2f6bd 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -55,6 +55,7 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint16_t len, uint64_t *start_list, uint64_t *length_list); +void qemu_savevm_send_colo_enable(QEMUFile *f); int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); diff --git a/migration/trace-events b/migration/trace-events index 9430f3cbe0..fa0ff3f3bf 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -37,6 +37,7 @@ savevm_send_ping(uint32_t val) "0x%x" savevm_send_postcopy_listen(void) "" savevm_send_postcopy_run(void) "" savevm_send_postcopy_resume(void) "" +savevm_send_colo_enable(void) "" savevm_send_recv_bitmap(char *name) "%s" savevm_state_setup(void) "" savevm_state_resume_prepare(void) "" diff --git a/vl.c b/vl.c index 16b913f9d5..a4e877e2ff 100644 --- a/vl.c +++ b/vl.c @@ -4392,8 +4392,6 @@ int main(int argc, char **argv, char **envp) #endif } - colo_info_init(); - if (net_init_clients(&err) < 0) { error_report_err(err); exit(1); From patchwork Sun Jul 22 19:33:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539389 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 74991174A for ; Sun, 22 Jul 2018 19:44:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 632222842E for ; Sun, 22 Jul 2018 19:44:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5512C28434; Sun, 22 Jul 2018 19:44:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9D9A12842E for ; Sun, 22 Jul 2018 19:44:36 +0000 (UTC) Received: from localhost ([::1]:57262 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKHT-0002NR-Pz for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:44:35 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37724) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK83-0002cd-Tz for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK82-0000dR-Jv for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:51 -0400 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:42138) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK82-0000cS-Bc for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:50 -0400 Received: by mail-pg1-x541.google.com with SMTP id y4-v6so10649781pgp.9 for ; Sun, 22 Jul 2018 12:34:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=W8JfVKmjPbrGiMOeI/vGJ6LI3Wusvl/osRJwUEswx5w=; b=RZjW0qfIUiGIrzk1PheO9OCMsIhid0Og9R+j+0WbjqB5luxxt8M39C7kbNk5S9aN37 MTpJP8wejLHUEoXmqPwtCK4bbpWZCX+dQM9Hn5UiWsn46bAV/UBYmC7k186Tm/oHFEYE HK1r+za79k6zva81T2P78GoBQ0ox66xTAJZk8+2GL13Myw+1Bk8FxF8iYd7T6djKD/Zb mhW8LDDyPfZM0/pROaXLrD3E7Oc6d75Ef9UHrdUDWUfAUMW6d/id1opb6eV5gT5SiP8z hV4lRzIcavLHPx5ap4+7mabqG5M6MuBazMdmTmErDpcTdC8+4lmpTXOsq0Vd9Avxme4K 8HMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=W8JfVKmjPbrGiMOeI/vGJ6LI3Wusvl/osRJwUEswx5w=; b=UP8zVyp1pE6CXE8FK8jpdjIgTyXRIPkfByp8ttpm5gAqEwlu60vmLESrczawQ/iqZn 5hCVJZXLujgVdM7AJPY5u6V428yPgxnvM5dhpUvITdPAqLcSxECaxVsXGKd+j95k3yOv qXWJ68XSWQZ2C+HbjQ/04eunQlgNaxZGuUZzxdoVr1p3evDoDmEk7hepLcDMnuE/V336 UaYbmzvkiLdPv7ZNsqZhroeZGoHBgTSxltXym1Q8C5wIKps7flSvdKfdbBOQYJBB189E UIEEUT1ZYa1pOw2DCk4RfRRjNYVhaTqtEqkLtzDWPYoloxDlYybwVJIWDA8XnR380MAy oPZg== X-Gm-Message-State: AOUpUlHsXFwCjs1NPtrDoug0NOPnd+R5pn2QsFaWuenPjSzfs1jAvbfP gJaaUnt25qnA/Yr0qySuweSSauim X-Google-Smtp-Source: AAOMgpcBo2VOI/IepzjXT02XFFBhsd97uOAM31K7PU76+1mBcavIYSsdYslZZMbXTwDxK8PJ4wfRNA== X-Received: by 2002:a65:6211:: with SMTP id d17-v6mr9759678pgv.450.1532288089063; Sun, 22 Jul 2018 12:34:49 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.34.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:34:48 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:37 +0800 Message-Id: <20180722193350.6028-8-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::541 Subject: [Qemu-devel] [PATCH V10 07/20] COLO: Load dirty pages into SVM's RAM cache firstly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP We should not load PVM's state directly into SVM, because there maybe some errors happen when SVM is receving data, which will break SVM. We need to ensure receving all data before load the state into SVM. We use an extra memory to cache these data (PVM's ram). The ram cache in secondary side is initially the same as SVM/PVM's memory. And in the process of checkpoint, we cache the dirty pages of PVM into this ram cache firstly, so this ram cache always the same as PVM's memory at every checkpoint, then we flush this cached ram to SVM after we receive all PVM's state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- include/exec/ram_addr.h | 1 + migration/migration.c | 6 +++ migration/ram.c | 83 ++++++++++++++++++++++++++++++++++++++++- migration/ram.h | 4 ++ migration/savevm.c | 2 +- 5 files changed, 93 insertions(+), 3 deletions(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index cf4ce06248..a78c1c99a7 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -27,6 +27,7 @@ struct RAMBlock { struct rcu_head rcu; struct MemoryRegion *mr; uint8_t *host; + uint8_t *colo_cache; /* For colo, VM's ram cache */ ram_addr_t offset; ram_addr_t used_length; ram_addr_t max_length; diff --git a/migration/migration.c b/migration/migration.c index c645f66f4e..d9683e06d3 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -441,6 +441,10 @@ static void process_incoming_migration_co(void *opaque) error_report_err(local_err); exit(EXIT_FAILURE); } + if (colo_init_ram_cache() < 0) { + error_report("Init ram cache failed"); + exit(EXIT_FAILURE); + } mis->migration_incoming_co = qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); @@ -449,6 +453,8 @@ static void process_incoming_migration_co(void *opaque) /* Wait checkpoint incoming thread exit before free resource */ qemu_thread_join(&mis->colo_incoming_thread); + /* We hold the global iothread lock, so it is safe here */ + colo_release_ram_cache(); } if (ret < 0) { diff --git a/migration/ram.c b/migration/ram.c index 52dd678092..33ebd09d70 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3314,6 +3314,20 @@ static inline void *host_from_ram_block_offset(RAMBlock *block, return block->host + offset; } +static inline void *colo_cache_from_block_offset(RAMBlock *block, + ram_addr_t offset) +{ + if (!offset_in_ramblock(block, offset)) { + return NULL; + } + if (!block->colo_cache) { + error_report("%s: colo_cache is NULL in block :%s", + __func__, block->idstr); + return NULL; + } + return block->colo_cache + offset; +} + /** * ram_handle_compressed: handle the zero page case * @@ -3518,6 +3532,58 @@ static void decompress_data_with_multi_threads(QEMUFile *f, qemu_mutex_unlock(&decomp_done_lock); } +/* + * colo cache: this is for secondary VM, we cache the whole + * memory of the secondary VM, it is need to hold the global lock + * to call this helper. + */ +int colo_init_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + block->colo_cache = qemu_anon_ram_alloc(block->used_length, + NULL, + false); + if (!block->colo_cache) { + error_report("%s: Can't alloc memory for COLO cache of block %s," + "size 0x" RAM_ADDR_FMT, __func__, block->idstr, + block->used_length); + goto out_locked; + } + memcpy(block->colo_cache, block->host, block->used_length); + } + rcu_read_unlock(); + return 0; + +out_locked: + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache = NULL; + } + } + + rcu_read_unlock(); + return -errno; +} + +/* It is need to hold the global lock to call this helper */ +void colo_release_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache = NULL; + } + } + rcu_read_unlock(); +} + /** * ram_load_setup: Setup RAM for migration incoming side * @@ -3534,6 +3600,7 @@ static int ram_load_setup(QEMUFile *f, void *opaque) xbzrle_load_setup(); ramblock_recv_map_init(); + return 0; } @@ -3547,6 +3614,7 @@ static int ram_load_cleanup(void *opaque) g_free(rb->receivedmap); rb->receivedmap = NULL; } + return 0; } @@ -3784,13 +3852,24 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { RAMBlock *block = ram_block_from_stream(f, flags); - host = host_from_ram_block_offset(block, addr); + /* + * After going into COLO, we should load the Page into colo_cache. + */ + if (migration_incoming_in_colo_state()) { + host = colo_cache_from_block_offset(block, addr); + } else { + host = host_from_ram_block_offset(block, addr); + } if (!host) { error_report("Illegal RAM offset " RAM_ADDR_FMT, addr); ret = -EINVAL; break; } - ramblock_recv_bitmap_set(block, host); + + if (!migration_incoming_in_colo_state()) { + ramblock_recv_bitmap_set(block, host); + } + trace_ram_load_loop(block->idstr, (uint64_t)addr, flags, host); } diff --git a/migration/ram.h b/migration/ram.h index 457bf54b8c..d009480494 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -70,4 +70,8 @@ int64_t ramblock_recv_bitmap_send(QEMUFile *file, const char *block_name); int ram_dirty_bitmap_reload(MigrationState *s, RAMBlock *rb); +/* ram cache */ +int colo_init_ram_cache(void); +void colo_release_ram_cache(void); + #endif diff --git a/migration/savevm.c b/migration/savevm.c index 437308877a..33e9e7cda0 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1929,7 +1929,7 @@ static int loadvm_handle_recv_bitmap(MigrationIncomingState *mis, static int loadvm_process_enable_colo(MigrationIncomingState *mis) { migration_incoming_enable_colo(); - return 0; + return colo_init_ram_cache(); } /* From patchwork Sun Jul 22 19:33:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539397 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E54C6112B for ; Sun, 22 Jul 2018 19:47:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C973B27F92 for ; Sun, 22 Jul 2018 19:47:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BA7E227FA1; Sun, 22 Jul 2018 19:47:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 173D92842E for ; Sun, 22 Jul 2018 19:47:34 +0000 (UTC) Received: from localhost ([::1]:57277 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKKL-0004qa-K3 for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:47:33 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37740) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK8B-0002iW-11 for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK87-0000fP-TO for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:59 -0400 Received: from mail-pl0-x244.google.com ([2607:f8b0:400e:c01::244]:36471) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK87-0000f6-N1 for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:34:55 -0400 Received: by mail-pl0-x244.google.com with SMTP id e11-v6so7297669plb.3 for ; Sun, 22 Jul 2018 12:34:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8tCTr8WWMvBgNBqYh56Canq/y8D1wo9Vrf1vFwhwvxM=; b=BYjbt/D31AfRjt0ypbw56SQCThTq3xJZEuzrufBarRp5l+bcnmrxGoOMks6W13Em7P As8Pf3E3TR4vBL/DQXbciKmQY+re6J2xOMdoJ+zU4xkTcay/XFSYbkdWeyCNQ1Zay8Hi S4AEbhwwhzcZ5+npDPxNQV1Su98skkP1oBz38bLyl5q4KrViuImxZhRv6uxntS5Int6v h/mMbOdEdnPF2v7rglGj2dXfEEgcIUFlIQKdSfqrlIAWI9cG/WFAQgfeswPX7choKLbx WyfCqwMyaq+EuKkgiDg1iv2KNoQM5KH/Ch9PSL5MnPSetsE1n3UB4VW7Po4OT2hlxsEB EWaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8tCTr8WWMvBgNBqYh56Canq/y8D1wo9Vrf1vFwhwvxM=; b=byQLB5KO6nJO9DT1Gw4Wcq8DWgFHXMcZKWXrJd6gSQzfG7hTD+9mh3HZlct0EtRet/ eDL9jUN4dxkv0KQmwn4hYAgWO1m0egNZU8IRlxFjKM68SK+V52wLxu25TA+ESLNXu1KV Lqg+BxKlm+Af4jD5zresJp9TvF6NNuiy2td66vWK1ksrORwh14epLa4qEUUNw3rdGLls Lo2LM+dnk+uZJJunkjqLh9MxKDKv05HOeXC6LPdX/sPuSZ6Fp4LxpkfqR/Hv8qepOz/K nRfNiCbDs69M/FRetM49FePX2pmVRCe3XfcoTY7Wf7NWD9IsqbWhSmHioszm/WGxxip5 LBvg== X-Gm-Message-State: AOUpUlFEQ9UM9JHQCCwhh5XP1iGw2fVraP5M/kEOCMgdTlOIGo1dOoCK HHy9opOEwey4mPy/QHNCMLv0LqPh X-Google-Smtp-Source: AAOMgpeDZGwwLv4qsxVQD8VVQU3TVK5td2Ixud/cw8bbR/Tp5R3j9HvrRfhUrP/qUNdvrEiyH2vPxA== X-Received: by 2002:a17:902:6ac3:: with SMTP id i3-v6mr10164882plt.252.1532288094508; Sun, 22 Jul 2018 12:34:54 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.34.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:34:53 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:38 +0800 Message-Id: <20180722193350.6028-9-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::244 Subject: [Qemu-devel] [PATCH V10 08/20] ram/COLO: Record the dirty pages that SVM received X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 33ebd09d70..d1060f1337 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3325,6 +3325,15 @@ static inline void *colo_cache_from_block_offset(RAMBlock *block, __func__, block->idstr); return NULL; } + + /* + * During colo checkpoint, we need bitmap of these migrated pages. + * It help us to decide which pages in ram cache should be flushed + * into VM's RAM later. + */ + if (!test_and_set_bit(offset >> TARGET_PAGE_BITS, block->bmap)) { + ram_state->migration_dirty_pages++; + } return block->colo_cache + offset; } @@ -3555,6 +3564,24 @@ int colo_init_ram_cache(void) memcpy(block->colo_cache, block->host, block->used_length); } rcu_read_unlock(); + /* + * Record the dirty pages that sent by PVM, we use this dirty bitmap together + * with to decide which page in cache should be flushed into SVM's RAM. Here + * we use the same name 'ram_bitmap' as for migration. + */ + if (ram_bytes_total()) { + RAMBlock *block; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + unsigned long pages = block->max_length >> TARGET_PAGE_BITS; + + block->bmap = bitmap_new(pages); + bitmap_set(block->bmap, 0, pages); + } + } + ram_state = g_new0(RAMState, 1); + ram_state->migration_dirty_pages = 0; + return 0; out_locked: @@ -3574,6 +3601,10 @@ void colo_release_ram_cache(void) { RAMBlock *block; + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + g_free(block->bmap); + block->bmap = NULL; + } rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -3582,6 +3613,8 @@ void colo_release_ram_cache(void) } } rcu_read_unlock(); + g_free(ram_state); + ram_state = NULL; } /** From patchwork Sun Jul 22 19:33:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539379 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0841B174A for ; Sun, 22 Jul 2018 19:39:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EE0662842E for ; Sun, 22 Jul 2018 19:39:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E229028434; Sun, 22 Jul 2018 19:39:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 703E92842E for ; Sun, 22 Jul 2018 19:39:15 +0000 (UTC) Received: from localhost ([::1]:57228 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKCI-00064w-IU for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:39:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37754) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK8E-0002kp-K2 for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK8D-0000h0-Ck for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:02 -0400 Received: from mail-pg1-x52d.google.com ([2607:f8b0:4864:20::52d]:41393) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK8D-0000gl-6n for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:01 -0400 Received: by mail-pg1-x52d.google.com with SMTP id z8-v6so10655531pgu.8 for ; Sun, 22 Jul 2018 12:35:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BEIbFZWpl6d3aI3vjmk78DkeNB/5U4L8tjZqNXpkXj8=; b=O5jKYkmqEsAhprJ3gHL/8DRxd/zY32HDzTfRb5PKkfJjOduro5j7e9GH3DZ8uuesJL BGutIWNccDZpsL8fUEpocDlOoOgk0IewhkeN8tXVZNT3CT80sLabqtUaxBygchwApDcc axGkUwBXl0Kcb4c1nFtJ73AjO+SzH3Z1uZLqN5jW7H79aOry2jUXM/ooSBJACL8Old/p 7n8Htg+rVpawaV2ZKjV28gBJe6MPFgTx5JqOhJEW/VlGegVJaMDtAkr6RG4D8NqNCtDu SIlMkvCFdQmXYSMrxnBpk21EVfCv4dcE5fNnNbK2XzdHjOgjR8WQfuUhOZQwe9IwHeba VrTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BEIbFZWpl6d3aI3vjmk78DkeNB/5U4L8tjZqNXpkXj8=; b=fRh9HseGo366XYrUxOxgeM410NOc+KxCMshQRPURp3ZBxTn+1HtXbniBBO3hdg9q3S yiHWncuHb1fjQVSArc8PIm5gRvT4YNdez1PedHoOZV8fTFMjXO+svROdUHsdrIkuZNYu 5jCqHLOXJBkL42qdgOyAwfMLrUhJ53rgN+n+LlQpGmZSqs+neRiNPm7FuvNYpiNyvJHm ZPCfcyA4UgXJE/PL6n5aTog/c77juYUOZrYQXJ9+0XwXW/wtZwHzPJgZG1Vqh0+2LOgb o404EtJbFI49d0GCFgjE/7D7c6f63b1aG23FdruepDxvkFX2a5t43aD2ybLq1bOI+LLg RXdA== X-Gm-Message-State: AOUpUlEMoGwF9hvpiLUe2v/i2pazQSgZY9urpnMr1XmIFTiDCfeXG8jG YZcEo6a88/Bd0FhdIyaIE3viIXn2 X-Google-Smtp-Source: AAOMgpdLde/bNsZZIY454p1KBOpCRsb3dPmT41/Nxh5HgKr1rrSGyLl+hrixR5VRc8H0o3vBQhRaFg== X-Received: by 2002:a65:6411:: with SMTP id a17-v6mr9374539pgv.287.1532288100000; Sun, 22 Jul 2018 12:35:00 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.34.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:34:59 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:39 +0800 Message-Id: <20180722193350.6028-10-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::52d Subject: [Qemu-devel] [PATCH V10 09/20] COLO: Flush memory data from ram cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint time. So, the content of SVM's RAM cache will always be same with PVM's memory after checkpoint. Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way: Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's. Besides, we must ensure flush RAM cache before load device state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 37 +++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 2 files changed, 39 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index d1060f1337..4df85d69c1 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3830,6 +3830,39 @@ static bool postcopy_is_running(void) return ps >= POSTCOPY_INCOMING_LISTENING && ps < POSTCOPY_INCOMING_END; } +/* + * Flush content of RAM cache into SVM's memory. + * Only flush the pages that be dirtied by PVM or SVM or both. + */ +static void colo_flush_ram_cache(void) +{ + RAMBlock *block = NULL; + void *dst_host; + void *src_host; + unsigned long offset = 0; + + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); + rcu_read_lock(); + block = QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset = migration_bitmap_find_dirty(ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >= block->used_length) { + offset = 0; + block = QLIST_NEXT_RCU(block, next); + } else { + migration_bitmap_clear_dirty(ram_state, block, offset); + dst_host = block->host + (offset << TARGET_PAGE_BITS); + src_host = block->colo_cache + (offset << TARGET_PAGE_BITS); + memcpy(dst_host, src_host, TARGET_PAGE_SIZE); + } + } + + rcu_read_unlock(); + trace_colo_flush_ram_cache_end(); +} + static int ram_load(QEMUFile *f, void *opaque, int version_id) { int flags = 0, ret = 0, invalid_flags = 0; @@ -4006,6 +4039,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) ret |= wait_for_decompress_done(); rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); + + if (!ret && migration_incoming_in_colo_state()) { + colo_flush_ram_cache(); + } return ret; } diff --git a/migration/trace-events b/migration/trace-events index fa0ff3f3bf..bd2d0cd25a 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -102,6 +102,8 @@ ram_dirty_bitmap_sync_start(void) "" ram_dirty_bitmap_sync_wait(void) "" ram_dirty_bitmap_sync_complete(void) "" ram_state_resume_prepare(uint64_t v) "%" PRId64 +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" # migration/migration.c await_return_path_close_on_source_close(void) "" From patchwork Sun Jul 22 19:33:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539375 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4A075174A for ; Sun, 22 Jul 2018 19:39:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3A1C62842E for ; Sun, 22 Jul 2018 19:39:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2E7FF28434; Sun, 22 Jul 2018 19:39:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7AE582842E for ; Sun, 22 Jul 2018 19:38:59 +0000 (UTC) Received: from localhost ([::1]:57226 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKC2-0005vB-NT for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:38:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37778) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK8L-0002pn-VI for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:12 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK8I-0000k3-Pb for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:09 -0400 Received: from mail-pl0-x242.google.com ([2607:f8b0:400e:c01::242]:44775) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK8I-0000jr-Iw for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:06 -0400 Received: by mail-pl0-x242.google.com with SMTP id m16-v6so7296054pls.11 for ; Sun, 22 Jul 2018 12:35:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=c1iNzgfPXEp/nvSlMaf31b6bPiaJ6p6YLDqzlK2hcxM=; b=BLYlD5uufO31YiHc2cMqermQj2hLvEbthajHYCCNtP0KppFGH9jP8r7M4o8tAlFWTI 4xqMQCKDliyZfZnIx6/Ef02ei1retWBYX4VvHdeI8EI2pAL/HuIKu81i3vHlx+JiNMta XSHtv4qFNJ2INxUHqa9j8m5L+ppBC3XtMw54n8IqZWev4t/bWCqoknuBaT6m+EeCnG/5 ysuqn9S6xTFeMa6Sxsh4qaFzwiCVqLIb5HcS5wIt0sS2gTJexqk8G5klOvZp5GftpjSo Saxrk1hYZ/KC6qzgQeVwPo885PlwdPHbBpyBn9WDHLtqRW1S2a4A1Sogv4630of1jViW lswQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=c1iNzgfPXEp/nvSlMaf31b6bPiaJ6p6YLDqzlK2hcxM=; b=B0fVe4Et5namBDr0zJzM5EXOOF/INmxa7Xv0qUyeKhnoCnwL8caG7Ji26ZsIVzP2TP K+Fbfv9eLGr2njK9wV8XohiMKTcLJQQzSnqBVO20uOCnzajjRFEinLq5t2VbEvXvDA40 AW6nqEeTmpffUwDiWCIdHU5DswFcvY3dOWClXFkLhyyixl2mrM8D/W9TCG0ZKmAyQIVl vq9/k/QTCqFGDkLRCLuUqYM5Lr+QaybZWZ2BMzf3lO10aI+nDw/YrOTHFxB9a7iEoK4m 4nzI68nm3C5le5q/ssBGZyOfliDdbJp7zY0nqu1vxhOSMCDmnXLHQnzuid7ODCYSidxC TrmQ== X-Gm-Message-State: AOUpUlG67QgQo7xu7CIePoJzO9jlDT63VQQJqm0RbjCEei27Ahr8pGDp uUpqDvsHQwc9SBLJYj2ER0WqzO3E X-Google-Smtp-Source: AAOMgpcjiwWONpnUCBU149JenKb6U0gcNLKNP8jC//kn6hSI/jeHcVxhqzTDKorzYaPzpXbSdiBW5g== X-Received: by 2002:a17:902:bb8d:: with SMTP id m13-v6mr9987583pls.46.1532288105363; Sun, 22 Jul 2018 12:35:05 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.35.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:35:04 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:40 +0800 Message-Id: <20180722193350.6028-11-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::242 Subject: [Qemu-devel] [PATCH V10 10/20] qmp event: Add COLO_EXIT event to notify users while exited COLO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x-colo-lost-heartbeat', Users can intervene in COLO's failover work immediately. If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Eric Blake Reviewed-by: Markus Armbruster --- migration/colo.c | 31 +++++++++++++++++++++++++++++++ qapi/migration.json | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index c083d3696f..ab484ad754 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -28,6 +28,7 @@ #include "net/colo-compare.h" #include "net/colo.h" #include "block/block.h" +#include "qapi/qapi-events-migration.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -514,6 +515,23 @@ out: qemu_fclose(fb); } + /* + * There are only two reasons we can get here, some error happened + * or the user triggered failover. + */ + switch (failover_get_state()) { + case FAILOVER_STATUS_NONE: + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_ERROR, NULL); + break; + case FAILOVER_STATUS_REQUIRE: + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_REQUEST, NULL); + break; + default: + abort(); + } + /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); @@ -745,6 +763,19 @@ out: error_report_err(local_err); } + switch (failover_get_state()) { + case FAILOVER_STATUS_NONE: + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_ERROR, NULL); + break; + case FAILOVER_STATUS_REQUIRE: + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_REQUEST, NULL); + break; + default: + abort(); + } + if (fb) { qemu_fclose(fb); } diff --git a/qapi/migration.json b/qapi/migration.json index 186e8a7303..05f65680e1 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -900,6 +900,44 @@ { 'enum': 'FailoverStatus', 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } +## +# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +# +# @mode: report COLO mode when COLO exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 3.0 +# +# Example: +# +# <- { "timestamp": {"seconds": 2032141960, "microseconds": 417172}, +# "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "request" } } +# +## +{ 'event': 'COLO_EXIT', + 'data': {'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## +# @COLOExitReason: +# +# The reason for a COLO exit +# +# @none: no failover has ever happened. This can't occur in the +# COLO_EXIT event, only in the result of query-colo-status. +# +# @request: COLO exit is due to an external request +# +# @error: COLO exit is due to an internal error +# +# Since: 3.0 +## +{ 'enum': 'COLOExitReason', + 'data': [ 'none', 'request', 'error' ] } + ## # @x-colo-lost-heartbeat: # From patchwork Sun Jul 22 19:33:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539385 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 42D141805 for ; Sun, 22 Jul 2018 19:42:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 315612842E for ; Sun, 22 Jul 2018 19:42:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2509D28434; Sun, 22 Jul 2018 19:42:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id BD0A12842E for ; Sun, 22 Jul 2018 19:42:09 +0000 (UTC) Received: from localhost ([::1]:57248 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKF6-0000MM-Vk for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:42:09 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37789) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK8P-0002qI-C0 for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK8O-0000le-E1 for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:13 -0400 Received: from mail-pl0-x233.google.com ([2607:f8b0:400e:c01::233]:40801) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK8O-0000lR-7q for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:12 -0400 Received: by mail-pl0-x233.google.com with SMTP id s17-v6so7293453plp.7 for ; Sun, 22 Jul 2018 12:35:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=HG3OIejVlOlbbFk1ae3D2HyIIKK9Mr0hXX2S1B5pkdk=; b=Mstx4op5ZDP7bCwKWD/6yGFioMc6d5fUZHMdPPEXNi8MFrNtv6dsKkCYqh1y8WWF/i RPuhL5ktyXRq0FDmVvJVWFs+jKCG18BrmASpeEpwVlWCFkWMU/apgiycWVnBRwEFcD2B ZCMTBhoCYxoQgREpffaB2VPt7sDjTE9kkYrysWthwUOSTSg7/qs8bTnBP/+XWx7GjM05 OTNIOhi7nqWIrcFpjjuJRIfgzOOMkewMYRvCxQoSZAs0W6H1HMqwgyce4oZXiSTilDK5 vuG6JcDbIgbAwwsTKaJOMcrH4v0eTYdMfVrytq+ykiUujo4YmG3eEufsiYZHKkBI48/E MOFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=HG3OIejVlOlbbFk1ae3D2HyIIKK9Mr0hXX2S1B5pkdk=; b=S0LWUhF1o8pkgCStYi2x3wJzSQV8oeUHnMA7BcshJcJvwTXDm3YwhYalbUA9F0VBIz 3gVU55qtGL/xIy15RtA1zfCelMEwLP/Hz071b0QURKIDBr1VgdyGlKpu7kmJIsVs8l33 E/gKTGTfPrdVc4WUYIdC2ke22IR1uZchvCUPqpg4Ku8OF1m4m9UWbOLoYHKqeJAA2nr2 8gPidBHRcMCU3lKOZjaN+cbJ69BU6zra+Z+9+KSJhpSQ8tB3632Ps8wzgdiypASmyuzd SydB2ayVpOKIx/GGW1OXJfGktfq18UZHizpCPjWQmMFVXXBVdKptumCX9XTAZ/fnl2x6 VARg== X-Gm-Message-State: AOUpUlGY/dWb8YOedpF259lRcwqxFuExMNC4mKoPHod2pizGJFN7p4Iv SEIqpQjAAnyoAyBn8MTXXfvqm68h X-Google-Smtp-Source: AAOMgpdVmq1JeNmJUw6w3m0SIQJe3T2XQRdSHX5yB1m3/oQHuKPzilQgSf8kBWs5v4xksATTnyliOw== X-Received: by 2002:a17:902:ab8e:: with SMTP id f14-v6mr192420plr.5.1532288111021; Sun, 22 Jul 2018 12:35:11 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.35.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:35:10 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:41 +0800 Message-Id: <20180722193350.6028-12-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::233 Subject: [Qemu-devel] [PATCH V10 11/20] qapi/migration.json: Rename COLO unknown mode to none mode. X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Zhang Chen Suggested by Markus Armbruster rename COLO unknown mode to none mode. Signed-off-by: Zhang Chen Reviewed-by: Eric Blake Reviewed-by: Markus Armbruster --- migration/colo-failover.c | 2 +- migration/colo.c | 2 +- qapi/migration.json | 10 +++++----- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/migration/colo-failover.c b/migration/colo-failover.c index 0ae0c41221..4854a96c92 100644 --- a/migration/colo-failover.c +++ b/migration/colo-failover.c @@ -77,7 +77,7 @@ FailoverStatus failover_get_state(void) void qmp_x_colo_lost_heartbeat(Error **errp) { - if (get_colo_mode() == COLO_MODE_UNKNOWN) { + if (get_colo_mode() == COLO_MODE_NONE) { error_setg(errp, QERR_FEATURE_DISABLED, "colo"); return; } diff --git a/migration/colo.c b/migration/colo.c index ab484ad754..8fdb79ac73 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -160,7 +160,7 @@ COLOMode get_colo_mode(void) } else if (migration_incoming_in_colo_state()) { return COLO_MODE_SECONDARY; } else { - return COLO_MODE_UNKNOWN; + return COLO_MODE_NONE; } } diff --git a/qapi/migration.json b/qapi/migration.json index 05f65680e1..cdae7ddccb 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -867,18 +867,18 @@ ## # @COLOMode: # -# The colo mode +# The COLO current mode. # -# @unknown: unknown mode +# @none: COLO is disabled. # -# @primary: master side +# @primary: COLO node in primary side. # -# @secondary: slave side +# @secondary: COLO node in slave side. # # Since: 2.8 ## { 'enum': 'COLOMode', - 'data': [ 'unknown', 'primary', 'secondary'] } + 'data': [ 'none', 'primary', 'secondary'] } ## # @FailoverStatus: From patchwork Sun Jul 22 19:33:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539401 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 01EBB112B for ; Sun, 22 Jul 2018 19:49:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E24F12842E for ; Sun, 22 Jul 2018 19:49:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D53C228434; Sun, 22 Jul 2018 19:49:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7B24B2842E for ; Sun, 22 Jul 2018 19:49:31 +0000 (UTC) Received: from localhost ([::1]:57283 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKME-0005nb-QD for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:49:30 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37817) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK8W-0002wQ-Ov for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK8T-0000o5-L8 for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:20 -0400 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:44098) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK8T-0000np-Ed for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:17 -0400 Received: by mail-pg1-x541.google.com with SMTP id r1-v6so10655761pgp.11 for ; Sun, 22 Jul 2018 12:35:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Zfh8Y4pwvfjhTw2l8csOLfzbGD+HGghoH1qZptJUvF8=; b=Xl1igTB9mFWKyZyOuKoL/6HajLCcAXDj94NA4DAtlOEGGhf+tkADn8X6t9Ow8ACLCT ExXIxoXq1f4asDJrPz11RInAw1Utrtlx9Af3bZfnMkyybfiQWiWhpo41qLtoyjpKHR2d qRK0gH/JKVTIehnYMnNZRGpv2V/sH9eDzGhCKLhshG3n9HvEFNmrsL+ReggWmh1Wz4Df T+wra0Cso01WjNWxfZXXdBPJrgm7OcFB74gRR9oITtHZs8ZrU0wh4Yu8tULbZ0q5a+G5 fodZxbqhL9foJLn2ioCgkYdULlNAORYioaf1NNU0GlcSA70NHJYM252ldI/8DW/ewl1K 8KLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Zfh8Y4pwvfjhTw2l8csOLfzbGD+HGghoH1qZptJUvF8=; b=ZvBQ4J4UGogqucRwz92elA7hdO+hndeKJcApzYvBJRC1TWqzvwHMwC+OPgb78jlwx7 xtLRZnAEUTVw2nFEF6/SM5PVkgMBQ5ce8VgP5hGd62TJHwTlAfzcFFwcVx7aMdRcqTzq zw6syuiekK0lFHDaU0Pj/1I025HniVwxCePkoaxhqvq8NftJuxV2JKSzJA6NvOKSySIy 5mJCL4t6X9uqcOQZQFscm07jBJKPyvEK4hDu8o5mwTHnqS2GdvzDCc/x3ivG6/HgH/zv WcCmx4oaT0VoU8SmrJU0JX456CIj4NZ9XsCFcMIYhJ6zooavfXlN80o5MuDTJ7YqoKHe XG3w== X-Gm-Message-State: AOUpUlHaV5l7AVP9UEUPkIN1YhsEfINI6Irg1y5vDxhGTwldxR7sxcYA VL2wx/oZ8IKsRJW1q6SwcogtLljU X-Google-Smtp-Source: AAOMgpcJ5xcvwswKE+kmtO4ATtvw68q/53xkmKR9HVL8GatiZglc9fepnD2xKbSZyy7awZUm5s1QyQ== X-Received: by 2002:a62:1016:: with SMTP id y22-v6mr10402357pfi.109.1532288116288; Sun, 22 Jul 2018 12:35:16 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.35.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:35:15 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:42 +0800 Message-Id: <20180722193350.6028-13-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::541 Subject: [Qemu-devel] [PATCH V10 12/20] qapi: Add new command to query colo status X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Libvirt or other high level software can use this command query colo status. You can test this command like that: {'execute':'query-colo-status'} Signed-off-by: Zhang Chen --- migration/colo.c | 21 +++++++++++++++++++++ qapi/migration.json | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 8fdb79ac73..b355de3f01 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -29,6 +29,7 @@ #include "net/colo.h" #include "block/block.h" #include "qapi/qapi-events-migration.h" +#include "qapi/qmp/qerror.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -237,6 +238,26 @@ void qmp_xen_colo_do_checkpoint(Error **errp) #endif } +COLOStatus *qmp_query_colo_status(Error **errp) +{ + COLOStatus *s = g_new0(COLOStatus, 1); + + s->mode = get_colo_mode(); + + switch (failover_get_state()) { + case FAILOVER_STATUS_NONE: + s->reason = COLO_EXIT_REASON_NONE; + break; + case FAILOVER_STATUS_REQUIRE: + s->reason = COLO_EXIT_REASON_REQUEST; + break; + default: + s->reason = COLO_EXIT_REASON_ERROR; + } + + return s; +} + static void colo_send_message(QEMUFile *f, COLOMessage msg, Error **errp) { diff --git a/qapi/migration.json b/qapi/migration.json index cdae7ddccb..21b9946595 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1251,6 +1251,38 @@ ## { 'command': 'xen-colo-do-checkpoint' } +## +# @COLOStatus: +# +# The result format for 'query-colo-status'. +# +# @mode: COLO running mode. If COLO is running, this field will return +# 'primary' or 'secondary'. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 3.0 +## +{ 'struct': 'COLOStatus', + 'data': { 'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## +# @query-colo-status: +# +# Query COLO status while the vm is running. +# +# Returns: A @COLOStatus object showing the status. +# +# Example: +# +# -> { "execute": "query-colo-status" } +# <- { "return": { "mode": "primary", "active": true, "reason": "request" } } +# +# Since: 3.0 +## +{ 'command': 'query-colo-status', + 'returns': 'COLOStatus' } + ## # @migrate-recover: # From patchwork Sun Jul 22 19:33:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539387 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B0AD81805 for ; Sun, 22 Jul 2018 19:43:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A02F02842E for ; Sun, 22 Jul 2018 19:43:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9302428434; Sun, 22 Jul 2018 19:43:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E534B2842E for ; Sun, 22 Jul 2018 19:43:10 +0000 (UTC) Received: from localhost ([::1]:57250 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKG6-0001Br-13 for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:43:10 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37832) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK8a-000325-If for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:25 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK8Z-0000qc-CQ for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:24 -0400 Received: from mail-pg1-x536.google.com ([2607:f8b0:4864:20::536]:39602) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK8Z-0000pj-2m for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:23 -0400 Received: by mail-pg1-x536.google.com with SMTP id g2-v6so10661717pgs.6 for ; Sun, 22 Jul 2018 12:35:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=y0NAphH1aCmODK4in8hkDuygrGgElI5zXGOFCtmLZW0=; b=cEIoPpKjTzGq7z7bKfQMuif/MAAwsxu/5ZYkhsC3eY6R7rU4r8Hes5iW7wSBcWQPCS 9cLgm5PO8737C/KuqUF0O8f/u5r5Nl4sdga2o+z2AKxiL9TibyXa2+bWYiTVaoSpPci+ UG6qaAjoKqVSNRx7PerMiDoLlj4J1zvA6L/IbGGn9BgCgb3LMbPc7sMcTSmc1SVlPNzP ih948AqXjc7wN2+0J15WJs7ihmwNVhpX1N2l8sLqHTWFSZM08JXCDvBNZFkZZbnIknbZ FzMEkMtYz35abchMxwPMpG/fO24wNDkYJAzz4S27uoxI4hGbzQwePFtnDvLEPiJQcyAW QVWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=y0NAphH1aCmODK4in8hkDuygrGgElI5zXGOFCtmLZW0=; b=LbbqDIYD6zkbUsZNdlQqG4LOenwoAtWKfYuHZ5/Bt+eeYS8lZVoXDo3l+Pi0ZFw8uM AqUlUS+MPupPykGwafALMQIhmBSnVA/bF+ZlNp68qgjsRVM3ZzmN2HBctI+lrnRbMv+x xhqxfK2r+n1xS+vFwndgszzT53dOJhu8aYLUTRUwPIK1wRqDDyL/HLmuXv7zhWfkP+PL gBNwZ/Rv6leZdHlbt6TSCnVb/t8sA+oK6z01p2TDnL614vEzn+t0xCeJ2VHT7GGEoTs8 bDROs+adh4oDdvR1LQ1j+FBEV1Jfps2fBBIwHp3SHwXTF2PGMggTcanP0MA5oGwidZz/ bCMA== X-Gm-Message-State: AOUpUlE4CFHyJnDIsOrRkAOLBNPdkffcVss0QlRlqcoOpAaNO711T+/C 67ptiP3L0tmleXOXIzeJodvR2pMs X-Google-Smtp-Source: AAOMgpdHeg5iIoOl94y0s7RNHavzCCtpNQDa1qdyGLh7bkv7YzrQQAoKn9mcHTltipcl1/t+ED3hqA== X-Received: by 2002:a62:b0c:: with SMTP id t12-v6mr10444692pfi.36.1532288121815; Sun, 22 Jul 2018 12:35:21 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.35.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:35:20 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:43 +0800 Message-Id: <20180722193350.6028-14-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::536 Subject: [Qemu-devel] [PATCH V10 13/20] savevm: split the process of different stages for loadvm/savevm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP There are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections. We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state. So we add three new helper functions: qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration. Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 41 ++++++++++++++++++++++++++++++++--------- migration/savevm.c | 36 +++++++++++++++++++++++++++++------- migration/savevm.h | 4 ++++ 3 files changed, 65 insertions(+), 16 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index b355de3f01..688d6f40b2 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -30,6 +30,7 @@ #include "block/block.h" #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" +#include "sysemu/cpus.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -401,23 +402,34 @@ static int colo_do_checkpoint_transaction(MigrationState *s, /* Disable block migration */ migrate_set_block_enabled(false, &local_err); - qemu_savevm_state_header(fb); - qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); replication_do_checkpoint_all(&local_err); if (local_err) { qemu_mutex_unlock_iothread(); goto out; } - qemu_savevm_state_complete_precopy(fb, false, false); - qemu_mutex_unlock_iothread(); - - qemu_fflush(fb); colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_err); if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* Note: device state is saved into buffer */ + ret = qemu_save_device_state(fb); + + qemu_mutex_unlock_iothread(); + if (ret < 0) { goto out; } + /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + + qemu_fflush(fb); + /* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read. @@ -635,6 +647,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t total_size; uint64_t value; Error *local_err = NULL; + int ret; qemu_sem_init(&mis->colo_incoming_sem, 0); @@ -707,6 +720,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + qemu_mutex_lock_iothread(); + cpu_synchronize_all_pre_loadvm(); + ret = qemu_loadvm_state_main(mis->from_src_file, mis); + qemu_mutex_unlock_iothread(); + + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } + value = colo_receive_message_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); if (local_err) { @@ -738,10 +761,10 @@ void *colo_process_incoming_thread(void *opaque) } qemu_mutex_lock_iothread(); - qemu_system_reset(SHUTDOWN_CAUSE_NONE); vmstate_loading = true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + ret = qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } diff --git a/migration/savevm.c b/migration/savevm.c index 33e9e7cda0..3a6aa747f9 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1378,13 +1378,21 @@ done: return ret; } -static int qemu_save_device_state(QEMUFile *f) +void qemu_savevm_live_state(QEMUFile *f) { - SaveStateEntry *se; + /* save QEMU_VM_SECTION_END section */ + qemu_savevm_state_complete_precopy(f, true, false); + qemu_put_byte(f, QEMU_VM_EOF); +} - qemu_put_be32(f, QEMU_VM_FILE_MAGIC); - qemu_put_be32(f, QEMU_VM_FILE_VERSION); +int qemu_save_device_state(QEMUFile *f) +{ + SaveStateEntry *se; + if (!migration_in_colo_state()) { + qemu_put_be32(f, QEMU_VM_FILE_MAGIC); + qemu_put_be32(f, QEMU_VM_FILE_VERSION); + } cpu_synchronize_all_states(); QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1440,8 +1448,6 @@ enum LoadVMExitCodes { LOADVM_QUIT = 1, }; -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); - /* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything @@ -2241,7 +2247,7 @@ static bool postcopy_pause_incoming(MigrationIncomingState *mis) return true; } -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) { uint8_t section_type; int ret = 0; @@ -2410,6 +2416,22 @@ int qemu_loadvm_state(QEMUFile *f) return ret; } +int qemu_load_device_state(QEMUFile *f) +{ + MigrationIncomingState *mis = migration_incoming_get_current(); + int ret; + + /* Load QEMU_VM_SECTION_FULL section */ + ret = qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to load device state: %d", ret); + return ret; + } + + cpu_synchronize_all_post_init(); + return 0; +} + int save_snapshot(const char *name, Error **errp) { BlockDriverState *bs, *bs1; diff --git a/migration/savevm.h b/migration/savevm.h index 8373c2f6bd..51a4b9caa8 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -56,8 +56,12 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint64_t *start_list, uint64_t *length_list); void qemu_savevm_send_colo_enable(QEMUFile *f); +void qemu_savevm_live_state(QEMUFile *f); +int qemu_save_device_state(QEMUFile *f); int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); +int qemu_load_device_state(QEMUFile *f); #endif From patchwork Sun Jul 22 19:33:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539403 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E3DB4112B for ; Sun, 22 Jul 2018 19:51:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C963E28420 for ; Sun, 22 Jul 2018 19:51:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BA9FC28434; Sun, 22 Jul 2018 19:51:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 67A9128420 for ; Sun, 22 Jul 2018 19:51:18 +0000 (UTC) Received: from localhost ([::1]:57299 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKNx-0007Im-LK for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:51:17 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37875) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK8i-0003BF-2J for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK8e-0000w7-Sr for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:32 -0400 Received: from mail-pg1-x544.google.com ([2607:f8b0:4864:20::544]:42144) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK8e-0000vj-Lg for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:28 -0400 Received: by mail-pg1-x544.google.com with SMTP id y4-v6so10650374pgp.9 for ; Sun, 22 Jul 2018 12:35:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=fAD/xEZYoJ4wnieoIOg5YBGlf+BOX1RbOrN8PsBOmkY=; b=cqDA7XJkyeAF1caWjtFc6zvbs/SG8kKAhAQT3tMqD85YegwS8vFAN15oQ6h96wBGMK 9Teao68rnxsdiUfyD9zgSEgz7xLZiAegQM0Ydi68dmN7jNN6LW38IiD4vYaLnxGB1mxS q2HMfeYCTRmFbo+0lLrVeDAlNQ/v+fYX9fRwklkL4xJQ6F0WNrp0kvqo1vZ+gn0ORW6r WpWi+aiY3Q7U+FM7g4mZ1LS59qMVA5fKTC3QsMnU6rAvJCdOuJHmmLZbPoWJefWjHoae mwW3GCZMBNGkL+djiOVGYRJ5aCmSnK2qruaCGR4QyjdZ6Ncz2d32FfLMIpzDHLN42GvC Gy4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=fAD/xEZYoJ4wnieoIOg5YBGlf+BOX1RbOrN8PsBOmkY=; b=XSy0UsOjBGJdcXjo9rpLKT2dKU00AkvnR2WSsnPzW2AhHVnxre+Lqr2DpyqX7v4LqL vXtaCJS/FlavCSWU/f4i/RwCnUbQtve7my1Vl021Sa0HJrnugmnJggQTeRTvJrcS2IZd VGsVKLMqjz5SyMT6Wd5qRoIWmKZJunL+Ik+LDm/7qDaUYrN0Gof9M4opWOLcR4CXi/V2 nEz93bOMOhes5nspYlnwiH8Sdk9DNteEr4J5erZbiBU3YEvvT3rJvgVRozTNYIBAR/pC NuOQLtjmVJvMe+s7tTJJgxL9W9aehAlw0Mn1iecp4WjXTOJ9oiXlg7j4bNfLq7TI6P0x H2Dw== X-Gm-Message-State: AOUpUlE67m/8BWlakalj0+DfQ3D/X59PKps6mFopEydBhNu23xSnnDS4 3b9KQEynTlO/Ffo8ouDO4u2fvSsW X-Google-Smtp-Source: AAOMgpc+jI/Z45wfWhALNuR2JEnMaTvQ6LLgVJUt/NEc4aRo4A5QnGim/lddtc+1ex3VE7YGMS8OfA== X-Received: by 2002:a62:ff0e:: with SMTP id b14-v6mr10385544pfn.135.1532288127454; Sun, 22 Jul 2018 12:35:27 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.35.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:35:26 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:44 +0800 Message-Id: <20180722193350.6028-15-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::544 Subject: [Qemu-devel] [PATCH V10 14/20] COLO: flush host dirty ram from cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 4df85d69c1..32bc141a2e 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3581,6 +3581,7 @@ int colo_init_ram_cache(void) } ram_state = g_new0(RAMState, 1); ram_state->migration_dirty_pages = 0; + memory_global_dirty_log_start(); return 0; @@ -3601,10 +3602,12 @@ void colo_release_ram_cache(void) { RAMBlock *block; + memory_global_dirty_log_stop(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { g_free(block->bmap); block->bmap = NULL; } + rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -3841,6 +3844,13 @@ static void colo_flush_ram_cache(void) void *src_host; unsigned long offset = 0; + memory_global_dirty_log_sync(); + rcu_read_lock(); + RAMBLOCK_FOREACH_MIGRATABLE(block) { + migration_bitmap_sync_range(ram_state, block, 0, block->used_length); + } + rcu_read_unlock(); + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); rcu_read_lock(); block = QLIST_FIRST_RCU(&ram_list.blocks); From patchwork Sun Jul 22 19:33:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539383 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D0EFB1805 for ; Sun, 22 Jul 2018 19:41:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C16BE2842E for ; Sun, 22 Jul 2018 19:41:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B3E2128434; Sun, 22 Jul 2018 19:41:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 517C32842E for ; Sun, 22 Jul 2018 19:41:55 +0000 (UTC) Received: from localhost ([::1]:57247 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKEs-0000Ay-BH for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:41:54 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37903) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK8l-0003Gs-Ph for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:36 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK8k-0000zq-PA for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:35 -0400 Received: from mail-pl0-x230.google.com ([2607:f8b0:400e:c01::230]:35438) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK8k-0000zS-FK for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:34 -0400 Received: by mail-pl0-x230.google.com with SMTP id w3-v6so7296826plq.2 for ; Sun, 22 Jul 2018 12:35:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=K3dkhi7bMX5ekmDfrikbh+64RrjWU3/pDvJ5q8A7RAU=; b=Ny0jFUJpCDONUYnniS3CgZxSK/SFhWh3uS3HSHu18uXydx6mwnDPlNQSiOo//pAPkX NaZawBtzsHaLVbO9666dnfmkQfxP1QiMtVerqsYzAjqxMexUtNFkq7yZzi1CT45Ybiyl IPDbRC4GG9WLKCEg+svHJtifzxdl3Vseaj4Uw5MamUiarrCBC2WT6rENyH+8RDTPi8w1 a7c5hKrIsWW6xN4/azlIurWHJFwqnMq1OpXu19bd2N9ND/hnuZbsigkrJhL3/R01+Cyj cibCd7KnE3rL06jh7s0wtPax7XXzDu/Y+oyxf1fqvuIa+Up8Y3oSuE/l8V3uk5I53PmX ZnBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=K3dkhi7bMX5ekmDfrikbh+64RrjWU3/pDvJ5q8A7RAU=; b=kB+JoqYDVNvGFH7qvGNsVA0cN86T/ie7SXXuT6rATjQ58EfcY+0hY7Hr+wUk7CJ1J2 D2Xi5FD8UzHJQ8wt/MwM778Dwdi/v5DadVvLCnLlPfLyaVld52kjQ/1ua+SMP4533J3I r2WwME/x2cpw6Lbf9zNn5dVOafdwTAVVwjpPbDFSJbmJ3amXyw9yVBGPBtK6AzxfgRIO ZmipytqxMg2BWFeceGDK629xfDvfrjFDtMQfSJst69f1t/bGkqy1fkNIHfDki11OvY4E VXkvDMQOdtJjGDc74yOzHyvVuwqOvY/j2W3CKB69I9n3RZa78+zb2yUXFDwqW02tQ5OB pNfw== X-Gm-Message-State: AOUpUlFNTx8natVfdJA5H453PuCRV1WWmziwk2MEJrUofjgDKlgwJcZK cA/YQwtLQqgncUwBb8B2O3LtScex X-Google-Smtp-Source: AAOMgpeRVoDeMUFoeXZV2qnQqTcE4tp4WVt4+i4Ct2i4TVfzISdDEALccAr1J0GRd06dV3jTPgrJQg== X-Received: by 2002:a17:902:542:: with SMTP id 60-v6mr10326832plf.122.1532288133206; Sun, 22 Jul 2018 12:35:33 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.35.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:35:32 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:45 +0800 Message-Id: <20180722193350.6028-16-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::230 Subject: [Qemu-devel] [PATCH V10 15/20] net/net.c: Add net client type check function for COLO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Zhang Chen We add is_colo_support_client_type() to check the net client type for COLO-compare. Currently we just support TAP. Suggested by Jason. Signed-off-by: Zhang Chen --- include/net/net.h | 1 + net/colo-compare.c | 5 +++++ net/net.c | 14 ++++++++++++++ 3 files changed, 20 insertions(+) diff --git a/include/net/net.h b/include/net/net.h index 1425960f76..dcbc7ba9c0 100644 --- a/include/net/net.h +++ b/include/net/net.h @@ -212,6 +212,7 @@ void hmp_host_net_add(Monitor *mon, const QDict *qdict); void hmp_host_net_remove(Monitor *mon, const QDict *qdict); void netdev_add(QemuOpts *opts, Error **errp); void qmp_netdev_add(QDict *qdict, QObject **ret, Error **errp); +bool is_colo_support_client_type(void); int net_hub_id_for_client(NetClientState *nc, int *id); NetClientState *net_hub_port_find(int hub_id); diff --git a/net/colo-compare.c b/net/colo-compare.c index 426eab5973..b8c0240725 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -996,6 +996,11 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) return; } + if (!is_colo_support_client_type()) { + error_setg(errp, "COLO-compare: Net client type is not supported"); + return; + } + net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); diff --git a/net/net.c b/net/net.c index 2a3133990c..a77ea88fff 100644 --- a/net/net.c +++ b/net/net.c @@ -1733,3 +1733,17 @@ int net_fill_rstate(SocketReadState *rs, const uint8_t *buf, int size) assert(size == 0); return 0; } + +/* Currently, COLO just support TAP */ +bool is_colo_support_client_type(void) +{ + NetClientState *nc; + + QTAILQ_FOREACH(nc, &net_clients, next) { + if (nc->info->type != NET_CLIENT_DRIVER_TAP) { + return false; + } + } + + return true; +} From patchwork Sun Jul 22 19:33:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539391 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BF1501805 for ; Sun, 22 Jul 2018 19:44:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AE0002842E for ; Sun, 22 Jul 2018 19:44:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A00AC28434; Sun, 22 Jul 2018 19:44:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3CD762842E for ; Sun, 22 Jul 2018 19:44:43 +0000 (UTC) Received: from localhost ([::1]:57263 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKHa-0002Sv-C5 for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:44:42 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37916) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK8r-0003LK-0y for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK8p-00011n-UA for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:41 -0400 Received: from mail-pg1-x529.google.com ([2607:f8b0:4864:20::529]:32976) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK8p-00011X-O7 for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:39 -0400 Received: by mail-pg1-x529.google.com with SMTP id r5-v6so10658752pgv.0 for ; Sun, 22 Jul 2018 12:35:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=CfEoMIqL7qqiMO3dLjhS4on0yvtvChay14/KE6chgvY=; b=ZDkAH6meq2PfurVWTB7c8uFNCeM619N+s0+AiA5pVs/WWIP2EKhpkzzVxey1+0H6Su myLCJLnZ7lN+ceUTO8aUJmJxKF/XZXALTJiS6sSD5r68TwOO1+J0NMPTTZ/MQqqJ7sTM 7gmhxBBCj8tKXYwbSyBrewWXTQz4lINQEJizrQX7ZHaI58cRFfTB5ccm63WlqOJICOsg R55EbRX4HqVKDVW3nRDzeiHe7v/840cT24OLmIeqx+Pi4Ftdd13YDk/UGZCzvK5tMUiX 0mlTXiOEyunoO//Eh6uCgiUJDd31yfrmKA7Seayw4Y2FnIe0pdAQylH9xqwh/TARDm6P WATQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=CfEoMIqL7qqiMO3dLjhS4on0yvtvChay14/KE6chgvY=; b=p47HsgWELkVlrmBtk7Uv3oqyfGuEFh7CjUugnUu13c3XSoVl6k8NIYMSzvXeYTiOMf 8x6JDEEVS/VuwfDGEWj5g4h4QpWx0QvUueSr7TuOkifat7Y0wlXbnQXr5ubo02Bmm3b5 ZQGpHJHHyCf56ZfiroUWTesf174UsT6pSMIyob0J8Aha1sOg7pR3eA2/trHg/Iyk+fYD 0NU8bLxErTKB0d8iP/c0F1elbRmNELs+iWJThg55lMafcwci+uwBuxDSDklNUYdxH+f9 BuCHUIulM4HVf5OGbd1S2bt/k+t0r/ZW4ygu3Ghv+x9IT5TMCpILTZTnTcb/Zh/tFmxL F0FA== X-Gm-Message-State: AOUpUlHSW1dbuUIN1bd2ujAU2kYd/prBgxnOV1TxXlwmJ4m0/IoSZpGk AOJtFzDRtzEjhex4wcs4aMhtSUvo X-Google-Smtp-Source: AAOMgpcFmIyNla/VrJJSKuk49bKjukX/C0MkZd4MNqhtyGBK/LOD6QeiJu+a8ypBBGsT7fJzStihTQ== X-Received: by 2002:a63:5143:: with SMTP id r3-v6mr9792523pgl.11.1532288138546; Sun, 22 Jul 2018 12:35:38 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.35.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:35:37 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:46 +0800 Message-Id: <20180722193350.6028-17-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::529 Subject: [Qemu-devel] [PATCH V10 16/20] filter: Add handle_event method for NetFilterClass X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Signed-off-by: zhanghailiang --- include/net/filter.h | 5 +++++ net/filter.c | 17 +++++++++++++++++ net/net.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+) diff --git a/include/net/filter.h b/include/net/filter.h index 435acd6f82..49da666ac0 100644 --- a/include/net/filter.h +++ b/include/net/filter.h @@ -38,6 +38,8 @@ typedef ssize_t (FilterReceiveIOV)(NetFilterState *nc, typedef void (FilterStatusChanged) (NetFilterState *nf, Error **errp); +typedef void (FilterHandleEvent) (NetFilterState *nf, int event, Error **errp); + typedef struct NetFilterClass { ObjectClass parent_class; @@ -45,6 +47,7 @@ typedef struct NetFilterClass { FilterSetup *setup; FilterCleanup *cleanup; FilterStatusChanged *status_changed; + FilterHandleEvent *handle_event; /* mandatory */ FilterReceiveIOV *receive_iov; } NetFilterClass; @@ -77,4 +80,6 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender, int iovcnt, void *opaque); +void colo_notify_filters_event(int event, Error **errp); + #endif /* QEMU_NET_FILTER_H */ diff --git a/net/filter.c b/net/filter.c index 2fd7d7d663..0f17eba143 100644 --- a/net/filter.c +++ b/net/filter.c @@ -17,6 +17,8 @@ #include "net/vhost_net.h" #include "qom/object_interfaces.h" #include "qemu/iov.h" +#include "net/colo.h" +#include "migration/colo.h" static inline bool qemu_can_skip_netfilter(NetFilterState *nf) { @@ -245,11 +247,26 @@ static void netfilter_finalize(Object *obj) g_free(nf->netdev_id); } +static void dummy_handle_event(NetFilterState *nf, int event, Error **errp) +{ + switch (event) { + case COLO_EVENT_CHECKPOINT: + break; + case COLO_EVENT_FAILOVER: + object_property_set_str(OBJECT(nf), "off", "status", errp); + break; + default: + break; + } +} + static void netfilter_class_init(ObjectClass *oc, void *data) { UserCreatableClass *ucc = USER_CREATABLE_CLASS(oc); + NetFilterClass *nfc = NETFILTER_CLASS(oc); ucc->complete = netfilter_complete; + nfc->handle_event = dummy_handle_event; } static const TypeInfo netfilter_info = { diff --git a/net/net.c b/net/net.c index a77ea88fff..b4f6a2efb2 100644 --- a/net/net.c +++ b/net/net.c @@ -1331,6 +1331,34 @@ void hmp_info_network(Monitor *mon, const QDict *qdict) } } +void colo_notify_filters_event(int event, Error **errp) +{ + NetClientState *nc, *peer; + NetClientDriver type; + NetFilterState *nf; + NetFilterClass *nfc = NULL; + Error *local_err = NULL; + + QTAILQ_FOREACH(nc, &net_clients, next) { + peer = nc->peer; + type = nc->info->type; + if (!peer || type != NET_CLIENT_DRIVER_TAP) { + continue; + } + QTAILQ_FOREACH(nf, &nc->filters, next) { + nfc = NETFILTER_GET_CLASS(OBJECT(nf)); + if (!nfc->handle_event) { + continue; + } + nfc->handle_event(nf, event, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + } + } +} + void qmp_set_link(const char *name, bool up, Error **errp) { NetClientState *ncs[MAX_QUEUE_NUM]; From patchwork Sun Jul 22 19:33:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539395 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E435B1805 for ; Sun, 22 Jul 2018 19:46:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D45EE2842E for ; Sun, 22 Jul 2018 19:46:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C68C228434; Sun, 22 Jul 2018 19:46:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 345BE2842E for ; Sun, 22 Jul 2018 19:46:04 +0000 (UTC) Received: from localhost ([::1]:57271 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKIt-0003xA-AA for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:46:03 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37940) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK8w-0003OI-Q7 for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK8v-00014E-N1 for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:46 -0400 Received: from mail-pf1-x441.google.com ([2607:f8b0:4864:20::441]:38518) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK8v-00013w-EW for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:45 -0400 Received: by mail-pf1-x441.google.com with SMTP id x17-v6so275448pfh.5 for ; Sun, 22 Jul 2018 12:35:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=EPQ74liOmsLk9XyE7a7NWXiXrd7bctJZVK71VNiFsSU=; b=ALZxsY1tAF6FIekAuehVhyaZ/8Zg+v/kOzqompnBO9H64wl02xRygwyYSUsMLqx0kz +izLep5JuniSlHyJCCFvcwnhaEGeeB36ADCWFtFf3b+t4QbXyzP2GuboVk0EK7Nx9rkS By+HUVqcYdhZaGdKZk6LrdtNFABbbQYQlCJqeknwHPwmUIYIws+PB1mZJvV18YIHgyMX 3xIzT8A1VqIS7tpn8VR2+/UFB0RVQemqGpTFOJgKc2dZ5BT8+v+KXSdCouGUnjLIj4AO 5ShvP6drIZwBEIPD2ioI4J9GKBirBmWl77dfK+0wO4GmSR0jucNOdQ7PCG7MdMzDUA+D Kfvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=EPQ74liOmsLk9XyE7a7NWXiXrd7bctJZVK71VNiFsSU=; b=Kubr9CsqIZPivQNTW25uGNQAfDq7CI/wW4RkLv9haSM1jS0j1xrE4UNeP34m4WTva6 oeLKxrFestTx/h2bEotBfTgSvKuiQnCxoa8B2jeFiMlrcCXyuIwUs/0wvQiZhQqDETsB UlwG91GJ9Ytk8Cvv/3YxHV0Bc73AJ5aPo/X/dJwZBl3T5kYM6Nc9Nlfzso/g0aVTE8dT baXD9117oV9GhIhAvK3P7wK0oe61ra+JCStOuT0MKvHQ2/de4b84xAjyM2tBlubwVQ1m yMp9Uwbjwbdfi5Q+1CTuNO6c9Tt6A11lLXOwDY5vdv5zSm7ENUXyA6X5WiF3hcSxJzTy dONg== X-Gm-Message-State: AOUpUlEZmTSmnhZepZYcF7ClWNADBZr01crSQvpbYOMTKeWqP9AKEtFr om4Pon1QlD9++6cPvOmCYlVOUYV3 X-Google-Smtp-Source: AAOMgpc9OTbKLkmfxLXzCV6Suwk4+5gx/vZ1ynsbodQSjlqXnFgdbgIThM5GWFxd2wRzth7X4vcsIg== X-Received: by 2002:a63:4c56:: with SMTP id m22-v6mr9364756pgl.299.1532288144211; Sun, 22 Jul 2018 12:35:44 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.35.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:35:43 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:47 +0800 Message-Id: <20180722193350.6028-18-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::441 Subject: [Qemu-devel] [PATCH V10 17/20] filter-rewriter: handle checkpoint and failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP After one round of checkpoint, the states between PVM and SVM become consistent, so it is unnecessary to adjust the sequence of net packets for old connections, besides, while failover happens, filter-rewriter will into failover mode that needn't handle the new TCP connection. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- net/colo-compare.c | 12 ++++----- net/colo.c | 8 ++++++ net/colo.h | 2 ++ net/filter-rewriter.c | 57 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 73 insertions(+), 6 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index b8c0240725..462e822be6 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -116,6 +116,12 @@ enum { SECONDARY_IN, }; +static void colo_compare_inconsistency_notify(void) +{ + notifier_list_notify(&colo_compare_notifiers, + migrate_get_current()); +} + static int compare_chr_send(CompareState *s, const uint8_t *buf, uint32_t size, @@ -562,12 +568,6 @@ static int colo_old_packet_check_one(Packet *pkt, int64_t *check_time) } } -static void colo_compare_inconsistency_notify(void) -{ - notifier_list_notify(&colo_compare_notifiers, - migrate_get_current()); -} - void colo_compare_register_notifier(Notifier *notify) { notifier_list_add(&colo_compare_notifiers, notify); diff --git a/net/colo.c b/net/colo.c index 97c8fc928f..49176bf07b 100644 --- a/net/colo.c +++ b/net/colo.c @@ -221,3 +221,11 @@ Connection *connection_get(GHashTable *connection_track_table, return conn; } + +bool connection_has_tracked(GHashTable *connection_track_table, + ConnectionKey *key) +{ + Connection *conn = g_hash_table_lookup(connection_track_table, key); + + return conn ? true : false; +} diff --git a/net/colo.h b/net/colo.h index 0277e0e9ba..11c5226488 100644 --- a/net/colo.h +++ b/net/colo.h @@ -98,6 +98,8 @@ void connection_destroy(void *opaque); Connection *connection_get(GHashTable *connection_track_table, ConnectionKey *key, GQueue *conn_list); +bool connection_has_tracked(GHashTable *connection_track_table, + ConnectionKey *key); void connection_hashtable_reset(GHashTable *connection_track_table); Packet *packet_new(const void *data, int size, int vnet_hdr_len); void packet_destroy(void *opaque, void *user_data); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index f18a71bf2e..c463a0c1d0 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -20,11 +20,15 @@ #include "qemu/main-loop.h" #include "qemu/iov.h" #include "net/checksum.h" +#include "net/colo.h" +#include "migration/colo.h" #define FILTER_COLO_REWRITER(obj) \ OBJECT_CHECK(RewriterState, (obj), TYPE_FILTER_REWRITER) #define TYPE_FILTER_REWRITER "filter-rewriter" +#define FAILOVER_MODE_ON true +#define FAILOVER_MODE_OFF false typedef struct RewriterState { NetFilterState parent_obj; @@ -32,8 +36,14 @@ typedef struct RewriterState { /* hashtable to save connection */ GHashTable *connection_track_table; bool vnet_hdr; + bool failover_mode; } RewriterState; +static void filter_rewriter_failover_mode(RewriterState *s) +{ + s->failover_mode = FAILOVER_MODE_ON; +} + static void filter_rewriter_flush(NetFilterState *nf) { RewriterState *s = FILTER_COLO_REWRITER(nf); @@ -269,6 +279,13 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, */ reverse_connection_key(&key); } + + /* After failover we needn't change new TCP packet */ + if (s->failover_mode && + connection_has_tracked(s->connection_track_table, &key)) { + goto out; + } + conn = connection_get(s->connection_track_table, &key, NULL); @@ -302,11 +319,49 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState *nf, } } +out: packet_destroy(pkt, NULL); pkt = NULL; return 0; } +static void reset_seq_offset(gpointer key, gpointer value, gpointer user_data) +{ + Connection *conn = (Connection *)value; + + conn->offset = 0; +} + +static gboolean offset_is_nonzero(gpointer key, + gpointer value, + gpointer user_data) +{ + Connection *conn = (Connection *)value; + + return conn->offset ? true : false; +} + +static void colo_rewriter_handle_event(NetFilterState *nf, int event, + Error **errp) +{ + RewriterState *rs = FILTER_COLO_REWRITER(nf); + + switch (event) { + case COLO_EVENT_CHECKPOINT: + g_hash_table_foreach(rs->connection_track_table, + reset_seq_offset, NULL); + break; + case COLO_EVENT_FAILOVER: + if (!g_hash_table_find(rs->connection_track_table, + offset_is_nonzero, NULL)) { + filter_rewriter_failover_mode(rs); + } + break; + default: + break; + } +} + static void colo_rewriter_cleanup(NetFilterState *nf) { RewriterState *s = FILTER_COLO_REWRITER(nf); @@ -350,6 +405,7 @@ static void filter_rewriter_init(Object *obj) RewriterState *s = FILTER_COLO_REWRITER(obj); s->vnet_hdr = false; + s->failover_mode = FAILOVER_MODE_OFF; object_property_add_bool(obj, "vnet_hdr_support", filter_rewriter_get_vnet_hdr, filter_rewriter_set_vnet_hdr, NULL); @@ -362,6 +418,7 @@ static void colo_rewriter_class_init(ObjectClass *oc, void *data) nfc->setup = colo_rewriter_setup; nfc->cleanup = colo_rewriter_cleanup; nfc->receive_iov = colo_rewriter_receive_iov; + nfc->handle_event = colo_rewriter_handle_event; } static const TypeInfo colo_rewriter_info = { From patchwork Sun Jul 22 19:33:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539405 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C06211805 for ; Sun, 22 Jul 2018 19:52:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A3D2C28420 for ; Sun, 22 Jul 2018 19:52:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 96D4E28434; Sun, 22 Jul 2018 19:52:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 09E7428420 for ; Sun, 22 Jul 2018 19:52:37 +0000 (UTC) Received: from localhost ([::1]:57304 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKPF-000830-0o for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:52:37 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37955) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK91-0003Qa-QM for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK91-00015W-10 for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:51 -0400 Received: from mail-pl0-x242.google.com ([2607:f8b0:400e:c01::242]:38590) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK90-00015O-Qi for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:50 -0400 Received: by mail-pl0-x242.google.com with SMTP id b1-v6so7294661pls.5 for ; Sun, 22 Jul 2018 12:35:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=O4LGoTN/7dnIo6QygGS9w7gOTri42IzEVdwEggBKL+4=; b=SrOKp5bhsY5mxN50Th7BdVV7ScHrtrfAcorb60sUWxT+fBQR1RfHILvHfwDACA8r/D 4WP/DLtRouyb8Hj5D9FWGr4KDF+mesCImlv5+WJGndXnPXVmv2SeIW8B/Jdq9Zc/DPy6 AcNQ2F6czLPJibpMsilmXRbQlcvJWapsMrdgpYFVCLhEG0JD1ia6mome5Od3L/GRuhuw FEfVILsQwl0NZPPmZPknR0N6JFET2MlqJZ+Utr2TsqCqfB4smaaDEOemQC9lzdPAbP0D q4J6fkKqSP8I06SFsUCtRAWpXEgkrw/nGIl52VOrtnRtjxTiBKssnahhy1FdCxPCtce/ uV4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=O4LGoTN/7dnIo6QygGS9w7gOTri42IzEVdwEggBKL+4=; b=ED2cwQfRv/Pd2v7TpRShB7twHWss5nhm3hAJob8A5zA4yQBDvtxr32NJcys2EL1jAa 3bg5KvMHA4649F6+2UUqFjrPdEDEcBVvwin11m5cYrBAGBDdg7t0LogWi8rGYfqBmIvx lVDupv17pSakaz5v7Da1J9byDXQt7DwiQfJPOqmpzd5MVCd8JwCrweB+ZwFCEca8GLC3 V7RCQdLdTgoc5tAENOr5DxKQkhyAQnbe3rI+i3jSiligw4HZNupATht2UBR8Kr/a/xa5 nUNK/bBWQGocKzbQhNpDmL8Uy5xjllzxG1Kcu4CPBx2V08YLfBTfHYGAQjGn7jY7+gAg qviA== X-Gm-Message-State: AOUpUlEHUfSoZtZ1xmcSQKgoNelrO1GL4b1rW8Em4rR651eZlgOYsC0O L4lYYieUgc00K9trRRSRWtc0KjTR X-Google-Smtp-Source: AAOMgpeLhQke4dWcEwqLtqtKFUY9DzfbmBI7HJdObvmVGEXetqIBO34dujbQxjQd+p4SVH0L/i0r+g== X-Received: by 2002:a17:902:3f81:: with SMTP id a1-v6mr10163842pld.29.1532288149561; Sun, 22 Jul 2018 12:35:49 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.35.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:35:48 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:48 +0800 Message-Id: <20180722193350.6028-19-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::242 Subject: [Qemu-devel] [PATCH V10 18/20] COLO: notify net filters about checkpoint/failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang Notify all net filters about the checkpoint and failover event. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 688d6f40b2..8221a18add 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -31,6 +31,7 @@ #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" #include "sysemu/cpus.h" +#include "net/filter.h" static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -83,6 +84,12 @@ static void secondary_vm_do_failover(void) error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } + if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side"); /* recover runstate to normal migration finish state */ @@ -781,6 +788,14 @@ void *colo_process_incoming_thread(void *opaque) goto out; } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading = false; vm_start(); trace_colo_vm_state_change("stop", "run"); From patchwork Sun Jul 22 19:33:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539393 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 57F581805 for ; Sun, 22 Jul 2018 19:44:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 46D3C2842E for ; Sun, 22 Jul 2018 19:44:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3B33C28434; Sun, 22 Jul 2018 19:44:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E466B2842E for ; Sun, 22 Jul 2018 19:44:54 +0000 (UTC) Received: from localhost ([::1]:57264 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKHm-0002cc-2e for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:44:54 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37984) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK99-0003We-B3 for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:36:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK96-00016k-7j for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:59 -0400 Received: from mail-pf1-x442.google.com ([2607:f8b0:4864:20::442]:33572) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK96-00016b-1t for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:35:56 -0400 Received: by mail-pf1-x442.google.com with SMTP id b17-v6so495856pfi.0 for ; Sun, 22 Jul 2018 12:35:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mU7awURrWAnfdKXiuwCryS0yrZb2zbth6Y2Fz4lEEgw=; b=i1+WA6vF5sIWUzFFcFQOtOEJpOc6yZMp162cxRitduNB86JCncLDoFWwLnut1ptvMO eLiw7MPwyQjtC9RkkCBYZufXc+VQLHr0CVborJzrOxpiNkNtQJZb+JaFI3/GqL3WyTbS EYJYqGDJLR2bccRM/I+dULh6bnHTkagmS0UqpLyjKW8umkoE8moqoYhvjabinmj3qHen K/W70ObjgQGu4JGatpXUS1ZFmVrYHc8fABu+dbkhxddi3eipCXpdpeuXk7qd/ClnUxvg 8B3ipD/u5qii8YbjJ2W5kH5WFy1Ai7Mq+K7H/8TMJJUi3dYodn4G3Km+Cs/Y/Nje4HRG sWfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mU7awURrWAnfdKXiuwCryS0yrZb2zbth6Y2Fz4lEEgw=; b=d4cwuKYgjfmAKLTy9ZqDeIhLC5Z79noVjze3ezJFwsTBUIps3qFa7AK6ToFdWBkEIq Vn155/KlJnG0Z1oOlsVHHJrcxKxJsnIWSqTK3iXKoWszeikpXzhtNq44kTkL/cR1OHaD 3ZuofcmL6t4Hf7dd7O6vNV7GaO5MV8p3nVHZxssKhHv6s51Hk4PrOH27YEgrO71bDnhN zNzrZ67g3Df0t9bnsIlViqVnZOiEh6TeO3dpG+rMYCY5/7eKvet2z9zcpHTRKy3Q7QCK nreH5Axha96Fb5jYBvV2T7IKGQtUKjbvDDDHzACjgDKAvOwb7sdDsvYTko/dJN/lmoo6 vong== X-Gm-Message-State: AOUpUlE64kfG1YBlctpJfr8n0r+oCv4cPVvh+2zRkF+MWrPa6RNk8btk 53jEL2g0hwGHduJkimID3hgROv/W X-Google-Smtp-Source: AAOMgpcAcMnZzEuNLUF0eh8qKLjF6MRELFofRArWdvcBBtW+ZWWQ7ySfoNL9qx5h8Vu3dGp+xJL30Q== X-Received: by 2002:a62:90d4:: with SMTP id q81-v6mr10484591pfk.37.1532288154885; Sun, 22 Jul 2018 12:35:54 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.35.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:35:53 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:49 +0800 Message-Id: <20180722193350.6028-20-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::442 Subject: [Qemu-devel] [PATCH V10 19/20] COLO: quick failover process by kick COLO thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: zhanghailiang COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 8221a18add..e951159071 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -131,6 +131,11 @@ static void primary_vm_do_failover(void) migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + /* + * kick COLO thread which might wait at + * qemu_sem_wait(&s->colo_checkpoint_sem). + */ + colo_checkpoint_notify(migrate_get_current()); /* * Wake up COLO thread which may blocked in recv() or send(), @@ -539,6 +544,9 @@ static void colo_process_checkpoint(MigrationState *s) qemu_sem_wait(&s->colo_checkpoint_sem); + if (s->state != MIGRATION_STATUS_COLO) { + goto out; + } ret = colo_do_checkpoint_transaction(s, bioc, fb); if (ret < 0) { goto out; From patchwork Sun Jul 22 19:33:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10539399 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 11BCA1805 for ; Sun, 22 Jul 2018 19:47:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 00FA127F92 for ; Sun, 22 Jul 2018 19:47:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E682F283BD; Sun, 22 Jul 2018 19:47:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 8E4A727F92 for ; Sun, 22 Jul 2018 19:47:49 +0000 (UTC) Received: from localhost ([::1]:57279 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhKKa-00050q-RZ for patchwork-qemu-devel@patchwork.kernel.org; Sun, 22 Jul 2018 15:47:48 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38005) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fhK9C-0003aC-Nl for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:36:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fhK9B-00018M-Ve for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:36:02 -0400 Received: from mail-pf1-x443.google.com ([2607:f8b0:4864:20::443]:33573) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fhK9B-000181-PT for qemu-devel@nongnu.org; Sun, 22 Jul 2018 15:36:01 -0400 Received: by mail-pf1-x443.google.com with SMTP id b17-v6so495873pfi.0 for ; Sun, 22 Jul 2018 12:36:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+PiF/0CCxP8AkFjhSAmThhHI/acJGqb7bzcME2c958A=; b=ZVmOJIMnbRAs8bkZZPC/gXSPJajjER1gp6UDJxu+bkXEFi4KHAD3RyVOfN8z0aMoFz XTXwC2M8QgzhDvCMWaIyDRCbzuQz3sxWY5d+lxtzrtngLEf/885t8dDsELcObCBcKmUl 5xTb/L22fOMfKVNVTV6SZX4jIGDowbVsRiLBoaeyUKhrNL4Dk1o8DpKETXgTHPGTw8Vs bpdT9ncHCyF6573ZBp/Wh87NKKompoGWTCoHhwHo+J4O3U+ygC9iatsOeMYi/jqlVjph /NKdECek8taHrbhNj98zRcfG0KXY9TcI+RjgjPC/YlGA8TuV3cgf8l/A0vmNq+aCL//v 9ksA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+PiF/0CCxP8AkFjhSAmThhHI/acJGqb7bzcME2c958A=; b=G0niJzAww/dsYC5+ZAp9Ir7Okff/r2dzfsEmwqD+wtQmJgUFxJzxh8EucftAHc3l/n HN887WGufGc7jXdat1mmYR0ijTHznqmWtT9dpcKwhEQ4kHyEMPQh2hXWRZFdso/8qK0H lDNuo+S3r8JWGVgNJ22YThS/hLOLwNavfKaZu2v5n/erWH/EvQm1ONLVFfSRceiO8d7Y DmDL6x7hvkfnNQw0cn23tNkkGBGhxRO8b+5Kcwo+8LBeK3JDnh45XAWc97mYSa961L5J ZdDOWYrCt4CqESOGiOeYZdU4sQ4qiAGBdCzt1tJczC7sJ82m39V6OKEvFVJfmrTZ4p10 NNzQ== X-Gm-Message-State: AOUpUlFsCoV5HoziTZsrDZ5PHroQVOMmQjpD2LXOCx7torO+7aIK3cYG DVOIdqIEjAM5Sp2XAM8YphoJQBdQ X-Google-Smtp-Source: AAOMgpeinyTn1tlKvSv9s61uX/t89K3wkKRKLPsGZDMF7G7q9haUzF9JgNjesQcHnzyvjUewNxxjmQ== X-Received: by 2002:a63:1b17:: with SMTP id b23-v6mr9601353pgb.275.1532288160614; Sun, 22 Jul 2018 12:36:00 -0700 (PDT) Received: from localhost.localdomain (155.150.229.35.bc.googleusercontent.com. [35.229.150.155]) by smtp.gmail.com with ESMTPSA id 87-v6sm14215353pfn.103.2018.07.22.12.35.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 12:35:59 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Mon, 23 Jul 2018 03:33:50 +0800 Message-Id: <20180722193350.6028-21-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180722193350.6028-1-zhangckid@gmail.com> References: <20180722193350.6028-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::443 Subject: [Qemu-devel] [PATCH V10 20/20] docs: Add COLO status diagram to COLO-FT.txt X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Zhang Chen This diagram make user better understand COLO. Suggested by Markus Armbruster. Signed-off-by: Zhang Chen --- docs/COLO-FT.txt | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/docs/COLO-FT.txt b/docs/COLO-FT.txt index d7c7dcda8f..d5007895d1 100644 --- a/docs/COLO-FT.txt +++ b/docs/COLO-FT.txt @@ -110,6 +110,40 @@ Note: HeartBeat has not been implemented yet, so you need to trigger failover process by using 'x-colo-lost-heartbeat' command. +== COLO operation status == + ++-----------------+ +| | +| Start COLO | +| | ++--------+--------+ + | + | Main qmp command: + | migrate-set-capabilities with x-colo + | migrate + | + v ++--------+--------+ +| | +| COLO running | +| | ++--------+--------+ + | + | Main qmp command: + | x-colo-lost-heartbeat + | or + | some error happened + v ++--------+--------+ +| | send qmp event: +| COLO failover | COLO_EXIT +| | ++-----------------+ + +COLO use the qmp command switching and report operation status. +The diagram just write the main qmp command, you can get the detail +in test procedure. + == Test procedure == 1. Startup qemu Primary: