From patchwork Sun Jun 3 05:05:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10445123 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6E8EE603B4 for ; Sun, 3 Jun 2018 05:11:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5D8F128A0A for ; Sun, 3 Jun 2018 05:11:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5242128AFB; Sun, 3 Jun 2018 05:11:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B6FAC28A0A for ; Sun, 3 Jun 2018 05:11:31 +0000 (UTC) Received: from localhost ([::1]:33654 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLIg-00034j-Rx for patchwork-qemu-devel@patchwork.kernel.org; Sun, 03 Jun 2018 01:11:30 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57747) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLDX-0008Aw-0v for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:12 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLDT-0000gF-Pd for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:11 -0400 Received: from mail-pl0-x234.google.com ([2607:f8b0:400e:c01::234]:35613) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLDT-0000fs-Hj for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:07 -0400 Received: by mail-pl0-x234.google.com with SMTP id i5-v6so17657621plt.2 for ; Sat, 02 Jun 2018 22:06:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=pUvZCAOlnw4NFtVLw+qbjIJRYnINssjK35yBI5gb4x8=; b=E51FiXmhRLpWDip6WSqw7Gsj3rjtY0jGK64YI3kk44SxigbZh44CqA6oU10f+L8Lkz GHde57fSETvysiDjflIAveb3SAMF5LN8z39cVYjUh1mCmXbIK8UB2bEEGCjQuRkwQiqI UhWZrCa+4HrEoDpT8EvHISQi4kbyumg2BAV6ieOv4iiZOwfToXSBm+uj7cpgb0yt1aYe hJrnAZvlKd6joPPYuuwDErHuYWTxv2yRCjrKRPZNpLUcnuoUOBmXcf9YD2EKsJcA05Au pjqwldSk0JwXWm38gdhiio/Rmh7uW4Y00+y1aujHW8TY94A1Ly5hjuXzUDePMsgKvcZ0 hG0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=pUvZCAOlnw4NFtVLw+qbjIJRYnINssjK35yBI5gb4x8=; b=spBv2ZmyoSfP/5VwTihLdXF3CetnGWuJOXa63jueIKsuIvXTobOTHpNMZXzHxIeaEh TN5BavNbJjE4bFYhxiZAEjPpGqHCTE7YfS4sgmwP2NxdBNBcvZIr2GuFy0bOmFnyG3Yr C95ETbpSmZSINe+CjoMvdxMBSbDAW4wpsMYMk+djfHIhh1PRwT2KU4HCSz5WKBolzzqu y/8xe8H5IygckAI9tR/KUvHBMDVPYY0srdD657iykMwc6RPyIi25xQagDQeGxWWCeDc3 zOLY8zIiwk5uYD+x8DxCG0T4qhCcoqvKJRHi9pRXu7W1xKEAh4vS6uUH97bIHc7n9Wue Wrvg== X-Gm-Message-State: ALKqPwcbPuHg1Wv+fwmHga9DwQ47GvSUhzrpLP7QOijgOoNrl4ahoFtA GNTEAqDrPWDwTn2D6riEXQ2LFw== X-Google-Smtp-Source: ADUXVKKDv9XGg9nEld5EKM1UOncXeyy1F0S9ane8otfmx2sRbaKBJw+8BW0phN+GXaCjZFH8TmXYBg== X-Received: by 2002:a17:902:7248:: with SMTP id c8-v6mr10309870pll.128.1528002366316; Sat, 02 Jun 2018 22:06:06 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.06.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:06:05 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:31 +0800 Message-Id: <20180603050546.6827-3-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::234 Subject: [Qemu-devel] [PATCH V8 02/17] colo-compare: implement the process of checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 6 ++++ net/colo-compare.c | 76 ++++++++++++++++++++++++++++++++++++++++ net/colo-compare.h | 22 ++++++++++++ 3 files changed, 104 insertions(+) create mode 100644 net/colo-compare.h diff --git a/include/migration/colo.h b/include/migration/colo.h index 2fe48ad353..fefb2fcf4c 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -16,6 +16,12 @@ #include "qemu-common.h" #include "qapi/qapi-types-migration.h" +enum colo_event { + COLO_EVENT_NONE, + COLO_EVENT_CHECKPOINT, + COLO_EVENT_FAILOVER, +}; + void colo_info_init(void); void migrate_start_colo_process(MigrationState *s); diff --git a/net/colo-compare.c b/net/colo-compare.c index 23b2d2c4cc..7ff3ae8904 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -27,11 +27,16 @@ #include "qemu/sockets.h" #include "net/colo.h" #include "sysemu/iothread.h" +#include "net/colo-compare.h" +#include "migration/colo.h" #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) +static QTAILQ_HEAD(, CompareState) net_compares = + QTAILQ_HEAD_INITIALIZER(net_compares); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 @@ -41,6 +46,10 @@ /* TODO: Should be configurable */ #define REGULAR_PACKET_CHECK_MS 3000 +static QemuMutex event_mtx; +static QemuCond event_complete_cond; +static int event_unhandled_count; + /* * + CompareState ++ * | | @@ -87,6 +96,11 @@ typedef struct CompareState { IOThread *iothread; GMainContext *worker_context; QEMUTimer *packet_check_timer; + + QEMUBH *event_bh; + enum colo_event event; + + QTAILQ_ENTRY(CompareState) next; } CompareState; typedef struct CompareClass { @@ -736,6 +750,25 @@ static void check_old_packet_regular(void *opaque) REGULAR_PACKET_CHECK_MS); } +/* Public API, Used for COLO frame to notify compare event */ +void colo_notify_compares_event(void *opaque, int event, Error **errp) +{ + CompareState *s; + + qemu_mutex_lock(&event_mtx); + QTAILQ_FOREACH(s, &net_compares, next) { + s->event = event; + qemu_bh_schedule(s->event_bh); + event_unhandled_count++; + } + /* Wait all compare threads to finish handling this event */ + while (event_unhandled_count > 0) { + qemu_cond_wait(&event_complete_cond, &event_mtx); + } + + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_timer_init(CompareState *s) { AioContext *ctx = iothread_get_aio_context(s->iothread); @@ -756,6 +789,28 @@ static void colo_compare_timer_del(CompareState *s) } } +static void colo_flush_packets(void *opaque, void *user_data); + +static void colo_compare_handle_event(void *opaque) +{ + CompareState *s = opaque; + + switch (s->event) { + case COLO_EVENT_CHECKPOINT: + g_queue_foreach(&s->conn_list, colo_flush_packets, s); + break; + case COLO_EVENT_FAILOVER: + break; + default: + break; + } + qemu_mutex_lock(&event_mtx); + assert(event_unhandled_count > 0); + event_unhandled_count--; + qemu_cond_broadcast(&event_complete_cond); + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_iothread(CompareState *s) { object_ref(OBJECT(s->iothread)); @@ -769,6 +824,7 @@ static void colo_compare_iothread(CompareState *s) s, s->worker_context, true); colo_compare_timer_init(s); + s->event_bh = qemu_bh_new(colo_compare_handle_event, s); } static char *compare_get_pri_indev(Object *obj, Error **errp) @@ -926,8 +982,13 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); + QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); + qemu_mutex_init(&event_mtx); + qemu_cond_init(&event_complete_cond); + s->connection_track_table = g_hash_table_new_full(connection_key_hash, connection_key_equal, g_free, @@ -990,6 +1051,7 @@ static void colo_compare_init(Object *obj) static void colo_compare_finalize(Object *obj) { CompareState *s = COLO_COMPARE(obj); + CompareState *tmp = NULL; qemu_chr_fe_deinit(&s->chr_pri_in, false); qemu_chr_fe_deinit(&s->chr_sec_in, false); @@ -997,6 +1059,16 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { colo_compare_timer_del(s); } + + qemu_bh_delete(s->event_bh); + + QTAILQ_FOREACH(tmp, &net_compares, next) { + if (!strcmp(tmp->outdev, s->outdev)) { + QTAILQ_REMOVE(&net_compares, s, next); + break; + } + } + /* Release all unhandled packets after compare thead exited */ g_queue_foreach(&s->conn_list, colo_flush_packets, s); @@ -1009,6 +1081,10 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { object_unref(OBJECT(s->iothread)); } + + qemu_mutex_destroy(&event_mtx); + qemu_cond_destroy(&event_complete_cond); + g_free(s->pri_indev); g_free(s->sec_indev); g_free(s->outdev); diff --git a/net/colo-compare.h b/net/colo-compare.h new file mode 100644 index 0000000000..1b1ce76aea --- /dev/null +++ b/net/colo-compare.h @@ -0,0 +1,22 @@ +/* + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) + * (a.k.a. Fault Tolerance or Continuous Replication) + * + * Copyright (c) 2017 HUAWEI TECHNOLOGIES CO., LTD. + * Copyright (c) 2017 FUJITSU LIMITED + * Copyright (c) 2017 Intel Corporation + * + * Authors: + * zhanghailiang + * Zhang Chen + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * later. See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event(void *opaque, int event, Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */