From patchwork Thu Jan 4 06:01:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 10143887 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id DA7736034B for ; Thu, 4 Jan 2018 06:05:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB6A526B41 for ; Thu, 4 Jan 2018 06:05:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BD40426CFC; Thu, 4 Jan 2018 06:05:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F40DB26B41 for ; Thu, 4 Jan 2018 06:05:57 +0000 (UTC) Received: from localhost ([::1]:47471 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWyf6-0007Gh-Uv for patchwork-qemu-devel@patchwork.kernel.org; Thu, 04 Jan 2018 01:05:56 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36715) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWyb9-0003U2-6v for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:01:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWyb7-0003vt-TU for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:01:51 -0500 Received: from mail-pg0-x242.google.com ([2607:f8b0:400e:c05::242]:41969) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWyb7-0003vV-LV for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:01:49 -0500 Received: by mail-pg0-x242.google.com with SMTP id 77so337307pgd.8 for ; Wed, 03 Jan 2018 22:01:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sKzd1WODysTlkeNRThruj5hxDE/MMDkcc/U2y8ce+fk=; b=hu+OpC/1Rh89/tvNAtJ8eZuzlOUKMXbqlkhJCAAH0wOJzQcsiOSfdiWBvg4vrcbJ1a D/CKP0UXnSQzPhWDoWrDTBgOWPiojN+30S1fJNsW4FlvTAHMp2EGKdhU5OAW8VdkBlCy mah1ijJ9f6hbOKea8Rbwae9lJ6HyTp9zdVcptXW/OGm4vxwDZQABDq4LSMiwmk94ypUa 49QlEBDMWg4hr+pq2f2+JcFhYa8OVFkvkpBblwhXHG2tvmsBj3xzNRJdvJJas5CxGApn wQofJ8AZTo3khGZJiajxSRZXg9dpJ5CVSH5RD/5MDQeO5++MgB7LZtZW1+q/PPOeP2cv ADzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sKzd1WODysTlkeNRThruj5hxDE/MMDkcc/U2y8ce+fk=; b=qNe3sSktrtK3OZixwK2hhnZZNjBT3J3nNnfgdNCMAzqBRU537h2R1CcSsltsEeTO1c KFxNlwyGNgu7j4k9kA6tzqsjkkjlv1AdDrBTA/yj323t+BN8Cc2S9PMyIIjpf/f95adI 4uBY5deDHoa3p4/GBetJeD/bAbBcaOuU94Y7fLh5nIVhRHKEvwqU3mVPGYXST2bCoSyW qfJaQwOLOpCAwcmh9wq9CCC3xMFhohxWhgIt2DaX7k6UAGdvCQ9a3h4mmaKWRSSNvuQh 1jKAnLOQ4PxX9Wh76PNWr01BdKP6bRv/usHT1ZUI/77CJUM74JDwZe4NvtuIfc2qjv3w XLsg== X-Gm-Message-State: AKGB3mKrzTwqG+T9wxcAxKeoPyIrNu/+1qOwwQmoNTaATdDhoBF/WJoM 2/FQevDZD3rW/eZpyeoXeF0UOOX4 X-Google-Smtp-Source: ACJfBotiyaE9XPsscHU+4sPdhd/BgUce41yLB3g2ZSNshebtx8J7MGG+oUJtGtiZ5oAkzjpDO20Trw== X-Received: by 10.98.34.85 with SMTP id i82mr3706821pfi.57.1515045708257; Wed, 03 Jan 2018 22:01:48 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.01.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:01:47 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:01 +0800 Message-Id: <1515045675-6993-3-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::242 Subject: [Qemu-devel] [PATCH RESEND V3 02/16] colo-compare: implement the process of checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , zhanghailiang , Zhang Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Cc: Jason Wang Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 6 ++++ net/colo-compare.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++++ net/colo-compare.h | 22 +++++++++++++++ 3 files changed, 99 insertions(+) create mode 100644 net/colo-compare.h diff --git a/include/migration/colo.h b/include/migration/colo.h index ff9874e..6adf3a5 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -15,6 +15,12 @@ #include "qemu-common.h" +enum colo_event { + COLO_EVENT_NONE, + COLO_EVENT_CHECKPOINT, + COLO_EVENT_FAILOVER, +}; + void colo_info_init(void); void migrate_start_colo_process(MigrationState *s); diff --git a/net/colo-compare.c b/net/colo-compare.c index 0ebdec9..e9cfca2 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -29,17 +29,28 @@ #include "qapi-visit.h" #include "net/colo.h" #include "sysemu/iothread.h" +#include "net/colo-compare.h" +#include "migration/colo.h" #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) +static QTAILQ_HEAD(, CompareState) net_compares = + QTAILQ_HEAD_INITIALIZER(net_compares); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 /* TODO: Should be configurable */ #define REGULAR_PACKET_CHECK_MS 3000 +static QemuMutex event_mtx = { .lock = PTHREAD_MUTEX_INITIALIZER, + .initialized = true }; +static QemuCond event_complete_cond = { .cond = PTHREAD_COND_INITIALIZER, + .initialized = true}; +static int event_unhandled_count; + /* * + CompareState ++ * | | @@ -86,6 +97,11 @@ typedef struct CompareState { IOThread *iothread; GMainContext *worker_context; QEMUTimer *packet_check_timer; + + QEMUBH *event_bh; + enum colo_event event; + + QTAILQ_ENTRY(CompareState) next; } CompareState; typedef struct CompareClass { @@ -631,6 +647,25 @@ static void check_old_packet_regular(void *opaque) REGULAR_PACKET_CHECK_MS); } +/* Public API, Used for COLO frame to notify compare event */ +void colo_notify_compares_event(void *opaque, int event, Error **errp) +{ + CompareState *s; + + qemu_mutex_lock(&event_mtx); + QTAILQ_FOREACH(s, &net_compares, next) { + s->event = event; + qemu_bh_schedule(s->event_bh); + event_unhandled_count++; + } + /* Wait all compare threads to finish handling this event */ + while (event_unhandled_count > 0) { + qemu_cond_wait(&event_complete_cond, &event_mtx); + } + + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_timer_init(CompareState *s) { AioContext *ctx = iothread_get_aio_context(s->iothread); @@ -651,6 +686,28 @@ static void colo_compare_timer_del(CompareState *s) } } +static void colo_flush_packets(void *opaque, void *user_data); + +static void colo_compare_handle_event(void *opaque) +{ + CompareState *s = opaque; + + switch (s->event) { + case COLO_EVENT_CHECKPOINT: + g_queue_foreach(&s->conn_list, colo_flush_packets, s); + break; + case COLO_EVENT_FAILOVER: + break; + default: + break; + } + qemu_mutex_lock(&event_mtx); + assert(event_unhandled_count > 0); + event_unhandled_count--; + qemu_cond_broadcast(&event_complete_cond); + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_iothread(CompareState *s) { object_ref(OBJECT(s->iothread)); @@ -664,6 +721,7 @@ static void colo_compare_iothread(CompareState *s) s, s->worker_context, true); colo_compare_timer_init(s); + s->event_bh = qemu_bh_new(colo_compare_handle_event, s); } static char *compare_get_pri_indev(Object *obj, Error **errp) @@ -821,6 +879,8 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); + QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); s->connection_track_table = g_hash_table_new_full(connection_key_hash, @@ -885,6 +945,7 @@ static void colo_compare_init(Object *obj) static void colo_compare_finalize(Object *obj) { CompareState *s = COLO_COMPARE(obj); + CompareState *tmp = NULL; qemu_chr_fe_deinit(&s->chr_pri_in, false); qemu_chr_fe_deinit(&s->chr_sec_in, false); @@ -892,6 +953,16 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { colo_compare_timer_del(s); } + + qemu_bh_delete(s->event_bh); + + QTAILQ_FOREACH(tmp, &net_compares, next) { + if (!strcmp(tmp->outdev, s->outdev)) { + QTAILQ_REMOVE(&net_compares, s, next); + break; + } + } + /* Release all unhandled packets after compare thead exited */ g_queue_foreach(&s->conn_list, colo_flush_packets, s); diff --git a/net/colo-compare.h b/net/colo-compare.h new file mode 100644 index 0000000..1b1ce76 --- /dev/null +++ b/net/colo-compare.h @@ -0,0 +1,22 @@ +/* + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) + * (a.k.a. Fault Tolerance or Continuous Replication) + * + * Copyright (c) 2017 HUAWEI TECHNOLOGIES CO., LTD. + * Copyright (c) 2017 FUJITSU LIMITED + * Copyright (c) 2017 Intel Corporation + * + * Authors: + * zhanghailiang + * Zhang Chen + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * later. See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event(void *opaque, int event, Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */