From patchwork Wed Feb 3 08:01:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B571C433DB for ; Wed, 3 Feb 2021 07:46:21 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1C46364E3E for ; Wed, 3 Feb 2021 07:46:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1C46364E3E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:58316 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Crk-0004gD-2S for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:46:20 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34218) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpe-0002xo-8I for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:10 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3358) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpP-0008EN-V8 for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:08 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr3XjKz163kw; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:36 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 01/18] migration/rdma: add the 'migrate_rdma_pin_all' function Date: Wed, 3 Feb 2021 16:01:34 +0800 Message-ID: <1612339311-114805-2-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng Reviewed-by: Dr. David Alan Gilbert --- migration/migration.c | 9 +++++++++ migration/migration.h | 1 + 2 files changed, 10 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 1986cb8..447dfb9 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2382,6 +2382,15 @@ bool migrate_use_events(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_EVENTS]; } +bool migrate_rdma_pin_all(void) +{ + MigrationState *s; + + s = migrate_get_current(); + + return s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL]; +} + bool migrate_use_multifd(void) { MigrationState *s; diff --git a/migration/migration.h b/migration/migration.h index d096b77..22b36f3 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -316,6 +316,7 @@ bool migrate_ignore_shared(void); bool migrate_validate_uuid(void); bool migrate_auto_converge(void); +bool migrate_rdma_pin_all(void); bool migrate_use_multifd(void); bool migrate_pause_before_switchover(void); int migrate_multifd_channels(void); From patchwork Wed Feb 3 08:01:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E381C433E6 for ; Wed, 3 Feb 2021 07:53:31 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EBE4F64E3E for ; Wed, 3 Feb 2021 07:53:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EBE4F64E3E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:55672 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Cyf-0006wF-Ou for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:53:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34296) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpj-00033r-0m for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:15 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3362) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpc-0008Lu-SW for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:14 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr4jk3z163x2; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:37 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 02/18] migration/rdma: judge whether or not the RDMA is used for migration Date: Wed, 3 Feb 2021 16:01:35 +0800 Message-ID: <1612339311-114805-3-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add enabled_rdma_migration into MigrationState to judge whether or not the RDMA is used for migration. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/migration.c | 13 +++++++++++++ migration/migration.h | 6 ++++++ 2 files changed, 19 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 447dfb9..129c81a 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -418,11 +418,13 @@ void migrate_add_address(SocketAddress *address) static void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p = NULL; + MigrationState *s = migrate_get_current(); if (!yank_register_instance(MIGRATION_YANK_INSTANCE, errp)) { return; } + s->enabled_rdma_migration = false; qapi_event_send_migration(MIGRATION_STATUS_SETUP); if (strstart(uri, "tcp:", &p) || strstart(uri, "unix:", NULL) || @@ -430,6 +432,7 @@ static void qemu_start_incoming_migration(const char *uri, Error **errp) socket_start_incoming_migration(p ? p : uri, errp); #ifdef CONFIG_RDMA } else if (strstart(uri, "rdma:", &p)) { + s->enabled_rdma_migration = true; rdma_start_incoming_migration(p, errp); #endif } else if (strstart(uri, "exec:", &p)) { @@ -1921,6 +1924,7 @@ void migrate_init(MigrationState *s) s->start_postcopy = false; s->postcopy_after_devices = false; s->migration_thread_running = false; + s->enabled_rdma_migration = false; error_free(s->error); s->error = NULL; s->hostname = NULL; @@ -2162,6 +2166,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, socket_start_outgoing_migration(s, p ? p : uri, &local_err); #ifdef CONFIG_RDMA } else if (strstart(uri, "rdma:", &p)) { + s->enabled_rdma_migration = true; rdma_start_outgoing_migration(s, p, &local_err); #endif } else if (strstart(uri, "exec:", &p)) { @@ -2391,6 +2396,14 @@ bool migrate_rdma_pin_all(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL]; } +bool migrate_use_rdma(void) +{ + MigrationState *s; + s = migrate_get_current(); + + return s->enabled_rdma_migration; +} + bool migrate_use_multifd(void) { MigrationState *s; diff --git a/migration/migration.h b/migration/migration.h index 22b36f3..da5681b 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -280,6 +280,11 @@ struct MigrationState { * This save hostname when out-going migration starts */ char *hostname; + + /* + * Enable RDMA migration + */ + bool enabled_rdma_migration; }; void migrate_set_state(int *state, int old_state, int new_state); @@ -317,6 +322,7 @@ bool migrate_validate_uuid(void); bool migrate_auto_converge(void); bool migrate_rdma_pin_all(void); +bool migrate_use_rdma(void); bool migrate_use_multifd(void); bool migrate_pause_before_switchover(void); int migrate_multifd_channels(void); From patchwork Wed Feb 3 08:01:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063611 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEBEAC433E0 for ; Wed, 3 Feb 2021 07:48:38 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1099564E43 for ; Wed, 3 Feb 2021 07:48:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1099564E43 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:38564 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Ctw-0008FI-Nr for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:48:36 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34302) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpj-00034J-8K for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:15 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3363) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpd-0008Lt-Hx for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:14 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr4FD4z163wm; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:38 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 03/18] migration/rdma: create multifd_setup_ops for Tx/Rx thread Date: Wed, 3 Feb 2021 16:01:36 +0800 Message-ID: <1612339311-114805-4-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Create multifd_setup_ops for TxRx thread, no logic change. Signed-off-by: Chuan Zheng --- migration/multifd.c | 44 +++++++++++++++++++++++++++++++++++++++----- migration/multifd.h | 7 +++++++ 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 1a1e589..cb1fc01 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -386,6 +386,8 @@ struct { int exiting; /* multifd ops */ MultiFDMethods *ops; + /* multifd setup ops */ + MultiFDSetup *setup_ops; } *multifd_send_state; /* @@ -805,8 +807,9 @@ static bool multifd_channel_connect(MultiFDSendParams *p, } else { /* update for tls qio channel */ p->c = ioc; - qemu_thread_create(&p->thread, p->name, multifd_send_thread, p, - QEMU_THREAD_JOINABLE); + qemu_thread_create(&p->thread, p->name, + multifd_send_state->setup_ops->send_thread, + p, QEMU_THREAD_JOINABLE); } return false; } @@ -854,6 +857,11 @@ cleanup: multifd_new_send_channel_cleanup(p, sioc, local_err); } +static void multifd_send_channel_setup(MultiFDSendParams *p) +{ + socket_send_channel_create(multifd_new_send_channel_async, p); +} + int multifd_save_setup(Error **errp) { int thread_count; @@ -871,6 +879,7 @@ int multifd_save_setup(Error **errp) multifd_send_state->pages = multifd_pages_init(page_count); qemu_sem_init(&multifd_send_state->channels_ready, 0); qatomic_set(&multifd_send_state->exiting, 0); + multifd_send_state->setup_ops = multifd_setup_ops_init(); multifd_send_state->ops = multifd_ops[migrate_multifd_compression()]; for (i = 0; i < thread_count; i++) { @@ -890,7 +899,7 @@ int multifd_save_setup(Error **errp) p->packet->version = cpu_to_be32(MULTIFD_VERSION); p->name = g_strdup_printf("multifdsend_%d", i); p->tls_hostname = g_strdup(s->hostname); - socket_send_channel_create(multifd_new_send_channel_async, p); + multifd_send_state->setup_ops->send_channel_setup(p); } for (i = 0; i < thread_count; i++) { @@ -917,6 +926,8 @@ struct { uint64_t packet_num; /* multifd ops */ MultiFDMethods *ops; + /* multifd setup ops */ + MultiFDSetup *setup_ops; } *multifd_recv_state; static void multifd_recv_terminate_threads(Error *err) @@ -1117,6 +1128,7 @@ int multifd_load_setup(Error **errp) multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count); qatomic_set(&multifd_recv_state->count, 0); qemu_sem_init(&multifd_recv_state->sem_sync, 0); + multifd_recv_state->setup_ops = multifd_setup_ops_init(); multifd_recv_state->ops = multifd_ops[migrate_multifd_compression()]; for (i = 0; i < thread_count; i++) { @@ -1195,9 +1207,31 @@ bool multifd_recv_new_channel(QIOChannel *ioc, Error **errp) p->num_packets = 1; p->running = true; - qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p, - QEMU_THREAD_JOINABLE); + multifd_recv_state->setup_ops->recv_channel_setup(ioc, p); + qemu_thread_create(&p->thread, p->name, + multifd_recv_state->setup_ops->recv_thread, + p, QEMU_THREAD_JOINABLE); qatomic_inc(&multifd_recv_state->count); return qatomic_read(&multifd_recv_state->count) == migrate_multifd_channels(); } + +static void multifd_recv_channel_setup(QIOChannel *ioc, MultiFDRecvParams *p) +{ + return; +} + +static MultiFDSetup multifd_socket_ops = { + .send_thread = multifd_send_thread, + .recv_thread = multifd_recv_thread, + .send_channel_setup = multifd_send_channel_setup, + .recv_channel_setup = multifd_recv_channel_setup +}; + +MultiFDSetup *multifd_setup_ops_init(void) +{ + MultiFDSetup *ops; + + ops = &multifd_socket_ops; + return ops; +} diff --git a/migration/multifd.h b/migration/multifd.h index 8d6751f..1d2dc90 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -166,6 +166,13 @@ typedef struct { int (*recv_pages)(MultiFDRecvParams *p, uint32_t used, Error **errp); } MultiFDMethods; +typedef struct { + void *(*send_thread)(void *opaque); + void *(*recv_thread)(void *opaque); + void (*send_channel_setup)(MultiFDSendParams *p); + void (*recv_channel_setup)(QIOChannel *ioc, MultiFDRecvParams *p); +} MultiFDSetup; + void multifd_register_ops(int method, MultiFDMethods *ops); #endif From patchwork Wed Feb 3 08:01:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 508B7C433E0 for ; Wed, 3 Feb 2021 07:46:21 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D278C64DD9 for ; Wed, 3 Feb 2021 07:46:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D278C64DD9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:58264 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Crj-0004f0-MN for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:46:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34174) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpa-0002xJ-BQ for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:07 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3360) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpO-0008ER-Ef for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:02 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr3n6xz163l7; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:38 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 04/18] migration/rdma: add multifd_setup_ops for rdma Date: Wed, 3 Feb 2021 16:01:37 +0800 Message-ID: <1612339311-114805-5-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Chuan Zheng Reviewed-by: Dr. David Alan Gilbert --- migration/multifd.c | 6 +++++ migration/multifd.h | 5 ++++ migration/rdma.c | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 82 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index cb1fc01..4820702 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -1232,6 +1232,12 @@ MultiFDSetup *multifd_setup_ops_init(void) { MultiFDSetup *ops; +#ifdef CONFIG_RDMA + if (migrate_use_rdma()) { + ops = &multifd_rdma_ops; + return ops; + } +#endif ops = &multifd_socket_ops; return ops; } diff --git a/migration/multifd.h b/migration/multifd.h index 1d2dc90..e3ab4b0 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -173,6 +173,11 @@ typedef struct { void (*recv_channel_setup)(QIOChannel *ioc, MultiFDRecvParams *p); } MultiFDSetup; +#ifdef CONFIG_RDMA +extern MultiFDSetup multifd_rdma_ops; +#endif +MultiFDSetup *multifd_setup_ops_init(void); + void multifd_register_ops(int method, MultiFDMethods *ops); #endif diff --git a/migration/rdma.c b/migration/rdma.c index 00eac34..e0ea86d 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -19,6 +19,7 @@ #include "qemu/cutils.h" #include "rdma.h" #include "migration.h" +#include "multifd.h" #include "qemu-file.h" #include "ram.h" #include "qemu-file-channel.h" @@ -4139,3 +4140,73 @@ err: g_free(rdma); g_free(rdma_return_path); } + +static void *multifd_rdma_send_thread(void *opaque) +{ + MultiFDSendParams *p = opaque; + + while (true) { + WITH_QEMU_LOCK_GUARD(&p->mutex) { + if (p->quit) { + break; + } + } + qemu_sem_wait(&p->sem); + } + + WITH_QEMU_LOCK_GUARD(&p->mutex) { + p->running = false; + } + + return NULL; +} + +static void multifd_rdma_send_channel_setup(MultiFDSendParams *p) +{ + Error *local_err = NULL; + + if (p->quit) { + error_setg(&local_err, "multifd: send id %d already quit", p->id); + return ; + } + p->running = true; + + qemu_thread_create(&p->thread, p->name, multifd_rdma_send_thread, p, + QEMU_THREAD_JOINABLE); +} + +static void *multifd_rdma_recv_thread(void *opaque) +{ + MultiFDRecvParams *p = opaque; + + while (true) { + WITH_QEMU_LOCK_GUARD(&p->mutex) { + if (p->quit) { + break; + } + } + qemu_sem_wait(&p->sem_sync); + } + + WITH_QEMU_LOCK_GUARD(&p->mutex) { + p->running = false; + } + + return NULL; +} + +static void multifd_rdma_recv_channel_setup(QIOChannel *ioc, + MultiFDRecvParams *p) +{ + QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); + + p->file = rioc->file; + return; +} + +MultiFDSetup multifd_rdma_ops = { + .send_thread = multifd_rdma_send_thread, + .recv_thread = multifd_rdma_recv_thread, + .send_channel_setup = multifd_rdma_send_channel_setup, + .recv_channel_setup = multifd_rdma_recv_channel_setup +}; From patchwork Wed Feb 3 08:01:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FB73C433E9 for ; Wed, 3 Feb 2021 07:51:09 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 07A6964E43 for ; Wed, 3 Feb 2021 07:51:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 07A6964E43 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:47086 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7CwN-0003MX-Pe for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:51:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34220) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpe-0002xq-Ag for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:11 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3361) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpW-0008FK-DK for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:10 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr40nsz163wM; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:39 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 05/18] migration/rdma: do not need sync main for rdma Date: Wed, 3 Feb 2021 16:01:38 +0800 Message-ID: <1612339311-114805-6-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/multifd.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index 4820702..5d34950 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -583,6 +583,10 @@ void multifd_send_sync_main(QEMUFile *f) if (!migrate_use_multifd()) { return; } + /* Do not need sync for rdma */ + if (migrate_use_rdma()) { + return; + } if (multifd_send_state->pages->used) { if (multifd_send_pages(f) < 0) { error_report("%s: multifd_send_pages fail", __func__); @@ -1024,6 +1028,10 @@ void multifd_recv_sync_main(void) if (!migrate_use_multifd()) { return; } + /* Do not need sync for rdma */ + if (migrate_use_rdma()) { + return; + } for (i = 0; i < migrate_multifd_channels(); i++) { MultiFDRecvParams *p = &multifd_recv_state->params[i]; From patchwork Wed Feb 3 08:01:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66485C433E9 for ; Wed, 3 Feb 2021 07:48:39 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7072964F5C for ; Wed, 3 Feb 2021 07:48:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7072964F5C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:38616 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Ctx-0008Gi-8Q for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:48:37 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34200) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpd-0002xZ-2S for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:09 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3357) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpO-0008EL-E0 for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:06 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr5Cfgz163x7; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:39 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 06/18] migration/rdma: export MultiFDSendParams/MultiFDRecvParams Date: Wed, 3 Feb 2021 16:01:39 +0800 Message-ID: <1612339311-114805-7-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" MultiFDSendParams and MultiFDRecvParams is need for rdma, export it Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/multifd.c | 26 ++++++++++++++++++++++++++ migration/multifd.h | 2 ++ 2 files changed, 28 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index 5d34950..ae0b7f0 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -390,6 +390,19 @@ struct { MultiFDSetup *setup_ops; } *multifd_send_state; +int get_multifd_send_param(int id, MultiFDSendParams **param) +{ + int ret = 0; + + if (id < 0 || id >= migrate_multifd_channels()) { + ret = -1; + } else { + *param = &(multifd_send_state->params[id]); + } + + return ret; +} + /* * How we use multifd_send_state->pages and channel->pages? * @@ -934,6 +947,19 @@ struct { MultiFDSetup *setup_ops; } *multifd_recv_state; +int get_multifd_recv_param(int id, MultiFDRecvParams **param) +{ + int ret = 0; + + if (id < 0 || id >= migrate_multifd_channels()) { + ret = -1; + } else { + *param = &(multifd_recv_state->params[id]); + } + + return ret; +} + static void multifd_recv_terminate_threads(Error *err) { int i; diff --git a/migration/multifd.h b/migration/multifd.h index e3ab4b0..d57756c 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -176,6 +176,8 @@ typedef struct { #ifdef CONFIG_RDMA extern MultiFDSetup multifd_rdma_ops; #endif +int get_multifd_send_param(int id, MultiFDSendParams **param); +int get_multifd_recv_param(int id, MultiFDRecvParams **param); MultiFDSetup *multifd_setup_ops_init(void); void multifd_register_ops(int method, MultiFDMethods *ops); From patchwork Wed Feb 3 08:01:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59244C433E0 for ; Wed, 3 Feb 2021 07:51:11 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E2A6064E43 for ; Wed, 3 Feb 2021 07:51:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E2A6064E43 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:47352 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7CwP-0003T1-Ss for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:51:09 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34282) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpi-00032n-Ha for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:14 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3359) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpW-0008EO-DN for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:14 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr4yj7z163x5; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:40 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 07/18] migration/rdma: add rdma field into multifd send/recv param Date: Wed, 3 Feb 2021 16:01:40 +0800 Message-ID: <1612339311-114805-8-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Note we do want to export any rdma struct, take void * instead. Signed-off-by: Chuan Zheng --- migration/multifd.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/multifd.h b/migration/multifd.h index d57756c..b17a2c1 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -108,6 +108,10 @@ typedef struct { QemuSemaphore sem_sync; /* used for compression methods */ void *data; + /* used for multifd rdma */ + void *rdma; + /* communication channel */ + QEMUFile *file; } MultiFDSendParams; typedef struct { @@ -147,6 +151,10 @@ typedef struct { QemuSemaphore sem_sync; /* used for de-compression methods */ void *data; + /* used for multifd rdma */ + void *rdma; + /* communication channel */ + QEMUFile *file; } MultiFDRecvParams; typedef struct { From patchwork Wed Feb 3 08:01:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0257C433DB for ; Wed, 3 Feb 2021 07:56:08 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 665F564E3E for ; Wed, 3 Feb 2021 07:56:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 665F564E3E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:34884 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7D1D-0001kn-1r for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:56:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34316) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpj-000361-RC for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:15 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:3364) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cph-0008MD-BI for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:15 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4DVttr5S49z163x8; Wed, 3 Feb 2021 15:42:28 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:41 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 08/18] migration/rdma: export getQIOChannel to get QIOchannel in rdma Date: Wed, 3 Feb 2021 16:01:41 +0800 Message-ID: <1612339311-114805-9-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.190; envelope-from=zhengchuan@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/qemu-file.c | 5 +++++ migration/qemu-file.h | 1 + 2 files changed, 6 insertions(+) diff --git a/migration/qemu-file.c b/migration/qemu-file.c index be21518..37f6201 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -260,6 +260,11 @@ void ram_control_before_iterate(QEMUFile *f, uint64_t flags) } } +void *getQIOChannel(QEMUFile *f) +{ + return f->opaque; +} + void ram_control_after_iterate(QEMUFile *f, uint64_t flags) { int ret = 0; diff --git a/migration/qemu-file.h b/migration/qemu-file.h index a9b6d6c..4cef043 100644 --- a/migration/qemu-file.h +++ b/migration/qemu-file.h @@ -165,6 +165,7 @@ void qemu_file_set_blocking(QEMUFile *f, bool block); void ram_control_before_iterate(QEMUFile *f, uint64_t flags); void ram_control_after_iterate(QEMUFile *f, uint64_t flags); void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data); +void *getQIOChannel(QEMUFile *f); /* Whenever this is found in the data stream, the flags * will be passed to ram_control_load_hook in the incoming-migration From patchwork Wed Feb 3 08:01:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063667 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D28FC433E0 for ; Wed, 3 Feb 2021 07:58:22 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CAE8D64E43 for ; Wed, 3 Feb 2021 07:58:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CAE8D64E43 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:40400 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7D3M-00045U-NV for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:58:20 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34330) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpl-0003BD-JN for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:17 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3476) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpj-0008MY-Px for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:17 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv06V3xzjJgc; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:41 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 09/18] migration/rdma: add multifd_rdma_load_setup() to setup multifd rdma Date: Wed, 3 Feb 2021 16:01:42 +0800 Message-ID: <1612339311-114805-10-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Chuan Zheng Reviewed-by: Dr. David Alan Gilbert --- migration/rdma.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/migration/rdma.c b/migration/rdma.c index e0ea86d..996afb0 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -4011,6 +4011,48 @@ static void rdma_accept_incoming_migration(void *opaque) } } +static bool multifd_rdma_load_setup(const char *host_port, + RDMAContext *rdma, Error **errp) +{ + int thread_count; + int i; + int idx; + MultiFDRecvParams *multifd_recv_param; + RDMAContext *multifd_rdma; + + if (!migrate_use_multifd()) { + return true; + } + + if (multifd_load_setup(errp) != 0) { + /* + * We haven't been able to create multifd threads + * nothing better to do + */ + return false; + } + + thread_count = migrate_multifd_channels(); + for (i = 0; i < thread_count; i++) { + if (get_multifd_recv_param(i, &multifd_recv_param) < 0) { + ERROR(errp, "rdma: error getting multifd_recv_param(%d)", i); + return false; + } + + multifd_rdma = qemu_rdma_data_init(host_port, errp); + for (idx = 0; idx < RDMA_WRID_MAX; idx++) { + multifd_rdma->wr_data[idx].control_len = 0; + multifd_rdma->wr_data[idx].control_curr = NULL; + } + /* the CM channel and CM id is shared */ + multifd_rdma->channel = rdma->channel; + multifd_rdma->listen_id = rdma->listen_id; + multifd_recv_param->rdma = (void *)multifd_rdma; + } + + return true; +} + void rdma_start_incoming_migration(const char *host_port, Error **errp) { int ret; @@ -4058,6 +4100,16 @@ void rdma_start_incoming_migration(const char *host_port, Error **errp) qemu_rdma_return_path_dest_init(rdma_return_path, rdma); } + /* multifd rdma setup */ + if (!multifd_rdma_load_setup(host_port, rdma, &local_err)) { + /* + * We haven't been able to create multifd threads + * nothing better to do + */ + error_report_err(local_err); + goto err; + } + qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migration, NULL, (void *)(intptr_t)rdma); return; From patchwork Wed Feb 3 08:01:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0616DC433E0 for ; Wed, 3 Feb 2021 07:53:31 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 68E2764DE7 for ; Wed, 3 Feb 2021 07:53:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 68E2764DE7 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:55584 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Cyf-0006tz-Br for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:53:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34320) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpk-00037v-EK for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:16 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3477) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpi-0008Ma-0h for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:16 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv06kvFzjJgj; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:42 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 10/18] migration/rdma: Create the multifd recv channels for RDMA Date: Wed, 3 Feb 2021 16:01:43 +0800 Message-ID: <1612339311-114805-11-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" We still don't transmit anything through them, and we only build the RDMA connections. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 67 insertions(+), 2 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index 996afb0..ed8a015 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3267,6 +3267,40 @@ static void rdma_cm_poll_handler(void *opaque) } } +static bool qemu_rdma_accept_setup(RDMAContext *rdma) +{ + RDMAContext *multifd_rdma = NULL; + int thread_count; + int i; + MultiFDRecvParams *multifd_recv_param; + thread_count = migrate_multifd_channels(); + /* create the multifd channels for RDMA */ + for (i = 0; i < thread_count; i++) { + if (get_multifd_recv_param(i, &multifd_recv_param) < 0) { + error_report("rdma: error getting multifd_recv_param(%d)", i); + return false; + } + + multifd_rdma = (RDMAContext *) multifd_recv_param->rdma; + if (multifd_rdma->cm_id == NULL) { + break; + } else { + multifd_rdma = NULL; + } + } + + if (multifd_rdma) { + qemu_set_fd_handler(rdma->channel->fd, + rdma_accept_incoming_migration, + NULL, (void *)(intptr_t)multifd_rdma); + } else { + qemu_set_fd_handler(rdma->channel->fd, rdma_cm_poll_handler, + NULL, rdma); + } + + return true; +} + static int qemu_rdma_accept(RDMAContext *rdma) { RDMACapabilities cap; @@ -3366,6 +3400,10 @@ static int qemu_rdma_accept(RDMAContext *rdma) qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migration, NULL, (void *)(intptr_t)rdma->return_path); + } else if (migrate_use_multifd()) { + if (!qemu_rdma_accept_setup(rdma)) { + goto err_rdma_dest_wait; + } } else { qemu_set_fd_handler(rdma->channel->fd, rdma_cm_poll_handler, NULL, rdma); @@ -3976,6 +4014,34 @@ static QEMUFile *qemu_fopen_rdma(RDMAContext *rdma, const char *mode) return rioc->file; } +static void migration_rdma_process_incoming(QEMUFile *f, + RDMAContext *rdma, Error **errp) +{ + MigrationIncomingState *mis = migration_incoming_get_current(); + QIOChannel *ioc = NULL; + bool start_migration = false; + + if (!migrate_use_multifd()) { + rdma->migration_started_on_destination = 1; + migration_fd_process_incoming(f, errp); + return; + } + + if (!mis->from_src_file) { + mis->from_src_file = f; + qemu_file_set_blocking(f, false); + } else { + ioc = QIO_CHANNEL(getQIOChannel(f)); + /* Multiple connections */ + assert(migrate_use_multifd()); + start_migration = multifd_recv_new_channel(ioc, errp); + } + + if (start_migration) { + migration_incoming_process(); + } +} + static void rdma_accept_incoming_migration(void *opaque) { RDMAContext *rdma = opaque; @@ -4004,8 +4070,7 @@ static void rdma_accept_incoming_migration(void *opaque) return; } - rdma->migration_started_on_destination = 1; - migration_fd_process_incoming(f, &local_err); + migration_rdma_process_incoming(f, rdma, &local_err); if (local_err) { error_reportf_err(local_err, "RDMA ERROR:"); } From patchwork Wed Feb 3 08:01:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53DD5C433DB for ; Wed, 3 Feb 2021 07:51:09 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BD93164E3E for ; Wed, 3 Feb 2021 07:51:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BD93164E3E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:47126 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7CwN-0003NW-KM for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:51:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34264) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cph-000317-Vu for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:14 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3038) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpb-0008J7-HU for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:13 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv05VLBzjJgZ; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:43 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 11/18] migration/rdma: record host_port for multifd RDMA Date: Wed, 3 Feb 2021 16:01:44 +0800 Message-ID: <1612339311-114805-12-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Chuan Zheng --- migration/migration.c | 1 + migration/migration.h | 3 +++ migration/rdma.c | 3 +++ 3 files changed, 7 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 129c81a..b8f4844 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1925,6 +1925,7 @@ void migrate_init(MigrationState *s) s->postcopy_after_devices = false; s->migration_thread_running = false; s->enabled_rdma_migration = false; + s->host_port = NULL; error_free(s->error); s->error = NULL; s->hostname = NULL; diff --git a/migration/migration.h b/migration/migration.h index da5681b..537ee09 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -285,6 +285,9 @@ struct MigrationState { * Enable RDMA migration */ bool enabled_rdma_migration; + + /* Need by Multi-RDMA */ + char *host_port; }; void migrate_set_state(int *state, int old_state, int new_state); diff --git a/migration/rdma.c b/migration/rdma.c index ed8a015..9654b87 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -4206,6 +4206,8 @@ void rdma_start_outgoing_migration(void *opaque, goto err; } + s->host_port = g_strdup(host_port); + ret = qemu_rdma_source_init(rdma, s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL], errp); @@ -4250,6 +4252,7 @@ void rdma_start_outgoing_migration(void *opaque, s->to_dst_file = qemu_fopen_rdma(rdma, "wb"); migrate_fd_connect(s, NULL); + g_free(s->host_port); return; return_path_err: qemu_rdma_cleanup(rdma); From patchwork Wed Feb 3 08:01:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08570C43381 for ; Wed, 3 Feb 2021 07:46:23 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 97A5464DD9 for ; Wed, 3 Feb 2021 07:46:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 97A5464DD9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:58494 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Crl-0004kS-Hu for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:46:21 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34256) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cph-00030Y-M2 for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:13 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3039) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpY-0008J8-4g for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:13 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv05kgfzjJgf; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:43 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 12/18] migration/rdma: Create the multifd send channels for RDMA Date: Wed, 3 Feb 2021 16:01:45 +0800 Message-ID: <1612339311-114805-13-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Chuan Zheng --- migration/multifd.c | 4 ++-- migration/multifd.h | 2 ++ migration/rdma.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 61 insertions(+), 2 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index ae0b7f0..919a414 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -176,7 +176,7 @@ void multifd_register_ops(int method, MultiFDMethods *ops) multifd_ops[method] = ops; } -static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) +int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) { MultiFDInit_t msg = {}; int ret; @@ -503,7 +503,7 @@ int multifd_queue_page(QEMUFile *f, RAMBlock *block, ram_addr_t offset) return 1; } -static void multifd_send_terminate_threads(Error *err) +void multifd_send_terminate_threads(Error *err) { int i; diff --git a/migration/multifd.h b/migration/multifd.h index b17a2c1..26d4489 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -184,6 +184,8 @@ typedef struct { #ifdef CONFIG_RDMA extern MultiFDSetup multifd_rdma_ops; #endif +void multifd_send_terminate_threads(Error *err); +int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp); int get_multifd_send_param(int id, MultiFDSendParams **param); int get_multifd_recv_param(int id, MultiFDRecvParams **param); MultiFDSetup *multifd_setup_ops_init(void); diff --git a/migration/rdma.c b/migration/rdma.c index 9654b87..cff9446 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -4261,9 +4261,54 @@ err: g_free(rdma_return_path); } +static int multifd_channel_rdma_connect(void *opaque) +{ + MultiFDSendParams *p = opaque; + Error *local_err = NULL; + int ret = 0; + MigrationState *s = migrate_get_current(); + + p->rdma = qemu_rdma_data_init(s->host_port, &local_err); + if (p->rdma == NULL) { + goto out; + } + + ret = qemu_rdma_source_init(p->rdma, + migrate_rdma_pin_all(), + &local_err); + if (ret) { + goto out; + } + + ret = qemu_rdma_connect(p->rdma, &local_err); + if (ret) { + goto out; + } + + p->file = qemu_fopen_rdma(p->rdma, "wb"); + if (p->file == NULL) { + goto out; + } + + p->c = QIO_CHANNEL(getQIOChannel(p->file)); + +out: + if (local_err) { + trace_multifd_send_error(p->id); + } + + return ret; +} + static void *multifd_rdma_send_thread(void *opaque) { MultiFDSendParams *p = opaque; + Error *local_err = NULL; + + trace_multifd_send_thread_start(p->id); + if (multifd_send_initial_packet(p, &local_err) < 0) { + goto out; + } while (true) { WITH_QEMU_LOCK_GUARD(&p->mutex) { @@ -4274,6 +4319,12 @@ static void *multifd_rdma_send_thread(void *opaque) qemu_sem_wait(&p->sem); } +out: + if (local_err) { + trace_multifd_send_error(p->id); + multifd_send_terminate_threads(local_err); + } + WITH_QEMU_LOCK_GUARD(&p->mutex) { p->running = false; } @@ -4285,6 +4336,12 @@ static void multifd_rdma_send_channel_setup(MultiFDSendParams *p) { Error *local_err = NULL; + if (multifd_channel_rdma_connect(p)) { + error_setg(&local_err, "multifd: rdma channel %d not established", + p->id); + return ; + } + if (p->quit) { error_setg(&local_err, "multifd: send id %d already quit", p->id); return ; From patchwork Wed Feb 3 08:01:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED82CC43381 for ; Wed, 3 Feb 2021 07:48:40 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7098E64E3E for ; Wed, 3 Feb 2021 07:48:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7098E64E3E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:38854 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Ctz-0008MK-7I for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:48:39 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34266) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpi-00031L-1t for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:14 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3474) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpW-0008J3-Nr for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:13 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv05zzBzjJgg; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:44 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 13/18] migration/rdma: Add the function for dynamic page registration Date: Wed, 3 Feb 2021 16:01:46 +0800 Message-ID: <1612339311-114805-14-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add the 'qemu_rdma_registration' function, multifd send threads call it to register memory. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/migration/rdma.c b/migration/rdma.c index cff9446..1095a8f 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3739,6 +3739,57 @@ out: return ret; } +/* + * Dynamic page registrations for multifd RDMA threads. + */ +static int qemu_rdma_registration(void *opaque) +{ + RDMAContext *rdma = opaque; + RDMAControlHeader resp = {.type = RDMA_CONTROL_RAM_BLOCKS_RESULT }; + RDMALocalBlocks *local = &rdma->local_ram_blocks; + int reg_result_idx, i, nb_dest_blocks; + RDMAControlHeader head = { .len = 0, .repeat = 1 }; + int ret = 0; + + head.type = RDMA_CONTROL_RAM_BLOCKS_REQUEST; + + ret = qemu_rdma_exchange_send(rdma, &head, NULL, &resp, + ®_result_idx, rdma->pin_all ? + qemu_rdma_reg_whole_ram_blocks : NULL); + if (ret < 0) { + goto out; + } + + nb_dest_blocks = resp.len / sizeof(RDMADestBlock); + + if (local->nb_blocks != nb_dest_blocks) { + rdma->error_state = -EINVAL; + ret = -1; + goto out; + } + + qemu_rdma_move_header(rdma, reg_result_idx, &resp); + memcpy(rdma->dest_blocks, + rdma->wr_data[reg_result_idx].control_curr, resp.len); + + for (i = 0; i < nb_dest_blocks; i++) { + network_to_dest_block(&rdma->dest_blocks[i]); + + /* We require that the blocks are in the same order */ + if (rdma->dest_blocks[i].length != local->block[i].length) { + rdma->error_state = -EINVAL; + ret = -1; + goto out; + } + local->block[i].remote_host_addr = + rdma->dest_blocks[i].remote_host_addr; + local->block[i].remote_rkey = rdma->dest_blocks[i].remote_rkey; + } + +out: + return ret; +} + /* Destination: * Called via a ram_control_load_hook during the initial RAM load section which * lists the RAMBlocks by name. This lets us know the order of the RAMBlocks From patchwork Wed Feb 3 08:01:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0C71C433DB for ; Wed, 3 Feb 2021 07:53:30 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3838664E3E for ; Wed, 3 Feb 2021 07:53:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3838664E3E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:55612 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Cyf-0006ue-49 for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:53:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34234) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpf-0002yJ-D1 for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:11 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3475) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpW-0008J4-DN for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:11 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv06DphzjJgh; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:44 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 14/18] migration/rdma: register memory for multifd RDMA channels Date: Wed, 3 Feb 2021 16:01:47 +0800 Message-ID: <1612339311-114805-15-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/multifd.c | 3 ++ migration/rdma.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 93 insertions(+), 2 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 919a414..1186246 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -537,6 +537,9 @@ void multifd_send_terminate_threads(Error *err) qemu_mutex_lock(&p->mutex); p->quit = true; qemu_sem_post(&p->sem); + if (migrate_use_rdma()) { + qemu_sem_post(&p->sem_sync); + } qemu_mutex_unlock(&p->mutex); } } diff --git a/migration/rdma.c b/migration/rdma.c index 1095a8f..c906cc7 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3838,6 +3838,19 @@ static int rdma_load_hook(QEMUFile *f, void *opaque, uint64_t flags, void *data) return rdma_block_notification_handle(opaque, data); case RAM_CONTROL_HOOK: + if (migrate_use_multifd()) { + int i; + MultiFDRecvParams *multifd_recv_param = NULL; + int thread_count = migrate_multifd_channels(); + /* Inform dest recv_thread to poll */ + for (i = 0; i < thread_count; i++) { + if (get_multifd_recv_param(i, &multifd_recv_param)) { + return -1; + } + qemu_sem_post(&multifd_recv_param->sem_sync); + } + } + return qemu_rdma_registration_handle(f, opaque); default: @@ -3910,6 +3923,24 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, head.type = RDMA_CONTROL_RAM_BLOCKS_REQUEST; trace_qemu_rdma_registration_stop_ram(); + if (migrate_use_multifd()) { + /* + * Inform the multifd channels to register memory + */ + int i; + int thread_count = migrate_multifd_channels(); + MultiFDSendParams *multifd_send_param = NULL; + for (i = 0; i < thread_count; i++) { + ret = get_multifd_send_param(i, &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd(%d)", i); + return ret; + } + + qemu_sem_post(&multifd_send_param->sem_sync); + } + } + /* * Make sure that we parallelize the pinning on both sides. * For very large guests, doing this serially takes a really @@ -3968,6 +3999,21 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, rdma->dest_blocks[i].remote_host_addr; local->block[i].remote_rkey = rdma->dest_blocks[i].remote_rkey; } + /* Wait for all multifd channels to complete registration */ + if (migrate_use_multifd()) { + int i; + int thread_count = migrate_multifd_channels(); + MultiFDSendParams *multifd_send_param = NULL; + for (i = 0; i < thread_count; i++) { + ret = get_multifd_send_param(i, &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd(%d)", i); + return ret; + } + + qemu_sem_wait(&multifd_send_param->sem); + } + } } trace_qemu_rdma_registration_stop(flags); @@ -3979,6 +4025,24 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, goto err; } + if (migrate_use_multifd()) { + /* + * Inform src send_thread to send FINISHED signal. + * Wait for multifd RDMA send threads to poll the CQE. + */ + int i; + int thread_count = migrate_multifd_channels(); + MultiFDSendParams *multifd_send_param = NULL; + for (i = 0; i < thread_count; i++) { + ret = get_multifd_send_param(i, &multifd_send_param); + if (ret < 0) { + goto err; + } + + qemu_sem_post(&multifd_send_param->sem_sync); + } + } + return 0; err: rdma->error_state = ret; @@ -4355,19 +4419,37 @@ static void *multifd_rdma_send_thread(void *opaque) { MultiFDSendParams *p = opaque; Error *local_err = NULL; + int ret = 0; + RDMAControlHeader head = { .len = 0, .repeat = 1 }; trace_multifd_send_thread_start(p->id); if (multifd_send_initial_packet(p, &local_err) < 0) { goto out; } + /* wait for semaphore notification to register memory */ + qemu_sem_wait(&p->sem_sync); + if (qemu_rdma_registration(p->rdma) < 0) { + goto out; + } + /* + * Inform the main RDMA thread to run when multifd + * RDMA thread have completed registration. + */ + qemu_sem_post(&p->sem); while (true) { + qemu_sem_wait(&p->sem_sync); WITH_QEMU_LOCK_GUARD(&p->mutex) { if (p->quit) { break; } } - qemu_sem_wait(&p->sem); + /* Send FINISHED to the destination */ + head.type = RDMA_CONTROL_REGISTER_FINISHED; + ret = qemu_rdma_exchange_send(p->rdma, &head, NULL, NULL, NULL, NULL); + if (ret < 0) { + return NULL; + } } out: @@ -4406,14 +4488,20 @@ static void multifd_rdma_send_channel_setup(MultiFDSendParams *p) static void *multifd_rdma_recv_thread(void *opaque) { MultiFDRecvParams *p = opaque; + int ret = 0; while (true) { + qemu_sem_wait(&p->sem_sync); WITH_QEMU_LOCK_GUARD(&p->mutex) { if (p->quit) { break; } } - qemu_sem_wait(&p->sem_sync); + ret = qemu_rdma_registration_handle(p->file, p->c); + if (ret < 0) { + qemu_file_set_error(p->file, ret); + break; + } } WITH_QEMU_LOCK_GUARD(&p->mutex) { From patchwork Wed Feb 3 08:01:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 685B6C433E6 for ; Wed, 3 Feb 2021 07:51:09 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AD95B64F5C for ; Wed, 3 Feb 2021 07:51:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AD95B64F5C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:47094 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7CwN-0003Mh-JA for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:51:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34308) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpj-00034j-AX for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:15 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3037) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpW-0008J2-Dk for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:15 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv05FPBzjJgS; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:45 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 15/18] migration/rdma: only register the memory for multifd channels Date: Wed, 3 Feb 2021 16:01:48 +0800 Message-ID: <1612339311-114805-16-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" All data is sent by multifd Channels, so we only register its for multifd channels and main channel don't register its. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/rdma.c b/migration/rdma.c index c906cc7..f5eb563 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3939,6 +3939,12 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, qemu_sem_post(&multifd_send_param->sem_sync); } + + /* + * Use multifd to migrate, we only register memory for + * multifd RDMA channel and main channel don't register it. + */ + goto wait_reg_complete; } /* @@ -3999,6 +4005,8 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, rdma->dest_blocks[i].remote_host_addr; local->block[i].remote_rkey = rdma->dest_blocks[i].remote_rkey; } + +wait_reg_complete: /* Wait for all multifd channels to complete registration */ if (migrate_use_multifd()) { int i; From patchwork Wed Feb 3 08:01:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E125EC433E6 for ; Wed, 3 Feb 2021 07:56:09 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A01D364F67 for ; Wed, 3 Feb 2021 07:56:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A01D364F67 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:34874 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7D1D-0001kZ-1d for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:56:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34358) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpn-0003FL-6P for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:19 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3478) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpk-0008N7-7a for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:18 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv108PJzjJgW; Wed, 3 Feb 2021 15:42:37 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:45 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 16/18] migration/rdma: add rdma_channel into Migrationstate field Date: Wed, 3 Feb 2021 16:01:49 +0800 Message-ID: <1612339311-114805-17-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Multifd RDMA is need to poll when we send data, record it. Signed-off-by: Chuan Zheng --- migration/migration.c | 1 + migration/migration.h | 1 + migration/rdma.c | 14 ++++++++++++++ 3 files changed, 16 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index b8f4844..47bd11d 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1926,6 +1926,7 @@ void migrate_init(MigrationState *s) s->migration_thread_running = false; s->enabled_rdma_migration = false; s->host_port = NULL; + s->rdma_channel = 0; error_free(s->error); s->error = NULL; s->hostname = NULL; diff --git a/migration/migration.h b/migration/migration.h index 537ee09..5ff46e6 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -288,6 +288,7 @@ struct MigrationState { /* Need by Multi-RDMA */ char *host_port; + int rdma_channel; }; void migrate_set_state(int *state, int old_state, int new_state); diff --git a/migration/rdma.c b/migration/rdma.c index f5eb563..2097839 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -183,6 +183,20 @@ typedef struct { } RDMAWorkRequestData; /* + * Get the multifd RDMA channel used to send data. + */ +static int get_multifd_RDMA_channel(void) +{ + int thread_count = migrate_multifd_channels(); + MigrationState *s = migrate_get_current(); + + s->rdma_channel++; + s->rdma_channel %= thread_count; + + return s->rdma_channel; +} + +/* * Negotiate RDMA capabilities during connection-setup time. */ typedef struct { From patchwork Wed Feb 3 08:01:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39930C433E9 for ; Wed, 3 Feb 2021 07:53:33 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9CB4F64DE7 for ; Wed, 3 Feb 2021 07:53:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9CB4F64DE7 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:55870 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Cyh-00070s-H1 for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:53:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34362) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpn-0003Fr-DY for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:19 -0500 Received: from szxga05-in.huawei.com ([45.249.212.191]:3479) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cpl-0008N6-25 for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:19 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.58]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4DVtv070QGzjJgd; Wed, 3 Feb 2021 15:42:36 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:46 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 17/18] migration/rdma: send data for both rdma-pin-all and NOT rdma-pin-all mode Date: Wed, 3 Feb 2021 16:01:50 +0800 Message-ID: <1612339311-114805-18-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 61 insertions(+), 4 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index 2097839..c19a91f 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2002,6 +2002,20 @@ static int qemu_rdma_write_one(QEMUFile *f, RDMAContext *rdma, .repeat = 1, }; + /* use multifd to send data */ + if (migrate_use_multifd()) { + int channel = get_multifd_RDMA_channel(); + int ret = 0; + MultiFDSendParams *multifd_send_param = NULL; + ret = get_multifd_send_param(channel, &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd_send_param(%d)", channel); + return -EINVAL; + } + rdma = (RDMAContext *)multifd_send_param->rdma; + block = &(rdma->local_ram_blocks.block[current_index]); + } + retry: sge.addr = (uintptr_t)(block->local_host_addr + (current_addr - block->offset)); @@ -2197,6 +2211,27 @@ retry: return 0; } +static int multifd_rdma_write_flush(void) +{ + /* The multifd RDMA threads send data */ + MultiFDSendParams *multifd_send_param = NULL; + RDMAContext *rdma = NULL; + MigrationState *s = migrate_get_current(); + int ret = 0; + + ret = get_multifd_send_param(s->rdma_channel, + &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd_send_param(%d)", + s->rdma_channel); + return ret; + } + rdma = (RDMAContext *)(multifd_send_param->rdma); + rdma->nb_sent++; + + return ret; +} + /* * Push out any unwritten RDMA operations. * @@ -2219,8 +2254,15 @@ static int qemu_rdma_write_flush(QEMUFile *f, RDMAContext *rdma) } if (ret == 0) { - rdma->nb_sent++; - trace_qemu_rdma_write_flush(rdma->nb_sent); + if (migrate_use_multifd()) { + ret = multifd_rdma_write_flush(); + if (ret) { + return ret; + } + } else { + rdma->nb_sent++; + trace_qemu_rdma_write_flush(rdma->nb_sent); + } } rdma->current_length = 0; @@ -4062,6 +4104,7 @@ wait_reg_complete: } qemu_sem_post(&multifd_send_param->sem_sync); + qemu_sem_wait(&multifd_send_param->sem); } } @@ -4443,6 +4486,7 @@ static void *multifd_rdma_send_thread(void *opaque) Error *local_err = NULL; int ret = 0; RDMAControlHeader head = { .len = 0, .repeat = 1 }; + RDMAContext *rdma = p->rdma; trace_multifd_send_thread_start(p->id); if (multifd_send_initial_packet(p, &local_err) < 0) { @@ -4451,7 +4495,7 @@ static void *multifd_rdma_send_thread(void *opaque) /* wait for semaphore notification to register memory */ qemu_sem_wait(&p->sem_sync); - if (qemu_rdma_registration(p->rdma) < 0) { + if (qemu_rdma_registration(rdma) < 0) { goto out; } /* @@ -4466,12 +4510,25 @@ static void *multifd_rdma_send_thread(void *opaque) break; } } + /* To complete polling(CQE) */ + while (rdma->nb_sent) { + ret = qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, NULL); + if (ret < 0) { + error_report("multifd RDMA migration: " + "complete polling error!"); + return NULL; + } + } /* Send FINISHED to the destination */ head.type = RDMA_CONTROL_REGISTER_FINISHED; - ret = qemu_rdma_exchange_send(p->rdma, &head, NULL, NULL, NULL, NULL); + ret = qemu_rdma_exchange_send(rdma, &head, NULL, NULL, NULL, NULL); if (ret < 0) { + error_report("multifd RDMA migration: " + "sending remote error!"); return NULL; } + /* sync main thread */ + qemu_sem_post(&p->sem); } out: From patchwork Wed Feb 3 08:01:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 12063613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 105B8C433DB for ; Wed, 3 Feb 2021 07:48:39 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 770EE64F61 for ; Wed, 3 Feb 2021 07:48:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 770EE64F61 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:38614 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7Ctx-0008Gf-B3 for qemu-devel@archiver.kernel.org; Wed, 03 Feb 2021 02:48:37 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34254) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7Cph-00030Q-IC for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:13 -0500 Received: from szxga06-in.huawei.com ([45.249.212.32]:2605) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7CpZ-0008JH-VU for qemu-devel@nongnu.org; Wed, 03 Feb 2021 02:44:12 -0500 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga06-in.huawei.com (SkyGuard) with ESMTP id 4DVtvK4qkCzjHYD; Wed, 3 Feb 2021 15:42:53 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.498.0; Wed, 3 Feb 2021 15:43:46 +0800 From: Chuan Zheng To: , , Subject: [PATCH v4 18/18] migration/rdma: RDMA cleanup for multifd migration Date: Wed, 3 Feb 2021 16:01:51 +0800 Message-ID: <1612339311-114805-19-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> References: <1612339311-114805-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.32; envelope-from=zhengchuan@huawei.com; helo=szxga06-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Chuan Zheng --- migration/multifd.c | 6 ++++++ migration/multifd.h | 1 + migration/rdma.c | 16 +++++++++++++++- 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/migration/multifd.c b/migration/multifd.c index 1186246..4031648 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -577,6 +577,9 @@ void multifd_save_cleanup(void) p->packet_len = 0; g_free(p->packet); p->packet = NULL; +#ifdef CONFIG_RDMA + multifd_rdma_cleanup(p->rdma); +#endif multifd_send_state->ops->send_cleanup(p, &local_err); if (local_err) { migrate_set_error(migrate_get_current(), local_err); @@ -1039,6 +1042,9 @@ int multifd_load_cleanup(Error **errp) p->packet_len = 0; g_free(p->packet); p->packet = NULL; +#ifdef CONFIG_RDMA + multifd_rdma_cleanup(p->rdma); +#endif multifd_recv_state->ops->recv_cleanup(p); } qemu_sem_destroy(&multifd_recv_state->sem_sync); diff --git a/migration/multifd.h b/migration/multifd.h index 26d4489..0ecec5e 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -183,6 +183,7 @@ typedef struct { #ifdef CONFIG_RDMA extern MultiFDSetup multifd_rdma_ops; +void multifd_rdma_cleanup(void *opaque); #endif void multifd_send_terminate_threads(Error *err); int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp); diff --git a/migration/rdma.c b/migration/rdma.c index c19a91f..f14357f 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2369,7 +2369,7 @@ static void qemu_rdma_cleanup(RDMAContext *rdma) { int idx; - if (rdma->cm_id && rdma->connected) { + if (rdma->channel && rdma->cm_id && rdma->connected) { if ((rdma->error_state || migrate_get_current()->state == MIGRATION_STATUS_CANCELLING) && !rdma->received_error) { @@ -4599,6 +4599,20 @@ static void multifd_rdma_recv_channel_setup(QIOChannel *ioc, return; } +void multifd_rdma_cleanup(void *opaque) +{ + RDMAContext *rdma = (RDMAContext *)opaque; + + if (!migrate_use_rdma()) { + return; + } + + rdma->listen_id = NULL; + rdma->channel = NULL; + qemu_rdma_cleanup(rdma); + g_free(rdma); +} + MultiFDSetup multifd_rdma_ops = { .send_thread = multifd_rdma_send_thread, .recv_thread = multifd_rdma_recv_thread,