From patchwork Sat Oct 17 04:25:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842519 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4F62417CA for ; Sat, 17 Oct 2020 04:11:21 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D81622075E for ; Sat, 17 Oct 2020 04:11:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D81622075E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:58666 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdYt-0007aB-Rd for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:11:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39868) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXd-0005rN-I6 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:01 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5234 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXa-0007BP-ES for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:01 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 89CCADA78D38A7C10B9E; Sat, 17 Oct 2020 12:09:46 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:40 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 01/18] migration/rdma: add the 'migrate_use_rdma_pin_all' function Date: Sat, 17 Oct 2020 12:25:31 +0800 Message-ID: <1602908748-43335-2-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng Reviewed-by: Dr. David Alan Gilbert --- migration/migration.c | 9 +++++++++ migration/migration.h | 1 + 2 files changed, 10 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 0575ecb..64ae417 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2329,6 +2329,15 @@ bool migrate_use_events(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_EVENTS]; } +bool migrate_use_rdma_pin_all(void) +{ + MigrationState *s; + + s = migrate_get_current(); + + return s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL]; +} + bool migrate_use_multifd(void) { MigrationState *s; diff --git a/migration/migration.h b/migration/migration.h index deb411a..74fd790 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -300,6 +300,7 @@ bool migrate_ignore_shared(void); bool migrate_validate_uuid(void); bool migrate_auto_converge(void); +bool migrate_use_rdma_pin_all(void); bool migrate_use_multifd(void); bool migrate_pause_before_switchover(void); int migrate_multifd_channels(void); From patchwork Sat Oct 17 04:25:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842541 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 056D417C9 for ; Sat, 17 Oct 2020 04:18:47 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 978BD20791 for ; Sat, 17 Oct 2020 04:18:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 978BD20791 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:53096 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdg5-0008Tk-Ir for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:18:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39936) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXg-0005td-Ho for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:04 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5239 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXe-0007Gb-DE for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:04 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 7B63CD1119B5D6BEA59B; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:41 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 02/18] migration/rdma: judge whether or not the RDMA is used for migration Date: Sat, 17 Oct 2020 12:25:32 +0800 Message-ID: <1602908748-43335-3-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Add enabled_rdma_migration into MigrationState to judge whether or not the RDMA is used for migration. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/migration.c | 13 +++++++++++++ migration/migration.h | 6 ++++++ 2 files changed, 19 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 64ae417..be6166a 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -389,7 +389,9 @@ void migrate_add_address(SocketAddress *address) void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p = NULL; + MigrationState *s = migrate_get_current(); + s->enabled_rdma_migration = false; qapi_event_send_migration(MIGRATION_STATUS_SETUP); if (!strcmp(uri, "defer")) { deferred_incoming_migration(errp); @@ -399,6 +401,7 @@ void qemu_start_incoming_migration(const char *uri, Error **errp) socket_start_incoming_migration(p ? p : uri, errp); #ifdef CONFIG_RDMA } else if (strstart(uri, "rdma:", &p)) { + s->enabled_rdma_migration = true; rdma_start_incoming_migration(p, errp); #endif } else if (strstart(uri, "exec:", &p)) { @@ -1887,6 +1890,7 @@ void migrate_init(MigrationState *s) s->start_postcopy = false; s->postcopy_after_devices = false; s->migration_thread_running = false; + s->enabled_rdma_migration = false; error_free(s->error); s->error = NULL; s->hostname = NULL; @@ -2115,6 +2119,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, socket_start_outgoing_migration(s, p ? p : uri, &local_err); #ifdef CONFIG_RDMA } else if (strstart(uri, "rdma:", &p)) { + s->enabled_rdma_migration = true; rdma_start_outgoing_migration(s, p, &local_err); #endif } else if (strstart(uri, "exec:", &p)) { @@ -2329,6 +2334,14 @@ bool migrate_use_events(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_EVENTS]; } +bool migrate_use_rdma(void) +{ + MigrationState *s; + s = migrate_get_current(); + + return s->enabled_rdma_migration; +} + bool migrate_use_rdma_pin_all(void) { MigrationState *s; diff --git a/migration/migration.h b/migration/migration.h index 74fd790..e92eb29 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -264,6 +264,11 @@ struct MigrationState * This save hostname when out-going migration starts */ char *hostname; + + /* + * Enable RDMA migration + */ + bool enabled_rdma_migration; }; void migrate_set_state(int *state, int old_state, int new_state); @@ -300,6 +305,7 @@ bool migrate_ignore_shared(void); bool migrate_validate_uuid(void); bool migrate_auto_converge(void); +bool migrate_use_rdma(void); bool migrate_use_rdma_pin_all(void); bool migrate_use_multifd(void); bool migrate_pause_before_switchover(void); From patchwork Sat Oct 17 04:25:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842549 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4E13914B4 for ; Sat, 17 Oct 2020 04:22:53 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 902BC20791 for ; Sat, 17 Oct 2020 04:22:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 902BC20791 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:38000 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdk3-0005b3-FS for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:22:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39968) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXh-0005w6-NP for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:05 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5241 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXe-0007Ge-MH for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:05 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 8168DF0A678DFFE6DC66; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:41 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 03/18] migration/rdma: create multifd_setup_ops for Tx/Rx thread Date: Sat, 17 Oct 2020 12:25:33 +0800 Message-ID: <1602908748-43335-4-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Create multifd_setup_ops for TxRx thread, no logic change. Signed-off-by: Chuan Zheng --- migration/multifd.c | 44 +++++++++++++++++++++++++++++++++++++++----- migration/multifd.h | 7 +++++++ 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 68b171f..1f82307 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -383,6 +383,8 @@ struct { int exiting; /* multifd ops */ MultiFDMethods *ops; + /* multifd setup ops */ + MultiFDSetup *setup_ops; } *multifd_send_state; /* @@ -790,8 +792,9 @@ static bool multifd_channel_connect(MultiFDSendParams *p, } else { /* update for tls qio channel */ p->c = ioc; - qemu_thread_create(&p->thread, p->name, multifd_send_thread, p, - QEMU_THREAD_JOINABLE); + qemu_thread_create(&p->thread, p->name, + multifd_send_state->setup_ops->send_thread_setup, + p, QEMU_THREAD_JOINABLE); } return false; } @@ -839,6 +842,11 @@ cleanup: multifd_new_send_channel_cleanup(p, sioc, local_err); } +static void multifd_send_channel_setup(MultiFDSendParams *p) +{ + socket_send_channel_create(multifd_new_send_channel_async, p); +} + int multifd_save_setup(Error **errp) { int thread_count; @@ -856,6 +864,7 @@ int multifd_save_setup(Error **errp) multifd_send_state->pages = multifd_pages_init(page_count); qemu_sem_init(&multifd_send_state->channels_ready, 0); qatomic_set(&multifd_send_state->exiting, 0); + multifd_send_state->setup_ops = multifd_setup_ops_init(); multifd_send_state->ops = multifd_ops[migrate_multifd_compression()]; for (i = 0; i < thread_count; i++) { @@ -875,7 +884,7 @@ int multifd_save_setup(Error **errp) p->packet->version = cpu_to_be32(MULTIFD_VERSION); p->name = g_strdup_printf("multifdsend_%d", i); p->tls_hostname = g_strdup(s->hostname); - socket_send_channel_create(multifd_new_send_channel_async, p); + multifd_send_state->setup_ops->send_channel_setup(p); } for (i = 0; i < thread_count; i++) { @@ -902,6 +911,8 @@ struct { uint64_t packet_num; /* multifd ops */ MultiFDMethods *ops; + /* multifd setup ops */ + MultiFDSetup *setup_ops; } *multifd_recv_state; static void multifd_recv_terminate_threads(Error *err) @@ -1095,6 +1106,7 @@ int multifd_load_setup(Error **errp) multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count); qatomic_set(&multifd_recv_state->count, 0); qemu_sem_init(&multifd_recv_state->sem_sync, 0); + multifd_recv_state->setup_ops = multifd_setup_ops_init(); multifd_recv_state->ops = multifd_ops[migrate_multifd_compression()]; for (i = 0; i < thread_count; i++) { @@ -1173,9 +1185,31 @@ bool multifd_recv_new_channel(QIOChannel *ioc, Error **errp) p->num_packets = 1; p->running = true; - qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p, - QEMU_THREAD_JOINABLE); + multifd_recv_state->setup_ops->recv_channel_setup(ioc, p); + qemu_thread_create(&p->thread, p->name, + multifd_recv_state->setup_ops->recv_thread_setup, + p, QEMU_THREAD_JOINABLE); qatomic_inc(&multifd_recv_state->count); return qatomic_read(&multifd_recv_state->count) == migrate_multifd_channels(); } + +static void multifd_recv_channel_setup(QIOChannel *ioc, MultiFDRecvParams *p) +{ + return; +} + +static MultiFDSetup multifd_socket_ops = { + .send_thread_setup = multifd_send_thread, + .recv_thread_setup = multifd_recv_thread, + .send_channel_setup = multifd_send_channel_setup, + .recv_channel_setup = multifd_recv_channel_setup +}; + +MultiFDSetup *multifd_setup_ops_init(void) +{ + MultiFDSetup *ops; + + ops = &multifd_socket_ops; + return ops; +} diff --git a/migration/multifd.h b/migration/multifd.h index 8d6751f..446315b 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -166,6 +166,13 @@ typedef struct { int (*recv_pages)(MultiFDRecvParams *p, uint32_t used, Error **errp); } MultiFDMethods; +typedef struct { + void *(*send_thread_setup)(void *opaque); + void *(*recv_thread_setup)(void *opaque); + void (*send_channel_setup)(MultiFDSendParams *p); + void (*recv_channel_setup)(QIOChannel *ioc, MultiFDRecvParams *p); +} MultiFDSetup; + void multifd_register_ops(int method, MultiFDMethods *ops); #endif From patchwork Sat Oct 17 04:25:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842517 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CB83214B5 for ; Sat, 17 Oct 2020 04:11:20 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 640B72075E for ; Sat, 17 Oct 2020 04:11:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 640B72075E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:58598 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdYt-0007YX-2q for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:11:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39842) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXc-0005r7-KO for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:00 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5235 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXa-0007CO-74 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:00 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 673E8B90808B864C7865; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:42 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 04/18] migration/rdma: add multifd_setup_ops for rdma Date: Sat, 17 Oct 2020 12:25:34 +0800 Message-ID: <1602908748-43335-5-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Chuan Zheng --- migration/multifd.c | 6 ++++ migration/multifd.h | 4 +++ migration/rdma.c | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 92 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index 1f82307..0d494df 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -1210,6 +1210,12 @@ MultiFDSetup *multifd_setup_ops_init(void) { MultiFDSetup *ops; +#ifdef CONFIG_RDMA + if (migrate_use_rdma()) { + ops = multifd_rdma_setup(); + return ops; + } +#endif ops = &multifd_socket_ops; return ops; } diff --git a/migration/multifd.h b/migration/multifd.h index 446315b..62a0b2a 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -173,6 +173,10 @@ typedef struct { void (*recv_channel_setup)(QIOChannel *ioc, MultiFDRecvParams *p); } MultiFDSetup; +#ifdef CONFIG_RDMA +MultiFDSetup *multifd_rdma_setup(void); +#endif +MultiFDSetup *multifd_setup_ops_init(void); void multifd_register_ops(int method, MultiFDMethods *ops); #endif diff --git a/migration/rdma.c b/migration/rdma.c index 0340841..ad4e4ba 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -19,6 +19,7 @@ #include "qemu/cutils.h" #include "rdma.h" #include "migration.h" +#include "multifd.h" #include "qemu-file.h" #include "ram.h" #include "qemu-file-channel.h" @@ -4138,3 +4139,84 @@ err: g_free(rdma); g_free(rdma_return_path); } + +static void *multifd_rdma_send_thread(void *opaque) +{ + MultiFDSendParams *p = opaque; + + while (true) { + qemu_mutex_lock(&p->mutex); + if (p->quit) { + qemu_mutex_unlock(&p->mutex); + break; + } + qemu_mutex_unlock(&p->mutex); + qemu_sem_wait(&p->sem); + } + + qemu_mutex_lock(&p->mutex); + p->running = false; + qemu_mutex_unlock(&p->mutex); + + return NULL; +} + +static void multifd_rdma_send_channel_setup(MultiFDSendParams *p) +{ + Error *local_err = NULL; + + if (p->quit) { + error_setg(&local_err, "multifd: send id %d already quit", p->id); + return ; + } + p->running = true; + + qemu_thread_create(&p->thread, p->name, multifd_rdma_send_thread, p, + QEMU_THREAD_JOINABLE); +} + +static void *multifd_rdma_recv_thread(void *opaque) +{ + MultiFDRecvParams *p = opaque; + + while (true) { + qemu_mutex_lock(&p->mutex); + if (p->quit) { + qemu_mutex_unlock(&p->mutex); + break; + } + qemu_mutex_unlock(&p->mutex); + qemu_sem_wait(&p->sem_sync); + } + + qemu_mutex_lock(&p->mutex); + p->running = false; + qemu_mutex_unlock(&p->mutex); + + return NULL; +} + +static void multifd_rdma_recv_channel_setup(QIOChannel *ioc, + MultiFDRecvParams *p) +{ + QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); + + p->file = rioc->file; + return; +} + +static MultiFDSetup multifd_rdma_ops = { + .send_thread_setup = multifd_rdma_send_thread, + .recv_thread_setup = multifd_rdma_recv_thread, + .send_channel_setup = multifd_rdma_send_channel_setup, + .recv_channel_setup = multifd_rdma_recv_channel_setup +}; + +MultiFDSetup *multifd_rdma_setup(void) +{ + MultiFDSetup *rdma_ops; + + rdma_ops = &multifd_rdma_ops; + + return rdma_ops; +} From patchwork Sat Oct 17 04:25:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842547 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 50B8517C9 for ; Sat, 17 Oct 2020 04:20:29 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 13EF020791 for ; Sat, 17 Oct 2020 04:20:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 13EF020791 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:59606 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdhj-0002jz-SV for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:20:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39952) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXg-0005uH-TT for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:04 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5240 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXe-0007GZ-Cu for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:04 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 77AEC1621E9FF7088B2F; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:43 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 05/18] migration/rdma: do not need sync main for rdma Date: Sat, 17 Oct 2020 12:25:35 +0800 Message-ID: <1602908748-43335-6-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/multifd.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index 0d494df..8ccfd46 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -580,6 +580,10 @@ void multifd_send_sync_main(QEMUFile *f) if (!migrate_use_multifd()) { return; } + /* Do not need sync for rdma */ + if (migrate_use_rdma()) { + return; + } if (multifd_send_state->pages->used) { if (multifd_send_pages(f) < 0) { error_report("%s: multifd_send_pages fail", __func__); @@ -1002,6 +1006,10 @@ void multifd_recv_sync_main(void) if (!migrate_use_multifd()) { return; } + /* Do not need sync for rdma */ + if (migrate_use_rdma()) { + return; + } for (i = 0; i < migrate_multifd_channels(); i++) { MultiFDRecvParams *p = &multifd_recv_state->params[i]; From patchwork Sat Oct 17 04:25:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842523 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DF51E14B5 for ; Sat, 17 Oct 2020 04:11:41 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 69D3B2075E for ; Sat, 17 Oct 2020 04:11:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 69D3B2075E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:59200 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdZE-0007nA-DW for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:11:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39940) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXg-0005tn-Lx for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:04 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5242 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXe-0007Gf-M6 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:04 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 8606FA20CCE762B3E224; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:43 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 06/18] migration/rdma: export MultiFDSendParams/MultiFDRecvParams Date: Sat, 17 Oct 2020 12:25:36 +0800 Message-ID: <1602908748-43335-7-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" MultiFDSendParams and MultiFDRecvParams is need for rdma, export it Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/multifd.c | 26 ++++++++++++++++++++++++++ migration/multifd.h | 2 ++ 2 files changed, 28 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index 8ccfd46..03f3a1e 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -387,6 +387,19 @@ struct { MultiFDSetup *setup_ops; } *multifd_send_state; +int get_multifd_send_param(int id, MultiFDSendParams **param) +{ + int ret = 0; + + if (id < 0 || id >= migrate_multifd_channels()) { + ret = -1; + } else { + *param = &(multifd_send_state->params[id]); + } + + return ret; +} + /* * How we use multifd_send_state->pages and channel->pages? * @@ -919,6 +932,19 @@ struct { MultiFDSetup *setup_ops; } *multifd_recv_state; +int get_multifd_recv_param(int id, MultiFDRecvParams **param) +{ + int ret = 0; + + if (id < 0 || id >= migrate_multifd_channels()) { + ret = -1; + } else { + *param = &(multifd_recv_state->params[id]); + } + + return ret; +} + static void multifd_recv_terminate_threads(Error *err) { int i; diff --git a/migration/multifd.h b/migration/multifd.h index 62a0b2a..2f4e585 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -176,6 +176,8 @@ typedef struct { #ifdef CONFIG_RDMA MultiFDSetup *multifd_rdma_setup(void); #endif +int get_multifd_send_param(int id, MultiFDSendParams **param); +int get_multifd_recv_param(int id, MultiFDRecvParams **param); MultiFDSetup *multifd_setup_ops_init(void); void multifd_register_ops(int method, MultiFDMethods *ops); From patchwork Sat Oct 17 04:25:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842521 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0B1EB14B5 for ; Sat, 17 Oct 2020 04:11:24 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 862982075E for ; Sat, 17 Oct 2020 04:11:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 862982075E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:58856 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdYw-0007em-Ig for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:11:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39838) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXc-0005r6-K3 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:00 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5236 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXa-0007CP-EO for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:00 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 6C1B6CA8D488661B70AD; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:44 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 07/18] migration/rdma: add rdma field into multifd send/recv param Date: Sat, 17 Oct 2020 12:25:37 +0800 Message-ID: <1602908748-43335-8-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Note we do want to export any rdma struct, take void * instead. Signed-off-by: Chuan Zheng --- migration/multifd.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/multifd.h b/migration/multifd.h index 2f4e585..ff80bd5 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -108,6 +108,10 @@ typedef struct { QemuSemaphore sem_sync; /* used for compression methods */ void *data; + /* used for multifd rdma */ + void *rdma; + /* communication channel */ + QEMUFile *file; } MultiFDSendParams; typedef struct { @@ -147,6 +151,10 @@ typedef struct { QemuSemaphore sem_sync; /* used for de-compression methods */ void *data; + /* used for multifd rdma */ + void *rdma; + /* communication channel */ + QEMUFile *file; } MultiFDRecvParams; typedef struct { From patchwork Sat Oct 17 04:25:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842525 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2C77B14B5 for ; Sat, 17 Oct 2020 04:13:35 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E2C742076C for ; Sat, 17 Oct 2020 04:13:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E2C742076C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:38814 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdb3-0002Xz-V0 for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:13:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39904) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXf-0005rb-5P for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:03 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5237 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXd-0007CT-Dt for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:02 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 6FBC4D0D39749B1F3E5F; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:45 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 08/18] migration/rdma: export getQIOChannel to get QIOchannel in rdma Date: Sat, 17 Oct 2020 12:25:38 +0800 Message-ID: <1602908748-43335-9-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/qemu-file.c | 5 +++++ migration/qemu-file.h | 1 + 2 files changed, 6 insertions(+) diff --git a/migration/qemu-file.c b/migration/qemu-file.c index be21518..37f6201 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -260,6 +260,11 @@ void ram_control_before_iterate(QEMUFile *f, uint64_t flags) } } +void *getQIOChannel(QEMUFile *f) +{ + return f->opaque; +} + void ram_control_after_iterate(QEMUFile *f, uint64_t flags) { int ret = 0; diff --git a/migration/qemu-file.h b/migration/qemu-file.h index a9b6d6c..4cef043 100644 --- a/migration/qemu-file.h +++ b/migration/qemu-file.h @@ -165,6 +165,7 @@ void qemu_file_set_blocking(QEMUFile *f, bool block); void ram_control_before_iterate(QEMUFile *f, uint64_t flags); void ram_control_after_iterate(QEMUFile *f, uint64_t flags); void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data); +void *getQIOChannel(QEMUFile *f); /* Whenever this is found in the data stream, the flags * will be passed to ram_control_load_hook in the incoming-migration From patchwork Sat Oct 17 04:25:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842535 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 196D314B4 for ; Sat, 17 Oct 2020 04:16:27 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DAE8420791 for ; Sat, 17 Oct 2020 04:16:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DAE8420791 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:46768 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTddp-0005tu-SO for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:16:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39932) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXg-0005t0-4T for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:04 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5238 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXe-0007CU-9D for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:03 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 7393A80C7702E1FEBCA2; Sat, 17 Oct 2020 12:09:51 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:45 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 09/18] migration/rdma: add multifd_rdma_load_setup() to setup multifd rdma Date: Sat, 17 Oct 2020 12:25:39 +0800 Message-ID: <1602908748-43335-10-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Chuan Zheng Reviewed-by: Dr. David Alan Gilbert --- migration/rdma.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/migration/rdma.c b/migration/rdma.c index ad4e4ba..2baa933 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -4010,6 +4010,48 @@ static void rdma_accept_incoming_migration(void *opaque) } } +static bool multifd_rdma_load_setup(const char *host_port, + RDMAContext *rdma, Error **errp) +{ + int thread_count; + int i; + int idx; + MultiFDRecvParams *multifd_recv_param; + RDMAContext *multifd_rdma; + + if (!migrate_use_multifd()) { + return true; + } + + if (multifd_load_setup(errp) != 0) { + /* + * We haven't been able to create multifd threads + * nothing better to do + */ + return false; + } + + thread_count = migrate_multifd_channels(); + for (i = 0; i < thread_count; i++) { + if (get_multifd_recv_param(i, &multifd_recv_param) < 0) { + ERROR(errp, "rdma: error getting multifd_recv_param(%d)", i); + return false; + } + + multifd_rdma = qemu_rdma_data_init(host_port, errp); + for (idx = 0; idx < RDMA_WRID_MAX; idx++) { + multifd_rdma->wr_data[idx].control_len = 0; + multifd_rdma->wr_data[idx].control_curr = NULL; + } + /* the CM channel and CM id is shared */ + multifd_rdma->channel = rdma->channel; + multifd_rdma->listen_id = rdma->listen_id; + multifd_recv_param->rdma = (void *)multifd_rdma; + } + + return true; +} + void rdma_start_incoming_migration(const char *host_port, Error **errp) { int ret; @@ -4057,6 +4099,16 @@ void rdma_start_incoming_migration(const char *host_port, Error **errp) qemu_rdma_return_path_dest_init(rdma_return_path, rdma); } + /* multifd rdma setup */ + if (!multifd_rdma_load_setup(host_port, rdma, &local_err)) { + /* + * We haven't been able to create multifd threads + * nothing better to do + */ + error_report_err(local_err); + goto err; + } + qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migration, NULL, (void *)(intptr_t)rdma); return; From patchwork Sat Oct 17 04:25:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842537 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1ABCB17C9 for ; Sat, 17 Oct 2020 04:16:28 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B22E520791 for ; Sat, 17 Oct 2020 04:16:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B22E520791 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:46838 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTddq-0005vX-Ou for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:16:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40074) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXu-0006Gg-8P for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:19 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5248 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXs-0007Iy-4e for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:17 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 87BF5CC55B61983A2968; Sat, 17 Oct 2020 12:09:56 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:46 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 10/18] migration/rdma: Create the multifd recv channels for RDMA Date: Sat, 17 Oct 2020 12:25:40 +0800 Message-ID: <1602908748-43335-11-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" We still don't transmit anything through them, and we only build the RDMA connections. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 68 insertions(+), 2 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index 2baa933..63559f1 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3266,6 +3266,40 @@ static void rdma_cm_poll_handler(void *opaque) } } +static bool qemu_rdma_accept_setup(RDMAContext *rdma) +{ + RDMAContext *multifd_rdma = NULL; + int thread_count; + int i; + MultiFDRecvParams *multifd_recv_param; + thread_count = migrate_multifd_channels(); + /* create the multifd channels for RDMA */ + for (i = 0; i < thread_count; i++) { + if (get_multifd_recv_param(i, &multifd_recv_param) < 0) { + error_report("rdma: error getting multifd_recv_param(%d)", i); + return false; + } + + multifd_rdma = (RDMAContext *) multifd_recv_param->rdma; + if (multifd_rdma->cm_id == NULL) { + break; + } else { + multifd_rdma = NULL; + } + } + + if (multifd_rdma) { + qemu_set_fd_handler(rdma->channel->fd, + rdma_accept_incoming_migration, + NULL, (void *)(intptr_t)multifd_rdma); + } else { + qemu_set_fd_handler(rdma->channel->fd, rdma_cm_poll_handler, + NULL, rdma); + } + + return true; +} + static int qemu_rdma_accept(RDMAContext *rdma) { RDMACapabilities cap; @@ -3365,6 +3399,10 @@ static int qemu_rdma_accept(RDMAContext *rdma) qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migration, NULL, (void *)(intptr_t)rdma->return_path); + } else if (migrate_use_multifd()) { + if (!qemu_rdma_accept_setup(rdma)) { + goto err_rdma_dest_wait; + } } else { qemu_set_fd_handler(rdma->channel->fd, rdma_cm_poll_handler, NULL, rdma); @@ -3975,6 +4013,35 @@ static QEMUFile *qemu_fopen_rdma(RDMAContext *rdma, const char *mode) return rioc->file; } +static void migration_rdma_process_incoming(QEMUFile *f, + RDMAContext *rdma, Error **errp) +{ + MigrationIncomingState *mis = migration_incoming_get_current(); + QIOChannel *ioc = NULL; + bool start_migration = false; + + /* FIXME: Need refactor */ + if (!migrate_use_multifd()) { + rdma->migration_started_on_destination = 1; + migration_fd_process_incoming(f, errp); + return; + } + + if (!mis->from_src_file) { + mis->from_src_file = f; + qemu_file_set_blocking(f, false); + } else { + ioc = QIO_CHANNEL(getQIOChannel(f)); + /* Multiple connections */ + assert(migrate_use_multifd()); + start_migration = multifd_recv_new_channel(ioc, errp); + } + + if (start_migration) { + migration_incoming_process(); + } +} + static void rdma_accept_incoming_migration(void *opaque) { RDMAContext *rdma = opaque; @@ -4003,8 +4070,7 @@ static void rdma_accept_incoming_migration(void *opaque) return; } - rdma->migration_started_on_destination = 1; - migration_fd_process_incoming(f, &local_err); + migration_rdma_process_incoming(f, rdma, &local_err); if (local_err) { error_reportf_err(local_err, "RDMA ERROR:"); } From patchwork Sat Oct 17 04:25:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842545 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8B1D517C9 for ; Sat, 17 Oct 2020 04:20:26 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5484120791 for ; Sat, 17 Oct 2020 04:20:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5484120791 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:59436 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdhh-0002fj-Ce for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:20:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40104) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXy-0006JF-D5 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:23 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5249 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXt-0007Jg-Ny for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:22 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 900DEFD1854B5AB8137D; Sat, 17 Oct 2020 12:09:56 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:47 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 11/18] migration/rdma: record host_port for multifd RDMA Date: Sat, 17 Oct 2020 12:25:41 +0800 Message-ID: <1602908748-43335-12-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Chuan Zheng --- migration/migration.c | 1 + migration/migration.h | 3 +++ migration/rdma.c | 3 +++ 3 files changed, 7 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index be6166a..7061410 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1891,6 +1891,7 @@ void migrate_init(MigrationState *s) s->postcopy_after_devices = false; s->migration_thread_running = false; s->enabled_rdma_migration = false; + s->host_port = NULL; error_free(s->error); s->error = NULL; s->hostname = NULL; diff --git a/migration/migration.h b/migration/migration.h index e92eb29..fea63de 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -269,6 +269,9 @@ struct MigrationState * Enable RDMA migration */ bool enabled_rdma_migration; + + /* Need by Multi-RDMA */ + char *host_port; }; void migrate_set_state(int *state, int old_state, int new_state); diff --git a/migration/rdma.c b/migration/rdma.c index 63559f1..dd9f705 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -4206,6 +4206,8 @@ void rdma_start_outgoing_migration(void *opaque, goto err; } + s->host_port = g_strdup(host_port); + ret = qemu_rdma_source_init(rdma, s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL], errp); @@ -4250,6 +4252,7 @@ void rdma_start_outgoing_migration(void *opaque, s->to_dst_file = qemu_fopen_rdma(rdma, "wb"); migrate_fd_connect(s, NULL); + g_free(s->host_port); return; return_path_err: qemu_rdma_cleanup(rdma); From patchwork Sat Oct 17 04:25:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842543 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CF99E17C9 for ; Sat, 17 Oct 2020 04:18:47 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 86DA520791 for ; Sat, 17 Oct 2020 04:18:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 86DA520791 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:53182 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdg6-00005P-E2 for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:18:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40100) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXv-0006HD-J5 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:19 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5245 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXr-0007Ij-5e for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:19 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 7F60251D1C4A51D745D8; Sat, 17 Oct 2020 12:09:56 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:47 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 12/18] migration/rdma: Create the multifd send channels for RDMA Date: Sat, 17 Oct 2020 12:25:42 +0800 Message-ID: <1602908748-43335-13-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Chuan Zheng --- migration/multifd.c | 4 ++-- migration/multifd.h | 2 ++ migration/rdma.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 60 insertions(+), 2 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 03f3a1e..9439b3c 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -173,7 +173,7 @@ void multifd_register_ops(int method, MultiFDMethods *ops) multifd_ops[method] = ops; } -static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) +int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) { MultiFDInit_t msg = {}; int ret; @@ -500,7 +500,7 @@ int multifd_queue_page(QEMUFile *f, RAMBlock *block, ram_addr_t offset) return 1; } -static void multifd_send_terminate_threads(Error *err) +void multifd_send_terminate_threads(Error *err) { int i; diff --git a/migration/multifd.h b/migration/multifd.h index ff80bd5..ec9e897 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -184,6 +184,8 @@ typedef struct { #ifdef CONFIG_RDMA MultiFDSetup *multifd_rdma_setup(void); #endif +void multifd_send_terminate_threads(Error *err); +int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp); int get_multifd_send_param(int id, MultiFDSendParams **param); int get_multifd_recv_param(int id, MultiFDRecvParams **param); MultiFDSetup *multifd_setup_ops_init(void); diff --git a/migration/rdma.c b/migration/rdma.c index dd9f705..1af81f5 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -4261,9 +4261,54 @@ err: g_free(rdma_return_path); } +static int multifd_channel_rdma_connect(void *opaque) +{ + MultiFDSendParams *p = opaque; + Error *local_err = NULL; + int ret = 0; + MigrationState *s = migrate_get_current(); + + p->rdma = qemu_rdma_data_init(s->host_port, &local_err); + if (p->rdma == NULL) { + goto out; + } + + ret = qemu_rdma_source_init(p->rdma, + migrate_use_rdma_pin_all(), + &local_err); + if (ret) { + goto out; + } + + ret = qemu_rdma_connect(p->rdma, &local_err); + if (ret) { + goto out; + } + + p->file = qemu_fopen_rdma(p->rdma, "wb"); + if (p->file == NULL) { + goto out; + } + + p->c = QIO_CHANNEL(getQIOChannel(p->file)); + +out: + if (local_err) { + trace_multifd_send_error(p->id); + } + + return ret; +} + static void *multifd_rdma_send_thread(void *opaque) { MultiFDSendParams *p = opaque; + Error *local_err = NULL; + + trace_multifd_send_thread_start(p->id); + if (multifd_send_initial_packet(p, &local_err) < 0) { + goto out; + } while (true) { qemu_mutex_lock(&p->mutex); @@ -4275,6 +4320,11 @@ static void *multifd_rdma_send_thread(void *opaque) qemu_sem_wait(&p->sem); } +out: + if (local_err) { + trace_multifd_send_error(p->id); + multifd_send_terminate_threads(local_err); + } qemu_mutex_lock(&p->mutex); p->running = false; qemu_mutex_unlock(&p->mutex); @@ -4286,6 +4336,12 @@ static void multifd_rdma_send_channel_setup(MultiFDSendParams *p) { Error *local_err = NULL; + if (multifd_channel_rdma_connect(p)) { + error_setg(&local_err, "multifd: rdma channel %d not established", + p->id); + return ; + } + if (p->quit) { error_setg(&local_err, "multifd: send id %d already quit", p->id); return ; From patchwork Sat Oct 17 04:25:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842539 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 19D8114B4 for ; Sat, 17 Oct 2020 04:18:45 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CF46020791 for ; Sat, 17 Oct 2020 04:18:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CF46020791 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:52960 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdg3-0008QR-VR for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:18:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40018) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXr-0006Db-LD for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:15 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5243 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXp-0007Hx-JL for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:15 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 7B2445C3648B4EC72B5E; Sat, 17 Oct 2020 12:09:56 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:48 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 13/18] migration/rdma: Add the function for dynamic page registration Date: Sat, 17 Oct 2020 12:25:43 +0800 Message-ID: <1602908748-43335-14-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Add the 'qemu_rdma_registration' function, multifd send threads call it to register memory. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/migration/rdma.c b/migration/rdma.c index 1af81f5..a366849 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3738,6 +3738,57 @@ out: return ret; } +/* + * Dynamic page registrations for multifd RDMA threads. + */ +static int qemu_rdma_registration(void *opaque) +{ + RDMAContext *rdma = opaque; + RDMAControlHeader resp = {.type = RDMA_CONTROL_RAM_BLOCKS_RESULT }; + RDMALocalBlocks *local = &rdma->local_ram_blocks; + int reg_result_idx, i, nb_dest_blocks; + RDMAControlHeader head = { .len = 0, .repeat = 1 }; + int ret = 0; + + head.type = RDMA_CONTROL_RAM_BLOCKS_REQUEST; + + ret = qemu_rdma_exchange_send(rdma, &head, NULL, &resp, + ®_result_idx, rdma->pin_all ? + qemu_rdma_reg_whole_ram_blocks : NULL); + if (ret < 0) { + goto out; + } + + nb_dest_blocks = resp.len / sizeof(RDMADestBlock); + + if (local->nb_blocks != nb_dest_blocks) { + rdma->error_state = -EINVAL; + ret = -1; + goto out; + } + + qemu_rdma_move_header(rdma, reg_result_idx, &resp); + memcpy(rdma->dest_blocks, + rdma->wr_data[reg_result_idx].control_curr, resp.len); + + for (i = 0; i < nb_dest_blocks; i++) { + network_to_dest_block(&rdma->dest_blocks[i]); + + /* We require that the blocks are in the same order */ + if (rdma->dest_blocks[i].length != local->block[i].length) { + rdma->error_state = -EINVAL; + ret = -1; + goto out; + } + local->block[i].remote_host_addr = + rdma->dest_blocks[i].remote_host_addr; + local->block[i].remote_rkey = rdma->dest_blocks[i].remote_rkey; + } + +out: + return ret; +} + /* Destination: * Called via a ram_control_load_hook during the initial RAM load section which * lists the RAMBlocks by name. This lets us know the order of the RAMBlocks From patchwork Sat Oct 17 04:25:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842531 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4B2F414B5 for ; Sat, 17 Oct 2020 04:13:40 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 197372076C for ; Sat, 17 Oct 2020 04:13:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 197372076C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:39374 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdb9-0002lY-4u for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:13:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39978) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXh-0005wQ-Sr for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:05 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5244 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXf-0007Hy-I0 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:05 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 7619A73645F7C5433B5A; Sat, 17 Oct 2020 12:09:56 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:49 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 14/18] migration/rdma: register memory for multifd RDMA channels Date: Sat, 17 Oct 2020 12:25:44 +0800 Message-ID: <1602908748-43335-15-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/multifd.c | 3 ++ migration/rdma.c | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 95 insertions(+), 2 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 9439b3c..c4d90ef 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -534,6 +534,9 @@ void multifd_send_terminate_threads(Error *err) qemu_mutex_lock(&p->mutex); p->quit = true; qemu_sem_post(&p->sem); + if (migrate_use_rdma()) { + qemu_sem_post(&p->sem_sync); + } qemu_mutex_unlock(&p->mutex); } } diff --git a/migration/rdma.c b/migration/rdma.c index a366849..3210e6e 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3837,6 +3837,19 @@ static int rdma_load_hook(QEMUFile *f, void *opaque, uint64_t flags, void *data) return rdma_block_notification_handle(opaque, data); case RAM_CONTROL_HOOK: + if (migrate_use_multifd()) { + int i; + MultiFDRecvParams *multifd_recv_param = NULL; + int thread_count = migrate_multifd_channels(); + /* Inform dest recv_thread to poll */ + for (i = 0; i < thread_count; i++) { + if (get_multifd_recv_param(i, &multifd_recv_param)) { + return -1; + } + qemu_sem_post(&multifd_recv_param->sem_sync); + } + } + return qemu_rdma_registration_handle(f, opaque); default: @@ -3909,6 +3922,24 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, head.type = RDMA_CONTROL_RAM_BLOCKS_REQUEST; trace_qemu_rdma_registration_stop_ram(); + if (migrate_use_multifd()) { + /* + * Inform the multifd channels to register memory + */ + int i; + int thread_count = migrate_multifd_channels(); + MultiFDSendParams *multifd_send_param = NULL; + for (i = 0; i < thread_count; i++) { + ret = get_multifd_send_param(i, &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd(%d)", i); + return ret; + } + + qemu_sem_post(&multifd_send_param->sem_sync); + } + } + /* * Make sure that we parallelize the pinning on both sides. * For very large guests, doing this serially takes a really @@ -3967,6 +3998,21 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, rdma->dest_blocks[i].remote_host_addr; local->block[i].remote_rkey = rdma->dest_blocks[i].remote_rkey; } + /* Wait for all multifd channels to complete registration */ + if (migrate_use_multifd()) { + int i; + int thread_count = migrate_multifd_channels(); + MultiFDSendParams *multifd_send_param = NULL; + for (i = 0; i < thread_count; i++) { + ret = get_multifd_send_param(i, &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd(%d)", i); + return ret; + } + + qemu_sem_wait(&multifd_send_param->sem); + } + } } trace_qemu_rdma_registration_stop(flags); @@ -3978,6 +4024,24 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, goto err; } + if (migrate_use_multifd()) { + /* + * Inform src send_thread to send FINISHED signal. + * Wait for multifd RDMA send threads to poll the CQE. + */ + int i; + int thread_count = migrate_multifd_channels(); + MultiFDSendParams *multifd_send_param = NULL; + for (i = 0; i < thread_count; i++) { + ret = get_multifd_send_param(i, &multifd_send_param); + if (ret < 0) { + goto err; + } + + qemu_sem_post(&multifd_send_param->sem_sync); + } + } + return 0; err: rdma->error_state = ret; @@ -4355,20 +4419,39 @@ static void *multifd_rdma_send_thread(void *opaque) { MultiFDSendParams *p = opaque; Error *local_err = NULL; + int ret = 0; + RDMAControlHeader head = { .len = 0, .repeat = 1 }; trace_multifd_send_thread_start(p->id); if (multifd_send_initial_packet(p, &local_err) < 0) { goto out; } + /* wait for semaphore notification to register memory */ + qemu_sem_wait(&p->sem_sync); + if (qemu_rdma_registration(p->rdma) < 0) { + goto out; + } + /* + * Inform the main RDMA thread to run when multifd + * RDMA thread have completed registration. + */ + qemu_sem_post(&p->sem); while (true) { + qemu_sem_wait(&p->sem_sync); qemu_mutex_lock(&p->mutex); if (p->quit) { qemu_mutex_unlock(&p->mutex); break; } qemu_mutex_unlock(&p->mutex); - qemu_sem_wait(&p->sem); + + /* Send FINISHED to the destination */ + head.type = RDMA_CONTROL_REGISTER_FINISHED; + ret = qemu_rdma_exchange_send(p->rdma, &head, NULL, NULL, NULL, NULL); + if (ret < 0) { + return NULL; + } } out: @@ -4406,15 +4489,22 @@ static void multifd_rdma_send_channel_setup(MultiFDSendParams *p) static void *multifd_rdma_recv_thread(void *opaque) { MultiFDRecvParams *p = opaque; + int ret = 0; while (true) { + qemu_sem_wait(&p->sem_sync); + qemu_mutex_lock(&p->mutex); if (p->quit) { qemu_mutex_unlock(&p->mutex); break; } qemu_mutex_unlock(&p->mutex); - qemu_sem_wait(&p->sem_sync); + ret = qemu_rdma_registration_handle(p->file, p->c); + if (ret < 0) { + qemu_file_set_error(p->file, ret); + break; + } } qemu_mutex_lock(&p->mutex); From patchwork Sat Oct 17 04:25:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842529 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6624614B5 for ; Sat, 17 Oct 2020 04:13:37 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3199F2076C for ; Sat, 17 Oct 2020 04:13:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3199F2076C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:39060 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdb6-0002ds-9q for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:13:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40070) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXu-0006GX-3q for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:19 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5246 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXr-0007Ig-5U for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:17 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 83D996FE1DD8D46EF0EE; Sat, 17 Oct 2020 12:09:56 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:50 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 15/18] migration/rdma: only register the memory for multifd channels Date: Sat, 17 Oct 2020 12:25:45 +0800 Message-ID: <1602908748-43335-16-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" All data is sent by multifd Channels, so we only register its for multifd channels and main channel don't register its. Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/rdma.c b/migration/rdma.c index 3210e6e..d5d6364 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3938,6 +3938,12 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, qemu_sem_post(&multifd_send_param->sem_sync); } + + /* + * Use multifd to migrate, we only register memory for + * multifd RDMA channel and main channel don't register it. + */ + goto wait_reg_complete; } /* @@ -3998,6 +4004,8 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, rdma->dest_blocks[i].remote_host_addr; local->block[i].remote_rkey = rdma->dest_blocks[i].remote_rkey; } + +wait_reg_complete: /* Wait for all multifd channels to complete registration */ if (migrate_use_multifd()) { int i; From patchwork Sat Oct 17 04:25:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842553 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 06C8B17C9 for ; Sat, 17 Oct 2020 04:27:23 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 972E920720 for ; Sat, 17 Oct 2020 04:27:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 972E920720 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:53138 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdoP-0003U2-KJ for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:27:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40102) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXy-0006JE-1G for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:23 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]:5247 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXs-0007Iz-66 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:21 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 8C29B7C90A39DFCFFA82; Sat, 17 Oct 2020 12:09:56 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:50 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 16/18] migration/rdma: add rdma_channel into Migrationstate field Date: Sat, 17 Oct 2020 12:25:46 +0800 Message-ID: <1602908748-43335-17-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.191; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:09:47 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Multifd RDMA is need to poll when we send data, record it. Signed-off-by: Chuan Zheng --- migration/migration.c | 1 + migration/migration.h | 1 + migration/rdma.c | 14 ++++++++++++++ 3 files changed, 16 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 7061410..1ec1dc9 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1892,6 +1892,7 @@ void migrate_init(MigrationState *s) s->migration_thread_running = false; s->enabled_rdma_migration = false; s->host_port = NULL; + s->rdma_channel = 0; error_free(s->error); s->error = NULL; s->hostname = NULL; diff --git a/migration/migration.h b/migration/migration.h index fea63de..5676b23 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -272,6 +272,7 @@ struct MigrationState /* Need by Multi-RDMA */ char *host_port; + int rdma_channel; }; void migrate_set_state(int *state, int old_state, int new_state); diff --git a/migration/rdma.c b/migration/rdma.c index d5d6364..327f80f 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -183,6 +183,20 @@ typedef struct { } RDMAWorkRequestData; /* + * Get the multifd RDMA channel used to send data. + */ +static int get_multifd_RDMA_channel(void) +{ + int thread_count = migrate_multifd_channels(); + MigrationState *s = migrate_get_current(); + + s->rdma_channel++; + s->rdma_channel %= thread_count; + + return s->rdma_channel; +} + +/* * Negotiate RDMA capabilities during connection-setup time. */ typedef struct { From patchwork Sat Oct 17 04:25:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842533 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9AF1E17C9 for ; Sat, 17 Oct 2020 04:16:26 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3B43820791 for ; Sat, 17 Oct 2020 04:16:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3B43820791 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:46638 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTddo-0005qZ-OC for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:16:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39988) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXk-000618-BT for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:08 -0400 Received: from szxga07-in.huawei.com ([45.249.212.35]:36936 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXg-0007I3-Q8 for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:08 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 8252E61335AD4405501E; Sat, 17 Oct 2020 12:10:01 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:51 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 17/18] migration/rdma: send data for both rdma-pin-all and NOT rdma-pin-all mode Date: Sat, 17 Oct 2020 12:25:47 +0800 Message-ID: <1602908748-43335-18-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.35; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:10:01 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Zhimin Feng Signed-off-by: Chuan Zheng --- migration/rdma.c | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 62 insertions(+), 5 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index 327f80f..519fa7a 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2001,6 +2001,20 @@ static int qemu_rdma_write_one(QEMUFile *f, RDMAContext *rdma, .repeat = 1, }; + /* use multifd to send data */ + if (migrate_use_multifd()) { + int channel = get_multifd_RDMA_channel(); + int ret = 0; + MultiFDSendParams *multifd_send_param = NULL; + ret = get_multifd_send_param(channel, &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd_send_param(%d)", channel); + return -EINVAL; + } + rdma = (RDMAContext *)multifd_send_param->rdma; + block = &(rdma->local_ram_blocks.block[current_index]); + } + retry: sge.addr = (uintptr_t)(block->local_host_addr + (current_addr - block->offset)); @@ -2196,6 +2210,27 @@ retry: return 0; } +static int multifd_rdma_write_flush(void) +{ + /* The multifd RDMA threads send data */ + MultiFDSendParams *multifd_send_param = NULL; + RDMAContext *rdma = NULL; + MigrationState *s = migrate_get_current(); + int ret = 0; + + ret = get_multifd_send_param(s->rdma_channel, + &multifd_send_param); + if (ret) { + error_report("rdma: error getting multifd_send_param(%d)", + s->rdma_channel); + return ret; + } + rdma = (RDMAContext *)(multifd_send_param->rdma); + rdma->nb_sent++; + + return ret; +} + /* * Push out any unwritten RDMA operations. * @@ -2218,8 +2253,15 @@ static int qemu_rdma_write_flush(QEMUFile *f, RDMAContext *rdma) } if (ret == 0) { - rdma->nb_sent++; - trace_qemu_rdma_write_flush(rdma->nb_sent); + if (migrate_use_multifd()) { + ret = multifd_rdma_write_flush(); + if (ret) { + return ret; + } + } else { + rdma->nb_sent++; + trace_qemu_rdma_write_flush(rdma->nb_sent); + } } rdma->current_length = 0; @@ -4061,6 +4103,7 @@ wait_reg_complete: } qemu_sem_post(&multifd_send_param->sem_sync); + qemu_sem_wait(&multifd_send_param->sem); } } @@ -4443,6 +4486,7 @@ static void *multifd_rdma_send_thread(void *opaque) Error *local_err = NULL; int ret = 0; RDMAControlHeader head = { .len = 0, .repeat = 1 }; + RDMAContext *rdma = p->rdma; trace_multifd_send_thread_start(p->id); if (multifd_send_initial_packet(p, &local_err) < 0) { @@ -4451,7 +4495,7 @@ static void *multifd_rdma_send_thread(void *opaque) /* wait for semaphore notification to register memory */ qemu_sem_wait(&p->sem_sync); - if (qemu_rdma_registration(p->rdma) < 0) { + if (qemu_rdma_registration(rdma) < 0) { goto out; } /* @@ -4467,13 +4511,26 @@ static void *multifd_rdma_send_thread(void *opaque) break; } qemu_mutex_unlock(&p->mutex); - + /* To complete polling(CQE) */ + while (rdma->nb_sent) { + ret = qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, NULL); + if (ret < 0) { + error_report("multifd RDMA migration: " + "complete polling error!"); + return NULL; + } + } /* Send FINISHED to the destination */ head.type = RDMA_CONTROL_REGISTER_FINISHED; - ret = qemu_rdma_exchange_send(p->rdma, &head, NULL, NULL, NULL, NULL); + ret = qemu_rdma_exchange_send(rdma, &head, NULL, NULL, NULL, NULL); if (ret < 0) { + error_report("multifd RDMA migration: " + "sending remote error!"); return NULL; } + + /* sync main thread */ + qemu_sem_post(&p->sem); } out: From patchwork Sat Oct 17 04:25:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Chuan X-Patchwork-Id: 11842551 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5CEDF14B4 for ; Sat, 17 Oct 2020 04:25:24 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DBE5720791 for ; Sat, 17 Oct 2020 04:25:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DBE5720791 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:46638 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kTdmU-0000hq-R9 for patchwork-qemu-devel@patchwork.kernel.org; Sat, 17 Oct 2020 00:25:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39984) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXi-0005xv-SW for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:06 -0400 Received: from szxga07-in.huawei.com ([45.249.212.35]:36940 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kTdXg-0007I4-NY for qemu-devel@nongnu.org; Sat, 17 Oct 2020 00:10:06 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 87C02B2F8DAD10F3D499; Sat, 17 Oct 2020 12:10:01 +0800 (CST) Received: from huawei.com (10.175.101.6) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Sat, 17 Oct 2020 12:09:51 +0800 From: Chuan Zheng To: , Subject: [PATCH v3 18/18] migration/rdma: RDMA cleanup for multifd migration Date: Sat, 17 Oct 2020 12:25:48 +0800 Message-ID: <1602908748-43335-19-git-send-email-zhengchuan@huawei.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> References: <1602908748-43335-1-git-send-email-zhengchuan@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.101.6] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.35; envelope-from=zhengchuan@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/17 00:10:01 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, zhang.zhanghailiang@huawei.com, fengzhimin1@huawei.com, qemu-devel@nongnu.org, xiexiangyou@huawei.com, alex.chen@huawei.com, wanghao232@huawei.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Chuan Zheng --- migration/multifd.c | 6 ++++++ migration/multifd.h | 1 + migration/rdma.c | 16 +++++++++++++++- 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/migration/multifd.c b/migration/multifd.c index c4d90ef..f548122 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -574,6 +574,9 @@ void multifd_save_cleanup(void) p->packet_len = 0; g_free(p->packet); p->packet = NULL; +#ifdef CONFIG_RDMA + multifd_rdma_cleanup(p->rdma); +#endif multifd_send_state->ops->send_cleanup(p, &local_err); if (local_err) { migrate_set_error(migrate_get_current(), local_err); @@ -1017,6 +1020,9 @@ int multifd_load_cleanup(Error **errp) p->packet_len = 0; g_free(p->packet); p->packet = NULL; +#ifdef CONFIG_RDMA + multifd_rdma_cleanup(p->rdma); +#endif multifd_recv_state->ops->recv_cleanup(p); } qemu_sem_destroy(&multifd_recv_state->sem_sync); diff --git a/migration/multifd.h b/migration/multifd.h index ec9e897..6fddd4e 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -183,6 +183,7 @@ typedef struct { #ifdef CONFIG_RDMA MultiFDSetup *multifd_rdma_setup(void); +void multifd_rdma_cleanup(void *opaque); #endif void multifd_send_terminate_threads(Error *err); int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp); diff --git a/migration/rdma.c b/migration/rdma.c index 519fa7a..89bf54b 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2368,7 +2368,7 @@ static void qemu_rdma_cleanup(RDMAContext *rdma) { int idx; - if (rdma->cm_id && rdma->connected) { + if (rdma->channel && rdma->cm_id && rdma->connected) { if ((rdma->error_state || migrate_get_current()->state == MIGRATION_STATUS_CANCELLING) && !rdma->received_error) { @@ -4609,6 +4609,20 @@ static MultiFDSetup multifd_rdma_ops = { .recv_channel_setup = multifd_rdma_recv_channel_setup }; +void multifd_rdma_cleanup(void *opaque) +{ + RDMAContext *rdma = (RDMAContext *)opaque; + + if (!migrate_use_rdma()) { + return; + } + + rdma->listen_id = NULL; + rdma->channel = NULL; + qemu_rdma_cleanup(rdma); + g_free(rdma); +} + MultiFDSetup *multifd_rdma_setup(void) { MultiFDSetup *rdma_ops;