From patchwork Fri Aug 3 09:13:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 10554869 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A9CB59093 for ; Fri, 3 Aug 2018 09:22:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9F5642BE27 for ; Fri, 3 Aug 2018 09:22:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 937A12C1F3; Fri, 3 Aug 2018 09:22:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 128E62C239 for ; Fri, 3 Aug 2018 09:22:41 +0000 (UTC) Received: from localhost ([::1]:49788 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1flWIC-00005y-8j for patchwork-qemu-devel@patchwork.kernel.org; Fri, 03 Aug 2018 05:22:40 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52986) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1flWA9-0001Zm-3T for qemu-devel@nongnu.org; Fri, 03 Aug 2018 05:14:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1flWA7-0001tn-Sg for qemu-devel@nongnu.org; Fri, 03 Aug 2018 05:14:21 -0400 Received: from mail-pf1-x441.google.com ([2607:f8b0:4864:20::441]:36972) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1flWA7-0001ri-M6 for qemu-devel@nongnu.org; Fri, 03 Aug 2018 05:14:19 -0400 Received: by mail-pf1-x441.google.com with SMTP id a26-v6so2927509pfo.4 for ; Fri, 03 Aug 2018 02:14:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=AP9lCjBQnayg5NRv2Z3sgi+XbfOW1T9ZCaxxobQuIRg=; b=US4ciwJwS0doYzJb+2cFTnBDKomlM5fdo/P8K4LyvLi8ka0OgbnVxypu/4s6TjsFbq Z3nAfIQZVl+QlHnsC7mtTrM/cB/47L+U1wibAImBvIzzYaYHVoLiaWMVgqup7OQXeZHz 0Ur0XN0OKzeSTKCh78bNxf2n5vUVNrsryLbI99i7BBA/8uBK7A+dVOqDmQRV6NVBj51P i/C6m/SEMRfq0hKwB20Bw5OprAd/NqXFupeLMaT53SR6MOmeL9N6kvSTOzAYnHI/SF1o OJ4mnkzsn0nr2EpQDNPiFhMvFCrNjP5jwqMeLTFJ+C75vYSgq3a/Ui2L2QNjoMPBHXJD bi7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=AP9lCjBQnayg5NRv2Z3sgi+XbfOW1T9ZCaxxobQuIRg=; b=ir6s+hD7T+Db9k6n1sbyNMnZMLHZivP/I0bCBpd0kZsvPk27YUQ9oHV9wN1NHTtx7U EQyac/rvt84GNJ1UuFMgKNibT/CIyfJAycURrqiaXi32lmNuo9o8/OsYP21PJYYOmySy 3mQcQvTETShSAtm0vZHJrF3H94QcEbYIs7j+g0os7OZjDk43BdVCC5EgEoM6k+XZ/LgC uBCzDOolcqNyNxFCQEZTECP3J2AEJrcLdUOQVsLkd/bUS6B3dCTaUJCBOfjLmEpD/xtH UrsB+NzzZA82rfMLqbvPBCGLpNrQ8uavK2/sQxD5CJRLBGXqkTHJEok3fXZ+x0JFMjtq hSlw== X-Gm-Message-State: AOUpUlFZ7zc25BFdvvsDC6O3szYvqmo02JFPzjT5OyFIWOZ7huHRseiO srNKPT9WcbZygLG2+UifX4M= X-Google-Smtp-Source: AAOMgpe0nb3fPuA83zwtqLvx6cCexXcVRNGVmtFUf+d+dRXZEfSCTVJ6HbjN1PPD8cLwqGVfob7Q7A== X-Received: by 2002:a63:d04f:: with SMTP id s15-v6mr2986203pgi.42.1533287658900; Fri, 03 Aug 2018 02:14:18 -0700 (PDT) Received: from VM_120_46_centos.localdomain ([119.28.87.64]) by smtp.gmail.com with ESMTPSA id e21-v6sm7991352pfl.187.2018.08.03.02.14.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 03 Aug 2018 02:14:18 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: zhang.zhanghailiang@huawei.com, quintela@redhat.com, dgilbert@redhat.com Date: Fri, 3 Aug 2018 17:13:50 +0800 Message-Id: <1533287630-4221-13-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1533287630-4221-1-git-send-email-lidongchen@tencent.com> References: <1533287630-4221-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::441 Subject: [Qemu-devel] [PATCH v6 12/12] migration: create a dedicated thread to release rdma resource X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-devel@nongnu.org, Lidong Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP ibv_dereg_mr wait for a long time for big memory size virtual server. The test result is: 10GB 326ms 20GB 699ms 30GB 1021ms 40GB 1387ms 50GB 1712ms 60GB 2034ms 70GB 2457ms 80GB 2807ms 90GB 3107ms 100GB 3474ms 110GB 3735ms 120GB 4064ms 130GB 4567ms 140GB 4886ms this will cause the guest os hang for a while when migration finished. So create a dedicated thread to release rdma resource. Signed-off-by: Lidong Chen --- migration/migration.c | 6 ++++++ migration/migration.h | 3 +++ migration/rdma.c | 47 +++++++++++++++++++++++++++++++---------------- 3 files changed, 40 insertions(+), 16 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 360ee94..f58fe45 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1499,6 +1499,7 @@ void migrate_init(MigrationState *s) s->vm_was_running = false; s->iteration_initial_bytes = 0; s->threshold_size = 0; + s->rdma_cleanup_thread_quit = true; } static GSList *migration_blockers; @@ -1660,6 +1661,10 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc, return false; } + if (s->rdma_cleanup_thread_quit != true) { + return false; + } + if (runstate_check(RUN_STATE_INMIGRATE)) { error_setg(errp, "Guest is waiting for an incoming migration"); return false; @@ -3214,6 +3219,7 @@ static void migration_instance_init(Object *obj) ms->state = MIGRATION_STATUS_NONE; ms->mbps = -1; + ms->rdma_cleanup_thread_quit = true; qemu_sem_init(&ms->pause_sem, 0); qemu_mutex_init(&ms->error_mutex); diff --git a/migration/migration.h b/migration/migration.h index a50c2de..4d1be08 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -231,6 +231,9 @@ struct MigrationState * do not trigger spurious decompression errors. */ bool decompress_error_check; + + /* Set this when rdma resource have released */ + bool rdma_cleanup_thread_quit; }; void migrate_set_state(int *state, int old_state, int new_state); diff --git a/migration/rdma.c b/migration/rdma.c index 14cdf82..3d1a4ad 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2995,35 +2995,50 @@ static void qio_channel_rdma_set_aio_fd_handler(QIOChannel *ioc, } } -static int qio_channel_rdma_close(QIOChannel *ioc, - Error **errp) +static void *qio_channel_rdma_close_thread(void *arg) { - QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); - RDMAContext *rdmain, *rdmaout; - trace_qemu_rdma_close(); - - rdmain = rioc->rdmain; - if (rdmain) { - atomic_rcu_set(&rioc->rdmain, NULL); - } + RDMAContext **rdma = arg; + RDMAContext *rdmain = rdma[0]; + RDMAContext *rdmaout = rdma[1]; + MigrationState *s = migrate_get_current(); - rdmaout = rioc->rdmaout; - if (rdmaout) { - atomic_rcu_set(&rioc->rdmaout, NULL); - } + rcu_register_thread(); synchronize_rcu(); - if (rdmain) { qemu_rdma_cleanup(rdmain); } - if (rdmaout) { qemu_rdma_cleanup(rdmaout); } g_free(rdmain); g_free(rdmaout); + g_free(rdma); + + rcu_unregister_thread(); + s->rdma_cleanup_thread_quit = true; + return NULL; +} + +static int qio_channel_rdma_close(QIOChannel *ioc, + Error **errp) +{ + QemuThread t; + QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); + RDMAContext **rdma = g_new0(RDMAContext*, 2); + MigrationState *s = migrate_get_current(); + + trace_qemu_rdma_close(); + if (rioc->rdmain || rioc->rdmaout) { + rdma[0] = rioc->rdmain; + rdma[1] = rioc->rdmaout; + atomic_rcu_set(&rioc->rdmain, NULL); + atomic_rcu_set(&rioc->rdmaout, NULL); + s->rdma_cleanup_thread_quit = false; + qemu_thread_create(&t, "rdma cleanup", qio_channel_rdma_close_thread, + rdma, QEMU_THREAD_DETACHED); + } return 0; }