From patchwork Tue Jun 5 15:28:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 10448495 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id EA8C4600CA for ; Tue, 5 Jun 2018 15:30:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D6F1C29793 for ; Tue, 5 Jun 2018 15:30:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CCEEC297BF; Tue, 5 Jun 2018 15:30:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4B2B5297BF for ; Tue, 5 Jun 2018 15:30:03 +0000 (UTC) Received: from localhost ([::1]:47509 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fQDuM-0001aw-HP for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 Jun 2018 11:30:02 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:40175) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fQDt2-0008Ix-0z for qemu-devel@nongnu.org; Tue, 05 Jun 2018 11:28:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fQDt1-0003xY-1m for qemu-devel@nongnu.org; Tue, 05 Jun 2018 11:28:40 -0400 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]:41673) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fQDt0-0003x5-OW for qemu-devel@nongnu.org; Tue, 05 Jun 2018 11:28:38 -0400 Received: by mail-pf0-x243.google.com with SMTP id a11-v6so1484134pff.8 for ; Tue, 05 Jun 2018 08:28:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MiVhFCP4aas7BoWLX4QmXx/HnZ+qlLWa1ujmNDXEV3c=; b=qwa352qOnCNCrZVsAPjmRd4oX+axa8wqbuQvcJtZHJcaabdnmsxK464Zm0R+lAZxg9 B/EDEJbRYpTpe5hE7SvWxhS8BHLawzQor9zobRK97qD2z4qlieDgr6GOb/qSWPDhZTuh LyA8BUDKnYjxE7eIDl0njtkP90EGAs6oRscoqRmsk5fXhtNHOJbg4WkH8z1+TU5p9bWg j/j2XnJ0lif1maxWbf/KCyk2/R47uUeZzvdwLuVw9SyNoKd5CXZ70tDFnCZ50FxhiTYX HSj/itY47IZWcgbDMJZHS+ZnUORqrpwE4tEmxHpXMgNVw8zzWisE1TsPCUNbL/dwRnLx U5qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=MiVhFCP4aas7BoWLX4QmXx/HnZ+qlLWa1ujmNDXEV3c=; b=PH/c7b0goaPWj9cXMkzcPztG6r9s14ruPKeDjJOJyZSK0cgA0Ewo7oaRZbNpwyfzEp maK8Kjy5sWv5ZQmBHdSrbEdec6c7xIvUYfyDgPgyNsTyUUIjaSTLvfto+QgNIMvEId5X YH4WTVzL4RPS1F+bODrsEd13tbNyfUBtQZXk69vIp8xmTqUci11Z5XUmQOWa15Kw2pZP s4fjuEHDPrsHC2l928R2XQ+36rjpKNDMBZuOJS7Rb5mp2LE9096F9zIe4Lqi6cdxMkgG pyF7DlwnMwtHUe+/g5jkQV80BkgxL4Z0+LU17QJ2oAIFdo8biOH+qXE87Bq3y6FjhcZC xPHA== X-Gm-Message-State: ALKqPweElfkeetO/OMEsb70al8C+skovbDgcpwu0A5cvfvQKVXAa/JrN xQez8v63bY2TE4iIJ4/YXAw= X-Google-Smtp-Source: ADUXVKLmssD1+fo0JfHaT+Mwz8fKnSFI6jzZcflO3Cq6PTqqVqx7gT3Z1lEmgtrOs2ToAC5+PUGgkA== X-Received: by 2002:a63:a60a:: with SMTP id t10-v6mr21171234pge.351.1528212517962; Tue, 05 Jun 2018 08:28:37 -0700 (PDT) Received: from VM_93_245_centos.localdomain ([150.109.57.149]) by smtp.gmail.com with ESMTPSA id t14-v6sm101952007pfh.109.2018.06.05.08.28.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 05 Jun 2018 08:28:37 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: zhang.zhanghailiang@huawei.com, quintela@redhat.com, dgilbert@redhat.com, berrange@redhat.com, aviadye@mellanox.com, pbonzini@redhat.com Date: Tue, 5 Jun 2018 23:28:07 +0800 Message-Id: <1528212489-19137-9-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1528212489-19137-1-git-send-email-lidongchen@tencent.com> References: <1528212489-19137-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::243 Subject: [Qemu-devel] [PATCH v5 08/10] migration: create a dedicated thread to release rdma resource X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: galsha@mellanox.com, adido@mellanox.com, qemu-devel@nongnu.org, Lidong Chen Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP ibv_dereg_mr wait for a long time for big memory size virtual server. The test result is: 10GB 326ms 20GB 699ms 30GB 1021ms 40GB 1387ms 50GB 1712ms 60GB 2034ms 70GB 2457ms 80GB 2807ms 90GB 3107ms 100GB 3474ms 110GB 3735ms 120GB 4064ms 130GB 4567ms 140GB 4886ms this will cause the guest os hang for a while when migration finished. So create a dedicated thread to release rdma resource. Signed-off-by: Lidong Chen --- migration/rdma.c | 43 +++++++++++++++++++++++++++---------------- 1 file changed, 27 insertions(+), 16 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index dfa4f77..f12e8d5 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2979,35 +2979,46 @@ static void qio_channel_rdma_set_aio_fd_handler(QIOChannel *ioc, } } -static int qio_channel_rdma_close(QIOChannel *ioc, - Error **errp) +static void *qio_channel_rdma_close_thread(void *arg) { - QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); - RDMAContext *rdmain, *rdmaout; - trace_qemu_rdma_close(); + RDMAContext **rdma = arg; + RDMAContext *rdmain = rdma[0]; + RDMAContext *rdmaout = rdma[1]; - rdmain = rioc->rdmain; - if (rdmain) { - atomic_rcu_set(&rioc->rdmain, NULL); - } - - rdmaout = rioc->rdmaout; - if (rdmaout) { - atomic_rcu_set(&rioc->rdmaout, NULL); - } + rcu_register_thread(); synchronize_rcu(); - if (rdmain) { qemu_rdma_cleanup(rdmain); } - if (rdmaout) { qemu_rdma_cleanup(rdmaout); } g_free(rdmain); g_free(rdmaout); + g_free(rdma); + + rcu_unregister_thread(); + return NULL; +} + +static int qio_channel_rdma_close(QIOChannel *ioc, + Error **errp) +{ + QemuThread t; + QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); + RDMAContext **rdma = g_new0(RDMAContext*, 2); + + trace_qemu_rdma_close(); + if (rioc->rdmain || rioc->rdmaout) { + rdma[0] = rioc->rdmain; + rdma[1] = rioc->rdmaout; + qemu_thread_create(&t, "rdma cleanup", qio_channel_rdma_close_thread, + rdma, QEMU_THREAD_DETACHED); + atomic_rcu_set(&rioc->rdmain, NULL); + atomic_rcu_set(&rioc->rdmaout, NULL); + } return 0; }