From patchwork Mon Jun 7 13:57:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?TElaSEFPWElOMSBb5p2O54Wn6ZGrXQ==?= X-Patchwork-Id: 12303621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.6 required=3.0 tests=BAYES_00, CHARSET_FARAWAY_HEADER,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E515C47094 for ; Mon, 7 Jun 2021 14:13:20 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E2DCB60FEA for ; Mon, 7 Jun 2021 14:13:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E2DCB60FEA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kingsoft.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:42322 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lqG0C-0003wb-Uj for qemu-devel@archiver.kernel.org; Mon, 07 Jun 2021 10:13:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54300) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lqFzS-0003GW-BA for qemu-devel@nongnu.org; Mon, 07 Jun 2021 10:12:30 -0400 Received: from mail.kingsoft.com ([114.255.44.145]:42942) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lqFzK-00043P-Uu for qemu-devel@nongnu.org; Mon, 07 Jun 2021 10:12:29 -0400 X-AuditID: 0a580155-3d1ff7000005011f-0d-60be25ae49a8 Received: from mail.kingsoft.com (localhost [10.88.1.79]) (using TLS with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mail.kingsoft.com (SMG-2-NODE-85) with SMTP id 91.75.00287.EA52EB06; Mon, 7 Jun 2021 21:57:02 +0800 (HKT) Received: from KSbjmail3.kingsoft.cn (10.88.1.78) by KSBJMAIL4.kingsoft.cn (10.88.1.79) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Mon, 7 Jun 2021 21:57:02 +0800 Received: from KSbjmail3.kingsoft.cn ([fe80::a143:8393:3ff1:cd3]) by KSBJMAIL3.kingsoft.cn ([fe80::a143:8393:3ff1:cd3%10]) with mapi id 15.01.2176.014; Mon, 7 Jun 2021 21:57:02 +0800 From: =?eucgb2312_cn?b?TElaSEFPWElOMSBbwO7V1fbOXQ==?= To: "qemu-devel@nongnu.org" , "quintela@redhat.com" , "dgilbert@redhat.com" Subject: [PATCH] migration/rdma: Use huge page register VM memory Thread-Topic: [PATCH] migration/rdma: Use huge page register VM memory Thread-Index: AddbpN+kEXe4Oe1WQuqS3KFy+fOdUQ== Date: Mon, 7 Jun 2021 13:57:02 +0000 Message-ID: <51819991cecb42f6a619768bc61d0bfd@kingsoft.com> Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.88.1.106] MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsXCFcHor7tOdV+CwbXLMha92+6xWxzv3cFi cWdLH5MDs8eTa5uZPN7vu8oWwBTFZZOSmpNZllqkb5fAlfGxYzVzwTOJiu/N/9kaGKdIdDFy ckgImEi0t69m6mLk4hASmM4ksX7fcxYI5zmjxMlplxlBqoQEdjFKTFjJBmKzCXhKfFp1hg2k SESgn1HixvoHYO3MAj1MEr/ftLKCVAkLOEosXz+BBcQWEXCT+LW6E8rWk7i38jZQNwcHi4CK xOurSSBhXgFriYW7X4EtYxSQlZj26D4TiM0sIC4xd9osVohTBSSW7DnPDGGLSrx8/I8VZIyE gLzE6o/CEOVaEvMafkO1KkpM6X7IDjFeUOLkzCcsExhFZiGZOgtJyywkLbOQtCxgZFnFyFKc m260iRES+KE7GGc0fdQ7xMjEwXiIUYKDWUmE10tmT4IQb0piZVVqUX58UWlOavEhRmkOFiVx 3rVGOxKEBNITS1KzU1MLUotgskwcnFINTLymH9KFn5r1pz6bktLaz75j068nClFMPTJ/9Xeb 6Rieu3YgPiqLx/aNq9Bk1RnmMbNsJ0+/ndqmcG35j2SevvN3krhMLZ2it35odm9Y2PvSlS8u aUn70xv7zmvPeLJCWnebb+Ur/yvPzTYlC/3otP12/esXlSQLxdlfsjY9Pn6xY53+x8sJZQsm py3Mmf5/01aGo548jsyPr0xNuLBUcpLT7sKUJY/V02ujlP1FF/x7Fvf99fZzjEvqd4lpiBqp KiQKGGbVnp1kGXK9I1RV5eHvCfPe3g599qu4Z41P3p5TybsXcZu2XH6wq+Hq6VDDvbPmbyvc d41Dwehb0sajXlwB5wRveuzjWMOi99Jvx3IlluKMREMt5qLiRABbqqSn6wIAAA== Received-SPF: pass client-ip=114.255.44.145; envelope-from=LIZHAOXIN1@kingsoft.com; helo=mail.kingsoft.com X-Spam_score_int: 13 X-Spam_score: 1.3 X-Spam_bar: + X-Spam_report: (1.3 / 5.0 requ) BAYES_00=-1.9, CHARSET_FARAWAY_HEADER=3.2, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?eucgb2312_cn?b?TElaSEFPWElOMSBbwO7V1fbOXQ==?= , =?eucgb2312_cn?b?c3VuaGFvMiBby+/qu10=?= , =?eucgb2312_cn?b?REVOR0xJTldFTiBbtcvB1s7EXQ==?= , =?eucgb2312_cn?b?WUFOR0ZFTkcxIFvR7rflXQ==?= Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" When using libvirt for RDMA live migration, if the VM memory is too large, it will take a lot of time to deregister the VM at the source side, resulting in a long downtime (VM 64G, deregister vm time is about 400ms). Although the VM's memory uses 2M huge pages, the MLNX driver still uses 4K pages for pin memory, as well as for unpin. So we use huge pages to skip the process of pin memory and unpin memory to reduce downtime. The test environment: kernel: linux-5.12 MLNX: ConnectX-4 LX libvirt command: virsh migrate --live --p2p --persistent --copy-storage-inc --listen-address \ 0.0.0.0 --rdma-pin-all --migrateuri rdma://192.168.0.2 [VM] qemu+tcp://192.168.0.2/system Signed-off-by: lizhaoxin diff --git a/migration/rdma.c b/migration/rdma.c index 1cdb4561f3..9823449297 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -1123,13 +1123,26 @@ static int qemu_rdma_reg_whole_ram_blocks(RDMAContext *rdma) RDMALocalBlocks *local = &rdma->local_ram_blocks; for (i = 0; i < local->nb_blocks; i++) { - local->block[i].mr = - ibv_reg_mr(rdma->pd, - local->block[i].local_host_addr, - local->block[i].length, - IBV_ACCESS_LOCAL_WRITE | - IBV_ACCESS_REMOTE_WRITE - ); + if (strcmp(local->block[i].block_name,"pc.ram") == 0) { + local->block[i].mr = + ibv_reg_mr(rdma->pd, + local->block[i].local_host_addr, + local->block[i].length, + IBV_ACCESS_LOCAL_WRITE | + IBV_ACCESS_REMOTE_WRITE | + IBV_ACCESS_ON_DEMAND | + IBV_ACCESS_HUGETLB + ); + } else { + local->block[i].mr = + ibv_reg_mr(rdma->pd, + local->block[i].local_host_addr, + local->block[i].length, + IBV_ACCESS_LOCAL_WRITE | + IBV_ACCESS_REMOTE_WRITE + ); + } + if (!local->block[i].mr) { perror("Failed to register local dest ram block!\n"); break;