From patchwork Mon Oct 30 01:11:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13439947 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCBDEC4332F for ; Mon, 30 Oct 2023 01:12:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 318A66B0181; Sun, 29 Oct 2023 21:12:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C8C96B0182; Sun, 29 Oct 2023 21:12:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DE6B6B0183; Sun, 29 Oct 2023 21:12:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0E1D36B0181 for ; Sun, 29 Oct 2023 21:12:09 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D7EF68055E for ; Mon, 30 Oct 2023 01:12:08 +0000 (UTC) X-FDA: 81400351536.04.550424C Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) by imf29.hostedemail.com (Postfix) with ESMTP id 2179D120015 for ; Mon, 30 Oct 2023 01:12:04 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf29.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.130 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698628327; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=JXanDA/ABYcoIZYeJL4efLBSA/his1lJ3Qr9kTwd/O4=; b=uBwl69KSSFf+TxSHi25j/Btw9EsZEPmhxIyAJNhj0xtIoYXA01Azqrr31YADHGj+DuqlW0 s8rG+IC8xBriMxgRYwqh+mhJbmrZLzbmqKaPgTRAXyMTbMmh/PoQ0m2hNwk4UiaFExS4Qe CEnrTMwR9Il3nsO7lcnjLWU4SDxaZso= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf29.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.130 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698628327; a=rsa-sha256; cv=none; b=MR6CeZkG0wZQLIPRrapjaw9J7L2whJem0ooZZQTQDpN3H/iUPBHPzzfoiWnsEuNcGHWkCm 2LtCE+HwW0nkdtNHJ+/GXQbZw1tsIdrYE6RWv5NSV1+MtceNznNI/9cAzDXozKWB4j3HrL lcfzAe8erNu2AAz2ac1DX29YQrnG8EI= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R381e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045192;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0Vv4ZEFb_1698628318; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0Vv4ZEFb_1698628318) by smtp.aliyun-inc.com; Mon, 30 Oct 2023 09:11:59 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: shy828301@gmail.com, ying.huang@intel.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH] mm: huge_memory: batch tlb flush when splitting a pte-mapped THP Date: Mon, 30 Oct 2023 09:11:47 +0800 Message-Id: <431d9fb6823036369dcb1d3b2f63732f01df21a7.1698488264.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 MIME-Version: 1.0 X-Rspamd-Queue-Id: 2179D120015 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: b5tqbef14d9mhuienjgdec9hb3pf9aeo X-HE-Tag: 1698628324-109065 X-HE-Meta: U2FsdGVkX1/yBlvYZ9mSdOWF5e9jB4RZ7auMNyFtgIJ5/SJop1h0palnC9HwP0FQTtG0aIW1N7o7V0XOGSpSAk27zgeMn0M4OaiakrZxXKn3IAebofVFBJ29isYsCBiKw7YccjuHhN0rV0lTPO8aq86xHfXTe5QircY8AH1dAQgqviWruzQNZVCPV9gGKvXbf42rR+wPFue8+RauwMgy6wGwQkZgFSU3g9nFoIs/Kb+cHvGeXzJMZhcS+CsjjiBFyI/6Sl7qjruAok8C2HSIcPllOostIB3azuzg8W89ey3gP4B3mUeNF+gvxqpzTU4SBguW8mK0fToOVd+wrIuzy/eNEM222EzqACECBFxMAe/JcL6d/FvG4n1uRC1uMArU5AjlWNtn+4iqByaea6OZR/Nv7Coy247MIicEHQuhH3+fN0JKA0oqa8ywdFPmBYhPmYVvdNztNn4KMwQkeM4MEM5PpxLwB3E+DD+GCd/DLreL5rj/o9IshwcB/EFq4q2oG1ndwGpis5ApRLZJ81qNE35BE/I36JxMgDMtJg1tJiU33kij3wnPTQH9f614Zk1GXIQa2IxWBjppBQGQdzGvN4CmRT+vH3BSkmSNlnCAdph627roOiaPXN2XDIVeY8RTmk4mAAhR0DH9JzK/FUvZVKp3bJ2D/uzX1SR3nbI+J/Gli5TBXWowghFfjBXbEX5OangfCbTa9EiikNtNw+sivelBh5z2gaGh4MHmEJ9pA+SaQIRSayGLjoeh/dQo1kllnerTEMrcwpEbEQ5aU1TiLha8vABojjRz4K3qe2j4Bx+UudvfxFIPdItXcCR3PXtVkKHcTarudhZqUKXgfnN0TVrShQdiv5FpkTcBAYcvsvs7O9+dMUejsd1MbrfjdpkrYASkJ6KPWgjqxgOyuWFgQSAX+3E8nHwowNmzdsojHDlp+ToNCrdNbtyD2M0lEyH2BunyWChh+wYHLYZHl2Y gn7c5zxt tlUd1Qc5hFsKt1ZLnDIwmmbE97XrkdtN9APNGNYdU6XuLslO3+ouPQQiWZia4i75jluwIVPQYu8//8ulFswzBvpbeIh0ke804xXP0xC6WEuT4UcX+B6wXUfXHMas04xqgKJSY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: I can observe an obvious tlb flush hotpot when splitting a pte-mapped THP on my ARM64 server, and the distribution of this hotspot is as follows: - 16.85% split_huge_page_to_list + 7.80% down_write - 7.49% try_to_migrate - 7.48% rmap_walk_anon 7.23% ptep_clear_flush + 1.52% __split_huge_page The reason is that the split_huge_page_to_list() will build migration entries for each subpage of a pte-mapped Anon THP by try_to_migrate(), or unmap for file THP, and it will clear and tlb flush for each subpage's pte. Moreover, the split_huge_page_to_list() will set TTU_SPLIT_HUGE_PMD flag to ensure the THP is already a pte-mapped THP before splitting it to some normal pages. Actually, there is no need to flush tlb for each subpage immediately, instead we can batch tlb flush for the pte-mapped THP to improve the performance. After this patch, we can see the batch tlb flush can improve the latency obviously when running thpscale. k6.5-base patched Amean fault-both-1 1071.17 ( 0.00%) 901.83 * 15.81%* Amean fault-both-3 2386.08 ( 0.00%) 1865.32 * 21.82%* Amean fault-both-5 2851.10 ( 0.00%) 2273.84 * 20.25%* Amean fault-both-7 3679.91 ( 0.00%) 2881.66 * 21.69%* Amean fault-both-12 5916.66 ( 0.00%) 4369.55 * 26.15%* Amean fault-both-18 7981.36 ( 0.00%) 6303.57 * 21.02%* Amean fault-both-24 10950.79 ( 0.00%) 8752.56 * 20.07%* Amean fault-both-30 14077.35 ( 0.00%) 10170.01 * 27.76%* Amean fault-both-32 13061.57 ( 0.00%) 11630.08 * 10.96%* Signed-off-by: Baolin Wang Reviewed-by: "Huang, Ying" Reviewed-by: Yang Shi Reviewed-by: Alistair Popple --- mm/huge_memory.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f31f02472396..0e4c14bf6872 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2379,7 +2379,7 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma, static void unmap_folio(struct folio *folio) { enum ttu_flags ttu_flags = TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD | - TTU_SYNC; + TTU_SYNC | TTU_BATCH_FLUSH; VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); @@ -2392,6 +2392,8 @@ static void unmap_folio(struct folio *folio) try_to_migrate(folio, ttu_flags); else try_to_unmap(folio, ttu_flags | TTU_IGNORE_MLOCK); + + try_to_unmap_flush(); } static void remap_page(struct folio *folio, unsigned long nr)