From patchwork Fri Jun 9 09:45:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13273592 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B21FC7EE2E for ; Fri, 9 Jun 2023 09:46:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 071BE8E0003; Fri, 9 Jun 2023 05:46:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 021EB8E0001; Fri, 9 Jun 2023 05:46:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E52718E0003; Fri, 9 Jun 2023 05:46:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id CF61B8E0001 for ; Fri, 9 Jun 2023 05:46:11 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 93D5E1601B4 for ; Fri, 9 Jun 2023 09:46:11 +0000 (UTC) X-FDA: 80882728542.06.45B4144 Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) by imf05.hostedemail.com (Postfix) with ESMTP id ACD83100012 for ; Fri, 9 Jun 2023 09:46:08 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.97 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686303970; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=Cr4TxD19N2hmRtd9m5/QpmZoQQU0Pc8wo9ceWnUXc/g=; b=CxygAu8N98nciRxrY3MUixZOKALyxjnJLDiq1+Y3YdEvFDb61mJFosSG56OhO/5eHkXpdC EHugzs2oz80L1Z+Qx6Taau8WIOGo+EhUd6VeO25JR+YFVAMKCJ3rP2g/9awGa3ej40x5sd uP9kSR+hH0Orulayqt3BASBPaPmRyvc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686303970; a=rsa-sha256; cv=none; b=s5c2sI4JdNGNMsuPfIGYStIi4KJgWYVzHJGs4P/jcIih3Geq9Ho5mhVfnOAbuKoi7YsaOa YzUyyROh0WXziqUFL9LIYRWarNxcHBygymkMnUy26DTUFTZddogUt18NxKU3GjWS8G5MNm 5/OvR7FoAe/LnNaSQmwJB+hoUgb1qPg= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.97 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R311e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046051;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=7;SR=0;TI=SMTPD_---0VkhjX-o_1686303963; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VkhjX-o_1686303963) by smtp.aliyun-inc.com; Fri, 09 Jun 2023 17:46:04 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: mgorman@techsingularity.net, vbabka@suse.cz, david@redhat.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH] mm: compaction: skip memory hole rapidly when isolating migratable pages Date: Fri, 9 Jun 2023 17:45:54 +0800 Message-Id: <8cc668b77c8eb2fa78058b3d81386ebed9c5a9cd.1686294549.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 X-Rspamd-Queue-Id: ACD83100012 X-Rspam-User: X-Stat-Signature: 18au6i6u37ohrxmazb67eak3znzp594g X-Rspamd-Server: rspam03 X-HE-Tag: 1686303968-542679 X-HE-Meta: U2FsdGVkX18W1boQJFjEPN3I7z+v95uhnWA9IABuZDF4tsrjmuD/Po2tW14tY0KiJFYi13sBmASJ4fy/pdo5mAPPKqthY7onncLMN/EPTAlCY/JKYMwW2bJgbnYRfltq+s2II/pf8PrqKGPhSYVQ82g2tFlc6RuN5rqT0/Geh/2VzFVsEbuD/+BzeNKJ8NEGM514GErQ9nT5yY0I/v8Jx/9p+7H07LRjFQ2CCjeeErDLpGRC18RZRV91kzWTvz/AUlZNs479YhlRDcbfrR0F9fPZI1nPA27bRTVQlbwFlkxROh4li+JEQqBm8DWgaWDuDAbu4mgmDgzCaRLzMmxi7058WWB4qhbBtLxWg8s8rRSAhWqs/Bz1rgs0GSsqT27t71OSuBeaRHDiRx8JkMv0Oqohhsn66MFwnCsfKwFkCvLLGD6Y+zmprsEz9wikeszHm8O8oZ2ihoL6Ty53g3uHSN1xa/9oDNsHGjWlcuV93fjUYwZuKhisLs0Y7mVm8W4oP1m36xtDiLpOstqFKOKeKUOLvkJggdmsOUd/x8oPOumHeOyaI6vyy5lgBaCTt5sBUTf7jPdXRtPD19zy+KWNzqoqFnn8N/JcwK73i3pu8r0lPaAhODNWWXkQbICCJO9HRRbsUsivFpvBxWPOVDNz5Z+rqeIyWuqwNM/zrueOA6eCNOQHH/kp+AM9KHXoLQwJmX8k/qceA9pYpbl5S6DxevM32hcIcTLVgnnJkwBS8KLJcXfTzH3AROjjeCKQaMdRx/4VoRvqwJj+5bkn4J4xvTrdBpWuS2pXQX8zeB2jPPnxeXTy8KBJIXNceLu61J30gCXBi7uSjqct4sPVoDNhT+cxsmFvUehPXrDYY16/gGSeZPt1gRJb3TlvsCNwx6rYhCwY9b741BmNmRMwZwPxR+XiGmI4VyiVLKMYJ2OUOwyJIR+G/ELS/K4HnIvYFB8vDy6I1vPDEOKSNh7YBXF eZipgoRa TZ/MrENZn8cKkSg+U+6bqQ3IsLHoZwkv6ZHYJ1c14MB4Uy2tjxa+8hiM6nl28Q/tnjeNwnT3/bQOKi6RtaMqL33gcOojSlKKBu/+vNpFFmXSY2KzRyg/WXYnXozM1ha75Z6nVWFH5+HRHF8c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On some machines, the normal zone can have a large memory hole like below memory layout, and we can see the range from 0x100000000 to 0x1800000000 is a hole. So when isolating some migratable pages, the scanner can meet the hole and it will take more time to skip the large hole. From my measurement, I can see the isolation scanner will take 80us ~ 100us to skip the large hole [0x100000000 - 0x1800000000]. So adding a new helper to fast search next online memory section to skip the large hole can help to find next suitable pageblock efficiently. With this patch, I can see the large hole scanning only takes < 1us. [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000040000000-0x00000000ffffffff] [ 0.000000] DMA32 empty [ 0.000000] Normal [mem 0x0000000100000000-0x0000001fa7ffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000040000000-0x0000000fffffffff] [ 0.000000] node 0: [mem 0x0000001800000000-0x0000001fa3c7ffff] [ 0.000000] node 0: [mem 0x0000001fa3c80000-0x0000001fa3ffffff] [ 0.000000] node 0: [mem 0x0000001fa4000000-0x0000001fa402ffff] [ 0.000000] node 0: [mem 0x0000001fa4030000-0x0000001fa40effff] [ 0.000000] node 0: [mem 0x0000001fa40f0000-0x0000001fa73cffff] [ 0.000000] node 0: [mem 0x0000001fa73d0000-0x0000001fa745ffff] [ 0.000000] node 0: [mem 0x0000001fa7460000-0x0000001fa746ffff] [ 0.000000] node 0: [mem 0x0000001fa7470000-0x0000001fa758ffff] [ 0.000000] node 0: [mem 0x0000001fa7590000-0x0000001fa7ffffff] Signed-off-by: Baolin Wang --- include/linux/mmzone.h | 10 ++++++++++ mm/compaction.c | 23 ++++++++++++++++++++++- 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 5a7ada0413da..87e6c535d895 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2000,6 +2000,16 @@ static inline unsigned long next_present_section_nr(unsigned long section_nr) return -1; } +static inline unsigned long next_online_section_nr(unsigned long section_nr) +{ + while (++section_nr <= __highest_present_section_nr) { + if (online_section_nr(section_nr)) + return section_nr; + } + + return -1UL; +} + /* * These are _only_ used during initialisation, therefore they * can use __initdata ... They could have names to indicate diff --git a/mm/compaction.c b/mm/compaction.c index 3398ef3a55fe..3a55fdd20c49 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -229,6 +229,21 @@ static void reset_cached_positions(struct zone *zone) pageblock_start_pfn(zone_end_pfn(zone) - 1); } +static unsigned long skip_hole_pageblock(unsigned long start_pfn) +{ + unsigned long next_online_nr; + unsigned long start_nr = pfn_to_section_nr(start_pfn); + + if (online_section_nr(start_nr)) + return -1UL; + + next_online_nr = next_online_section_nr(start_nr); + if (next_online_nr != -1UL) + return section_nr_to_pfn(next_online_nr); + + return -1UL; +} + /* * Compound pages of >= pageblock_order should consistently be skipped until * released. It is always pointless to compact pages of such order (if they are @@ -1991,8 +2006,14 @@ static isolate_migrate_t isolate_migratepages(struct compact_control *cc) page = pageblock_pfn_to_page(block_start_pfn, block_end_pfn, cc->zone); - if (!page) + if (!page) { + unsigned long next_pfn; + + next_pfn = skip_hole_pageblock(block_start_pfn); + if (next_pfn != -1UL) + block_end_pfn = next_pfn; continue; + } /* * If isolation recently failed, do not retry. Only check the