From patchwork Mon Jan 24 17:59:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 12722646 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6BBAC433EF for ; Mon, 24 Jan 2022 18:00:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 602646B0099; Mon, 24 Jan 2022 13:00:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5B3916B009C; Mon, 24 Jan 2022 13:00:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 454AA6B00A0; Mon, 24 Jan 2022 13:00:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0143.hostedemail.com [216.40.44.143]) by kanga.kvack.org (Postfix) with ESMTP id 35C896B0099 for ; Mon, 24 Jan 2022 13:00:20 -0500 (EST) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id DF8D692DE8 for ; Mon, 24 Jan 2022 18:00:19 +0000 (UTC) X-FDA: 79065944958.27.E925BBC Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by imf03.hostedemail.com (Postfix) with ESMTP id 09AEA20091 for ; Mon, 24 Jan 2022 18:00:11 +0000 (UTC) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id A192D5C0185; Mon, 24 Jan 2022 13:00:11 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Mon, 24 Jan 2022 13:00:11 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:date:date:from:from:in-reply-to :message-id:mime-version:reply-to:reply-to:sender:subject :subject:to:to; s=fm2; bh=9BrsfCn9GsbUXkbFEX9irdBIf/KH/bhctKCCvd Cg07k=; b=UDVNR59FzCrmekkSFf9nlBqhIeYL5fwLKwYUWcjdA/EzpS5oq6b0GZ DnxH7b3zhhwWOhv2bdwRwgHHAqMVOaxiVNcM6TQVtmL2xwyBjHI0Ou7CDC8zbbkh Ml0wc9IaYopbl2dgH0wgCche6RQrdiT9BD982pf/MhA7qQ99xKtNhuD1PFgwjVhL 0s/BmiZE/OZhf1//5cOSSbJHS38IJGcGZOcOsvAPYz+RyCrkapQ5u48lzoKh8i+y w3tE+yphSFjUy0lYpeTfkayo3Ja96pOs5VfJQig1tGzQ2XzuAiTa7gEMuRMsnk07 Eo+utaZ8mwFE2hDU2GqbQuG58H+4HnOg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :from:from:in-reply-to:message-id:mime-version:reply-to:reply-to :sender:subject:subject:to:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=9BrsfCn9GsbUXkbFEX9irdBIf/KH/ bhctKCCvdCg07k=; b=d2s/UswaHW+LI5SwBcV6QOLHf+n/f+3vMQSg+VCLQgdbu +WbqBeV5IkczRYziH4HY0OqXH1hkXbszcuomvd89hiWjYIPXdr3CsmyqigEMJlSC M3cybLBo4F+xM8FQanh9FTSctJLsTpR5YxJW0XAicNTEh2fs7huP9DFlmaLCeQfg +6n/gAucDQb+BTqzgXQl+0g4tP8s+zlp/ki/fA7lVKlfep7RUf9VqVhJP8Ep3/66 GBQWkU0IEghs+NY6mLH6IIxqI+T5yBaq6t1LNNuwvfDZN9lg1XDKRuQyKOn4HyKP N0zphFm4Qhow7NIjrO/+tZI+SzVA03g9tKQ4Y9XwQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrvdeigddutdelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkfforhgggfestdhqredtredttdenucfhrhhomhepkghiucgjrghn uceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepteeiiedtje fghfefueekjedvueeftddvudelhfetudffiefgfeetheeghfdvgedunecuffhomhgrihhn pehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmh grihhlfhhrohhmpeiiihdrhigrnhesshgvnhhtrdgtohhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 24 Jan 2022 13:00:10 -0500 (EST) From: Zi Yan To: Mel Gorman , linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, David Hildenbrand , Vlastimil Babka , Mike Rapoport , Andrew Morton , Zi Yan Subject: [PATCH] mm: page_alloc: avoid merging non-fallbackable pageblocks with others. Date: Mon, 24 Jan 2022 12:59:57 -0500 Message-Id: <20220124175957.1261961-1-zi.yan@sent.com> X-Mailer: git-send-email 2.34.1 Reply-To: Zi Yan MIME-Version: 1.0 X-Stat-Signature: zgpauj15y13pe1cdu7wros8twmasfp4h Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=UDVNR59F; dkim=pass header.d=messagingengine.com header.s=fm1 header.b="d2s/Uswa"; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf03.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.28 as permitted sender) smtp.mailfrom=zi.yan@sent.com X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 09AEA20091 X-HE-Tag: 1643047211-141576 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan This is done in addition to MIGRATE_ISOLATE pageblock merge avoidance. It prepares for the upcoming removal of the MAX_ORDER-1 alignment requirement for CMA and alloc_contig_range(). MIGRARTE_HIGHATOMIC should not merge with other migratetypes like MIGRATE_ISOLATE and MIGRARTE_CMA[1], so this commit prevents that too. Remove MIGRARTE_CMA and MIGRATE_ISOLATE from fallbacks list, since they are never used. [1] https://lore.kernel.org/linux-mm/20211130100853.GP3366@techsingularity.net/ Signed-off-by: Zi Yan Acked-by: Mel Gorman Acked-by: David Hildenbrand Reviewed-by: Oscar Salvador Acked-by: Mike Rapoport Acked-by: Vlastimil Babka --- include/linux/mmzone.h | 11 +++++++++++ mm/page_alloc.c | 44 ++++++++++++++++++++---------------------- 2 files changed, 32 insertions(+), 23 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index aed44e9b5d89..71b77aab748d 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -83,6 +83,17 @@ static inline bool is_migrate_movable(int mt) return is_migrate_cma(mt) || mt == MIGRATE_MOVABLE; } +/* + * Check whether a migratetype can be merged with another migratetype. + * + * It is only mergeable when it can fall back to other migratetypes for + * allocation. See fallbacks[MIGRATE_TYPES][3] in page_alloc.c. + */ +static inline bool migratetype_is_mergeable(int mt) +{ + return mt < MIGRATE_PCPTYPES; +} + #define for_each_migratetype_order(order, type) \ for (order = 0; order < MAX_ORDER; order++) \ for (type = 0; type < MIGRATE_TYPES; type++) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8dd6399bafb5..5a91d4d9279a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1117,25 +1117,24 @@ static inline void __free_one_page(struct page *page, } if (order < MAX_ORDER - 1) { /* If we are here, it means order is >= pageblock_order. - * We want to prevent merge between freepages on isolate - * pageblock and normal pageblock. Without this, pageblock - * isolation could cause incorrect freepage or CMA accounting. + * We want to prevent merge between freepages on pageblock + * without fallbacks and normal pageblock. Without this, + * pageblock isolation could cause incorrect freepage or CMA + * accounting or HIGHATOMIC accounting. * * We don't want to hit this code for the more frequent * low-order merging. */ - if (unlikely(has_isolate_pageblock(zone))) { - int buddy_mt; + int buddy_mt; - buddy_pfn = __find_buddy_pfn(pfn, order); - buddy = page + (buddy_pfn - pfn); - buddy_mt = get_pageblock_migratetype(buddy); + buddy_pfn = __find_buddy_pfn(pfn, order); + buddy = page + (buddy_pfn - pfn); + buddy_mt = get_pageblock_migratetype(buddy); - if (migratetype != buddy_mt - && (is_migrate_isolate(migratetype) || - is_migrate_isolate(buddy_mt))) - goto done_merging; - } + if (migratetype != buddy_mt + && (!migratetype_is_mergeable(migratetype) || + !migratetype_is_mergeable(buddy_mt))) + goto done_merging; max_order = order + 1; goto continue_merging; } @@ -2479,17 +2478,13 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, /* * This array describes the order lists are fallen back to when * the free lists for the desirable migrate type are depleted + * + * The other migratetypes do not have fallbacks. */ static int fallbacks[MIGRATE_TYPES][3] = { [MIGRATE_UNMOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, MIGRATE_TYPES }, [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_TYPES }, [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_TYPES }, -#ifdef CONFIG_CMA - [MIGRATE_CMA] = { MIGRATE_TYPES }, /* Never used */ -#endif -#ifdef CONFIG_MEMORY_ISOLATION - [MIGRATE_ISOLATE] = { MIGRATE_TYPES }, /* Never used */ -#endif }; #ifdef CONFIG_CMA @@ -2795,8 +2790,8 @@ static void reserve_highatomic_pageblock(struct page *page, struct zone *zone, /* Yoink! */ mt = get_pageblock_migratetype(page); - if (!is_migrate_highatomic(mt) && !is_migrate_isolate(mt) - && !is_migrate_cma(mt)) { + /* Only reserve normal pageblocks (i.e., they can merge with others) */ + if (migratetype_is_mergeable(mt)) { zone->nr_reserved_highatomic += pageblock_nr_pages; set_pageblock_migratetype(page, MIGRATE_HIGHATOMIC); move_freepages_block(zone, page, MIGRATE_HIGHATOMIC, NULL); @@ -3545,8 +3540,11 @@ int __isolate_free_page(struct page *page, unsigned int order) struct page *endpage = page + (1 << order) - 1; for (; page < endpage; page += pageblock_nr_pages) { int mt = get_pageblock_migratetype(page); - if (!is_migrate_isolate(mt) && !is_migrate_cma(mt) - && !is_migrate_highatomic(mt)) + /* + * Only change normal pageblocks (i.e., they can merge + * with others) + */ + if (migratetype_is_mergeable(mt)) set_pageblock_migratetype(page, MIGRATE_MOVABLE); }