From patchwork Tue Feb 15 14:51:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 12747203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08FB2C433F5 for ; Tue, 15 Feb 2022 14:51:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9835B6B0080; Tue, 15 Feb 2022 09:51:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 932B06B0081; Tue, 15 Feb 2022 09:51:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7FA696B0082; Tue, 15 Feb 2022 09:51:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id 713A76B0080 for ; Tue, 15 Feb 2022 09:51:55 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 415206205D for ; Tue, 15 Feb 2022 14:51:55 +0000 (UTC) X-FDA: 79145303790.04.AE98DDC Received: from outbound-smtp18.blacknight.com (outbound-smtp18.blacknight.com [46.22.139.245]) by imf15.hostedemail.com (Postfix) with ESMTP id 87904A0009 for ; Tue, 15 Feb 2022 14:51:54 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail04.blacknight.ie [81.17.254.17]) by outbound-smtp18.blacknight.com (Postfix) with ESMTPS id 385171C3825 for ; Tue, 15 Feb 2022 14:51:53 +0000 (GMT) Received: (qmail 15449 invoked from network); 15 Feb 2022 14:51:53 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.17.223]) by 81.17.254.9 with ESMTPA; 15 Feb 2022 14:51:52 -0000 From: Mel Gorman To: Andrew Morton Cc: Aaron Lu , Dave Hansen , Vlastimil Babka , Michal Hocko , Jesper Dangaard Brouer , LKML , Linux-MM , Mel Gorman Subject: [PATCH 3/5] mm/page_alloc: Simplify how many pages are selected per pcp list during bulk free Date: Tue, 15 Feb 2022 14:51:09 +0000 Message-Id: <20220215145111.27082-4-mgorman@techsingularity.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220215145111.27082-1-mgorman@techsingularity.net> References: <20220215145111.27082-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 87904A0009 X-Stat-Signature: wkkmhs3eipar6cuyr8ys4ftsu9tzm57o Authentication-Results: imf15.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf15.hostedemail.com: domain of mgorman@techsingularity.net designates 46.22.139.245 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-HE-Tag: 1644936714-860891 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_pcppages_bulk() selects pages to free by round-robining between lists. Originally this was to evenly shrink pages by migratetype but uneven freeing is inevitable due to high pages. Simplify list selection by starting with a list that definitely has pages on it in free_unref_page_commit() and for drain, it does not matter where draining starts as all pages are removed. Signed-off-by: Mel Gorman Reviewed-by: Vlastimil Babka --- mm/page_alloc.c | 34 +++++++++++----------------------- 1 file changed, 11 insertions(+), 23 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c5110fdeb115..5e8c7cbe7a41 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1447,13 +1447,11 @@ static inline void prefetch_buddy(struct page *page, unsigned int order) * count is the number of pages to free. */ static void free_pcppages_bulk(struct zone *zone, int count, - struct per_cpu_pages *pcp) + struct per_cpu_pages *pcp, + int pindex) { - int pindex = 0; int min_pindex = 0; int max_pindex = NR_PCP_LISTS - 1; - int batch_free = 0; - int nr_freed = 0; unsigned int order; int prefetch_nr = READ_ONCE(pcp->batch); bool isolated_pageblocks; @@ -1467,16 +1465,10 @@ static void free_pcppages_bulk(struct zone *zone, int count, count = min(pcp->count, count); while (count > 0) { struct list_head *list; + int nr_pages; - /* - * Remove pages from lists in a round-robin fashion. A - * batch_free count is maintained that is incremented when an - * empty list is encountered. This is so more pages are freed - * off fuller lists instead of spinning excessively around empty - * lists - */ + /* Remove pages from lists in a round-robin fashion. */ do { - batch_free++; if (++pindex == NR_PCP_LISTS) pindex = 0; list = &pcp->lists[pindex]; @@ -1489,18 +1481,15 @@ static void free_pcppages_bulk(struct zone *zone, int count, min_pindex++; } while (1); - /* This is the only non-empty list. Free them all. */ - if (batch_free >= max_pindex - min_pindex) - batch_free = count; - order = pindex_to_order(pindex); + nr_pages = 1 << order; BUILD_BUG_ON(MAX_ORDER >= (1<lru); - nr_freed += 1 << order; - count -= 1 << order; + count -= nr_pages; + pcp->count -= nr_pages; if (bulkfree_pcp_prepare(page)) continue; @@ -1524,9 +1513,8 @@ static void free_pcppages_bulk(struct zone *zone, int count, prefetch_buddy(page, order); prefetch_nr--; } - } while (count > 0 && --batch_free && !list_empty(list)); + } while (count > 0 && !list_empty(list)); } - pcp->count -= nr_freed; /* * local_lock_irq held so equivalent to spin_lock_irqsave for @@ -3095,7 +3083,7 @@ void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp) batch = READ_ONCE(pcp->batch); to_drain = min(pcp->count, batch); if (to_drain > 0) - free_pcppages_bulk(zone, to_drain, pcp); + free_pcppages_bulk(zone, to_drain, pcp, 0); local_unlock_irqrestore(&pagesets.lock, flags); } #endif @@ -3116,7 +3104,7 @@ static void drain_pages_zone(unsigned int cpu, struct zone *zone) pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); if (pcp->count) - free_pcppages_bulk(zone, pcp->count, pcp); + free_pcppages_bulk(zone, pcp->count, pcp, 0); local_unlock_irqrestore(&pagesets.lock, flags); } @@ -3397,7 +3385,7 @@ static void free_unref_page_commit(struct page *page, unsigned long pfn, if (pcp->count >= high) { int batch = READ_ONCE(pcp->batch); - free_pcppages_bulk(zone, nr_pcp_free(pcp, high, batch), pcp); + free_pcppages_bulk(zone, nr_pcp_free(pcp, high, batch), pcp, pindex); } }