From patchwork Tue Mar 22 21:43:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 12789139 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77D52C433F5 for ; Tue, 22 Mar 2022 21:43:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 11C1D6B00EA; Tue, 22 Mar 2022 17:43:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0A56F6B00EB; Tue, 22 Mar 2022 17:43:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED4AF6B00EC; Tue, 22 Mar 2022 17:43:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id DB8896B00EA for ; Tue, 22 Mar 2022 17:43:39 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B9DE722C37 for ; Tue, 22 Mar 2022 21:43:39 +0000 (UTC) X-FDA: 79273349358.02.7D6C669 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf02.hostedemail.com (Postfix) with ESMTP id 1EDEE80020 for ; Tue, 22 Mar 2022 21:43:38 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id DFABBB81D9E; Tue, 22 Mar 2022 21:43:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 94B20C340EC; Tue, 22 Mar 2022 21:43:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1647985416; bh=oTGU3v+FDaYpYNMTFZCrhPgnhnewJiycMKKjLxSu9UU=; h=Date:To:From:In-Reply-To:Subject:From; b=w23Cyq1+EgB/5f5QLoQy7WpBKbgeT+xcpWc9zaWc8A9DZ4uCvUSig2l54Hce2lU7p lxmVG1ncXorc9wcKMux3+u2AuNQip0ZNvR8xkrF3V9EsqpMsah0XAJqJSOxoKv5ejV bVPjb9TRKj9RgdRKBzTu+2D5zYxbl5EREcV+DNZw= Date: Tue, 22 Mar 2022 14:43:36 -0700 To: vbabka@suse.cz,mhocko@kernel.org,dave.hansen@linux.intel.com,brouer@redhat.com,aaron.lu@intel.com,mgorman@techsingularity.net,akpm@linux-foundation.org,patches@lists.linux.dev,linux-mm@kvack.org,mm-commits@vger.kernel.org,torvalds@linux-foundation.org,akpm@linux-foundation.org From: Andrew Morton In-Reply-To: <20220322143803.04a5e59a07e48284f196a2f9@linux-foundation.org> Subject: [patch 100/227] mm/page_alloc: simplify how many pages are selected per pcp list during bulk free Message-Id: <20220322214336.94B20C340EC@smtp.kernel.org> X-Stat-Signature: gafyg7i6acthp7rgo741biachbb9mhb5 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b=w23Cyq1+; spf=pass (imf02.hostedemail.com: domain of akpm@linux-foundation.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org; dmarc=none X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1EDEE80020 X-HE-Tag: 1647985418-497238 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mel Gorman Subject: mm/page_alloc: simplify how many pages are selected per pcp list during bulk free free_pcppages_bulk() selects pages to free by round-robining between lists. Originally this was to evenly shrink pages by migratetype but uneven freeing is inevitable due to high pages. Simplify list selection by starting with a list that definitely has pages on it in free_unref_page_commit() and for drain, it does not matter where draining starts as all pages are removed. Link: https://lkml.kernel.org/r/20220217002227.5739-4-mgorman@techsingularity.net Signed-off-by: Mel Gorman Reviewed-by: Vlastimil Babka Tested-by: Aaron Lu Cc: Dave Hansen Cc: Jesper Dangaard Brouer Cc: Michal Hocko Signed-off-by: Andrew Morton --- mm/page_alloc.c | 34 +++++++++++----------------------- 1 file changed, 11 insertions(+), 23 deletions(-) --- a/mm/page_alloc.c~mm-page_alloc-simplify-how-many-pages-are-selected-per-pcp-list-during-bulk-free +++ a/mm/page_alloc.c @@ -1444,13 +1444,11 @@ static inline void prefetch_buddy(struct * count is the number of pages to free. */ static void free_pcppages_bulk(struct zone *zone, int count, - struct per_cpu_pages *pcp) + struct per_cpu_pages *pcp, + int pindex) { - int pindex = 0; int min_pindex = 0; int max_pindex = NR_PCP_LISTS - 1; - int batch_free = 0; - int nr_freed = 0; unsigned int order; int prefetch_nr = READ_ONCE(pcp->batch); bool isolated_pageblocks; @@ -1464,16 +1462,10 @@ static void free_pcppages_bulk(struct zo count = min(pcp->count, count); while (count > 0) { struct list_head *list; + int nr_pages; - /* - * Remove pages from lists in a round-robin fashion. A - * batch_free count is maintained that is incremented when an - * empty list is encountered. This is so more pages are freed - * off fuller lists instead of spinning excessively around empty - * lists - */ + /* Remove pages from lists in a round-robin fashion. */ do { - batch_free++; if (++pindex > max_pindex) pindex = min_pindex; list = &pcp->lists[pindex]; @@ -1486,18 +1478,15 @@ static void free_pcppages_bulk(struct zo min_pindex++; } while (1); - /* This is the only non-empty list. Free them all. */ - if (batch_free >= max_pindex - min_pindex) - batch_free = count; - order = pindex_to_order(pindex); + nr_pages = 1 << order; BUILD_BUG_ON(MAX_ORDER >= (1<lru); - nr_freed += 1 << order; - count -= 1 << order; + count -= nr_pages; + pcp->count -= nr_pages; if (bulkfree_pcp_prepare(page)) continue; @@ -1521,9 +1510,8 @@ static void free_pcppages_bulk(struct zo prefetch_buddy(page, order); prefetch_nr--; } - } while (count > 0 && --batch_free && !list_empty(list)); + } while (count > 0 && !list_empty(list)); } - pcp->count -= nr_freed; /* * local_lock_irq held so equivalent to spin_lock_irqsave for @@ -3077,7 +3065,7 @@ void drain_zone_pages(struct zone *zone, batch = READ_ONCE(pcp->batch); to_drain = min(pcp->count, batch); if (to_drain > 0) - free_pcppages_bulk(zone, to_drain, pcp); + free_pcppages_bulk(zone, to_drain, pcp, 0); local_unlock_irqrestore(&pagesets.lock, flags); } #endif @@ -3098,7 +3086,7 @@ static void drain_pages_zone(unsigned in pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); if (pcp->count) - free_pcppages_bulk(zone, pcp->count, pcp); + free_pcppages_bulk(zone, pcp->count, pcp, 0); local_unlock_irqrestore(&pagesets.lock, flags); } @@ -3379,7 +3367,7 @@ static void free_unref_page_commit(struc if (pcp->count >= high) { int batch = READ_ONCE(pcp->batch); - free_pcppages_bulk(zone, nr_pcp_free(pcp, high, batch), pcp); + free_pcppages_bulk(zone, nr_pcp_free(pcp, high, batch), pcp, pindex); } }