From patchwork Tue Mar 22 21:43:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 12789138 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60DB7C433F5 for ; Tue, 22 Mar 2022 21:43:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E61986B00E8; Tue, 22 Mar 2022 17:43:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E113E6B00E9; Tue, 22 Mar 2022 17:43:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB3786B00EA; Tue, 22 Mar 2022 17:43:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0009.hostedemail.com [216.40.44.9]) by kanga.kvack.org (Postfix) with ESMTP id B41E46B00E8 for ; Tue, 22 Mar 2022 17:43:36 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 6F02818265D22 for ; Tue, 22 Mar 2022 21:43:36 +0000 (UTC) X-FDA: 79273349232.26.F5594A9 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf04.hostedemail.com (Postfix) with ESMTP id EBB2C40032 for ; Tue, 22 Mar 2022 21:43:35 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id DB7CDB81DB3; Tue, 22 Mar 2022 21:43:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 98B0DC340F3; Tue, 22 Mar 2022 21:43:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1647985413; bh=6rFJhdM8DpdUmiWloX+OzeAm5fksq9BQ8wrsG4qfAa8=; h=Date:To:From:In-Reply-To:Subject:From; b=jWC7TeGPiqTuOVK4FvS7uYiFQn+RRjUWSjjmwX83IlZBrdfcs9usbJ3sOxSapLi3B 4u0Tuw855LHK3sHcP22fJJ6t5MlqWPn+vt9nQmJoysibiIMtQ4FLhLP6zctzB2tRbB wKYP8cUq7TmzqRBOqf1PiFtw+eJXTGtkk7gWRH7s= Date: Tue, 22 Mar 2022 14:43:33 -0700 To: vbabka@suse.cz,mhocko@kernel.org,dave.hansen@linux.intel.com,brouer@redhat.com,aaron.lu@intel.com,mgorman@techsingularity.net,akpm@linux-foundation.org,patches@lists.linux.dev,linux-mm@kvack.org,mm-commits@vger.kernel.org,torvalds@linux-foundation.org,akpm@linux-foundation.org From: Andrew Morton In-Reply-To: <20220322143803.04a5e59a07e48284f196a2f9@linux-foundation.org> Subject: [patch 099/227] mm/page_alloc: track range of active PCP lists during bulk free Message-Id: <20220322214333.98B0DC340F3@smtp.kernel.org> X-Rspam-User: X-Stat-Signature: ojre56k3yfyurwjfjstd7ypfnwk41pxy Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b=jWC7TeGP; spf=pass (imf04.hostedemail.com: domain of akpm@linux-foundation.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org; dmarc=none X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: EBB2C40032 X-HE-Tag: 1647985415-530114 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mel Gorman Subject: mm/page_alloc: track range of active PCP lists during bulk free free_pcppages_bulk() frees pages in a round-robin fashion. Originally, this was dealing only with migratetypes but storing high-order pages means that there can be many more empty lists that are uselessly checked. Track the minimum and maximum active pindex to reduce the search space. Link: https://lkml.kernel.org/r/20220217002227.5739-3-mgorman@techsingularity.net Signed-off-by: Mel Gorman Reviewed-by: Vlastimil Babka Tested-by: Aaron Lu Cc: Dave Hansen Cc: Jesper Dangaard Brouer Cc: Michal Hocko Signed-off-by: Andrew Morton --- mm/page_alloc.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) --- a/mm/page_alloc.c~mm-page_alloc-track-range-of-active-pcp-lists-during-bulk-free +++ a/mm/page_alloc.c @@ -1447,6 +1447,8 @@ static void free_pcppages_bulk(struct zo struct per_cpu_pages *pcp) { int pindex = 0; + int min_pindex = 0; + int max_pindex = NR_PCP_LISTS - 1; int batch_free = 0; int nr_freed = 0; unsigned int order; @@ -1472,13 +1474,20 @@ static void free_pcppages_bulk(struct zo */ do { batch_free++; - if (++pindex == NR_PCP_LISTS) - pindex = 0; + if (++pindex > max_pindex) + pindex = min_pindex; list = &pcp->lists[pindex]; - } while (list_empty(list)); + if (!list_empty(list)) + break; + + if (pindex == max_pindex) + max_pindex--; + if (pindex == min_pindex) + min_pindex++; + } while (1); /* This is the only non-empty list. Free them all. */ - if (batch_free == NR_PCP_LISTS) + if (batch_free >= max_pindex - min_pindex) batch_free = count; order = pindex_to_order(pindex);