From patchwork Mon Oct 21 09:48:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11201705 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5AB1F1599 for ; Mon, 21 Oct 2019 09:48:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3256220830 for ; Mon, 21 Oct 2019 09:48:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3256220830 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 32E406B0007; Mon, 21 Oct 2019 05:48:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 12A286B000C; Mon, 21 Oct 2019 05:48:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EBD5B6B0007; Mon, 21 Oct 2019 05:48:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0095.hostedemail.com [216.40.44.95]) by kanga.kvack.org (Postfix) with ESMTP id B9B5D6B0008 for ; Mon, 21 Oct 2019 05:48:13 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id 4F5B98249980 for ; Mon, 21 Oct 2019 09:48:13 +0000 (UTC) X-FDA: 76067316066.10.key74_2a9942ee59725 X-Spam-Summary: 2,0,0,ea6bfc050a789281,d41d8cd98f00b204,mgorman@techsingularity.net,:akpm@linux-foundation.org:mhocko@suse.com:vbabka@suse.cz:tglx@linutronix.de:matt@codeblueprint.co.uk:bp@alien8.de::linux-kernel@vger.kernel.org:mgorman@techsingularity.net,RULES_HIT:41:355:379:421:541:800:960:966:973:988:989:1260:1345:1359:1437:1534:1543:1711:1730:1747:1777:1792:1801:2196:2199:2393:2559:2562:2693:2895:3138:3139:3140:3141:3142:3355:3740:3865:3866:3867:3868:3870:3871:3872:3874:4250:4321:4385:4605:5007:6119:6261:7774:7875:7903:8603:8634:8666:9010:9121:10004:11026:11537:11658:11914:12043:12297:12438:12517:12519:12522:12555:12895:13151:13161:13228:13229:13255:14096:14181:14394:14721:14819:21080:21222:21324:21451:21627:21740:21795:21809:30034:30051:30054:30060,0,RBL:81.17.249.16:@techsingularity.net:.lbl8.mailshell.net-62.2.114.100 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fs,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:27,LUA_S UMMARY:n X-HE-Tag: key74_2a9942ee59725 X-Filterd-Recvd-Size: 4842 Received: from outbound-smtp03.blacknight.com (outbound-smtp03.blacknight.com [81.17.249.16]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Mon, 21 Oct 2019 09:48:12 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp03.blacknight.com (Postfix) with ESMTPS id 986F298905 for ; Mon, 21 Oct 2019 10:48:10 +0100 (IST) Received: (qmail 32579 invoked from network); 21 Oct 2019 09:48:10 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.19.210]) by 81.17.254.9 with ESMTPA; 21 Oct 2019 09:48:10 -0000 From: Mel Gorman To: Andrew Morton Cc: Michal Hocko , Vlastimil Babka , Thomas Gleixner , Matt Fleming , Borislav Petkov , Linux-MM , Linux Kernel Mailing List , Mel Gorman Subject: [PATCH 1/3] mm, meminit: Recalculate pcpu batch and high limits after init completes Date: Mon, 21 Oct 2019 10:48:06 +0100 Message-Id: <20191021094808.28824-2-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20191021094808.28824-1-mgorman@techsingularity.net> References: <20191021094808.28824-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Deferred memory initialisation updates zone->managed_pages during the initialisation phase but before that finishes, the per-cpu page allocator (pcpu) calculates the number of pages allocated/freed in batches as well as the maximum number of pages allowed on a per-cpu list. As zone->managed_pages is not up to date yet, the pcpu initialisation calculates inappropriately low batch and high values. This increases zone lock contention quite severely in some cases with the degree of severity depending on how many CPUs share a local zone and the size of the zone. A private report indicated that kernel build times were excessive with extremely high system CPU usage. A perf profile indicated that a large chunk of time was lost on zone->lock contention. This patch recalculates the pcpu batch and high values after deferred initialisation completes for every populated zone in the system. It was tested on a 2-socket AMD EPYC 2 machine using a kernel compilation workload -- allmodconfig and all available CPUs. mmtests configuration: config-workload-kernbench-max Configuration was modified to build on a fresh XFS partition. kernbench 5.4.0-rc3 5.4.0-rc3 vanilla resetpcpu-v2 Amean user-256 13249.50 ( 0.00%) 16401.31 * -23.79%* Amean syst-256 14760.30 ( 0.00%) 4448.39 * 69.86%* Amean elsp-256 162.42 ( 0.00%) 119.13 * 26.65%* Stddev user-256 42.97 ( 0.00%) 19.15 ( 55.43%) Stddev syst-256 336.87 ( 0.00%) 6.71 ( 98.01%) Stddev elsp-256 2.46 ( 0.00%) 0.39 ( 84.03%) 5.4.0-rc3 5.4.0-rc3 vanilla resetpcpu-v2 Duration User 39766.24 49221.79 Duration System 44298.10 13361.67 Duration Elapsed 519.11 388.87 The patch reduces system CPU usage by 69.86% and total build time by 26.65%. The variance of system CPU usage is also much reduced. Before, this was the breakdown of batch and high values over all zones was. 256 batch: 1 256 batch: 63 512 batch: 7 256 high: 0 256 high: 378 512 high: 42 512 pcpu pagesets had a batch limit of 7 and a high limit of 42. After the patch 256 batch: 1 768 batch: 63 256 high: 0 768 high: 378 Cc: stable@vger.kernel.org # v4.1+ Signed-off-by: Mel Gorman Acked-by: Michal Hocko Acked-by: Vlastimil Babka --- mm/page_alloc.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c0b2e0306720..f972076d0f6b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1947,6 +1947,14 @@ void __init page_alloc_init_late(void) /* Block until all are initialised */ wait_for_completion(&pgdat_init_all_done_comp); + /* + * The number of managed pages has changed due to the initialisation + * so the pcpu batch and high limits needs to be updated or the limits + * will be artificially small. + */ + for_each_populated_zone(zone) + zone_pcp_update(zone); + /* * We initialized the rest of the deferred pages. Permanently disable * on-demand struct page initialization. From patchwork Mon Oct 21 09:48:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11201703 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 730981390 for ; Mon, 21 Oct 2019 09:48:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4A63F21721 for ; Mon, 21 Oct 2019 09:48:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4A63F21721 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EBF366B000D; Mon, 21 Oct 2019 05:48:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E50A06B000A; Mon, 21 Oct 2019 05:48:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D10256B000C; Mon, 21 Oct 2019 05:48:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0122.hostedemail.com [216.40.44.122]) by kanga.kvack.org (Postfix) with ESMTP id AC8666B0007 for ; Mon, 21 Oct 2019 05:48:13 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id 52668181AC9CC for ; Mon, 21 Oct 2019 09:48:13 +0000 (UTC) X-FDA: 76067316066.14.burn20_2a99f3267365c X-Spam-Summary: 2,0,0,0bbef80688e5e552,d41d8cd98f00b204,mgorman@techsingularity.net,:akpm@linux-foundation.org:mhocko@suse.com:vbabka@suse.cz:tglx@linutronix.de:matt@codeblueprint.co.uk:bp@alien8.de::linux-kernel@vger.kernel.org:mgorman@techsingularity.net,RULES_HIT:41:69:355:379:541:800:960:966:973:988:989:1260:1345:1359:1437:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:3874:4321:4385:5007:6261:7875:8634:9592:10004:11026:11537:11658:11914:12043:12296:12297:12438:12517:12519:12522:12555:12679:12895:12986:13069:13255:13311:13357:14096:14181:14384:14394:14721:21080:21222:21451:21627:30012:30054:30060:30070,0,RBL:81.17.249.38:@techsingularity.net:.lbl8.mailshell.net-62.2.114.100 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fs,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: burn20_2a99f3267365c X-Filterd-Recvd-Size: 3385 Received: from outbound-smtp05.blacknight.com (outbound-smtp05.blacknight.com [81.17.249.38]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Mon, 21 Oct 2019 09:48:12 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp05.blacknight.com (Postfix) with ESMTPS id 49B9198920 for ; Mon, 21 Oct 2019 10:48:11 +0100 (IST) Received: (qmail 32608 invoked from network); 21 Oct 2019 09:48:11 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.19.210]) by 81.17.254.9 with ESMTPA; 21 Oct 2019 09:48:10 -0000 From: Mel Gorman To: Andrew Morton Cc: Michal Hocko , Vlastimil Babka , Thomas Gleixner , Matt Fleming , Borislav Petkov , Linux-MM , Linux Kernel Mailing List , Mel Gorman Subject: [PATCH 2/3] mm, pcp: Share common code between memory hotplug and percpu sysctl handler Date: Mon, 21 Oct 2019 10:48:07 +0100 Message-Id: <20191021094808.28824-3-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20191021094808.28824-1-mgorman@techsingularity.net> References: <20191021094808.28824-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Both the percpu_pagelist_fraction sysctl handler and memory hotplug have a common requirement of updating the pcpu page allocation batch and high values. Split the relevant helper to share common code. No functional change. Signed-off-by: Mel Gorman Acked-by: Michal Hocko Acked-by: Vlastimil Babka --- mm/page_alloc.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f972076d0f6b..4179376bb336 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7991,6 +7991,15 @@ int lowmem_reserve_ratio_sysctl_handler(struct ctl_table *table, int write, return 0; } +static void __zone_pcp_update(struct zone *zone) +{ + unsigned int cpu; + + for_each_possible_cpu(cpu) + pageset_set_high_and_batch(zone, + per_cpu_ptr(zone->pageset, cpu)); +} + /* * percpu_pagelist_fraction - changes the pcp->high for each zone on each * cpu. It is the fraction of total pages in each zone that a hot per cpu @@ -8022,13 +8031,8 @@ int percpu_pagelist_fraction_sysctl_handler(struct ctl_table *table, int write, if (percpu_pagelist_fraction == old_percpu_pagelist_fraction) goto out; - for_each_populated_zone(zone) { - unsigned int cpu; - - for_each_possible_cpu(cpu) - pageset_set_high_and_batch(zone, - per_cpu_ptr(zone->pageset, cpu)); - } + for_each_populated_zone(zone) + __zone_pcp_update(zone); out: mutex_unlock(&pcp_batch_high_lock); return ret; @@ -8527,11 +8531,8 @@ void free_contig_range(unsigned long pfn, unsigned int nr_pages) */ void __meminit zone_pcp_update(struct zone *zone) { - unsigned cpu; mutex_lock(&pcp_batch_high_lock); - for_each_possible_cpu(cpu) - pageset_set_high_and_batch(zone, - per_cpu_ptr(zone->pageset, cpu)); + __zone_pcp_update(zone); mutex_unlock(&pcp_batch_high_lock); } #endif From patchwork Mon Oct 21 09:48:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11201707 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4A28D1515 for ; Mon, 21 Oct 2019 09:48:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 115EA20830 for ; Mon, 21 Oct 2019 09:48:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 115EA20830 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C4F126B0008; Mon, 21 Oct 2019 05:48:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BB0B66B000A; Mon, 21 Oct 2019 05:48:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A79486B000C; Mon, 21 Oct 2019 05:48:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0145.hostedemail.com [216.40.44.145]) by kanga.kvack.org (Postfix) with ESMTP id 6794C6B0008 for ; Mon, 21 Oct 2019 05:48:14 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id C64DB180ABF75 for ; Mon, 21 Oct 2019 09:48:13 +0000 (UTC) X-FDA: 76067316066.16.bread54_2aaf0cbc1735e X-Spam-Summary: 2,0,0,5e0bcd7d6fd791a2,d41d8cd98f00b204,mgorman@techsingularity.net,:akpm@linux-foundation.org:mhocko@suse.com:vbabka@suse.cz:tglx@linutronix.de:matt@codeblueprint.co.uk:bp@alien8.de::linux-kernel@vger.kernel.org:mgorman@techsingularity.net,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1345:1359:1437:1534:1541:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2731:3138:3139:3140:3141:3142:3352:3865:3867:3868:3871:4321:4385:4605:5007:6120:6261:7901:8634:9592:10004:11026:11537:11658:11914:12296:12297:12438:12517:12519:12522:12555:12895:12986:13069:13161:13229:13311:13357:14096:14181:14384:14394:14721:21080:21451:21627:30054:30060:30070,0,RBL:46.22.139.14:@techsingularity.net:.lbl8.mailshell.net-62.2.114.100 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fs,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: bread54_2aaf0cbc1735e X-Filterd-Recvd-Size: 2767 Received: from outbound-smtp09.blacknight.com (outbound-smtp09.blacknight.com [46.22.139.14]) by imf46.hostedemail.com (Postfix) with ESMTP for ; Mon, 21 Oct 2019 09:48:13 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp09.blacknight.com (Postfix) with ESMTPS id AB18C1C285C for ; Mon, 21 Oct 2019 10:48:11 +0100 (IST) Received: (qmail 32640 invoked from network); 21 Oct 2019 09:48:11 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.19.210]) by 81.17.254.9 with ESMTPA; 21 Oct 2019 09:48:11 -0000 From: Mel Gorman To: Andrew Morton Cc: Michal Hocko , Vlastimil Babka , Thomas Gleixner , Matt Fleming , Borislav Petkov , Linux-MM , Linux Kernel Mailing List , Mel Gorman Subject: [PATCH 3/3] mm, pcpu: Make zone pcp updates and reset internal to the mm Date: Mon, 21 Oct 2019 10:48:08 +0100 Message-Id: <20191021094808.28824-4-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20191021094808.28824-1-mgorman@techsingularity.net> References: <20191021094808.28824-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Memory hotplug needs to be able to reset and reinit the pcpu allocator batch and high limits but this action is internal to the VM. Move the declaration to internal.h Signed-off-by: Mel Gorman Acked-by: Michal Hocko Acked-by: Vlastimil Babka --- include/linux/mm.h | 3 --- mm/internal.h | 3 +++ 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index cc292273e6ba..22d6104f2341 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2219,9 +2219,6 @@ void warn_alloc(gfp_t gfp_mask, nodemask_t *nodemask, const char *fmt, ...); extern void setup_per_cpu_pageset(void); -extern void zone_pcp_update(struct zone *zone); -extern void zone_pcp_reset(struct zone *zone); - /* page_alloc.c */ extern int min_free_kbytes; extern int watermark_boost_factor; diff --git a/mm/internal.h b/mm/internal.h index 0d5f720c75ab..0a3d41c7b3c5 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -165,6 +165,9 @@ extern void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags); extern int user_min_free_kbytes; +extern void zone_pcp_update(struct zone *zone); +extern void zone_pcp_reset(struct zone *zone); + #if defined CONFIG_COMPACTION || defined CONFIG_CMA /*