From patchwork Fri May 8 07:25:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feng Tang X-Patchwork-Id: 11535807 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9FE4492A for ; Fri, 8 May 2020 07:25:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 83CF720736 for ; Fri, 8 May 2020 07:25:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 83CF720736 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 28CDB900004; Fri, 8 May 2020 03:25:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 23ED2900002; Fri, 8 May 2020 03:25:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17B00900004; Fri, 8 May 2020 03:25:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0002.hostedemail.com [216.40.44.2]) by kanga.kvack.org (Postfix) with ESMTP id 015E9900002 for ; Fri, 8 May 2020 03:25:27 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id B1DA81F08 for ; Fri, 8 May 2020 07:25:27 +0000 (UTC) X-FDA: 76792716294.18.paper84_69bac43537d0e X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,feng.tang@intel.com,,RULES_HIT:30054:30064:30070,0,RBL:192.55.52.115:@intel.com:.lbl8.mailshell.net-62.18.0.100 64.95.201.95,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: paper84_69bac43537d0e X-Filterd-Recvd-Size: 2437 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Fri, 8 May 2020 07:25:26 +0000 (UTC) IronPort-SDR: A9FymHDiGbdUPs3oiEaS09UU95Totrb8xOZkLBiqlIJRLxuH9NtULBv1bFKCPqmBDpnQjiLEe8 kQDsdHNPHNZg== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2020 00:25:26 -0700 IronPort-SDR: BaPIIrbWpE7bV+c1ngNAq+lL2CKpewOcYZUiU9wU+ozDW06TmCvkmthWhmF308Lk3l4El1JixD 4YhgLc6czNIQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,366,1583222400"; d="scan'208";a="264210532" Received: from shbuild999.sh.intel.com ([10.239.147.136]) by orsmga006.jf.intel.com with ESMTP; 08 May 2020 00:25:22 -0700 From: Feng Tang To: Andrew Morton , Michal Hocko , Johannes Weiner , Matthew Wilcox , Mel Gorman , Kees Cook , Luis Chamberlain , Iurii Zaikin , andi.kleen@intel.com, tim.c.chen@intel.com, dave.hansen@intel.com, ying.huang@intel.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Feng Tang Subject: [PATCH 1/3] proc/meminfo: avoid open coded reading of vm_committed_as Date: Fri, 8 May 2020 15:25:15 +0800 Message-Id: <1588922717-63697-2-git-send-email-feng.tang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1588922717-63697-1-git-send-email-feng.tang@intel.com> References: <1588922717-63697-1-git-send-email-feng.tang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the existing vm_memory_committed() instead, which is also convenient for future change. Signed-off-by: Feng Tang Acked-by: Michal Hocko --- fs/proc/meminfo.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c index 8c1f1bb..578c0b8 100644 --- a/fs/proc/meminfo.c +++ b/fs/proc/meminfo.c @@ -42,7 +42,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v) si_meminfo(&i); si_swapinfo(&i); - committed = percpu_counter_read_positive(&vm_committed_as); + committed = vm_memory_committed(); cached = global_node_page_state(NR_FILE_PAGES) - total_swapcache_pages() - i.bufferram; From patchwork Fri May 8 07:25:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feng Tang X-Patchwork-Id: 11535809 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7853F15AB for ; Fri, 8 May 2020 07:25:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4F428208D6 for ; Fri, 8 May 2020 07:25:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4F428208D6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 76F02900005; Fri, 8 May 2020 03:25:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7476B900002; Fri, 8 May 2020 03:25:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65DD2900005; Fri, 8 May 2020 03:25:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0159.hostedemail.com [216.40.44.159]) by kanga.kvack.org (Postfix) with ESMTP id 4609C900002 for ; Fri, 8 May 2020 03:25:32 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 0DBD44995E3 for ; Fri, 8 May 2020 07:25:32 +0000 (UTC) X-FDA: 76792716504.23.lead87_6a4f2ae6edb16 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,feng.tang@intel.com,,RULES_HIT:30046:30054:30064:30075,0,RBL:192.55.52.115:@intel.com:.lbl8.mailshell.net-64.95.201.95 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: lead87_6a4f2ae6edb16 X-Filterd-Recvd-Size: 2709 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Fri, 8 May 2020 07:25:30 +0000 (UTC) IronPort-SDR: QlHmQIgtGyHByp5btHqZhTZ/fT0FTavaAShfLS1hkqUqE7nOpg0sbWVBb6UVyRn2CJc8vdxzv7 WEDrODdj0gOA== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2020 00:25:30 -0700 IronPort-SDR: XTxoPgiIpK+yKm21wvSPK3ZDNzwkPuJQf+V+gxo+7RrjNFtdNo8zJdE2Rt+Y3Xk+SDBDr0k++H 2sAn7N1ZXtdw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,366,1583222400"; d="scan'208";a="264210556" Received: from shbuild999.sh.intel.com ([10.239.147.136]) by orsmga006.jf.intel.com with ESMTP; 08 May 2020 00:25:26 -0700 From: Feng Tang To: Andrew Morton , Michal Hocko , Johannes Weiner , Matthew Wilcox , Mel Gorman , Kees Cook , Luis Chamberlain , Iurii Zaikin , andi.kleen@intel.com, tim.c.chen@intel.com, dave.hansen@intel.com, ying.huang@intel.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Feng Tang Subject: [PATCH 2/3] mm/util.c: make vm_memory_committed() more accurate Date: Fri, 8 May 2020 15:25:16 +0800 Message-Id: <1588922717-63697-3-git-send-email-feng.tang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1588922717-63697-1-git-send-email-feng.tang@intel.com> References: <1588922717-63697-1-git-send-email-feng.tang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: percpu_counter_sum_positive() will provide more accurate info. Its time cost is about 800 nanoseconds on a 2C/4T platform and 2~3 microseconds on a 2S/36C/72T server in normal case, and in worst case where vm_committed_as's spinlock is under severe contention, it costs 30~40 microseconds for the 2S/36C/72T sever, which should be fine for its two users: /proc/meminfo and HV ballon driver's status trace per second. Signed-off-by: Feng Tang --- mm/util.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/util.c b/mm/util.c index 988d11e..3de78e9 100644 --- a/mm/util.c +++ b/mm/util.c @@ -774,7 +774,7 @@ struct percpu_counter vm_committed_as ____cacheline_aligned_in_smp; */ unsigned long vm_memory_committed(void) { - return percpu_counter_read_positive(&vm_committed_as); + return percpu_counter_sum_positive(&vm_committed_as); } EXPORT_SYMBOL_GPL(vm_memory_committed); From patchwork Fri May 8 07:25:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feng Tang X-Patchwork-Id: 11535811 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2E05E15AB for ; Fri, 8 May 2020 07:25:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EB71120736 for ; Fri, 8 May 2020 07:25:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EB71120736 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E99AA900006; Fri, 8 May 2020 03:25:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E4AEC900002; Fri, 8 May 2020 03:25:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D60EA900006; Fri, 8 May 2020 03:25:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0066.hostedemail.com [216.40.44.66]) by kanga.kvack.org (Postfix) with ESMTP id C0061900002 for ; Fri, 8 May 2020 03:25:35 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 770DA8248047 for ; Fri, 8 May 2020 07:25:35 +0000 (UTC) X-FDA: 76792716630.22.house42_6ae499b4e971c X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,feng.tang@intel.com,,RULES_HIT:30034:30051:30054:30064:30070,0,RBL:192.55.52.115:@intel.com:.lbl8.mailshell.net-62.18.0.100 64.95.201.95,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: house42_6ae499b4e971c X-Filterd-Recvd-Size: 6738 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Fri, 8 May 2020 07:25:34 +0000 (UTC) IronPort-SDR: VyesaUFCEVogo5rdl9Q/y0TeNx+e2D3uL+pTP2EY9eOLWQNl31xeEjC2Nj3H/7Ve8u/DprljQv YaEFfWzEZwvg== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2020 00:25:34 -0700 IronPort-SDR: /EANI/ZJmWkl7bx+3saQMpoxOdCPopKbVG3WLhHGTLeN3GmCsrNXocVaE8sZZUBI/7nST1nL0S mLkl5NLQugfQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,366,1583222400"; d="scan'208";a="264210569" Received: from shbuild999.sh.intel.com ([10.239.147.136]) by orsmga006.jf.intel.com with ESMTP; 08 May 2020 00:25:30 -0700 From: Feng Tang To: Andrew Morton , Michal Hocko , Johannes Weiner , Matthew Wilcox , Mel Gorman , Kees Cook , Luis Chamberlain , Iurii Zaikin , andi.kleen@intel.com, tim.c.chen@intel.com, dave.hansen@intel.com, ying.huang@intel.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Feng Tang Subject: [PATCH 3/3] mm: adjust vm_committed_as_batch according to vm overcommit policy Date: Fri, 8 May 2020 15:25:17 +0800 Message-Id: <1588922717-63697-4-git-send-email-feng.tang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1588922717-63697-1-git-send-email-feng.tang@intel.com> References: <1588922717-63697-1-git-send-email-feng.tang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When checking a performance change for will-it-scale scalability mmap test [1], we found very high lock contention for spinlock of percpu counter 'vm_committed_as': 94.14% 0.35% [kernel.kallsyms] [k] _raw_spin_lock_irqsave 48.21% _raw_spin_lock_irqsave;percpu_counter_add_batch;__vm_enough_memory;mmap_region;do_mmap; 45.91% _raw_spin_lock_irqsave;percpu_counter_add_batch;__do_munmap; Actually this heavy lock contention is not always necessary. The 'vm_committed_as' needs to be very precise when the strict OVERCOMMIT_NEVER policy is set, which requires a rather small batch number for the percpu counter. So lift the batch number to 16X for OVERCOMMIT_ALWAYS and OVERCOMMIT_GUESS policies, and add a sysctl handler to adjust it when the policy is reconfigured. Benchmark with the same testcase in [1] shows 53% improvement on a 8C/16T desktop, and 2097%(20X) on a 4S/72C/144T server. And no change for some platforms, due to the test mmap size of the case is bigger than the batch number computed, though the patch will help mmap/munmap generally. [1] https://lkml.org/lkml/2020/3/5/57 Signed-off-by: Feng Tang --- include/linux/mm.h | 2 ++ include/linux/mman.h | 4 ++++ kernel/sysctl.c | 2 +- mm/mm_init.c | 19 +++++++++++++++---- mm/util.c | 13 +++++++++++++ 5 files changed, 35 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 5a32342..bc3722f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -205,6 +205,8 @@ extern int overcommit_ratio_handler(struct ctl_table *, int, void __user *, size_t *, loff_t *); extern int overcommit_kbytes_handler(struct ctl_table *, int, void __user *, size_t *, loff_t *); +extern int overcommit_policy_handler(struct ctl_table *, int, void __user *, + size_t *, loff_t *); #define nth_page(page,n) pfn_to_page(page_to_pfn((page)) + (n)) diff --git a/include/linux/mman.h b/include/linux/mman.h index 4b08e9c..91c93c1 100644 --- a/include/linux/mman.h +++ b/include/linux/mman.h @@ -57,8 +57,12 @@ extern struct percpu_counter vm_committed_as; #ifdef CONFIG_SMP extern s32 vm_committed_as_batch; +extern void mm_compute_batch(void); #else #define vm_committed_as_batch 0 +static inline void mm_compute_batch(void) +{ +} #endif unsigned long vm_memory_committed(void); diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 8a176d8..6fa552d 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -1278,7 +1278,7 @@ static struct ctl_table vm_table[] = { .data = &sysctl_overcommit_memory, .maxlen = sizeof(sysctl_overcommit_memory), .mode = 0644, - .proc_handler = proc_dointvec_minmax, + .proc_handler = overcommit_policy_handler, .extra1 = SYSCTL_ZERO, .extra2 = &two, }, diff --git a/mm/mm_init.c b/mm/mm_init.c index 7da6991..1654358 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -13,6 +13,7 @@ #include #include #include +#include #include "internal.h" #ifdef CONFIG_DEBUG_MEMORY_INIT @@ -140,16 +141,26 @@ EXPORT_SYMBOL_GPL(mm_kobj); #ifdef CONFIG_SMP s32 vm_committed_as_batch = 32; -static void __meminit mm_compute_batch(void) +void mm_compute_batch(void) { u64 memsized_batch; s32 nr = num_present_cpus(); s32 batch = max_t(s32, nr*2, 32); - - /* batch size set to 0.4% of (total memory/#cpus), or max int32 */ - memsized_batch = min_t(u64, (totalram_pages()/nr)/256, 0x7fffffff); + unsigned long ram_pages = totalram_pages(); + + /* + * For policy of OVERCOMMIT_NEVER, set batch size to 0.4% + * of (total memory/#cpus), and lift it to 6.25% for other + * policies to easy the possible lock contention for percpu_counter + * vm_committed_as, while the max limit is INT_MAX + */ + if (sysctl_overcommit_memory == OVERCOMMIT_NEVER) + memsized_batch = min_t(u64, ram_pages/nr/256, INT_MAX); + else + memsized_batch = min_t(u64, ram_pages/nr/16, INT_MAX); vm_committed_as_batch = max_t(s32, memsized_batch, batch); + printk("vm_committed_as_batch = %d\n", vm_committed_as_batch); } static int __meminit mm_compute_batch_notifier(struct notifier_block *self, diff --git a/mm/util.c b/mm/util.c index 3de78e9..99936d3 100644 --- a/mm/util.c +++ b/mm/util.c @@ -729,6 +729,19 @@ int overcommit_ratio_handler(struct ctl_table *table, int write, return ret; } +int overcommit_policy_handler(struct ctl_table *table, int write, + void __user *buffer, size_t *lenp, + loff_t *ppos) +{ + int ret; + + ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos); + if (ret == 0 && write) + mm_compute_batch(); + + return ret; +} + int overcommit_kbytes_handler(struct ctl_table *table, int write, void __user *buffer, size_t *lenp, loff_t *ppos)