From patchwork Fri Jun 26 21:23:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Chen X-Patchwork-Id: 11628945 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9E4F7913 for ; Fri, 26 Jun 2020 22:20:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6B6D3206BE for ; Fri, 26 Jun 2020 22:20:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6B6D3206BE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 78AB36B0003; Fri, 26 Jun 2020 18:20:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 714346B0005; Fri, 26 Jun 2020 18:20:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5DBB56B0006; Fri, 26 Jun 2020 18:20:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 429606B0003 for ; Fri, 26 Jun 2020 18:20:53 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 076D8181AC9CC for ; Fri, 26 Jun 2020 22:20:53 +0000 (UTC) X-FDA: 76972783986.03.table66_2109d9a26e59 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id D0D5528A4E8 for ; Fri, 26 Jun 2020 22:20:52 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,tim.c.chen@linux.intel.com,,RULES_HIT:30054:30056,0,RBL:192.55.52.43:@linux.intel.com:.lbl8.mailshell.net-62.18.0.100 64.95.201.95;04y8xid74mcyu6synkmk3cqo1q75aopwcwag9ybjifjzcdicd1o5paaj6iqspsk.nrqixgafgkiwopu79z5uq8kkycrs6u68fdqxpzqmfbhim1y4w9n5o6ptenkqn41.n-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: table66_2109d9a26e59 X-Filterd-Recvd-Size: 3956 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Fri, 26 Jun 2020 22:20:52 +0000 (UTC) IronPort-SDR: 9GvmpfKKYkLEWFn4QgS1vcdl0C1S7STdbu0HGSHSMUbLLSt8oiAIorH0tN5BGZUGe4pdFz2XZE bsaRTSyRceVg== X-IronPort-AV: E=McAfee;i="6000,8403,9664"; a="230273454" X-IronPort-AV: E=Sophos;i="5.75,285,1589266800"; d="scan'208";a="230273454" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Jun 2020 15:20:50 -0700 IronPort-SDR: ZRK/Scww/abTcQTZCtwAphfFzhtSJkBRgtZAxUb2dx6iS2Om0bFchbahkgMl53kPUpZvPDt6/8 HI26odnZzifw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,285,1589266800"; d="scan'208";a="480143426" Received: from skl-02.jf.intel.com ([10.54.74.28]) by fmsmga005.fm.intel.com with ESMTP; 26 Jun 2020 15:20:50 -0700 From: Tim Chen To: Andrew Morton , Vladimir Davydov , Johannes Weiner , Michal Hocko Cc: Tim Chen , Dave Hansen , Ying Huang , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [Patch] mm: Increase pagevec size on large system Date: Fri, 26 Jun 2020 14:23:03 -0700 Message-Id: X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: D0D5528A4E8 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Pages to be added to a LRU are first cached in pagevec before being added to LRU in a batch via pagevec_lru_move_fn. By adding the pages in a batch with pagevec, the contention on LRU lock is mitigated. Currently the pagevec size is defined to be 15. We found during testing on a large SMT system with 2 sockets, 48 cores and 96 CPU threads, the pagevec size of 15 is too small for workload that caused frequent page additions to LRU. With pmbench, 8.9% of the CPU cycles are spent contending for the LRU lock. 12.92% pmbench [kernel.kallsyms] [k] queued_spin_lock_slowpath | --12.92%--0x5555555582f2 | --12.92%--page_fault do_page_fault __do_page_fault handle_mm_fault __handle_mm_fault | |--8.90%--__lru_cache_add | pagevec_lru_move_fn | | | --8.90%--_raw_spin_lock_irqsave | queued_spin_lock_slowpat Enlarge the pagevec size to 31 to reduce LRU lock contention for large systems. The LRU lock contention is reduced from 8.9% of total CPU cycles to 2.2% of CPU cyles. And the pmbench throughput increases from 88.8 Mpages/sec to 95.1 Mpages/sec. Signed-off-by: Tim Chen --- include/linux/pagevec.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h index 081d934eda64..466ebcdd190d 100644 --- a/include/linux/pagevec.h +++ b/include/linux/pagevec.h @@ -11,8 +11,16 @@ #include +#if CONFIG_NR_CPUS > 64 +/* + * Use larger size to reduce lru lock contention on large system. + * 31 pointers + header align the pagevec structure to a power of two + */ +#define PAGEVEC_SIZE 31 +#else /* 15 pointers + header align the pagevec structure to a power of two */ #define PAGEVEC_SIZE 15 +#endif struct page; struct address_space;