From patchwork Thu Mar 11 00:44:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 12129969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3646CC433DB for ; Thu, 11 Mar 2021 00:45:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B2ED464FC4 for ; Thu, 11 Mar 2021 00:45:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B2ED464FC4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 347F08D0258; Wed, 10 Mar 2021 19:45:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F7AC8D0250; Wed, 10 Mar 2021 19:45:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 199488D0258; Wed, 10 Mar 2021 19:45:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0052.hostedemail.com [216.40.44.52]) by kanga.kvack.org (Postfix) with ESMTP id F34BB8D0250 for ; Wed, 10 Mar 2021 19:45:16 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 9A0A1181AF5C2 for ; Thu, 11 Mar 2021 00:45:16 +0000 (UTC) X-FDA: 77905749432.19.C6BA1E2 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf03.hostedemail.com (Postfix) with ESMTP id 26585C0007D1 for ; Thu, 11 Mar 2021 00:45:12 +0000 (UTC) IronPort-SDR: vCZEsuMe4ugkq8OgPOygFNlFy4zAQ3vczFkGGCIBmVUXOijFCAgEpWxBqdRNo3gLI2V5LqTg0G 4qPbKTSAv50g== X-IronPort-AV: E=McAfee;i="6000,8403,9919"; a="185234946" X-IronPort-AV: E=Sophos;i="5.81,238,1610438400"; d="scan'208";a="185234946" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Mar 2021 16:45:12 -0800 IronPort-SDR: Ps234FXqzH4qwxm4AKvRuzA80q4gQ8Wdd9S7lx4ZBji95xYi9ILFaMyZlRvDQqFfxdfEDPEo9s nRl5arLtZJtg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,238,1610438400"; d="scan'208";a="603321625" Received: from yhuang-dev.sh.intel.com ([10.239.159.130]) by fmsmga005.fm.intel.com with ESMTP; 10 Mar 2021 16:45:09 -0800 From: "Huang, Ying" To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Mel Gorman , Johannes Weiner , Vladimir Davydov , Michal Hocko , Joonsoo Kim Subject: [PATCH] vmscan: retry without cache trim mode if nothing scanned Date: Thu, 11 Mar 2021 08:44:49 +0800 Message-Id: <20210311004449.1170308-1-ying.huang@intel.com> X-Mailer: git-send-email 2.30.1 MIME-Version: 1.0 X-Stat-Signature: ngiok96ut83wkdxji793ydnck1ehygxp X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 26585C0007D1 Received-SPF: none (intel.com>: No applicable sender policy available) receiver=imf03; identity=mailfrom; envelope-from=""; helo=mga11.intel.com; client-ip=192.55.52.93 X-HE-DKIM-Result: none/none X-HE-Tag: 1615423512-723994 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Huang Ying In shrink_node(), to determine whether to enable cache trim mode, the LRU size is gotten via lruvec_page_state(). That gets the value from a per-CPU counter (mem_cgroup_per_node->lruvec_stat[]). The error of the per-CPU counter from CPU local counting and the descendant memory cgroups may cause some issues. We run into this in 0-Day performance test. 0-Day uses the RAM file system as root file system, so the number of the reclaimable file pages is very small. In the swap testing, the inactive file LRU list will become almost empty soon. But the size of the inactive file LRU list gotten from the per-CPU counter may keep a much larger value (say, 33, 50, etc.). This will enable cache trim mode, but nothing can be scanned in fact. The following pattern repeats for long time in the test, priority inactive_file_size cache_trim_mode 12 33 0 11 33 0 ... 6 33 0 5 33 1 ... 1 33 1 That is, the cache_trim_mode will be enabled wrongly when the scan priority decreases to 5. And the problem will not be recovered for long time. It's hard to get the more accurate size of the inactive file list without much more overhead. And it's hard to estimate the error of the per-CPU counter too, because there may be many descendant memory cgroups. But after the actual scanning, if nothing can be scanned with the cache trim mode, it should be wrong to enable the cache trim mode. So we can retry with the cache trim mode disabled. This patch implement this policy. The test results for pmbench with normal access address distribution and 2 NVMe disks as swap on a Intel server for the base and patched kernel are as follows. base change patched ---------- ------ --------- 122100000 18.6% 144800000 pmbench.throughput.aps 164078 -56.3% 71686 vmstat.swap.si 166511 -20.8% 131957 vmstat.swap.so 1992 -52.2% 953.17 proc-vmstat.kswapd_high_wmark_hit_quickly 230.20 -63.8% 83.33 proc-vmstat.kswapd_low_wmark_hit_quickly 2318 -41.4% 1358 proc-vmstat.pageoutrun From the above table, with the patch, the page reclaiming algorithm runs more smoothly, the hot/cold pages distinguishing works better, so the swap in/out throughput decreases considerably, and the benchmark throughput increases 18.6%. Signed-off-by: "Huang, Ying" Cc: Mel Gorman Cc: Johannes Weiner Cc: Vladimir Davydov Cc: Michal Hocko Cc: Joonsoo Kim --- mm/vmscan.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index fea6b43bc1f9..1304e3c98a14 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2606,7 +2606,8 @@ static inline bool should_continue_reclaim(struct pglist_data *pgdat, return inactive_lru_pages > pages_for_compaction; } -static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) +static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc, + bool skip_slab) { struct mem_cgroup *target_memcg = sc->target_mem_cgroup; struct mem_cgroup *memcg; @@ -2652,8 +2653,9 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) shrink_lruvec(lruvec, sc); - shrink_slab(sc->gfp_mask, pgdat->node_id, memcg, - sc->priority); + if (!skip_slab) + shrink_slab(sc->gfp_mask, pgdat->node_id, memcg, + sc->priority); /* Record the group's reclaim efficiency */ vmpressure(sc->gfp_mask, memcg, false, @@ -2669,6 +2671,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) unsigned long nr_reclaimed, nr_scanned; struct lruvec *target_lruvec; bool reclaimable = false; + bool skip_slab; unsigned long file; target_lruvec = mem_cgroup_lruvec(sc->target_mem_cgroup, pgdat); @@ -2767,7 +2770,15 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) anon >> sc->priority; } - shrink_node_memcgs(pgdat, sc); + skip_slab = false; +retry: + shrink_node_memcgs(pgdat, sc, skip_slab); + /* Nothing can be scanned with cache trim mode, retry without it */ + if (sc->cache_trim_mode && sc->nr_scanned == nr_scanned) { + sc->cache_trim_mode = 0; + skip_slab = true; + goto retry; + } if (reclaim_state) { sc->nr_reclaimed += reclaim_state->reclaimed_slab;