From patchwork Mon Mar 4 02:30:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13579978 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 722C9C48BF6 for ; Mon, 4 Mar 2024 02:30:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D0CC26B008A; Sun, 3 Mar 2024 21:30:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C95B06B0092; Sun, 3 Mar 2024 21:30:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B358B6B009E; Sun, 3 Mar 2024 21:30:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9CFB06B008A for ; Sun, 3 Mar 2024 21:30:35 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4BB1B1407B5 for ; Mon, 4 Mar 2024 02:30:35 +0000 (UTC) X-FDA: 81857778030.02.F54F457 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf07.hostedemail.com (Postfix) with ESMTP id 829634000B for ; Mon, 4 Mar 2024 02:30:31 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; spf=temperror (imf07.hostedemail.com: error in processing during lookup of byungchul@sk.com: DNS error) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709519433; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=LeYc5mf2I8TsBO8azTzNqyDWC1zy6gm93RhZO4602bA=; b=RblYdEj2rQE2cr74M/rtEzEw5IdAtZW88bxUdrfjHTvxtOeO5a8w8fVBmVNKZiEg00NlqZ MtS5OU+5Mzey342lxCkfRvyRMn4nRTOadmjMfqMT6n5vQq5emtW8bOqv7aHsYEwKwW7WIX ZJtDSyUFyOtJm6uyK/MPIU8TfT706RI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709519433; a=rsa-sha256; cv=none; b=8Q/gvC53Wi0QSp0LdUZTzk20WOoUF/EwXbajYEBK0DAtG/wjK7OLIQIjEQLy8d4dCqzrlJ 8s6lqiQ3oJdsYsm6kGE8H4ZMkf1lSexnhq6tqyNNb4mhG1yDrfrxacW6wvs66yn60k1poM +jz2TJLg1hsSDwDJ8jNoqTH63+1K+Gg= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; spf=temperror (imf07.hostedemail.com: error in processing during lookup of byungchul@sk.com: DNS error) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d6dff70000001748-e8-65e53244aee4 From: Byungchul Park To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel_team@skhynix.com, yuzhao@google.com, ying.huang@intel.com, hannes@cmpxchg.org Subject: [PATCH v5] mm, vmscan: retry kswapd's priority loop with cache_trim_mode off on failure Date: Mon, 4 Mar 2024 11:30:18 +0900 Message-Id: <20240304023018.69705-1-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrOJMWRmVeSWpSXmKPExsXC9ZZnka6L0dNUg6eXVSzmrF/DZrF6k6/F 5V1z2CzurfnPanFy1mQWi3cTvrA6sHkcfvOe2WPBplKPxXteMnls+jSJ3ePEjN8sHp83yQWw RXHZpKTmZJalFunbJXBldP97xlSw2qHi8aLZjA2MfQZdjJwcEgImEgeeHGOGsV+cW8kOYrMJ qEvcuPETLC4iICsx9e95li5GLg5mgSmMEivutbOBJIQFEiRWPd3IAmKzCKhKzN/+BczmFTCV eLBqFgvEUHmJ1RsOMIM0SwjcZZX42fufCSIhKXFwxQ2WCYzcCxgZVjEKZeaV5SZm5pjoZVTm ZVboJefnbmIEBsey2j/ROxg/XQg+xCjAwajEw5vR+SRViDWxrLgy9xCjBAezkghvzS+gEG9K YmVValF+fFFpTmrxIUZpDhYlcV6jb+UpQgLpiSWp2ampBalFMFkmDk6pBkYu9xPmyrl+qlYO x5xCKy7bPg2areLyQ65xelZ3l5j+7c+KIecfbG9fp7TA6Bnvt0tfU2syTHiELtwOYJZQPiYa HLD6bKLPya9JW/puTEy93+lbUfQj483xZXvnPbBb59akyCV+RmThvb4OSyO3EwfnW65vr2Cu X1AZITHzEYsDs1jCMo6I/0osxRmJhlrMRcWJAJMvXxQKAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrHJMWRmVeSWpSXmKPExsXC5WfdrOti9DTVYP9GMYs569ewWaze5Gtx eO5JVovLu+awWdxb85/V4uSsySwW7yZ8YXVg9zj85j2zx4JNpR6L97xk8tj0aRK7x4kZv1k8 Fr/4wOTxeZNcAHsUl01Kak5mWWqRvl0CV0b3v2dMBasdKh4vms3YwNhn0MXIySEhYCLx4txK dhCbTUBd4saNn8wgtoiArMTUv+dZuhi5OJgFpjBKrLjXzgaSEBZIkFj1dCMLiM0ioCoxf/sX MJtXwFTiwapZLBBD5SVWbzjAPIGRYwEjwypGkcy8stzEzBxTveLsjMq8zAq95PzcTYxAXy+r /TNxB+OXy+6HGAU4GJV4eCeseZIqxJpYVlyZe4hRgoNZSYS35hdQiDclsbIqtSg/vqg0J7X4 EKM0B4uSOK9XeGqCkEB6YklqdmpqQWoRTJaJg1OqgbHZJUjak92xyWrSrOXXZbNaZx0sNW9Z uEp1v0PSjIrw2DOf/n8Wt3qSnbRlPbtUfvT/Z8X3zuomeJ/ZE21uWZAceurUgxuNAgvLlhWG tkdNUJh0KPT5pTl3HT2XVq62zZkRueTs0/v7bjSs6vu8jvtnjnNH8EqhfZG8X8RCFMILelvm JLesbVdiKc5INNRiLipOBAAMUnki8QEAAA== X-CFilter-Loop: Reflected X-Stat-Signature: 7rc5njampn4czt3cf98xf5ya881n8baj X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 829634000B X-Rspam-User: X-HE-Tag: 1709519431-316004 X-HE-Meta: U2FsdGVkX19Pn4mPvrOR72pZrQszP1uyM+Sb1VZ8zB26dt6X5KhM2GByIkNYysoxCf35j65Fhp4tB8RxYTesxxvI29pSvp3AOSAnZEJZERbbP1dQfInCP4mi3nwzAk6yEdXEw1/D+H6N3CvOkurAz9LeS4aJ12HMwhAMF0UaHFGKh+KyfSCXwuIQb2lXLqka/gjW3Ci5o87m4b8HZm9uq5NpEjx8y5hdctr0nbEOUnwJADDv8HvZZZ1Atjz+Dpb8j+rk55TFRjDfSVQG5MfhS/yOUSiMR7MW0DPI/x0I9jc5hiOEt7G9yuYckweUsUHAatv7WQlJh//ZnA8omQeXx/dvdMcuQXC57gqtymGEgVH3CAIQ2gn9s9cHszy+PX/b0AgkFaNdVbH5Q5XUxQDC0cd78pM/Mxn3pFsElFczxSubuffiUEo/QnCCKoUill2LrWgAT/73YxBzcSloQC7LUDpyLt+JihatZ6CRQSxwmoi5/sboC97VTKSUiie6ZMwldpo7GDpj3uKsw+yk45VGjp8b9VWg+gNxDGPR7F19sa8C85VRbV40D77L94CMeLVn4CDmrso17Ulp2RB33V/9yWoReqQHUeoHoJLrlcZ18Hf2n1dZY+StE0VdGC+ZhfBOB/ejTqnagrqxCyMJZsMO5DhE680gfjVucLwol8197sp//ll2jS1qkVEP73LmjNp/uHdMFbtRSyQ1ptlXxqcYOTk/TQesRbOrXDloXfuVD/P1HLIH8IE38W29f0VT14tbkw3rdSxOO1HkAxPpSczxuJxGgraCmon8iblwzrxkkL6VxY/TrE2M2847h2iRCtqg1zF3boTtzpLSAV8O3WKHxNWpnG0z7K99lLjpSOqDkbQpKrGGVXu/dQ1vzQ5ipQTw/vxd5fzohS4FsciORT0EuHQ6L9SohQFpPx0wZoBikRGo+eYr35Luy35VGagMtF/Ud0oMSEwoux1CmuVirEd eMUIjjET HCHp1CNvFd67336RmG39REvs4mRAQcvTB2/rU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Sorry for noise. I should've applied v5's change in v4. Changes from v4: 1. Make other scans start with may_cache_trim_mode = 1. Changes from v3: 1. Update the test result in the commit message with v4. 2. Retry the whole priority loop with cache_trim_mode off again, rather than forcing the mode off at the highest priority, when the mode doesn't work. (feedbacked by Johannes Weiner) Changes from v2: 1. Change the condition to stop cache_trim_mode. From - Stop it if it's at high scan priorities, 0 or 1. To - Stop it if it's at high scan priorities, 0 or 1, and the mode didn't work in the previous turn. (feedbacked by Huang Ying) 2. Change the test result in the commit message after testing with the new logic. Changes from v1: 1. Add a comment describing why this change is necessary in code and rewrite the commit message with how to reproduce and what the result is using vmstat. (feedbacked by Andrew Morton and Yu Zhao) 2. Change the condition to avoid cache_trim_mode from 'sc->priority != 1' to 'sc->priority > 1' to reflect cases where the priority goes to zero all the way. (feedbacked by Yu Zhao) --->8--- From 58f1a0e41b9feea72d7fd4bd7bed1ace592e6e4c Mon Sep 17 00:00:00 2001 From: Byungchul Park Date: Mon, 4 Mar 2024 11:24:40 +0900 Subject: [PATCH v5] mm, vmscan: retry kswapd's priority loop with cache_trim_mode off on failure With cache_trim_mode on, reclaim logic doesn't bother reclaiming anon pages. However, it should be more careful to use the mode because it's going to prevent anon pages from being reclaimed even if there are a huge number of anon pages that are cold and should be reclaimed. Even worse, that leads kswapd_failures to reach MAX_RECLAIM_RETRIES and stopping kswapd from functioning until direct reclaim eventually works to resume kswapd. So kswapd needs to retry its scan priority loop with cache_trim_mode off again if the mode doesn't work for reclaim. The problematic behavior can be reproduced by: CONFIG_NUMA_BALANCING enabled sysctl_numa_balancing_mode set to NUMA_BALANCING_MEMORY_TIERING numa node0 (8GB local memory, 16 CPUs) numa node1 (8GB slow tier memory, no CPUs) Sequence: 1) echo 3 > /proc/sys/vm/drop_caches 2) To emulate the system with full of cold memory in local DRAM, run the following dummy program and never touch the region: mmap(0, 8 * 1024 * 1024 * 1024, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE | MAP_POPULATE, -1, 0); 3) Run any memory intensive work e.g. XSBench. 4) Check if numa balancing is working e.i. promotion/demotion. 5) Iterate 1) ~ 4) until numa balancing stops. With this, you could see that promotion/demotion are not working because kswapd has stopped due to ->kswapd_failures >= MAX_RECLAIM_RETRIES. Interesting vmstat delta's differences between before and after are like: +-----------------------+-------------------------------+ | interesting vmstat | before | after | +-----------------------+-------------------------------+ | nr_inactive_anon | 321935 | 1646193 | | nr_active_anon | 1780700 | 456388 | | nr_inactive_file | 30425 | 27836 | | nr_active_file | 14961 | 1217 | | pgpromote_success | 356 | 1310120 | | pgpromote_candidate | 21953245 | 1736872 | | pgactivate | 1844523 | 3292443 | | pgdeactivate | 50634 | 1526701 | | pgfault | 31100294 | 6715375 | | pgdemote_kswapd | 30856 | 1954199 | | pgscan_kswapd | 1861981 | 7100099 | | pgscan_anon | 1822930 | 7061135 | | pgscan_file | 39051 | 38964 | | pgsteal_anon | 386 | 1925214 | | pgsteal_file | 30470 | 28985 | | pageoutrun | 30 | 500 | | numa_hint_faults | 27418279 | 3090773 | | numa_pages_migrated | 356 | 1310120 | +-----------------------+-------------------------------+ Signed-off-by: Byungchul Park --- mm/vmscan.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index bba207f41b14..77948b0f8b5b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -108,6 +108,9 @@ struct scan_control { /* Can folios be swapped as part of reclaim? */ unsigned int may_swap:1; + /* Can cache_trim_mode be turned on as part of reclaim? */ + unsigned int may_cache_trim_mode:1; + /* Proactive reclaim invoked by userspace through memory.reclaim */ unsigned int proactive:1; @@ -1500,6 +1503,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, struct scan_control sc = { .gfp_mask = GFP_KERNEL, .may_unmap = 1, + .may_cache_trim_mode = 1, }; struct reclaim_stat stat; unsigned int nr_reclaimed; @@ -2094,6 +2098,7 @@ static unsigned int reclaim_folio_list(struct list_head *folio_list, .may_writepage = 1, .may_unmap = 1, .may_swap = 1, + .may_cache_trim_mode = 1, .no_demotion = 1, }; @@ -2268,7 +2273,8 @@ static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) * anonymous pages. */ file = lruvec_page_state(target_lruvec, NR_INACTIVE_FILE); - if (file >> sc->priority && !(sc->may_deactivate & DEACTIVATE_FILE)) + if (file >> sc->priority && !(sc->may_deactivate & DEACTIVATE_FILE) && + sc->may_cache_trim_mode) sc->cache_trim_mode = 1; else sc->cache_trim_mode = 0; @@ -5435,6 +5441,7 @@ static ssize_t lru_gen_seq_write(struct file *file, const char __user *src, .may_writepage = true, .may_unmap = true, .may_swap = true, + .may_cache_trim_mode = 1, .reclaim_idx = MAX_NR_ZONES - 1, .gfp_mask = GFP_KERNEL, }; @@ -6394,6 +6401,7 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order, .may_writepage = !laptop_mode, .may_unmap = 1, .may_swap = 1, + .may_cache_trim_mode = 1, }; /* @@ -6439,6 +6447,7 @@ unsigned long mem_cgroup_shrink_node(struct mem_cgroup *memcg, .may_unmap = 1, .reclaim_idx = MAX_NR_ZONES - 1, .may_swap = !noswap, + .may_cache_trim_mode = 1, }; WARN_ON_ONCE(!current->reclaim_state); @@ -6482,6 +6491,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, .may_writepage = !laptop_mode, .may_unmap = 1, .may_swap = !!(reclaim_options & MEMCG_RECLAIM_MAY_SWAP), + .may_cache_trim_mode = 1, .proactive = !!(reclaim_options & MEMCG_RECLAIM_PROACTIVE), }; /* @@ -6744,6 +6754,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) .gfp_mask = GFP_KERNEL, .order = order, .may_unmap = 1, + .may_cache_trim_mode = 1, }; set_task_reclaim_state(current, &sc.reclaim_state); @@ -6898,8 +6909,14 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) sc.priority--; } while (sc.priority >= 1); - if (!sc.nr_reclaimed) + if (!sc.nr_reclaimed) { + if (sc.may_cache_trim_mode) { + sc.may_cache_trim_mode = 0; + goto restart; + } + pgdat->kswapd_failures++; + } out: clear_reclaim_active(pgdat, highest_zoneidx); @@ -7202,6 +7219,7 @@ unsigned long shrink_all_memory(unsigned long nr_to_reclaim) .may_writepage = 1, .may_unmap = 1, .may_swap = 1, + .may_cache_trim_mode = 1, .hibernation_mode = 1, }; struct zonelist *zonelist = node_zonelist(numa_node_id(), sc.gfp_mask); @@ -7360,6 +7378,7 @@ static int __node_reclaim(struct pglist_data *pgdat, gfp_t gfp_mask, unsigned in .may_writepage = !!(node_reclaim_mode & RECLAIM_WRITE), .may_unmap = !!(node_reclaim_mode & RECLAIM_UNMAP), .may_swap = 1, + .may_cache_trim_mode = 1, .reclaim_idx = gfp_zone(gfp_mask), }; unsigned long pflags;