From patchwork Mon Mar 4 02:09:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13579952 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CBFBC48BF6 for ; Mon, 4 Mar 2024 02:09:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2FE96B0082; Sun, 3 Mar 2024 21:09:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DE0056B009E; Sun, 3 Mar 2024 21:09:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA78A6B00A1; Sun, 3 Mar 2024 21:09:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id BB7CF6B0082 for ; Sun, 3 Mar 2024 21:09:27 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0C8DB8091A for ; Mon, 4 Mar 2024 02:09:27 +0000 (UTC) X-FDA: 81857724774.17.4E08357 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf29.hostedemail.com (Postfix) with ESMTP id 904B7120004 for ; Mon, 4 Mar 2024 02:09:23 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf29.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709518165; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=OaxkuBa+nJUGABukTx4PdgqoC3Ps4+uSGwu/Ttf2qOY=; b=Nv9MujJ8WK3t8JqnXo1F3WmhayBFMdTATEQ+6OwyOPCyJneJ9Muke4sRgJ0gw61nXPnG3w zwsxgS7Z8FzdEWk3GotGH7Tb8vSqQlUKUcIGuY43KlO2oBA5UrCsne8MKHIBsoZEMztWnn TD3hox/hMaqt2zYZMbtUZU7h12NdW7M= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf29.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709518165; a=rsa-sha256; cv=none; b=gFdcQvgkXg8C7Vi1poi3I94gljhnqz4zQfMNYsQUYEs7jaHd/sMzU1E5AwKFypHqFXKS/w y6i+oPl3MGznjJ+qbvzzWokNKQuiiHN4idp/WZ8kSzRDYbQOz5QYAGlnVn+dwfogHXJ3oa 0HrZd41cUJHkcchl8vOloXhfdFR8KIY= X-AuditID: a67dfc5b-d6dff70000001748-08-65e52d4db49e From: Byungchul Park To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel_team@skhynix.com, yuzhao@google.com, ying.huang@intel.com, hannes@cmpxchg.org Subject: [PATCH v4] mm, vmscan: retry kswapd's priority loop with cache_trim_mode off on failure Date: Mon, 4 Mar 2024 11:09:07 +0900 Message-Id: <20240304020907.12514-1-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrOJMWRmVeSWpSXmKPExsXC9ZZnka6v7tNUg7ZeVYs569ewWaze5Gtx edccNot7a/6zWpycNZnF4t2EL6wObB6H37xn9liwqdRj8Z6XTB6bPk1i9zgx4zeLx+dNcgFs UVw2Kak5mWWpRfp2CVwZHWf3MRbs1qtY9vYYUwPjF8UuRk4OCQETif+zf7DD2Of7LzGC2GwC 6hI3bvxkBrFFBGQlpv49z9LFyMXBLDCFUWLFvXY2kISwQILEhavXwWwWAVWJ6/d2gtm8AqYS J55OgBoqL7F6wwFmkGYJgZusEvdOtrJAJCQlDq64wTKBkXsBI8MqRqHMvLLcxMwcE72MyrzM Cr3k/NxNjMDgWFb7J3oH46cLwYcYBTgYlXh4MzqfpAqxJpYVV+YeYpTgYFYS4a35BRTiTUms rEotyo8vKs1JLT7EKM3BoiTOa/StPEVIID2xJDU7NbUgtQgmy8TBKdXAuHpRYMuheVUXU5/0 7YvdmTOX3fXqi2lc4RYVH5duTY624nQoff3wzwVWR8H0rP5mwxOfXjHu0HB5LheVJlRhflD4 1Rz2bI3Y39MFbq55tvSOw/cZJT8XLAostMzZc2jDsr93n4muW7+lLMFi68qcAP5svQO3f5vl WLN894ib77/gYWG54JcCOyWW4oxEQy3mouJEAIE4UioKAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrHJMWRmVeSWpSXmKPExsXC5WfdrOur+zTV4EmTpMWc9WvYLFZv8rU4 PPckq8XlXXPYLO6t+c9qcXLWZBaLdxO+sDqwexx+857ZY8GmUo/Fe14yeWz6NInd48SM3ywe i198YPL4vEkugD2KyyYlNSezLLVI3y6BK6Pj7D7Ggt16FcveHmNqYPyi2MXIySEhYCJxvv8S I4jNJqAucePGT2YQW0RAVmLq3/MsXYxcHMwCUxglVtxrZwNJCAskSFy4eh3MZhFQlbh+byeY zStgKnHi6QR2iKHyEqs3HGCewMixgJFhFaNIZl5ZbmJmjqlecXZGZV5mhV5yfu4mRqCvl9X+ mbiD8ctl90OMAhyMSjy8E9Y8SRViTSwrrsw9xCjBwawkwlvzCyjEm5JYWZValB9fVJqTWnyI UZqDRUmc1ys8NUFIID2xJDU7NbUgtQgmy8TBKdXAON/G4BXXp2OX3lzhec5WtuCUdcRG6WM6 X2cus14hETpBjbNPQuDL1PerZt2oeBATYO7usu26jGRE84W/gWmqa7/ppwg7nA+5KpF76o7V Hr/3vKLZZ3itnikwS7+rWTXtwNJeThkDC89QvWCDaR3PK72Xbr/99CErx4Fr36al/Nz+Uurw e8aGl0osxRmJhlrMRcWJALGGzI/xAQAA X-CFilter-Loop: Reflected X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 904B7120004 X-Stat-Signature: py3jqb6b6epdyfs1chu3ywu1srzek69e X-Rspam-User: X-HE-Tag: 1709518163-767495 X-HE-Meta: U2FsdGVkX19rH8Lr0tBmfbbxRJSKoZ5s6YGDT8t4nyJ7gkf6jXCpAQF0xXn2Ng5i/4VAWuby9W5jihVjQtAfVCDXpzlwXbzKf9n4D8+iCseo82ql+Y6Q9DRb8aDOWQ/4Jn7OPtcpCekNxvYhZWfR2PW5F6S7Va1F38vYVgZ4uYp+X/fXfyx4F+pFiNZboChKH5LtAcQi5Nx5oKbjiaiKEwar5A64w6SsuMzRAPUAWFXpVPubquwC4IFDGUW412bsh9fD8vPiFZ52S5JdziumIvnfEZB3tY+QKJRW+rpn0GmpQKwk64gB3s7T5zhw8JrWPBoXx9Sds7pCjvay9aaZLBveQb5XGcsVpd0Ex4s4oQCWrnXpFU7RqMXpZCZYchA3+OvSANLF/hTMf8gKx8q2b0iJ0bDg6XPfZpziJcBi4m/VArQQSpPThX+Qi63j+Z34TLZbEsU/UIj3z195iHc9t1ENtAtm6VhD0e6c71OdyrfV9GtB5jRkwzA32eQKo+kOsClTDTcostjm0EZ71s7c5W5WEoqI0MBOiOQ2Erj3nIxd1M7TS052yOdFS8PVh9bSYOICuLYA0GAYo1wATAoQo4p0vsGrrXSDy7JLxxIef6gl3C3GiRDxq3tTK+kAxzE9cBJLPu3sT+zLKTu9Brn6eo1rRJq20VDkPlIfMaGSK7mXE+eMAwJrg06nwKQD2aJ1R0OSLmRIQucKDb+O34bfxaviNf1kPznWZGdssJGRurSVDXLFbs1ckCYF2mP14tE1ioTO8F/U/S8hEWtBGLVm3+DxZcC2Onlj8/bR40HS34Y24eoToTpASChg2tdqhC2NO54JadL8UNFD3uN4Xt7d/Y70N9sHjdKNKy+bMONldbr77s7CrPCQ3pyTlUAKJcr0UGVx12eAYRvJ/he9oG9l6XEZQnlrgdEnVVbWrWaJAYW4LHUh+ML0D0vjUW+dRqBXEqoTe9UgEgHCSKDmiyp ti/s88fs 17JhDSpmalg0FAd4He/YsQ2WGXe8TItIz/fuB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Changes from v3: 1. Update the test result in the commit message with v4. 2. Retry the whole priority loop with cache_trim_mode off again, rather than forcing the mode off at the highest priority, when the mode doesn't work. (feedbacked by Johannes Weiner) Changes from v2: 1. Change the condition to stop cache_trim_mode. From - Stop it if it's at high scan priorities, 0 or 1. To - Stop it if it's at high scan priorities, 0 or 1, and the mode didn't work in the previous turn. (feedbacked by Huang Ying) 2. Change the test result in the commit message after testing with the new logic. Changes from v1: 1. Add a comment describing why this change is necessary in code and rewrite the commit message with how to reproduce and what the result is using vmstat. (feedbacked by Andrew Morton and Yu Zhao) 2. Change the condition to avoid cache_trim_mode from 'sc->priority != 1' to 'sc->priority > 1' to reflect cases where the priority goes to zero all the way. (feedbacked by Yu Zhao) --->8--- From bd275d8d8e9ed91869401f0e1d089c4a2aeeaac6 Mon Sep 17 00:00:00 2001 From: Byungchul Park Date: Thu, 29 Feb 2024 19:48:01 +0900 Subject: [PATCH v4] mm, vmscan: retry kswapd's priority loop with cache_trim_mode off on failure With cache_trim_mode on, reclaim logic doesn't bother reclaiming anon pages. However, it should be more careful to use the mode because it's going to prevent anon pages from being reclaimed even if there are a huge number of anon pages that are cold and should be reclaimed. Even worse, that leads kswapd_failures to reach MAX_RECLAIM_RETRIES and stopping kswapd from functioning until direct reclaim eventually works to resume kswapd. So kswapd needs to retry its scan priority loop with cache_trim_mode off again if the mode doesn't work for reclaim. The problematic behavior can be reproduced by: CONFIG_NUMA_BALANCING enabled sysctl_numa_balancing_mode set to NUMA_BALANCING_MEMORY_TIERING numa node0 (8GB local memory, 16 CPUs) numa node1 (8GB slow tier memory, no CPUs) Sequence: 1) echo 3 > /proc/sys/vm/drop_caches 2) To emulate the system with full of cold memory in local DRAM, run the following dummy program and never touch the region: mmap(0, 8 * 1024 * 1024 * 1024, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE | MAP_POPULATE, -1, 0); 3) Run any memory intensive work e.g. XSBench. 4) Check if numa balancing is working e.i. promotion/demotion. 5) Iterate 1) ~ 4) until numa balancing stops. With this, you could see that promotion/demotion are not working because kswapd has stopped due to ->kswapd_failures >= MAX_RECLAIM_RETRIES. Interesting vmstat delta's differences between before and after are like: +-----------------------+-------------------------------+ | interesting vmstat | before | after | +-----------------------+-------------------------------+ | nr_inactive_anon | 321935 | 1646193 | | nr_active_anon | 1780700 | 456388 | | nr_inactive_file | 30425 | 27836 | | nr_active_file | 14961 | 1217 | | pgpromote_success | 356 | 1310120 | | pgpromote_candidate | 21953245 | 1736872 | | pgactivate | 1844523 | 3292443 | | pgdeactivate | 50634 | 1526701 | | pgfault | 31100294 | 6715375 | | pgdemote_kswapd | 30856 | 1954199 | | pgscan_kswapd | 1861981 | 7100099 | | pgscan_anon | 1822930 | 7061135 | | pgscan_file | 39051 | 38964 | | pgsteal_anon | 386 | 1925214 | | pgsteal_file | 30470 | 28985 | | pageoutrun | 30 | 500 | | numa_hint_faults | 27418279 | 3090773 | | numa_pages_migrated | 356 | 1310120 | +-----------------------+-------------------------------+ Signed-off-by: Byungchul Park --- mm/vmscan.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index bba207f41b14..7ec1fbcba7d4 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -108,6 +108,9 @@ struct scan_control { /* Can folios be swapped as part of reclaim? */ unsigned int may_swap:1; + /* Can cache_trim_mode be turned on as part of reclaim? */ + unsigned int may_cache_trim_mode:1; + /* Proactive reclaim invoked by userspace through memory.reclaim */ unsigned int proactive:1; @@ -2268,7 +2271,8 @@ static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) * anonymous pages. */ file = lruvec_page_state(target_lruvec, NR_INACTIVE_FILE); - if (file >> sc->priority && !(sc->may_deactivate & DEACTIVATE_FILE)) + if (file >> sc->priority && !(sc->may_deactivate & DEACTIVATE_FILE) && + sc->may_cache_trim_mode) sc->cache_trim_mode = 1; else sc->cache_trim_mode = 0; @@ -6744,6 +6748,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) .gfp_mask = GFP_KERNEL, .order = order, .may_unmap = 1, + .may_cache_trim_mode = 1, }; set_task_reclaim_state(current, &sc.reclaim_state); @@ -6898,8 +6903,14 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) sc.priority--; } while (sc.priority >= 1); - if (!sc.nr_reclaimed) + if (!sc.nr_reclaimed) { + if (sc.may_cache_trim_mode) { + sc.may_cache_trim_mode = 0; + goto restart; + } + pgdat->kswapd_failures++; + } out: clear_reclaim_active(pgdat, highest_zoneidx);