From patchwork Fri Feb 23 05:44:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13568572 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54057C48BC4 for ; Fri, 23 Feb 2024 05:44:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C4F8E6B0072; Fri, 23 Feb 2024 00:44:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C006B6B0074; Fri, 23 Feb 2024 00:44:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC7C36B0075; Fri, 23 Feb 2024 00:44:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9E1686B0072 for ; Fri, 23 Feb 2024 00:44:28 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 461CEA183C for ; Fri, 23 Feb 2024 05:44:28 +0000 (UTC) X-FDA: 81821978616.15.46E81D4 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf23.hostedemail.com (Postfix) with ESMTP id 6DA76140006 for ; Fri, 23 Feb 2024 05:44:24 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=none; spf=pass (imf23.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708667067; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=hjBMID0Qrkm89h/viJJMMiXGALjsVOzL/sTJmY5heoE=; b=dtgAS2WmIuUrTne7oBrbzIrHtlBdXPmtLNsa/eO+nbx4Q9zlNO3VTtY0CWVWPA80hy58Tt k298+GKXFcvFCjP4mBmDnP8K41tL7g5cJjzHWH6/tFU5NAD8EIRNpVpcgiGxuRksLo+5mW yuC70xtHFkD5BHhnFsIVm0kBmGst8PU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708667067; a=rsa-sha256; cv=none; b=lVekPj7uNHpjjRxLu0ppcm/m9nx8akFZYA6sG6b1l6INfj4GuPMWSLN/yLX5EfKE71wKc4 3zFtlPw99JfeqXvlwERreQc3296NxfLIWbNVD6hyUEY9IhYBFKYXx7oPH6aof2hnGzeRpH trNMWbCpyQFf/QQqebhrGEWJf4dqDYE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=none; spf=pass (imf23.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d6dff70000001748-a5-65d830b2bc15 From: Byungchul Park To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel_team@skhynix.com, yuzhao@google.com, ying.huang@intel.com Subject: [PATCH v3] mm, vmscan: do not turn on cache_trim_mode if it doesn't work Date: Fri, 23 Feb 2024 14:44:07 +0900 Message-Id: <20240223054407.14829-1-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrLJMWRmVeSWpSXmKPExsXC9ZZnoe4mgxupBl9blSzmrF/DZnF51xw2 i3tr/rNanJw1mcXi3YQvrA6sHgs2lXos3vOSyWPTp0nsHidm/Gbx+LxJLoA1issmJTUnsyy1 SN8ugStj3dF9rAUnjCqezvvI3MB4S62LkZNDQsBE4mV3IyuMfWjKL0YQm01AXeLGjZ/MILaI gKzE1L/nWUBsZoE6ieX/PoLFhQUCJd7cWQrWyyKgKvHz9lywXl4BU4nO5j4WiJnyEqs3HACq 5wKyT7JKXJm/kR0iISlxcMUNlgmM3AsYGVYxCmXmleUmZuaY6GVU5mVW6CXn525iBAbCsto/ 0TsYP10IPsQowMGoxMNbkHg9VYg1say4MvcQowQHs5IIL0v5lVQh3pTEyqrUovz4otKc1OJD jNIcLErivEbfylOEBNITS1KzU1MLUotgskwcnFINjBvuLt5VE8b2d0/iuU+Tgksa+aQM2xd0 PNO3ywppCHhgp9v+4KCVervGr5d6lxTy9unlFJ65wOVTudWX69r2z5FWjx4drT8f92xNd4Xi MeYv+5gf5qRuXVV8ZkvGkjPXShbva3ZUKlv575D6E7kWjdT2jqeMZSoBJfzJ6d3K86YdMWE5 NXOljhJLcUaioRZzUXEiAFKlVn0AAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrCJMWRmVeSWpSXmKPExsXC5WfdrLvJ4EaqQedfLos569ewWRyee5LV 4vKuOWwW99b8Z7U4OWsyi8W7CV9YHdg8Fmwq9Vi85yWTx6ZPk9g9Tsz4zeKx+MUHJo/Pm+QC 2KK4bFJSczLLUov07RK4MtYd3cdacMKo4um8j8wNjLfUuhg5OSQETCQOTfnFCGKzCahL3Ljx kxnEFhGQlZj69zwLiM0sUCex/N9HsLiwQKDEmztLWUFsFgFViZ+354L18gqYSnQ297FAzJSX WL3hAPMERo4FjAyrGEUy88pyEzNzTPWKszMq8zIr9JLzczcxAv26rPbPxB2MXy67H2IU4GBU 4uEtSLyeKsSaWFZcmXuIUYKDWUmEl6X8SqoQb0piZVVqUX58UWlOavEhRmkOFiVxXq/w1AQh gfTEktTs1NSC1CKYLBMHp1QD47lHS198ffNu7cx3l0NljpV2OIZs9v46V9vL0rj7kIPz7S67 hurvX17mHWYpYHu8TeO80h6+SLP8R0e4Gkq5Dgs5la/ICe2YFD9d05Tbtl4kYbGf6yRXya1L dNsZogPvP5grwhd2qbCw/cPc5PKfPXfXljxf1tMVsuBTJ48M7/XV9TdUWzUKlViKMxINtZiL ihMBAND6VucBAAA= X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 6DA76140006 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: f8u7fjqpbhwc4nqydje4fcpotuska9oo X-HE-Tag: 1708667064-949361 X-HE-Meta: U2FsdGVkX19Bc4nnf5f8eYY5zxpiSTBiPFWooMwjugD+ErjWgKD132UItn7jjardZH16vTt6/ubokC+JWjvRjEv2pWymiy/dI4AJvz6XbbFlAZIuV3pl72UXndFbuathCAo1cruKtk2ct2a6JeZBW5g9MblWR2f1Hk2iX7c+lc8t6x18P5wZSFADds5NLVI1xYBY5HLNh/3nmhDgoi0I4JCnmwwmzsiBhlGkyeDBaC3hjuUAadWWUn6aZJbgT3/nL5WbgHd1Y74A5kObUN5Juul5ygm7ZZLFfUVUbqjHsFZumxvwnf+mnzP6l12Ejsp0LiyfItZ6nTzAyJxlgkeNAS4vUztJg4mxzvzblPW5v8EYssAbALgGH7FYg2aGyghCLMRNDqB7kvn3nCoMNoSA0uoUvNcdizDOIoyBKD1DNSbpePpiisUySiW2meiiI8Et2a7eG5yF0z1TLGd0SbuJN3Dl+V4AzuK0SBujpK0e9Fb2IpPYd7MtQN6DEUVS6xsJyzYFeQN6BRKBdUzKHIl6qEAJyC3CdKwZdKljgj5N8IfCNXEzjIXbjyBmPgHtItNhjgfqC81sP88YnV2pszbq/9gj/R+Ci8bO54vTRUrQAR8yx+RvBQbs0PZDESSAu2cLYjjHdiWW3eCB0v8/W+foiB15Qh0nYmBtXAGvzHN9fY91gr6D8myh+9b6V7e/CJDnEgaZwG9SsEaNZxoHvV2TICKzFVLCPD6nJkDRoMkCd6+AFsxfS9ocup7U7lJE7aSC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Changes from v2: 1. Change the condition to stop cache_trim_mode. From - Stop it if it's at high scan priorities, 0 or 1. To - Stop it if it's at high scan priorities, 0 or 1, and the mode didn't work in the previous turn. (feedbacked by Huang Ying) 2. Change the test result in the commit message after testing with the new logic. Changes from v1: 1. Add a comment describing why this change is necessary in code and rewrite the commit message with how to reproduce and what the result is using vmstat. (feedbacked by Andrew Morton and Yu Zhao) 2. Change the condition to avoid cache_trim_mode from 'sc->priority != 1' to 'sc->priority > 1' to reflect cases where the priority goes to zero all the way. (feedbacked by Yu Zhao) --->8--- From 05846e34bf02ac9b3e254324dc2d7afd97a025d9 Mon Sep 17 00:00:00 2001 From: Byungchul Park Date: Fri, 23 Feb 2024 13:47:16 +0900 Subject: [PATCH v3] mm, vmscan: do not turn on cache_trim_mode if it doesn't work With cache_trim_mode on, reclaim logic doesn't bother reclaiming anon pages. However, it should be more careful to turn on the mode because it's going to prevent anon pages from being reclaimed even if there are a huge number of anon pages that are cold and should be reclaimed. Even worse, that leads kswapd_failures to reach MAX_RECLAIM_RETRIES and stopping kswapd from functioning until direct reclaim eventually works to resume kswapd. So do not turn on cache_trim_mode if the mode doesn't work, especially while the sytem is struggling against reclaim. The problematic behavior can be reproduced by: CONFIG_NUMA_BALANCING enabled sysctl_numa_balancing_mode set to NUMA_BALANCING_MEMORY_TIERING numa node0 (8GB local memory, 16 CPUs) numa node1 (8GB slow tier memory, no CPUs) Sequence: 1) echo 3 > /proc/sys/vm/drop_caches 2) To emulate the system with full of cold memory in local DRAM, run the following dummy program and never touch the region: mmap(0, 8 * 1024 * 1024 * 1024, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE | MAP_POPULATE, -1, 0); 3) Run any memory intensive work e.g. XSBench. 4) Check if numa balancing is working e.i. promotion/demotion. 5) Iterate 1) ~ 4) until numa balancing stops. With this, you could see that promotion/demotion are not working because kswapd has stopped due to ->kswapd_failures >= MAX_RECLAIM_RETRIES. Interesting vmstat delta's differences between before and after are like: +-----------------------+-------------------------------+ | interesting vmstat | before | after | +-----------------------+-------------------------------+ | nr_inactive_anon | 321935 | 1636737 | | nr_active_anon | 1780700 | 465913 | | nr_inactive_file | 30425 | 35711 | | nr_active_file | 14961 | 8698 | | pgpromote_success | 356 | 1267785 | | pgpromote_candidate | 21953245 | 1745631 | | pgactivate | 1844523 | 3309867 | | pgdeactivate | 50634 | 1545041 | | pgfault | 31100294 | 6411036 | | pgdemote_kswapd | 30856 | 2267467 | | pgscan_kswapd | 1861981 | 7729231 | | pgscan_anon | 1822930 | 7667544 | | pgscan_file | 39051 | 61687 | | pgsteal_anon | 386 | 2227217 | | pgsteal_file | 30470 | 40250 | | pageoutrun | 30 | 457 | | numa_hint_faults | 27418279 | 2752289 | | numa_pages_migrated | 356 | 1267785 | +-----------------------+-------------------------------+ Signed-off-by: Byungchul Park Acked-by: "Huang, Ying" --- mm/vmscan.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index bba207f41b14..f7312d831fed 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -127,6 +127,9 @@ struct scan_control { /* One of the zones is ready for compaction */ unsigned int compaction_ready:1; + /* If the last try was reclaimable */ + unsigned int reclaimable:1; + /* There is easily reclaimable cold cache in the current node */ unsigned int cache_trim_mode:1; @@ -2266,9 +2269,14 @@ static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) * If we have plenty of inactive file pages that aren't * thrashing, try to reclaim those first before touching * anonymous pages. + * + * It doesn't make sense to keep cache_trim_mode on if the mode + * is not working while struggling against reclaim. So do not + * turn it on if so. Note the highest priority of kswapd is 1. */ file = lruvec_page_state(target_lruvec, NR_INACTIVE_FILE); - if (file >> sc->priority && !(sc->may_deactivate & DEACTIVATE_FILE)) + if (file >> sc->priority && !(sc->may_deactivate & DEACTIVATE_FILE) && + !(sc->cache_trim_mode && !sc->reclaimable && sc->priority <= 1)) sc->cache_trim_mode = 1; else sc->cache_trim_mode = 0; @@ -5862,7 +5870,6 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) { unsigned long nr_reclaimed, nr_scanned, nr_node_reclaimed; struct lruvec *target_lruvec; - bool reclaimable = false; if (lru_gen_enabled() && root_reclaim(sc)) { lru_gen_shrink_node(pgdat, sc); @@ -5877,6 +5884,14 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) nr_reclaimed = sc->nr_reclaimed; nr_scanned = sc->nr_scanned; + /* + * Reset to the default values at the start. + */ + if (sc->priority == DEF_PRIORITY) { + sc->reclaimable = 1; + sc->cache_trim_mode = 0; + } + prepare_scan_control(pgdat, sc); shrink_node_memcgs(pgdat, sc); @@ -5890,8 +5905,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) vmpressure(sc->gfp_mask, sc->target_mem_cgroup, true, sc->nr_scanned - nr_scanned, nr_node_reclaimed); - if (nr_node_reclaimed) - reclaimable = true; + sc->reclaimable = !!nr_node_reclaimed; if (current_is_kswapd()) { /* @@ -5965,7 +5979,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) * sleep. On reclaim progress, reset the failure counter. A * successful direct reclaim run will revive a dormant kswapd. */ - if (reclaimable) + if (sc->reclaimable) pgdat->kswapd_failures = 0; }