From patchwork Fri Apr 5 06:08:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Honggyu Kim X-Patchwork-Id: 13618553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46AB2CD11C2 for ; Fri, 5 Apr 2024 06:09:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73ED36B00F3; Fri, 5 Apr 2024 02:09:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6EE476B00F4; Fri, 5 Apr 2024 02:09:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F1996B00F5; Fri, 5 Apr 2024 02:09:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 30F846B00F3 for ; Fri, 5 Apr 2024 02:09:16 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F40BBA17A3 for ; Fri, 5 Apr 2024 06:09:15 +0000 (UTC) X-FDA: 81974450670.15.E465807 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf07.hostedemail.com (Postfix) with ESMTP id E720F40022 for ; Fri, 5 Apr 2024 06:09:13 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf07.hostedemail.com: domain of honggyu.kim@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=honggyu.kim@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712297354; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ESIWyjaNk5BwYYiYQCvAAQVOsQgbekkiQzOqtALzHTI=; b=xtYCZ3pYUidzIp27oRptJYD/LQkXWrMKIHkGV1GQLypKzlFGrwAJHP2pdzAgxb67kR+jnw 792U8gYpbBmsagETFYMDUq1noSornxiEnLxBDtOiVJmESYWBsDumgIZQNuKQ/v+rBz/52R 3Q+fBZfwZ6/scbO3qzdthrxkzd//Pnc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf07.hostedemail.com: domain of honggyu.kim@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=honggyu.kim@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712297354; a=rsa-sha256; cv=none; b=CUP5sq4hiCnvSBV0912XKwNeV6UREklK+LAsYicnAiIwbhqRw4J/iCGC7DuvIrhEei52KG y+G9U28F8DTW329DGashgBsY6CoPXBkLqjkMBpSsZm7EYkj6uC0I7NhhQkGAhSHLhPENVW kLUwn1u0JoNIwatDLqV8/gUgvmwQNBg= X-AuditID: a67dfc5b-d6dff70000001748-bf-660f9584a273 From: Honggyu Kim To: sj@kernel.org, damon@lists.linux.dev, linux-mm@kvack.org Cc: akpm@linux-foundation.org, apopple@nvidia.com, baolin.wang@linux.alibaba.com, dave.jiang@intel.com, honggyu.kim@sk.com, hyeongtak.ji@sk.com, kernel_team@skhynix.com, linmiaohe@huawei.com, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, mathieu.desnoyers@efficios.com, mhiramat@kernel.org, rakie.kim@sk.com, rostedt@goodmis.org, surenb@google.com, yangx.jy@fujitsu.com, ying.huang@intel.com, ziy@nvidia.com, 42.hyeyoo@gmail.com, art.jeongseob@gmail.com Subject: [RFC PATCH v3 5/7] mm/damon/paddr: introduce DAMOS_MIGRATE_COLD action for demotion Date: Fri, 5 Apr 2024 15:08:54 +0900 Message-ID: <20240405060858.2818-6-honggyu.kim@sk.com> X-Mailer: git-send-email 2.43.0.windows.1 In-Reply-To: <20240405060858.2818-1-honggyu.kim@sk.com> References: <20240405060858.2818-1-honggyu.kim@sk.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrCIsWRmVeSWpSXmKPExsXC9ZZnkW7rVP40g/cv2S0m9hhYzFm/hs1i 140Qi/sPXrNb/N97jNHiyf/frBYnbjayWXR+X8picXnXHDaLe2v+s1ocWX+WxWLz2TPMFouX q1ns63jAZHH46xsmi8mXFrBZvJhyhtHi5KzJLBazj95jdxD2WHr6DZvHhiYgsXPWXXaPln23 2D0WbCr1aDnyltVj8Z6XTB6bVnWyeWz6NInd48SM3yweOx9aerzYPJPRo7f5HZvH501yAXxR XDYpqTmZZalF+nYJXBlzt39lKXjmWHFx+wPWBsafJl2MnBwSAiYS1yY8ZIWxv8xYwQRiswmo SVx5OQnI5uAQEXCQWPVVoYuRi4NZ4D+zxOXeX2D1wgIxEltu9bGA2CwCqhLv23+xg9i8AmYS 6w5MYYSYqSnxePtPsDingLnEvMcbwGwhoJpPB49B1QtKnJz5BGwOs4C8RPPW2cwgyyQEjrFL dPcuZ4EYJClxcMUNlgmM/LOQ9MxC0rOAkWkVo1BmXlluYmaOiV5GZV5mhV5yfu4mRmDcLav9 E72D8dOF4EOMAhyMSjy8HnP50oRYE8uKK3MPMUpwMCuJ8HY78KYJ8aYkVlalFuXHF5XmpBYf YpTmYFES5zX6Vp4iJJCeWJKanZpakFoEk2Xi4JRqYGTO1Z7VPcNyR8qcL2IbfwY1nGD8ukzz +eKnGe9WXD7HyMGgt2rulqqdLZvMHSYqXVwqe+C3yd11/tf+fNp1klX585qpDUsnG79RadJT fXK06keZ/Zeea8XyZxRCi++/PGd4V3NX6LmrjzSqmBgmH+lY+Vb+b2CvjcWd/XNPMzpsqb9j Pvuk/9oCJZbijERDLeai4kQAHlV+d7cCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprHIsWRmVeSWpSXmKPExsXCNUNLT7dlKn+awYajxhYTewws5qxfw2ax 60aIxf0Hr9kt/u89xmjx5P9vVosTNxvZLD4/e81s0fnkO6PF4bknWS06vy9lsbi8aw6bxb01 /1ktjqw/y2Kx+ewZZovFy9UsDl17zmqxr+MBk8Xhr2+YLCZfWsBm8WLKGUaLk7Mms1jMPnqP 3UHcY+npN2weG5qAxM5Zd9k9WvbdYvdYsKnUo+XIW1aPxXteMnlsWtXJ5rHp0yR2jxMzfrN4 7Hxo6fFi80xGj97md2we3257eCx+8YHJ4/MmuQCBKC6blNSczLLUIn27BK6Mudu/shQ8c6y4 uP0BawPjT5MuRk4OCQETiS8zVjCB2GwCahJXXk4Csjk4RAQcJFZ9Vehi5OJgFvjPLHG59xcr SI2wQIzEllt9LCA2i4CqxPv2X+wgNq+AmcS6A1MYIWZqSjze/hMszilgLjHv8QYwWwio5tPB Y1D1ghInZz4Bm8MsIC/RvHU28wRGnllIUrOQpBYwMq1iFMnMK8tNzMwx1SvOzqjMy6zQS87P 3cQIjLVltX8m7mD8ctn9EKMAB6MSD6/HXL40IdbEsuLK3EOMEhzMSiK83Q68aUK8KYmVValF +fFFpTmpxYcYpTlYlMR5vcJTE4QE0hNLUrNTUwtSi2CyTBycUg2MrXcDg/iWGvI5yvDcEPSw P7yqU3jJ5C2Pvy4L8fm1582vb8cbBHa9dWFsOCph3xcfE7ij53oE2/o5kXf2dcY+e1Nr+oMr 4Hrr5n1XHJdsEYkI9Jt05UHSp92tM6sTGmznMuj59LzSXeVnl+zzf8/UavV1Jhr7TaNcmH7q CmYbMthL8AfIuscosRRnJBpqMRcVJwIAr8ixFrECAAA= X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: E720F40022 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: mtnqam31qk3bk5fq8t77rfrqnku3z9d4 X-HE-Tag: 1712297353-172573 X-HE-Meta: U2FsdGVkX19X8NtJSrKVUXTXEGyiKMjVuvrYMwO9a/J1w5l/EY+QeOa1tsY8Ji6xINU16zBWP4MakGuKpSkWEEmzeUdJdxeI0brfSOi9ocZ4fY21dOgGinsLtg1fttw8mxHXd1Cc4i+ndJ0w1zJ8fynl4PydaBC9b/0iuP1rT6ksNQJXTEccrFBTmSuUOuPTRmOKTOc3tE7kD/c250+TSvavnA3VF1rcOJ2B7SYeLO1sm/j6CG/8TuM8SixcvMCxfBGVembd3wiVk96RULzQBDWy1qP5Ui1FedgK1KCxy0lcd5xci0Ix7Xi8yaNT2b5ZYHO1sr7v6Oxgq7+Y03zFXADQ7q8DD7Im5lFyTmoSP1P/AeG3i1Ax1SRr19YlMKbnWlfD6j3rYaASnPJ9b3+UESW5ZX2sy9INddmhvKpqHPAOkHFddnd9/VpAzWcIKoIX0m0c1dikEQH/7cwlEzicJRO+5V1++Odi1FXxRlA8O9oENi1cbJAQXWLQhwyuitMdT3fRBXSaWkdgNsSMZGNS9YWDvsKLikXL8GpkM7UNil4SCZb1TrWVegu2F1y6Cy2kAjH+I3aJaDOsnlazHA2RSFP8rsCrK3GO49xycaI0Zy4NkwFgURnQsmutIaq3PcE8wLV9MUH0WM5QQvtBblTUOUmtk+JnIMXv4RPFFSCoAV7Yd/2DY4LAVEv8v3Wwu0cMUp8yu9Z0IoRsItr7e/+JGPtGt1II5RBu5FRLz+AYdItdLV2AeJGyWSRsrruUbItMbJbNuSB7Drc4A6haZAQcnRwMPMAc+eOjKT5AcPjEEMOhHO0mlaH4Rp8K7Ub+9+n3i/cJLnZ1gIPMGDn8rl0d67R4jMoUsNfj8IRPhny489vxazK8KxfL+kbrTv9CjXFcyMzN0HG9gvlWT3bBst7VVNyzj/fPJmHEyDHaqxg9Z+w85cq0Oo6YZxzJ5Xh1fGcY/GJHKWjxuVxbFjXb826 l9xC/J7B W5YDWL0EsJuIPqYxPwH6jXACbKtXdWTd05OIFGYamnckMfE6AwDLFakaV96L1CyfaGsC8YGLGD/CkhxWNUhTOsfh+TsHLaupF5jAXaTkPXoej4HAu9QqpP/sL1K2ZLwLvEruKSuAPOie4gzt1W3JEPjLQFg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch introduces DAMOS_MIGRATE_COLD action, which is similar to DAMOS_PAGEOUT, but migrate folios to the given 'target_nid' in the sysfs instead of swapping them out. The 'target_nid' sysfs knob is created by this patch to inform the migration target node ID. Here is one of the example usage of this 'migrate_cold' action. $ cd /sys/kernel/mm/damon/admin/kdamonds/ $ cat contexts//schemes//action migrate_cold $ echo 2 > contexts//schemes//target_nid $ echo commit > state $ numactl -p 0 ./hot_cold 500M 600M & $ numastat -c -p hot_cold Per-node process memory usage (in MBs) PID Node 0 Node 1 Node 2 Total -------------- ------ ------ ------ ----- 701 (hot_cold) 501 0 601 1101 Since there are some common routines with pageout, many functions have similar logics between pageout and migrate cold. damon_pa_migrate_folio_list() is a minimized version of shrink_folio_list(), but it's minified only for demotion. Signed-off-by: Honggyu Kim Signed-off-by: Hyeongtak Ji --- include/linux/damon.h | 2 + mm/damon/paddr.c | 146 ++++++++++++++++++++++++++++++++++++++- mm/damon/sysfs-schemes.c | 4 ++ 3 files changed, 151 insertions(+), 1 deletion(-) diff --git a/include/linux/damon.h b/include/linux/damon.h index 24ea33a03d5d..df8671e69a70 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -105,6 +105,7 @@ struct damon_target { * @DAMOS_NOHUGEPAGE: Call ``madvise()`` for the region with MADV_NOHUGEPAGE. * @DAMOS_LRU_PRIO: Prioritize the region on its LRU lists. * @DAMOS_LRU_DEPRIO: Deprioritize the region on its LRU lists. + * @DAMOS_MIGRATE_COLD: Migrate for the given cold region. * @DAMOS_STAT: Do nothing but count the stat. * @NR_DAMOS_ACTIONS: Total number of DAMOS actions * @@ -122,6 +123,7 @@ enum damos_action { DAMOS_NOHUGEPAGE, DAMOS_LRU_PRIO, DAMOS_LRU_DEPRIO, + DAMOS_MIGRATE_COLD, DAMOS_STAT, /* Do nothing but only record the stat */ NR_DAMOS_ACTIONS, }; diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index 277a1c4d833c..fe217a26f788 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -12,6 +12,9 @@ #include #include #include +#include +#include +#include #include "../internal.h" #include "ops-common.h" @@ -226,8 +229,137 @@ static bool damos_pa_filter_out(struct damos *scheme, struct folio *folio) enum migration_mode { MIG_PAGEOUT, + MIG_MIGRATE_COLD, }; +static unsigned int migrate_folio_list(struct list_head *migrate_folios, + struct pglist_data *pgdat, + int target_nid) +{ + unsigned int nr_succeeded; + nodemask_t allowed_mask = NODE_MASK_NONE; + + struct migration_target_control mtc = { + /* + * Allocate from 'node', or fail quickly and quietly. + * When this happens, 'page' will likely just be discarded + * instead of migrated. + */ + .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOWARN | + __GFP_NOMEMALLOC | GFP_NOWAIT, + .nid = target_nid, + .nmask = &allowed_mask + }; + + if (pgdat->node_id == target_nid || target_nid == NUMA_NO_NODE) + return 0; + + if (list_empty(migrate_folios)) + return 0; + + /* Migration ignores all cpuset and mempolicy settings */ + migrate_pages(migrate_folios, alloc_migrate_folio, NULL, + (unsigned long)&mtc, MIGRATE_ASYNC, MR_DAMON, + &nr_succeeded); + + return nr_succeeded; +} + +static unsigned int damon_pa_migrate_folio_list(struct list_head *folio_list, + struct pglist_data *pgdat, + enum migration_mode mm, + int target_nid) +{ + unsigned int nr_migrated = 0; + struct folio *folio; + LIST_HEAD(ret_folios); + LIST_HEAD(migrate_folios); + + cond_resched(); + + while (!list_empty(folio_list)) { + struct folio *folio; + + cond_resched(); + + folio = lru_to_folio(folio_list); + list_del(&folio->lru); + + if (!folio_trylock(folio)) + goto keep; + + VM_BUG_ON_FOLIO(folio_test_active(folio), folio); + + /* Relocate its contents to another node. */ + list_add(&folio->lru, &migrate_folios); + folio_unlock(folio); + continue; +keep: + list_add(&folio->lru, &ret_folios); + VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); + } + /* 'folio_list' is always empty here */ + + /* Migrate folios selected for migration */ + nr_migrated += migrate_folio_list(&migrate_folios, pgdat, target_nid); + /* Folios that could not be migrated are still in @migrate_folios */ + if (!list_empty(&migrate_folios)) { + /* Folios which weren't migrated go back on @folio_list */ + list_splice_init(&migrate_folios, folio_list); + } + + try_to_unmap_flush(); + + list_splice(&ret_folios, folio_list); + + while (!list_empty(folio_list)) { + folio = lru_to_folio(folio_list); + list_del(&folio->lru); + folio_putback_lru(folio); + } + + return nr_migrated; +} + +static unsigned long damon_pa_migrate_pages(struct list_head *folio_list, + enum migration_mode mm, + int target_nid) +{ + int nid; + unsigned int nr_migrated = 0; + LIST_HEAD(node_folio_list); + unsigned int noreclaim_flag; + + if (list_empty(folio_list)) + return nr_migrated; + + noreclaim_flag = memalloc_noreclaim_save(); + + nid = folio_nid(lru_to_folio(folio_list)); + do { + struct folio *folio = lru_to_folio(folio_list); + + if (nid == folio_nid(folio)) { + folio_clear_active(folio); + list_move(&folio->lru, &node_folio_list); + continue; + } + + nr_migrated += damon_pa_migrate_folio_list(&node_folio_list, + NODE_DATA(nid), mm, + target_nid); + nid = folio_nid(lru_to_folio(folio_list)); + } while (!list_empty(folio_list)); + + nr_migrated += damon_pa_migrate_folio_list(&node_folio_list, + NODE_DATA(nid), mm, + target_nid); + + memalloc_noreclaim_restore(noreclaim_flag); + + return nr_migrated; +} + static unsigned long damon_pa_migrate(struct damon_region *r, struct damos *s, enum migration_mode mm) { @@ -247,7 +379,11 @@ static unsigned long damon_pa_migrate(struct damon_region *r, struct damos *s, folio_test_clear_young(folio); if (!folio_isolate_lru(folio)) goto put_folio; - if (folio_test_unevictable(folio)) + /* + * Since unevictable folios can be demoted or promoted, + * unevictable test is needed only for pageout. + */ + if (mm == MIG_PAGEOUT && folio_test_unevictable(folio)) folio_putback_lru(folio); else list_add(&folio->lru, &folio_list); @@ -258,6 +394,10 @@ static unsigned long damon_pa_migrate(struct damon_region *r, struct damos *s, case MIG_PAGEOUT: applied = reclaim_pages(&folio_list); break; + case MIG_MIGRATE_COLD: + applied = damon_pa_migrate_pages(&folio_list, mm, + s->target_nid); + break; default: /* Unexpected migration mode. */ return 0; @@ -314,6 +454,8 @@ static unsigned long damon_pa_apply_scheme(struct damon_ctx *ctx, return damon_pa_mark_accessed(r, scheme); case DAMOS_LRU_DEPRIO: return damon_pa_deactivate_pages(r, scheme); + case DAMOS_MIGRATE_COLD: + return damon_pa_migrate(r, scheme, MIG_MIGRATE_COLD); case DAMOS_STAT: break; default: @@ -334,6 +476,8 @@ static int damon_pa_scheme_score(struct damon_ctx *context, return damon_hot_score(context, r, scheme); case DAMOS_LRU_DEPRIO: return damon_cold_score(context, r, scheme); + case DAMOS_MIGRATE_COLD: + return damon_cold_score(context, r, scheme); default: break; } diff --git a/mm/damon/sysfs-schemes.c b/mm/damon/sysfs-schemes.c index 1a30ea82c890..18b7d054c748 100644 --- a/mm/damon/sysfs-schemes.c +++ b/mm/damon/sysfs-schemes.c @@ -1406,6 +1406,7 @@ static const char * const damon_sysfs_damos_action_strs[] = { "nohugepage", "lru_prio", "lru_deprio", + "migrate_cold", "stat", }; @@ -1659,6 +1660,9 @@ static ssize_t target_nid_store(struct kobject *kobj, struct damon_sysfs_scheme, kobj); int err = 0; + if (scheme->action != DAMOS_MIGRATE_COLD) + return -EINVAL; + /* TODO: error handling for target_nid range. */ err = kstrtoint(buf, 0, &scheme->target_nid);