From patchwork Wed Nov 27 02:57:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13886511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AE02D66BAE for ; Wed, 27 Nov 2024 02:57:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01B908D0005; Tue, 26 Nov 2024 21:57:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F0DCA8D0001; Tue, 26 Nov 2024 21:57:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D3B8F8D0005; Tue, 26 Nov 2024 21:57:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AEEAC8D0001 for ; Tue, 26 Nov 2024 21:57:55 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3D301AEE5E for ; Wed, 27 Nov 2024 02:57:55 +0000 (UTC) X-FDA: 82830364932.20.A4CCC8F Received: from mail-oa1-f74.google.com (mail-oa1-f74.google.com [209.85.160.74]) by imf23.hostedemail.com (Postfix) with ESMTP id 049B8140015 for ; Wed, 27 Nov 2024 02:57:49 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=RFI5M5MS; spf=pass (imf23.hostedemail.com: domain of 3sIpGZwcKCCEVR7K9ERDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuanchu.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3sIpGZwcKCCEVR7K9ERDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732676269; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t/WPTXcvcabaljBBrVkxMdtvNE4T0NFrHkMqJHgXQGk=; b=qnH4FmI5vKA59U/dwRgY61vaVoFfnBdAfTGlljI1DJ6SY/ZEVV5dvhjYpYZZKRButC48tR Ir2mNV8ynoR+tG3Eyw+PuiIpP7Mh1ZVilZzbIMVrcl7i4LaMklUlAl6hDyxGWgYZ32XF61 I2oBHJkHslFsnrMgIkgkswZ3wKiQjrQ= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=RFI5M5MS; spf=pass (imf23.hostedemail.com: domain of 3sIpGZwcKCCEVR7K9ERDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuanchu.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3sIpGZwcKCCEVR7K9ERDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732676269; a=rsa-sha256; cv=none; b=WoTIl80nx1xgKrCQFsO21h5Bt6yxuU0pKKez6wttTCuXkaQQvHrnBcALfH9Qtwu77k7uAc nzshwGN3KzDxBfbqXUSbpL6WVlabPFL6SvSvR3F1Xp2/e8tsrGnN3gBG4ztcv9eaqK+MRl +Fysmrnqxov+oZ+Xh4xxw+QTy8M4Nlg= Received: by mail-oa1-f74.google.com with SMTP id 586e51a60fabf-29678d451b1so3433414fac.3 for ; Tue, 26 Nov 2024 18:57:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1732676272; x=1733281072; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=t/WPTXcvcabaljBBrVkxMdtvNE4T0NFrHkMqJHgXQGk=; b=RFI5M5MSy2fKdSRxGP37cLehQ6cg2M0GWzC7M/shZXpIMJ/oO+sNLwohXDacwBkh5D eLFGP7oqW7OJWNvBtE3Esc05ce3Dx/FYX0FfsenktjP0N1jJZL2Om/H7pj+3rWnBwXDt OrNWtCvNQcxCL13AEX1zVub+BePxALobp89tRU16DikVyjPzRp6iWGw4Yy6n/piBSIQ1 5qk7qAB3hachoWCzPKcuZJ7ieGIWxQtUalFPh1vsOIH9zgqGqUtSvOnyjLlJzJ6oZCjN jjfH5PgkbanElc3V8Wckiiw4Sx5Pwi//haLXcBJ02fnL9fzqRny89l7nI8PvJcz+aOHo EoCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732676272; x=1733281072; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=t/WPTXcvcabaljBBrVkxMdtvNE4T0NFrHkMqJHgXQGk=; b=ny9WqoUVrXDk14enT0lCPsSR+xkXLwKiRf7mIBKY/xlXgEeRGNcfXsnRAF5+CKK9jK HSPjOoY/KEWveMFTMmj4xDgt4s4F6Voa6x8P3ah3W4ZHDXFkRXxDCyOBVOPexLyrJsRw bUI7BNw74vUUMus8fe4ISmGnStpllC1Pyh31YJ/EqX62tuMx22naCbuoZ6GIRq0Sbr1h GGmW79UlamN3tXEQjci40PDBSCGxnWuUKoCLamUN1CLSDjp/d5hamOe2QIccCrSk4ogO yOMbr4eOGm/+qMj9ctbISVdMWM2FnKSFmik3YC95g6cH7Jm/3IVrf323ELWtMzJ9Scb0 oKkg== X-Forwarded-Encrypted: i=1; AJvYcCWuewfwxSpwN80JMcxxrD472oC6Bwl2fnRLW1XX73MVCzyxT+jTS0clvzUeWQFD1nqFt6DaDtKK6w==@kvack.org X-Gm-Message-State: AOJu0Yw34RXkF+69sy+MsiTDxOW+rmFisLopvmaQwuYV+WFlcV6BvOvw MiI5fNHg+ueYvQVWJLAn0kczqfdxCkBGguAJrFfHOeVSgy8YhU1swtnYgxkAJYZ7ZqrQM6a4nkE aU+2xLA== X-Google-Smtp-Source: AGHT+IFRcRWDlCpSpvonVfz6gqukD5+iS1Nh+k6upjKjqm+Gv9S2zOic8Epnn/W/99f5T4mKjm5ZX+88GSwn X-Received: from oacnq1.prod.google.com ([2002:a05:6871:3781:b0:296:523f:4d02]) (user=yuanchu job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6870:4996:b0:297:2376:9b07 with SMTP id 586e51a60fabf-29dc400538dmr1178830fac.10.1732676272567; Tue, 26 Nov 2024 18:57:52 -0800 (PST) Date: Tue, 26 Nov 2024 18:57:24 -0800 In-Reply-To: <20241127025728.3689245-1-yuanchu@google.com> Mime-Version: 1.0 References: <20241127025728.3689245-1-yuanchu@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241127025728.3689245-6-yuanchu@google.com> Subject: [PATCH v4 5/9] mm: add kernel aging thread for workingset reporting From: Yuanchu Xie To: Andrew Morton , David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Tejun Heo , Johannes Weiner , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , " =?utf-8?q?Eugenio_P=C3=A9rez?= " , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Mike Rapoport , Shuah Khan , Christian Brauner , Daniel Watson , Yuanchu Xie , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, linux-mm@kvack.org, linux-kselftest@vger.kernel.org X-Rspamd-Queue-Id: 049B8140015 X-Rspamd-Server: rspam12 X-Stat-Signature: adwg83gd65jrqsnbcnbyst4qbfkf5oyh X-Rspam-User: X-HE-Tag: 1732676269-942221 X-HE-Meta: U2FsdGVkX1+tVgnbOHOCuMXsJOFFYl+eR2YLfXzZQOvv2A/dxnPBBYCn+3L31KsFja24LHoi2PTKYS7BJpQFxnnPsYXSfIhTErZUs0xXJhX0eIalztFx4BSQs97b4Z5kQOhAdnuS5H9waSL71jsjq06ccgSEHJGizxjszcDGGvKIwOJY3mgPhsW0iQqL3fWJyoe6mIduJGIAUA10dMCq4yJGfAYojNwR1DSqOECC+m1PJXRMl+6r3pO7fswAq1K4XIdlthiO8i5kLyus1BfiPV09DGlPD3FUvDZwPCVg8pUPzRXAmqLqG3vf6/1WrIOPhc3hatVzhBhvDZoxoqAXcVxIiL68eaMSGx2I8sgE6NNv+Bz+Q8d7ZXJj/yIMZLiCJ3Pynp+9M7NJQjGyF91cBiTrug33zKPwxIjBBDaPqCxb5raHYu7YOmvCTUv1U3bcvxV55+T8jtu3V5u4iB6J3e8ZCbohiT8ti2mO6A5RRUO1Z7adl2eDMJqvTj8akI4LAxmv/GpaEjmHRe7Y3tjZ3a+DlQs143g3l1gTDgjYEnjGuw5zGGRjGfPzCPMyesyI4k5ZPmfe8BGk/VW30yfsEAnEg1LWzSI8daLJF0HyFFb1osnZ0B2DzwK8S+m5sV84zhmyoNBgNeb1I8J/yNsWshm3yDUtEXuGmBWznlvQK2hoUgtmRyEkm7us66rNumcS8ARkHNX5Nv6pPZtP1oxVCAmZAiKBVxtwEdgV2GA4nL3P4QasAvXb+deeVnMzdsfSf4oRe269eJMi2X7+wZ6moAfNZxXfRma2DouK9lyOwVN9g4aaGSc6ul4TdlNusPYT5SV/eEFh0FUoOX1cIx9KBbPXxQEftd/mzsa2RapPY/Yu1BaZnzHcERb2mhrfAvim5VA+GrW5QigVQP8J2yyyy8ybKnHPNNCAzMvxSGIv35AJ5M2bIBK6YsfI8koGo0giYqjUuIkkfrwxU8WhjpN /y/qngPM jXW9Q9cqUE7sD1tUX6cC7jAZtCGcRMGFfyUZu4ALjg6vGdvFPe3DMkfyIEc949FBkR9Lmj3vcC0/a072i6+FgwwW6sadoQFHqltiqj/gk28yXHTlpRks3QU4BMNT1pvm92rfdcZZiylaMnuYfcu39jG9YADxG3SrOOdw7WE1x0BtCEZ8uvcw9Gm5PcUV6TDjUwBMZ2VwyiQhaQbIH1sjypRfq1xucI5sQU5ZDiPzrWXYSnNwm1vyoIsIc/E73f+HqEHlNdsgW/ncaESwjAEVa9LlVjQUw02VT1bxTflJ+XnUpbf1GQrE4DNGhJRWrNPypk/oLEyYL10QDlb6YMu4IixQ3ZMoC6ejn0oz3TyqspJtaraq+zISq3JlLk3CPe6l3yQdWkmt+iuTmHodxuFyJqd9y2oxH3crXlSAViae/e1OF2RyhFAqdG8TFTavkEEyy9ja4Zb0Ra8MWv2zNN/XxLcDFfU4AgwfIRLIeN/sEsB7nPmmBSvMTsNdJOn1csQ1fvLLVTtt4py43G+hG6eorfgiHLD2XMmWXS9DvFA35qA8Ptmo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For reliable and timely aging on memcgs, one has to read the page age histograms on time. A kernel thread makes it easier by aging memcgs with valid refresh_interval when they can be refreshed, and also reduces the latency of any userspace consumers of the page age histogram. The kerne aging thread is gated behind CONFIG_WORKINGSET_REPORT_AGING. Debugging stats may be added in the future for when aging cannot keep up with the configured refresh_interval. Signed-off-by: Yuanchu Xie --- include/linux/workingset_report.h | 10 ++- mm/Kconfig | 6 ++ mm/Makefile | 1 + mm/memcontrol.c | 2 +- mm/workingset_report.c | 13 ++- mm/workingset_report_aging.c | 127 ++++++++++++++++++++++++++++++ 6 files changed, 154 insertions(+), 5 deletions(-) create mode 100644 mm/workingset_report_aging.c diff --git a/include/linux/workingset_report.h b/include/linux/workingset_report.h index 616be6469768..f6bbde2a04c3 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -64,7 +64,15 @@ void wsr_remove_sysfs(struct node *node); * The next refresh time is stored in refresh_time. */ bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, - struct pglist_data *pgdat); + struct pglist_data *pgdat, unsigned long *refresh_time); + +#ifdef CONFIG_WORKINGSET_REPORT_AGING +void wsr_wakeup_aging_thread(void); +#else /* CONFIG_WORKINGSET_REPORT_AGING */ +static inline void wsr_wakeup_aging_thread(void) +{ +} +#endif /* CONFIG_WORKINGSET_REPORT_AGING */ int wsr_set_refresh_interval(struct wsr_state *wsr, unsigned long refresh_interval); diff --git a/mm/Kconfig b/mm/Kconfig index be949786796d..a8def8c65610 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1310,6 +1310,12 @@ config WORKINGSET_REPORT This option exports stats and events giving the user more insight into its memory working set. +config WORKINGSET_REPORT_AGING + bool "Workingset report kernel aging thread" + depends on WORKINGSET_REPORT + help + Performs aging on memcgs with their configured refresh intervals. + source "mm/damon/Kconfig" endmenu diff --git a/mm/Makefile b/mm/Makefile index f5ef0768253a..3a282510f960 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -99,6 +99,7 @@ obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o obj-$(CONFIG_WORKINGSET_REPORT) += workingset_report.o +obj-$(CONFIG_WORKINGSET_REPORT_AGING) += workingset_report_aging.o ifdef CONFIG_SWAP obj-$(CONFIG_MEMCG) += swap_cgroup.o endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d1032c6efc66..ea83f10b22a1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4462,7 +4462,7 @@ static int memory_ws_page_age_show(struct seq_file *m, void *v) if (!READ_ONCE(wsr->page_age)) continue; - wsr_refresh_report(wsr, memcg, NODE_DATA(nid)); + wsr_refresh_report(wsr, memcg, NODE_DATA(nid), NULL); mutex_lock(&wsr->page_age_lock); if (!wsr->page_age) goto unlock; diff --git a/mm/workingset_report.c b/mm/workingset_report.c index 1e1bdb8bf75b..dad539e602bb 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -283,7 +283,7 @@ static void copy_node_bins(struct pglist_data *pgdat, } bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, - struct pglist_data *pgdat) + struct pglist_data *pgdat, unsigned long *refresh_time) { struct wsr_page_age_histo *page_age; unsigned long refresh_interval = READ_ONCE(wsr->refresh_interval); @@ -300,10 +300,14 @@ bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, goto unlock; if (page_age->timestamp && time_is_after_jiffies(page_age->timestamp + refresh_interval)) - goto unlock; + goto time; refresh_scan(wsr, root, pgdat, refresh_interval); copy_node_bins(pgdat, page_age); refresh_aggregate(page_age, root, pgdat); + +time: + if (refresh_time) + *refresh_time = page_age->timestamp + refresh_interval; unlock: mutex_unlock(&wsr->page_age_lock); return !!page_age; @@ -386,6 +390,9 @@ int wsr_set_refresh_interval(struct wsr_state *wsr, WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(refresh_interval)); unlock: mutex_unlock(&wsr->page_age_lock); + if (!err && refresh_interval && + (!old_interval || jiffies_to_msecs(old_interval) > refresh_interval)) + wsr_wakeup_aging_thread(); return err; } @@ -491,7 +498,7 @@ static ssize_t page_age_show(struct kobject *kobj, struct kobj_attribute *attr, int ret = 0; struct wsr_state *wsr = kobj_to_wsr(kobj); - wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj)); + wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj), NULL); mutex_lock(&wsr->page_age_lock); if (!wsr->page_age) diff --git a/mm/workingset_report_aging.c b/mm/workingset_report_aging.c new file mode 100644 index 000000000000..91ad5020778a --- /dev/null +++ b/mm/workingset_report_aging.c @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Workingset report kernel aging thread + * + * Performs aging on behalf of memcgs with their configured refresh interval. + * While a userspace program can periodically read the page age breakdown + * per-memcg and trigger aging, the kernel performing aging is less overhead, + * more consistent, and more reliable for the use case where every memcg should + * be aged according to their refresh interval. + */ +#define pr_fmt(fmt) "workingset report aging: " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static DECLARE_WAIT_QUEUE_HEAD(aging_wait); +static bool refresh_pending; + +static bool do_aging_node(int nid, unsigned long *next_wake_time) +{ + struct mem_cgroup *memcg; + bool should_wait = true; + struct pglist_data *pgdat = NODE_DATA(nid); + + memcg = mem_cgroup_iter(NULL, NULL, NULL); + do { + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); + struct wsr_state *wsr = &lruvec->wsr; + unsigned long refresh_time; + + /* use returned time to decide when to wake up next */ + if (wsr_refresh_report(wsr, memcg, pgdat, &refresh_time)) { + if (should_wait) { + should_wait = false; + *next_wake_time = refresh_time; + } else if (time_before(refresh_time, *next_wake_time)) { + *next_wake_time = refresh_time; + } + } + + cond_resched(); + } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL))); + + return should_wait; +} + +static int do_aging(void *unused) +{ + while (!kthread_should_stop()) { + int nid; + long timeout_ticks; + unsigned long next_wake_time; + bool should_wait = true; + + WRITE_ONCE(refresh_pending, false); + for_each_node_state(nid, N_MEMORY) { + unsigned long node_next_wake_time; + + if (do_aging_node(nid, &node_next_wake_time)) + continue; + if (should_wait) { + should_wait = false; + next_wake_time = node_next_wake_time; + } else if (time_before(node_next_wake_time, + next_wake_time)) { + next_wake_time = node_next_wake_time; + } + } + + if (should_wait) { + wait_event_interruptible(aging_wait, refresh_pending); + continue; + } + + /* sleep until next aging */ + timeout_ticks = next_wake_time - jiffies; + if (timeout_ticks > 0 && + timeout_ticks != MAX_SCHEDULE_TIMEOUT) { + schedule_timeout_idle(timeout_ticks); + continue; + } + } + return 0; +} + +/* Invoked when refresh_interval shortens or changes to a non-zero value. */ +void wsr_wakeup_aging_thread(void) +{ + WRITE_ONCE(refresh_pending, true); + wake_up_interruptible(&aging_wait); +} + +static struct task_struct *aging_thread; + +static int aging_init(void) +{ + struct task_struct *task; + + task = kthread_run(do_aging, NULL, "kagingd"); + + if (IS_ERR(task)) { + pr_err("Failed to create aging kthread\n"); + return PTR_ERR(task); + } + + aging_thread = task; + pr_info("module loaded\n"); + return 0; +} + +static void aging_exit(void) +{ + kthread_stop(aging_thread); + aging_thread = NULL; + pr_info("module unloaded\n"); +} + +module_init(aging_init); +module_exit(aging_exit);