From patchwork Tue Aug 13 16:56:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13762344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6766AC52D7C for ; Tue, 13 Aug 2024 16:59:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 44D836B0098; Tue, 13 Aug 2024 12:59:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 408D96B0095; Tue, 13 Aug 2024 12:59:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2281C6B0098; Tue, 13 Aug 2024 12:59:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 010BD6B0092 for ; Tue, 13 Aug 2024 12:59:40 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id AE8FF1A08B7 for ; Tue, 13 Aug 2024 16:59:40 +0000 (UTC) X-FDA: 82447833720.17.A72B2EA Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf24.hostedemail.com (Postfix) with ESMTP id CD5DF180012 for ; Tue, 13 Aug 2024 16:59:38 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nIkn+EC6; spf=pass (imf24.hostedemail.com: domain of 3-ZC7ZgcKCEY62ivkp2owwotm.kwutqv25-uus3iks.wzo@flex--yuanchu.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3-ZC7ZgcKCEY62ivkp2owwotm.kwutqv25-uus3iks.wzo@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723568367; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=N9o3rrXvjK3EuUuaBsq4Dq2VUkpZajOoCi9sWYw3qPg=; b=vbQftkAurP5sf8rbzUk4vSjC7m5kP9IjEcgEzq4cALsIM07h11GPWOyhBj+glq8JLMKjbQ P2SsVgpIqjyXAQzPP/ct6BgfqM6OZyxR/Bs6k+iQNgpAOReDKcW99nZbAhlUgFidR1yNBI prH85xw7fyOXSIcRZ8ywl4k5rOeFRmM= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nIkn+EC6; spf=pass (imf24.hostedemail.com: domain of 3-ZC7ZgcKCEY62ivkp2owwotm.kwutqv25-uus3iks.wzo@flex--yuanchu.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3-ZC7ZgcKCEY62ivkp2owwotm.kwutqv25-uus3iks.wzo@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723568367; a=rsa-sha256; cv=none; b=FYb2zYPkPgYiP0JjXy6NuHauljLSSnzRNfK281Ds7/9/zMxFZMPTuUDsOYDhdW0fUAswGP MOTaL07zlAnjhDGb6FnH3COQwGOK54Gb5+B8/VAgwMQqZElCg8W1SM2aAVwgxo423TXxr7 b0EasReT16B7YatQUznD5TmXbS6R8QI= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-650b621f4cdso123686247b3.1 for ; Tue, 13 Aug 2024 09:59:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723568378; x=1724173178; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=N9o3rrXvjK3EuUuaBsq4Dq2VUkpZajOoCi9sWYw3qPg=; b=nIkn+EC6SzWkjbO7oWX1MHuMX395wMepVE0gdmsW3KRcLiFJ+XXH6tj2J+896T1+H6 O3VMV5deYK0nRW1L77BOKYdR0qodqE1zvRat/KDOQ1yz60V/xqfHTGXy80YOTmfuFheH 0P9Y9Ram14TNgUJ7+KIbxnUAv0yxLJcbWiF6HvGsfMlKAfYe5Ir711W+Ba1MTKN1YZLi i/bAMtl3tN9gzxng3dOxiz1oxIgvkdk+QWwB1mxjg+OATccrCmstveTxJ509i/aqWQoL 0MBlFUQZNj+BY20E01HQXa4/erF6tysGTBh3th2GSX35FFWdwaY6SEtqPAXCAV+NQb4p Ypzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723568378; x=1724173178; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=N9o3rrXvjK3EuUuaBsq4Dq2VUkpZajOoCi9sWYw3qPg=; b=jA0ohSf7YDmNNehnECO1/wq+f44mXArtTNlKb5E/5fIpgG3QyGr7EccjTOpjhhj4Jb RmygwhIQ3klZ8VEQYHV3NBJnbUNw3HHTEQ9cMRKUXC0KeZqzojzBCim+ALkZuPbksIeE uZ+1vO/UBc3i/zAbxktSWkhdB4CG447akRvYpn1IRJKKToz79Y79w34L6mVFtbOdLgx/ qVTFKMKSsCisZpJXwtuV4QJ6u0RDSijC1lHLH7hkaVNzNH+sM0SD6EuAmuU+1OAKDpxS AjkDXvsMRDRz/I+q5M1aU/nxKg1xuydFvWTh7Jh4EhNPxw9EvM/EPJ3r+PTgqSzJR/uW 14Mg== X-Forwarded-Encrypted: i=1; AJvYcCVZhopeUgWruYMnC4dVsHV+wki34iQg1455DPCx4IfYx8Fwm6RjkQ0mVync8V9J5YDu136dBFTWKAkaXigls6SJASE= X-Gm-Message-State: AOJu0YzXU3ha2TOlZPjtcYL4RvHeSq2ui8HLS8jLtpIEvmwt1VYaBWS4 j2M6MeH2BxxTLPSPygPwez+7qK5fBWccT89Gj0/LWUm+RU6Ygj3vbE36SlMSFZtX6qYhcTMsELe fBt9EfA== X-Google-Smtp-Source: AGHT+IFcT1PaXJoLmzamp9fHH9zjFph1X3u8jmQRIW7x8w/hZfTh+8gA42aT7LBM4EO5qJIAh6kA4AaV1iY6 X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:b50c:66e8:6532:a371]) (user=yuanchu job=sendgmr) by 2002:a81:8506:0:b0:691:2f66:4af9 with SMTP id 00721157ae682-6ac98b677b4mr1967b3.4.1723568377681; Tue, 13 Aug 2024 09:59:37 -0700 (PDT) Date: Tue, 13 Aug 2024 09:56:12 -0700 In-Reply-To: <20240813165619.748102-1-yuanchu@google.com> Mime-Version: 1.0 References: <20240813165619.748102-1-yuanchu@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813165619.748102-2-yuanchu@google.com> Subject: [PATCH v3 1/7] mm: aggregate working set information into histograms From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Andrew Morton , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org X-Rspam-User: X-Stat-Signature: e5x5ttir6qwejz7kpjnntbg871tycwur X-Rspamd-Queue-Id: CD5DF180012 X-Rspamd-Server: rspam11 X-HE-Tag: 1723568378-351472 X-HE-Meta: U2FsdGVkX18cdGm9Puxnfoa1Zw1dO15dFF+IKDYMwVv1Qf2Kd3JrK0mLoxxQEGZ7iBTL8mScsyWr/rg4/k7UBl6B1HXGMHUxaKsZrXIhvl9Pv8ZarnD18YPnWjb8O5wQ8Zx3HClLysx5ISnTGN3GGLwtxkurCVIctfbTnF+ECrNbkAVKe5eyNZohlJkXeQpVrvbNsj9f3kiEdwc0Z1o/y1gqoIjYlJuxqoF2e/ug5rJNaicN1f0vuliaDB8WTt3FeDsMQp60Spr+JAALV1KRk5CL6Mwti/4nRogY15wPC2fIuPkyqN8tYnpJ+UFpAyr58yJndI0gXSTx7sullS2MJknKDK5vQ32zwralnVw42pQx8VAvG85PDg7/FmPW5B/ITmOOEcO9KbyAegfPgoDhSU8OcthmJzmQQP6yoAP9EiCkbPj0zcgBjBQUrWxYND+8wLQyIqU6OsbKb/m1iJKRCrDOUMew3rExYfTe42t7Xgywx5b8rPSw/dZZkNPAoRJ/DWNaQ/8iL1maGNwziJWwfhAct05yU+Iy2fRETwoDzsPMRRfS35TdYg6DKCyVL+tu2mZ3Asdgk9Cpo2Az1rZZ/YJEccPdRk7klYzvNVADUQXRNuMrpBlQ9j8utv61g/DR4FAG46IZHMFjhNwQEdHFP1Clc+Xvc7KQ4nJ21oLe/gaD4J2+Ccgdb247dZQUhuEWJu2SEpX6yHNYMUipuNB3OVIsdpLNCtralzJ49FQ29+6haOhPnYe4a2vXFvlQPe7E5YDW0KXxUXRoiBu35wRMifWHUIhM29pir/Om+vchQRqU02nZUtUIOZaQ8msNvgTb7offl2+YKGzycwNOHkAXnuMNB96oOiR/dBWLJ29z6k+vFIpCEVSFH0HhgfKwVHZUa+q8EghZ1Xn+ckimkZ9KeBHQFkGCchjLUSeeJKwlH9LEe12t6EqWKkrqdpjoLmzmGlaj00OTHsaJK1lbGI+ KY7jVHo8 kkUqGKmf4OMmgVZ7cd9YfyYQKKdaBlkNNaTqkZqKIxgUw7W6Y912K5OgwMt9JLROYWcrC8eyH2yr1/D1JI66gQLb3dESwIm1xYVTiq+CiPgXF+ykOuwUGSNLECOF/CM0ZY8d2nvEirqz+QD3klDULoG/4Uj8LjtVu2glgraAXqgDrOS+o8vvVI2SR6LRsiBUBk2ZutIsFY1u+kd/AwudGv93jGZNu9x9J6dWd1ePL0aFqblb5mshTN1VFBbhaHBT254hXw4bWz6KEi6edxDzXpFplF5pVhcMhbIS0gZO8109qQ70GMXPbMRcJA0dMI1wmMa5GbG0j4Z4fwJE12r1g/fMHb9icMZmx7kSVj3eWdH62G0bPwYjm/F9aAOi3MyU7ix0atOaKeU8G8Eqp7irjPDcNxKwILeccdR58Tjb7TjSzpG/RP+G9qcLZYdoYadlEelnb/MVg84YTx3o/5/+ZoIytZ+sRbuajut+/GPvr4KqeGSvOhK6J45KrqHQzkj9EtG4Yn7eunpFEJ+FqJTXBK0Ll9i3NBkNXXyftoqH2YT3n/5+Y1A9V90SijkejYUnX0axAjNKRF5MQKSfJ8/nbnqmG8AmhS0iFMeaa X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hierarchically aggregate all memcgs' MGLRU generations and their page counts into working set page age histograms. The histograms break down the system's working set per-node, per-anon/file. The sysfs interfaces are as follows: /sys/devices/system/node/nodeX/workingset_report/page_age A per-node page age histogram, showing an aggregate of the node's lruvecs. The information is extracted from MGLRU's per-generation page counters. Reading this file causes a hierarchical aging of all lruvecs, scanning pages and creates a new generation in each lruvec. For example: 1000 anon=0 file=0 2000 anon=0 file=0 100000 anon=5533696 file=5566464 18446744073709551615 anon=0 file=0 /sys/devices/system/node/nodeX/workingset_report/page_age_interval A comma separated list of time in milliseconds that configures what the page age histogram uses for aggregation. Signed-off-by: Yuanchu Xie Change-Id: I88fb6183f511f5a697c91a14619ee8e71e98e553 --- drivers/base/node.c | 6 + include/linux/mmzone.h | 9 + include/linux/workingset_report.h | 79 ++++++ mm/Kconfig | 9 + mm/Makefile | 1 + mm/internal.h | 4 + mm/memcontrol.c | 2 + mm/mm_init.c | 2 + mm/mmzone.c | 2 + mm/vmscan.c | 10 +- mm/workingset_report.c | 451 ++++++++++++++++++++++++++++++ 11 files changed, 571 insertions(+), 4 deletions(-) create mode 100644 include/linux/workingset_report.h create mode 100644 mm/workingset_report.c diff --git a/drivers/base/node.c b/drivers/base/node.c index eb72580288e6..ba5b8720dbfa 100644 --- a/drivers/base/node.c +++ b/drivers/base/node.c @@ -20,6 +20,8 @@ #include #include #include +#include +#include static const struct bus_type node_subsys = { .name = "node", @@ -626,6 +628,7 @@ static int register_node(struct node *node, int num) } else { hugetlb_register_node(node); compaction_register_node(node); + wsr_init_sysfs(node); } return error; @@ -642,6 +645,9 @@ void unregister_node(struct node *node) { hugetlb_unregister_node(node); compaction_unregister_node(node); + wsr_remove_sysfs(node); + wsr_destroy_lruvec(mem_cgroup_lruvec(NULL, NODE_DATA(node->dev.id))); + wsr_destroy_pgdat(NODE_DATA(node->dev.id)); node_remove_accesses(node); node_remove_caches(node); device_unregister(&node->dev); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 41458892bc8a..57d485f73363 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -24,6 +24,7 @@ #include #include #include +#include /* Free memory management - zoned buddy allocator. */ #ifndef CONFIG_ARCH_FORCE_MAX_ORDER @@ -633,6 +634,9 @@ struct lruvec { struct lru_gen_mm_state mm_state; #endif #endif /* CONFIG_LRU_GEN */ +#ifdef CONFIG_WORKINGSET_REPORT + struct wsr_state wsr; +#endif /* CONFIG_WORKINGSET_REPORT */ #ifdef CONFIG_MEMCG struct pglist_data *pgdat; #endif @@ -1405,6 +1409,11 @@ typedef struct pglist_data { struct lru_gen_memcg memcg_lru; #endif +#ifdef CONFIG_WORKINGSET_REPORT + struct mutex wsr_update_mutex; + struct wsr_report_bins __rcu *wsr_page_age_bins; +#endif + CACHELINE_PADDING(_pad2_); /* Per-node vmstats */ diff --git a/include/linux/workingset_report.h b/include/linux/workingset_report.h new file mode 100644 index 000000000000..d7c2ee14ec87 --- /dev/null +++ b/include/linux/workingset_report.h @@ -0,0 +1,79 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_WORKINGSET_REPORT_H +#define _LINUX_WORKINGSET_REPORT_H + +#include +#include + +struct mem_cgroup; +struct pglist_data; +struct node; +struct lruvec; + +#ifdef CONFIG_WORKINGSET_REPORT + +#define WORKINGSET_REPORT_MIN_NR_BINS 2 +#define WORKINGSET_REPORT_MAX_NR_BINS 32 + +#define WORKINGSET_INTERVAL_MAX ((unsigned long)-1) +#define ANON_AND_FILE 2 + +struct wsr_report_bin { + unsigned long idle_age; + unsigned long nr_pages[ANON_AND_FILE]; +}; + +struct wsr_report_bins { + /* excludes the WORKINGSET_INTERVAL_MAX bin */ + unsigned long nr_bins; + /* last bin contains WORKINGSET_INTERVAL_MAX */ + unsigned long idle_age[WORKINGSET_REPORT_MAX_NR_BINS]; + struct rcu_head rcu; +}; + +struct wsr_page_age_histo { + unsigned long timestamp; + struct wsr_report_bin bins[WORKINGSET_REPORT_MAX_NR_BINS]; +}; + +struct wsr_state { + /* breakdown of workingset by page age */ + struct mutex page_age_lock; + struct wsr_page_age_histo *page_age; +}; + +void wsr_init_lruvec(struct lruvec *lruvec); +void wsr_destroy_lruvec(struct lruvec *lruvec); +void wsr_init_pgdat(struct pglist_data *pgdat); +void wsr_destroy_pgdat(struct pglist_data *pgdat); +void wsr_init_sysfs(struct node *node); +void wsr_remove_sysfs(struct node *node); + +/* + * Returns true if the wsr is configured to be refreshed. + * The next refresh time is stored in refresh_time. + */ +bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, + struct pglist_data *pgdat); +#else +static inline void wsr_init_lruvec(struct lruvec *lruvec) +{ +} +static inline void wsr_destroy_lruvec(struct lruvec *lruvec) +{ +} +static inline void wsr_init_pgdat(struct pglist_data *pgdat) +{ +} +static inline void wsr_destroy_pgdat(struct pglist_data *pgdat) +{ +} +static inline void wsr_init_sysfs(struct node *node) +{ +} +static inline void wsr_remove_sysfs(struct node *node) +{ +} +#endif /* CONFIG_WORKINGSET_REPORT */ + +#endif /* _LINUX_WORKINGSET_REPORT_H */ diff --git a/mm/Kconfig b/mm/Kconfig index b72e7d040f78..951b4e7300d2 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1263,6 +1263,15 @@ config IOMMU_MM_DATA config EXECMEM bool +config WORKINGSET_REPORT + bool "Working set reporting" + depends on LRU_GEN && SYSFS + help + Report system and per-memcg working set to userspace. + + This option exports stats and events giving the user more insight + into its memory working set. + source "mm/damon/Kconfig" endmenu diff --git a/mm/Makefile b/mm/Makefile index d2915f8c9dc0..62cef72ce7f8 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -98,6 +98,7 @@ obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o +obj-$(CONFIG_WORKINGSET_REPORT) += workingset_report.o ifdef CONFIG_SWAP obj-$(CONFIG_MEMCG) += swap_cgroup.o endif diff --git a/mm/internal.h b/mm/internal.h index b4d86436565b..f7d790f5d41c 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -406,11 +406,15 @@ extern unsigned long highest_memmap_pfn; /* * in mm/vmscan.c: */ +struct scan_control; bool isolate_lru_page(struct page *page); bool folio_isolate_lru(struct folio *folio); void putback_lru_page(struct page *page); void folio_putback_lru(struct folio *folio); extern void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason); +bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, bool can_swap, + bool force_scan); +void set_task_reclaim_state(struct task_struct *task, struct reclaim_state *rs); /* * in mm/rmap.c: diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f29157288b7d..5d16fd9d2c3f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -61,6 +61,7 @@ #include #include #include +#include #include "internal.h" #include #include @@ -3504,6 +3505,7 @@ static void free_mem_cgroup_per_node_info(struct mem_cgroup *memcg, int node) if (!pn) return; + wsr_destroy_lruvec(&pn->lruvec); free_percpu(pn->lruvec_stats_percpu); kfree(pn->lruvec_stats); kfree(pn); diff --git a/mm/mm_init.c b/mm/mm_init.c index 75c3bd42799b..d8fe19fc1cef 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -30,6 +30,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -1378,6 +1379,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat) pgdat_page_ext_init(pgdat); lruvec_init(&pgdat->__lruvec); + wsr_init_pgdat(pgdat); } static void __meminit zone_init_internals(struct zone *zone, enum zone_type idx, int nid, diff --git a/mm/mmzone.c b/mm/mmzone.c index c01896eca736..477cd5ac1d78 100644 --- a/mm/mmzone.c +++ b/mm/mmzone.c @@ -90,6 +90,8 @@ void lruvec_init(struct lruvec *lruvec) */ list_del(&lruvec->lists[LRU_UNEVICTABLE]); + wsr_init_lruvec(lruvec); + lru_gen_init_lruvec(lruvec); } diff --git a/mm/vmscan.c b/mm/vmscan.c index cfa839284b92..8ab1b456d2cf 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -56,6 +56,7 @@ #include #include #include +#include #include #include @@ -270,8 +271,7 @@ static int sc_swappiness(struct scan_control *sc, struct mem_cgroup *memcg) } #endif -static void set_task_reclaim_state(struct task_struct *task, - struct reclaim_state *rs) +void set_task_reclaim_state(struct task_struct *task, struct reclaim_state *rs) { /* Check for an overwrite */ WARN_ON_ONCE(rs && task->reclaim_state); @@ -3857,8 +3857,8 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, return success; } -static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, - bool can_swap, bool force_scan) +bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, bool can_swap, + bool force_scan) { bool success; struct lru_gen_mm_walk *walk; @@ -5631,6 +5631,8 @@ static int __init init_lru_gen(void) if (sysfs_create_group(mm_kobj, &lru_gen_attr_group)) pr_err("lru_gen: failed to create sysfs group\n"); + wsr_init_sysfs(NULL); + debugfs_create_file("lru_gen", 0644, NULL, NULL, &lru_gen_rw_fops); debugfs_create_file("lru_gen_full", 0444, NULL, NULL, &lru_gen_ro_fops); diff --git a/mm/workingset_report.c b/mm/workingset_report.c new file mode 100644 index 000000000000..a4dcf62fcd96 --- /dev/null +++ b/mm/workingset_report.c @@ -0,0 +1,451 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "internal.h" + +void wsr_init_pgdat(struct pglist_data *pgdat) +{ + mutex_init(&pgdat->wsr_update_mutex); + RCU_INIT_POINTER(pgdat->wsr_page_age_bins, NULL); +} + +void wsr_destroy_pgdat(struct pglist_data *pgdat) +{ + struct wsr_report_bins __rcu *bins; + + mutex_lock(&pgdat->wsr_update_mutex); + bins = rcu_replace_pointer(pgdat->wsr_page_age_bins, NULL, + lockdep_is_held(&pgdat->wsr_update_mutex)); + kfree_rcu(bins, rcu); + mutex_unlock(&pgdat->wsr_update_mutex); + mutex_destroy(&pgdat->wsr_update_mutex); +} + +void wsr_init_lruvec(struct lruvec *lruvec) +{ + struct wsr_state *wsr = &lruvec->wsr; + + memset(wsr, 0, sizeof(*wsr)); + mutex_init(&wsr->page_age_lock); +} + +void wsr_destroy_lruvec(struct lruvec *lruvec) +{ + struct wsr_state *wsr = &lruvec->wsr; + + mutex_destroy(&wsr->page_age_lock); + kfree(wsr->page_age); + memset(wsr, 0, sizeof(*wsr)); +} + +static int workingset_report_intervals_parse(char *src, + struct wsr_report_bins *bins) +{ + int err = 0, i = 0; + char *cur, *next = strim(src); + + if (*next == '\0') + return 0; + + while ((cur = strsep(&next, ","))) { + unsigned int interval; + + err = kstrtouint(cur, 0, &interval); + if (err) + goto out; + + bins->idle_age[i] = msecs_to_jiffies(interval); + if (i > 0 && bins->idle_age[i] <= bins->idle_age[i - 1]) { + err = -EINVAL; + goto out; + } + + if (++i == WORKINGSET_REPORT_MAX_NR_BINS) { + err = -ERANGE; + goto out; + } + } + + if (i && i < WORKINGSET_REPORT_MIN_NR_BINS - 1) { + err = -ERANGE; + goto out; + } + + bins->nr_bins = i; + bins->idle_age[i] = WORKINGSET_INTERVAL_MAX; +out: + return err ?: i; +} + +static unsigned long get_gen_start_time(const struct lru_gen_folio *lrugen, + unsigned long seq, + unsigned long max_seq, + unsigned long curr_timestamp) +{ + int younger_gen; + + if (seq == max_seq) + return curr_timestamp; + younger_gen = lru_gen_from_seq(seq + 1); + return READ_ONCE(lrugen->timestamps[younger_gen]); +} + +static void collect_page_age_type(const struct lru_gen_folio *lrugen, + struct wsr_report_bin *bin, + unsigned long max_seq, unsigned long min_seq, + unsigned long curr_timestamp, int type) +{ + unsigned long seq; + + for (seq = max_seq; seq + 1 > min_seq; seq--) { + int gen, zone; + unsigned long gen_end, gen_start, size = 0; + + gen = lru_gen_from_seq(seq); + + for (zone = 0; zone < MAX_NR_ZONES; zone++) + size += max( + READ_ONCE(lrugen->nr_pages[gen][type][zone]), + 0L); + + gen_start = get_gen_start_time(lrugen, seq, max_seq, + curr_timestamp); + gen_end = READ_ONCE(lrugen->timestamps[gen]); + + while (bin->idle_age != WORKINGSET_INTERVAL_MAX && + time_before(gen_end + bin->idle_age, curr_timestamp)) { + unsigned long gen_in_bin = (long)gen_start - + (long)curr_timestamp + + (long)bin->idle_age; + unsigned long gen_len = (long)gen_start - (long)gen_end; + + if (!gen_len) + break; + if (gen_in_bin) { + unsigned long split_bin = + size / gen_len * gen_in_bin; + + bin->nr_pages[type] += split_bin; + size -= split_bin; + } + gen_start = curr_timestamp - bin->idle_age; + bin++; + } + bin->nr_pages[type] += size; + } +} + +/* + * proportionally aggregate Multi-gen LRU bins into a working set report + * MGLRU generations: + * current time + * | max_seq timestamp + * | | max_seq - 1 timestamp + * | | | unbounded + * | | | | + * -------------------------------- + * | max_seq | ... | ... | min_seq + * -------------------------------- + * + * Bins: + * + * current time + * | current - idle_age[0] + * | | current - idle_age[1] + * | | | unbounded + * | | | | + * ------------------------------ + * | bin 0 | ... | ... | bin n-1 + * ------------------------------ + * + * Assume the heuristic that pages are in the MGLRU generation + * through uniform accesses, so we can aggregate them + * proportionally into bins. + */ +static void collect_page_age(struct wsr_page_age_histo *page_age, + const struct lruvec *lruvec) +{ + int type; + const struct lru_gen_folio *lrugen = &lruvec->lrugen; + unsigned long curr_timestamp = jiffies; + unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq); + unsigned long min_seq[ANON_AND_FILE] = { + READ_ONCE(lruvec->lrugen.min_seq[LRU_GEN_ANON]), + READ_ONCE(lruvec->lrugen.min_seq[LRU_GEN_FILE]), + }; + struct wsr_report_bin *bin = &page_age->bins[0]; + + for (type = 0; type < ANON_AND_FILE; type++) + collect_page_age_type(lrugen, bin, max_seq, min_seq[type], + curr_timestamp, type); +} + +/* First step: hierarchically scan child memcgs. */ +static void refresh_scan(struct wsr_state *wsr, struct mem_cgroup *root, + struct pglist_data *pgdat) +{ + struct mem_cgroup *memcg; + unsigned int flags; + struct reclaim_state rs = { 0 }; + + set_task_reclaim_state(current, &rs); + flags = memalloc_noreclaim_save(); + + memcg = mem_cgroup_iter(root, NULL, NULL); + do { + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); + unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq); + + /* + * setting can_swap=true and force_scan=true ensures + * proper workingset stats when the system cannot swap. + */ + try_to_inc_max_seq(lruvec, max_seq, true, true); + cond_resched(); + } while ((memcg = mem_cgroup_iter(root, memcg, NULL))); + + memalloc_noreclaim_restore(flags); + set_task_reclaim_state(current, NULL); +} + +/* Second step: aggregate child memcgs into the page age histogram. */ +static void refresh_aggregate(struct wsr_page_age_histo *page_age, + struct mem_cgroup *root, + struct pglist_data *pgdat) +{ + struct mem_cgroup *memcg; + struct wsr_report_bin *bin; + + for (bin = page_age->bins; + bin->idle_age != WORKINGSET_INTERVAL_MAX; bin++) { + bin->nr_pages[0] = 0; + bin->nr_pages[1] = 0; + } + /* the last used bin has idle_age == WORKINGSET_INTERVAL_MAX. */ + bin->nr_pages[0] = 0; + bin->nr_pages[1] = 0; + + memcg = mem_cgroup_iter(root, NULL, NULL); + do { + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); + + collect_page_age(page_age, lruvec); + cond_resched(); + } while ((memcg = mem_cgroup_iter(root, memcg, NULL))); + WRITE_ONCE(page_age->timestamp, jiffies); +} + +static void copy_node_bins(struct pglist_data *pgdat, + struct wsr_page_age_histo *page_age) +{ + struct wsr_report_bins *node_page_age_bins; + int i = 0; + + rcu_read_lock(); + node_page_age_bins = rcu_dereference(pgdat->wsr_page_age_bins); + if (!node_page_age_bins) + goto nocopy; + for (i = 0; i < node_page_age_bins->nr_bins; ++i) + page_age->bins[i].idle_age = node_page_age_bins->idle_age[i]; + +nocopy: + page_age->bins[i].idle_age = WORKINGSET_INTERVAL_MAX; + rcu_read_unlock(); +} + +bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, + struct pglist_data *pgdat) +{ + struct wsr_page_age_histo *page_age; + + if (!READ_ONCE(wsr->page_age)) + return false; + + refresh_scan(wsr, root, pgdat); + mutex_lock(&wsr->page_age_lock); + page_age = READ_ONCE(wsr->page_age); + if (page_age) { + copy_node_bins(pgdat, page_age); + refresh_aggregate(page_age, root, pgdat); + } + mutex_unlock(&wsr->page_age_lock); + return !!page_age; +} +EXPORT_SYMBOL_GPL(wsr_refresh_report); + +static struct pglist_data *kobj_to_pgdat(struct kobject *kobj) +{ + int nid = IS_ENABLED(CONFIG_NUMA) ? kobj_to_dev(kobj)->id : + first_memory_node; + + return NODE_DATA(nid); +} + +static struct wsr_state *kobj_to_wsr(struct kobject *kobj) +{ + return &mem_cgroup_lruvec(NULL, kobj_to_pgdat(kobj))->wsr; +} + +static ssize_t page_age_intervals_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct wsr_report_bins *bins; + int len = 0; + struct pglist_data *pgdat = kobj_to_pgdat(kobj); + + rcu_read_lock(); + bins = rcu_dereference(pgdat->wsr_page_age_bins); + if (bins) { + int i; + int nr_bins = bins->nr_bins; + + for (i = 0; i < bins->nr_bins; ++i) { + len += sysfs_emit_at( + buf, len, "%u", + jiffies_to_msecs(bins->idle_age[i])); + if (i + 1 < nr_bins) + len += sysfs_emit_at(buf, len, ","); + } + } + len += sysfs_emit_at(buf, len, "\n"); + rcu_read_unlock(); + + return len; +} + +static ssize_t page_age_intervals_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *src, size_t len) +{ + struct wsr_report_bins *bins = NULL, __rcu *old; + char *buf = NULL; + int err = 0; + struct pglist_data *pgdat = kobj_to_pgdat(kobj); + + buf = kstrdup(src, GFP_KERNEL); + if (!buf) { + err = -ENOMEM; + goto failed; + } + + bins = + kzalloc(sizeof(struct wsr_report_bins), GFP_KERNEL); + + if (!bins) { + err = -ENOMEM; + goto failed; + } + + err = workingset_report_intervals_parse(buf, bins); + if (err < 0) + goto failed; + + if (err == 0) { + kfree(bins); + bins = NULL; + } + + mutex_lock(&pgdat->wsr_update_mutex); + old = rcu_replace_pointer(pgdat->wsr_page_age_bins, bins, + lockdep_is_held(&pgdat->wsr_update_mutex)); + mutex_unlock(&pgdat->wsr_update_mutex); + kfree_rcu(old, rcu); + kfree(buf); + return len; +failed: + kfree(bins); + kfree(buf); + + return err; +} + +static struct kobj_attribute page_age_intervals_attr = + __ATTR_RW(page_age_intervals); + +static ssize_t page_age_show(struct kobject *kobj, struct kobj_attribute *attr, + char *buf) +{ + struct wsr_report_bin *bin; + int ret = 0; + struct wsr_state *wsr = kobj_to_wsr(kobj); + + + mutex_lock(&wsr->page_age_lock); + if (!wsr->page_age) + wsr->page_age = + kzalloc(sizeof(struct wsr_page_age_histo), GFP_KERNEL); + mutex_unlock(&wsr->page_age_lock); + + wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj)); + + mutex_lock(&wsr->page_age_lock); + if (!wsr->page_age) + goto unlock; + for (bin = wsr->page_age->bins; + bin->idle_age != WORKINGSET_INTERVAL_MAX; bin++) + ret += sysfs_emit_at(buf, ret, "%u anon=%lu file=%lu\n", + jiffies_to_msecs(bin->idle_age), + bin->nr_pages[0] * PAGE_SIZE, + bin->nr_pages[1] * PAGE_SIZE); + + ret += sysfs_emit_at(buf, ret, "%lu anon=%lu file=%lu\n", + WORKINGSET_INTERVAL_MAX, + bin->nr_pages[0] * PAGE_SIZE, + bin->nr_pages[1] * PAGE_SIZE); + +unlock: + mutex_unlock(&wsr->page_age_lock); + return ret; +} + +static struct kobj_attribute page_age_attr = __ATTR_RO(page_age); + +static struct attribute *workingset_report_attrs[] = { + &page_age_intervals_attr.attr, &page_age_attr.attr, NULL +}; + +static const struct attribute_group workingset_report_attr_group = { + .name = "workingset_report", + .attrs = workingset_report_attrs, +}; + +void wsr_init_sysfs(struct node *node) +{ + struct kobject *kobj = node ? &node->dev.kobj : mm_kobj; + struct wsr_state *wsr; + + if (IS_ENABLED(CONFIG_NUMA) && !node) + return; + + wsr = kobj_to_wsr(kobj); + + if (sysfs_create_group(kobj, &workingset_report_attr_group)) + pr_warn("Workingset report failed to create sysfs files\n"); +} +EXPORT_SYMBOL_GPL(wsr_init_sysfs); + +void wsr_remove_sysfs(struct node *node) +{ + struct kobject *kobj = &node->dev.kobj; + struct wsr_state *wsr; + + if (IS_ENABLED(CONFIG_NUMA) && !node) + return; + + wsr = kobj_to_wsr(kobj); + sysfs_remove_group(kobj, &workingset_report_attr_group); +} +EXPORT_SYMBOL_GPL(wsr_remove_sysfs); From patchwork Tue Aug 13 16:56:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13762345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFAEEC52D7B for ; Tue, 13 Aug 2024 16:59:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 732D36B0092; Tue, 13 Aug 2024 12:59:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6BBA56B0095; Tue, 13 Aug 2024 12:59:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E74D6B009A; Tue, 13 Aug 2024 12:59:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2F42D6B0092 for ; Tue, 13 Aug 2024 12:59:43 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B467B80852 for ; Tue, 13 Aug 2024 16:59:42 +0000 (UTC) X-FDA: 82447833804.07.9F5B9A2 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf14.hostedemail.com (Postfix) with ESMTP id C739410001C for ; Tue, 13 Aug 2024 16:59:40 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=OXvX0Zg4; spf=pass (imf14.hostedemail.com: domain of 3-5C7ZgcKCEg84kxmr4qyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuanchu.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3-5C7ZgcKCEg84kxmr4qyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723568345; a=rsa-sha256; cv=none; b=tGDME7DqI4uuKh/FIZD+4EUuQlWmiZtuy30T8RzrR+IQqzZ8gL6bICCiy8nv2osAwTlmDw 0FpjBL1vYDI7ZlU5LHjo5zfqCQe3CucWnO52bXLkvMXJVTRHEgnGolxNLO8MqpqXzGLqE2 9yRO8nxiU/QJa1ysz3OZB28HIBdHuN0= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=OXvX0Zg4; spf=pass (imf14.hostedemail.com: domain of 3-5C7ZgcKCEg84kxmr4qyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuanchu.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3-5C7ZgcKCEg84kxmr4qyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723568345; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Hqd1jKx+q5wvRHFsKe7RtJaTNM99zPu8YKm8b61Swh4=; b=KXX3/vafDj81f6fY05dWM2G9KWsSou3MtHRLq9oDDEJyvbuyNuf5O5syfdhZToo54HTeiH uIhslM1dA6LJ1b2RuuGUI3uU2cLunVCvuZIKGJD4Ock44VUJDElwWGnwsqFtRG4P8vBanP Y0cvoqWuLCHH62OhoJ+aDwbl9RDzIN4= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-664fc7c4e51so116144817b3.3 for ; Tue, 13 Aug 2024 09:59:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723568380; x=1724173180; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Hqd1jKx+q5wvRHFsKe7RtJaTNM99zPu8YKm8b61Swh4=; b=OXvX0Zg4jf8p3FWqi9U8pm0NHcYzXmPk6WWMAU3p0pfv6NpbDgM2mynPmh56D9Kylk gtFBIDbkN1BWIBu9ZT+phF5H+APJaPUtpK0Z7wrb84KX2POMT7XOx1PNEOWt/mJUUjHM jbOTxjFOvZwefofHPxOkbJ8KYZCBZ3dIPHFVJOH4/feXBP8EyyYtYVJftspBM9dcmtza vPVegRIQpPIpJ3Y28N7VE8YPeOqwVUFGWEGHryQkpiYvsVJF0RRtjsWH5Yh+O6Akl6jl TCVHxw++Nc4M6bR3P9HbdBbSbCiZVDnSn2DKRf1GntsNCqerfGp6EFmAwzbb3se3udbg d80A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723568380; x=1724173180; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Hqd1jKx+q5wvRHFsKe7RtJaTNM99zPu8YKm8b61Swh4=; b=BKn4qeeiv0tcy94YYXHYJjz9xTVJ4IE3vC2iE+M6UjrV0IITCAnvPfFZAnZx/ln4q/ UBBE4uek/g2RRmQMcohRiy94oLlJFNXOJE3ZB7gH/RFYTidwdLxPmmRHmFSITwouoYoS RHYD5dHmvzpSVw3PZDOJHrsu7ihEi7VPeSiS/wkb1xxmhcyUNuLhi5Q002h4M+ZIFZ3m mGBVp4G8sDNAJwrnOzvOD1exXNdBosSOXSJPK78mPXueBWVb7dh8FNENnzM+zVagwRNc nikG1rsAtzykQtw9GlVbSfuJTwbj0crffHl9PPPrA3PsWCgfXEkwhkNjluSBWfv6ejvJ D05A== X-Forwarded-Encrypted: i=1; AJvYcCXtU6T2G5A2gBO7bf2mCV6Bs/Q4nNC6QRv+ISRpfNXikqnVEfdGpqq3hiZOTcjE+IY0zEi6X9ziz2UlIO0N6LKXf0Q= X-Gm-Message-State: AOJu0YyJS5isN0GAzuq3DDc7YmqXgCtQDN3O1buSX3sK+ejyD7Lv7/bD MdYq3l4OAh8kvOYbGJTqK0WjqCvicIUDfb8Cb+vDAgaczajxHADNYmiG/Yw79fTgVqdj4TQBzmO EGfo/pg== X-Google-Smtp-Source: AGHT+IHcXqA8D3zvHAIQfVpeQAXseyth/6ohOvX1sAG9G57GDA3x2krpy7DDbUSjGWDUUw5XpHr8rLgkLnqj X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:b50c:66e8:6532:a371]) (user=yuanchu job=sendgmr) by 2002:a0d:ce43:0:b0:62f:f535:f41 with SMTP id 00721157ae682-6ac9bd76ef6mr1297b3.9.1723568379605; Tue, 13 Aug 2024 09:59:39 -0700 (PDT) Date: Tue, 13 Aug 2024 09:56:13 -0700 In-Reply-To: <20240813165619.748102-1-yuanchu@google.com> Mime-Version: 1.0 References: <20240813165619.748102-1-yuanchu@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813165619.748102-3-yuanchu@google.com> Subject: [PATCH v3 2/7] mm: use refresh interval to rate-limit workingset report aggregation From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Andrew Morton , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org X-Stat-Signature: joh6xqyysc1sz9zhs5hfy97qch3rok5z X-Rspamd-Queue-Id: C739410001C X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1723568380-228505 X-HE-Meta: U2FsdGVkX18uTmZlhUFo9UXdvIftE/i7nAX+U2Zy/1BU31ivmPPNj5BQgm99oW5ZFVesC1gL1vLM14NaDM38M0yIOHjHmMXnsa1jIO/VeyIWZLiCZoSZOtElGk+k/9pzvJ86mXZqX5/5NEK+lFyGiTFHQ6ZcBJhOH8/L1VGerHw8fr+nm/yp3kWzQQ7tu1s7LpS9X0njgVPdv3NUzKf5pphveCfGCN3gUUfs/tUsScVK6/a2u98xoUeuOBykF8yLeJQajX6oXFFdhEF0JQaXv/Yc5CTDo6p2dERR2FRntJ3Op7GsJYPKINPPmR5o4Th5PQZ6rPwmrMjoyzayTxgAD8itJOXHbzX5KgmnFWUfpu1PEHlFs2ubbqtYZda2SfZ5dXsL2SDFMwKn/b8T+p2n5Yh9GyfT7vM3kIPc36rwIVSsfqrpKa3FVViml1AQS5A+oAfszq4uj2TXnQ9niZTWEWfp3ylQ4a5Rg7uIKamN0+apwaAv1DowKL3DRSgZ2Nba19WS3As6KxU0+HUgNRbPt05C6qFn2zNCNw67WmKiP1sLxDODlpSnG0ubLsEYDLuYn9t2HTXjSKH40YwKjCGYusB/7GuqtOSm+Z07ri9tD1HlDWqvJDXktQ8wIBE1Sr+wpi5eEZIXjsWGXmDaN06QAu6ku3Sd48irvVtRhRr3rwqmYPHZ2L7QRBSseWONNKt3I2/45xcGQhrLt16qDDHaTnIPqDRi+n0JosEWo8Jq1PAQKpLtnQXp7Iz8l4Layk0pIUWra0RFK3pIkwo2r18ZkYQRepfp7ropLTCKJljDU+P50/B7fll5r/vRIR8xmLHtsv8Zfgc1OGXW2ISXCwxArV5d4K4ZrOzSZj3uHtaD8UzjC8KY2grgssGvGDI57ybtG80mbGsiaXCFCVb3BPafI05Q95lcupcrWbun0mO1PTHujXp/JAjMLDC1E6ApQpsuhm8CfGaTNzy+sm27nvh W3hlFUWt +owMdddSV1w/duTSPC9T7mhMFFwSILDcjG28YqV+daJemU/Bk+ZTDaiYyF0Fp8km1J0L9nbCldZ3Mlr2WEms7WUSobS8ByLmrjPuaA1j23qd4JS9JlrRhegicCo7kfBNF5ATVGJSvcyDcgE+HKuRKEEgD/og4prIHA0JzmadR9L3trY8Yu3uLB0txlG9Ed3FQX4nfRRN/qQgR508xbaFPJ2IIN+uxKpDsgE28xpKqrAwu1N9L1sfCFbc3tvmU9T+Uf9/baMDGl0Bh0mJuPHg+G6nHr7lBxgvM4i5mPHFo2AMTWmxKlRUCRKnJu0cbWM7w/bWhDoTmYwcScZKIu2hJjjqoC0EzCm1yrYg9SnR2/waBioGXeuGWOpUUoHbY2N0B4KkJpdYhs4vHlBeDflHUXDr7MlgD3rzsIth7vuTJaLvuhmuxyUma1lzOOdc68v5F2O7SLcvOuASG+rau42wfAfaF8cJUwA1kUtKLyXXQH3ex1DFrdPffnwxXJkHtI4DEY36EyIaQNBbc9MI/YvGYw6UMFKXvjj2pCv8O0vvP2WuLkXQCQXAlMNMsoNPT5qUpZQF66JNo2r5UEkM5/hq5XMWFhh7O4kFhs9ZJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The refresh interval is a rate limiting factor to workingset page age histogram reads. When a workingset report is generated, a timestamp is noted, and the same report will be read until it expires beyond the refresh interval, at which point a new report is generated. Sysfs interface /sys/devices/system/node/nodeX/workingset_report/refresh_interval time in milliseconds specifying how long the report is valid for Signed-off-by: Yuanchu Xie Change-Id: Iabacec2845a7fac7280ffc5f4faed55186cfa7a2 --- include/linux/workingset_report.h | 1 + mm/workingset_report.c | 84 +++++++++++++++++++++++++------ 2 files changed, 70 insertions(+), 15 deletions(-) diff --git a/include/linux/workingset_report.h b/include/linux/workingset_report.h index d7c2ee14ec87..8bae6a600410 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -37,6 +37,7 @@ struct wsr_page_age_histo { }; struct wsr_state { + unsigned long refresh_interval; /* breakdown of workingset by page age */ struct mutex page_age_lock; struct wsr_page_age_histo *page_age; diff --git a/mm/workingset_report.c b/mm/workingset_report.c index a4dcf62fcd96..fe553c0a653e 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -195,7 +195,8 @@ static void collect_page_age(struct wsr_page_age_histo *page_age, /* First step: hierarchically scan child memcgs. */ static void refresh_scan(struct wsr_state *wsr, struct mem_cgroup *root, - struct pglist_data *pgdat) + struct pglist_data *pgdat, + unsigned long refresh_interval) { struct mem_cgroup *memcg; unsigned int flags; @@ -208,12 +209,15 @@ static void refresh_scan(struct wsr_state *wsr, struct mem_cgroup *root, do { struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq); + int gen = lru_gen_from_seq(max_seq); + unsigned long birth = READ_ONCE(lruvec->lrugen.timestamps[gen]); /* * setting can_swap=true and force_scan=true ensures * proper workingset stats when the system cannot swap. */ - try_to_inc_max_seq(lruvec, max_seq, true, true); + if (time_is_before_jiffies(birth + refresh_interval)) + try_to_inc_max_seq(lruvec, max_seq, true, true); cond_resched(); } while ((memcg = mem_cgroup_iter(root, memcg, NULL))); @@ -270,17 +274,25 @@ bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, struct pglist_data *pgdat) { struct wsr_page_age_histo *page_age; + unsigned long refresh_interval = READ_ONCE(wsr->refresh_interval); if (!READ_ONCE(wsr->page_age)) return false; - refresh_scan(wsr, root, pgdat); + if (!refresh_interval) + return false; + mutex_lock(&wsr->page_age_lock); page_age = READ_ONCE(wsr->page_age); - if (page_age) { - copy_node_bins(pgdat, page_age); - refresh_aggregate(page_age, root, pgdat); - } + if (!page_age) + goto unlock; + if (page_age->timestamp && + time_is_after_jiffies(page_age->timestamp + refresh_interval)) + goto unlock; + refresh_scan(wsr, root, pgdat, refresh_interval); + copy_node_bins(pgdat, page_age); + refresh_aggregate(page_age, root, pgdat); +unlock: mutex_unlock(&wsr->page_age_lock); return !!page_age; } @@ -299,6 +311,52 @@ static struct wsr_state *kobj_to_wsr(struct kobject *kobj) return &mem_cgroup_lruvec(NULL, kobj_to_pgdat(kobj))->wsr; } +static ssize_t refresh_interval_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct wsr_state *wsr = kobj_to_wsr(kobj); + unsigned int interval = READ_ONCE(wsr->refresh_interval); + + return sysfs_emit(buf, "%u\n", jiffies_to_msecs(interval)); +} + +static ssize_t refresh_interval_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t len) +{ + unsigned int interval; + int err; + struct wsr_state *wsr = kobj_to_wsr(kobj); + + err = kstrtouint(buf, 0, &interval); + if (err) + return err; + + mutex_lock(&wsr->page_age_lock); + if (interval && !wsr->page_age) { + struct wsr_page_age_histo *page_age = + kzalloc(sizeof(struct wsr_page_age_histo), GFP_KERNEL); + + if (!page_age) { + err = -ENOMEM; + goto unlock; + } + wsr->page_age = page_age; + } + if (!interval && wsr->page_age) { + kfree(wsr->page_age); + wsr->page_age = NULL; + } + + WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(interval)); +unlock: + mutex_unlock(&wsr->page_age_lock); + return err ?: len; +} + +static struct kobj_attribute refresh_interval_attr = + __ATTR_RW(refresh_interval); + static ssize_t page_age_intervals_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) { @@ -382,13 +440,6 @@ static ssize_t page_age_show(struct kobject *kobj, struct kobj_attribute *attr, int ret = 0; struct wsr_state *wsr = kobj_to_wsr(kobj); - - mutex_lock(&wsr->page_age_lock); - if (!wsr->page_age) - wsr->page_age = - kzalloc(sizeof(struct wsr_page_age_histo), GFP_KERNEL); - mutex_unlock(&wsr->page_age_lock); - wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj)); mutex_lock(&wsr->page_age_lock); @@ -414,7 +465,10 @@ static ssize_t page_age_show(struct kobject *kobj, struct kobj_attribute *attr, static struct kobj_attribute page_age_attr = __ATTR_RO(page_age); static struct attribute *workingset_report_attrs[] = { - &page_age_intervals_attr.attr, &page_age_attr.attr, NULL + &refresh_interval_attr.attr, + &page_age_intervals_attr.attr, + &page_age_attr.attr, + NULL }; static const struct attribute_group workingset_report_attr_group = { From patchwork Tue Aug 13 16:56:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13762346 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9667EC52D7F for ; Tue, 13 Aug 2024 16:59:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D2A246B009A; Tue, 13 Aug 2024 12:59:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CB0676B009E; Tue, 13 Aug 2024 12:59:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B29256B009F; Tue, 13 Aug 2024 12:59:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8F5926B009A for ; Tue, 13 Aug 2024 12:59:45 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id DCC4FA7827 for ; Tue, 13 Aug 2024 16:59:44 +0000 (UTC) X-FDA: 82447833888.29.650E7FB Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf27.hostedemail.com (Postfix) with ESMTP id D50E540020 for ; Tue, 13 Aug 2024 16:59:42 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZpQKte5M; spf=pass (imf27.hostedemail.com: domain of 3_ZC7ZgcKCEoA6mzot6s00sxq.o0yxuz69-yyw7mow.03s@flex--yuanchu.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3_ZC7ZgcKCEoA6mzot6s00sxq.o0yxuz69-yyw7mow.03s@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723568327; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I1JUyOR+BauD0beWrpDUPXKr0GpTowWtPDSwnxHbi4c=; b=ivLLcGwvTR/JLQuy7qwJ+hExMYw01MhqhfP4PldDCpOdeSK6KCH2uKzsFG9/qTjnE6YR4r 0jfHc7nOOvGY8vmD/zUIHvCvWG/KvjwwqYGd5HBiLW0YbgnPMYHNFlYgi7UeQb45sTfO8Z bGQ7Il+KMOCL+77bkUBimOxW4ywWYAE= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZpQKte5M; spf=pass (imf27.hostedemail.com: domain of 3_ZC7ZgcKCEoA6mzot6s00sxq.o0yxuz69-yyw7mow.03s@flex--yuanchu.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3_ZC7ZgcKCEoA6mzot6s00sxq.o0yxuz69-yyw7mow.03s@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723568327; a=rsa-sha256; cv=none; b=yc/+CadiK0YA8ALpyuhT09goblQb2HU8OalUUeyR+a79oAJnsCEpHSWdydUKLK0njCylrg dyFFLYuPucDEo3rlzgQpz3vVJZm9LBxYx/vWgwd/62BAIxVsFwigiU5nvluqg3JdNNXWK0 1y6Won6oxHNcE88zqKcW7ZSzHM5Nk1Y= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-666010fb35cso1071497b3.0 for ; Tue, 13 Aug 2024 09:59:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723568382; x=1724173182; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=I1JUyOR+BauD0beWrpDUPXKr0GpTowWtPDSwnxHbi4c=; b=ZpQKte5MIDJ/lnNynSUeHfoit1BCtiBNVRyjTRHG11g4uMwjA+8JUgvjja8EaqUmdJ zYaMLh316yi+kjPJU98DIV3xF9OlneTF7VA93xkFQ2Xcz6DsPA7gxGowVuPJEg6xJeYJ hriMgSy89AfmB3/483xLA0KZ2okIREY+t18DMKVuw8qQfc7lno0LvuLE1EsgdjmqIQ3d HNB0xQrzZoWQavGJvDknJvx8jnlQXOdWX/IZr/RsQnpFVchYiKd/TBnfC1JbaUnPlGes wgXTOEccdaJCATKu45M+4UEJ6excwFRM9HOcxvfmwfY7Z3zVrU6/paUoGLmUUIjL9S8n 04aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723568382; x=1724173182; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=I1JUyOR+BauD0beWrpDUPXKr0GpTowWtPDSwnxHbi4c=; b=hQBQCw8Rn6w3aOz/0WSUypBw5zSVW4RCwlLdEKiLbtmsW+yvCI0TNAGaxjM56FyDbP uYezyNbGYWUrYiO38FW8kC3S5i4PvYLtOpI9J6Hc6rMd55R2f3TwA6a4udaVccNLXTgQ E0d7Rtgm3k5Lm7B2VhtgPd+gKE04JaMHHQpR5DL/DH0S5ywneHXXk0rtb68sxrQHUwrf QeVD7jkAqQP7cpepD2nVTOEdR+OSrMcTU2kcVJt3BfKm1wPUZj5wvWDafjmyv6pdji4N xepmckKcYmsJp7miNFapkeRCgKz9HQurBNSqBeNrLlM4TId+bPn61TkBGNdhE1yie9AT kirw== X-Forwarded-Encrypted: i=1; AJvYcCWo42Tb2Aoa3cR/s6kONJgzAZD3AmPGhIIJTeS/Std/1j4Yman2KaIDicGuwvkWOYQlbdiwvuvT3NsRdf+eahVS4MA= X-Gm-Message-State: AOJu0YydB/p/o/6Ld5MIazvQBfBhlNyuTxin7Z2bFx7rY8fzQZlNJ1B2 X5iGd1MqKwwPvzLk+x6CewZskYaEpBvpz6IrtpkCeAThK+TtJhNPLb1dezf6DDsr/aRKMn2G3In P0ix1eQ== X-Google-Smtp-Source: AGHT+IGF6gcvOUuK0qRwt2F4MDLyEzr3H7dooXhW2VhDhQ7vMG1kJd5bUtwNOeRY1xrvYnGAuUjldLTxpbDJ X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:b50c:66e8:6532:a371]) (user=yuanchu job=sendgmr) by 2002:a0d:ce01:0:b0:6a1:741b:b723 with SMTP id 00721157ae682-6a9ea8e5238mr545567b3.5.1723568381863; Tue, 13 Aug 2024 09:59:41 -0700 (PDT) Date: Tue, 13 Aug 2024 09:56:14 -0700 In-Reply-To: <20240813165619.748102-1-yuanchu@google.com> Mime-Version: 1.0 References: <20240813165619.748102-1-yuanchu@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813165619.748102-4-yuanchu@google.com> Subject: [PATCH v3 3/7] mm: report workingset during memory pressure driven scanning From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Andrew Morton , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org X-Stat-Signature: bcjote3hawq9gfc6wnbewt33tc7hwu3x X-Rspam-User: X-Rspamd-Queue-Id: D50E540020 X-Rspamd-Server: rspam02 X-HE-Tag: 1723568382-102211 X-HE-Meta: U2FsdGVkX18XZ16qh+OJzmxLEydYjyd/jsqSV+KA6T2J+P+xYk50/7NYgcwdonZ4KeDCoCF7MP/YeEKauhGaeYuLcF6ilM9G1hMtYbJIgbNhh7GXoytBHHiRfv/A3q9iPMv5rQRrdJi98CBsCHKqpyHMhuft7ARG+toiHintcCAJxxJkG5MjcSV1482iBBb8m2iL17wyqPla6wRomi8czUri3nmR+FbtRvi2onC+0j/HuUw5LNNQX6l9dyxVxdB8wIEUV0+d96X0P/W0jSwk0eUTvpLvwwEgd315DmIeo55yJauwtpdC+qjVHPK7+ShO+ueEKAOPfAWBHhRfaiyfiRiSn04lmtt6WvUQbDVpzetVcbKHFN0II7gA89HKhKi2KV3pc6gGfqK84QmV8OIkDhYO43FcObVyUYtQkrMhRd2wVkw0L2ncVSUJueWOtTZLOAq/Ca0v6mtx2uHajgt21aVO2g5ADPwD6kr7MFD2rphIcQkY+gyLsfvVJVSxFkJRMRrPTQ7Rjr5cOuxluZfj8yBND0AVfsIVZBgdkBKh1GzDD1aqyrQfNQ6ZAnag0M+IKX3c+3Xs/pFrL2jis6e24IRm11niaPYDNwNsqEfIBMKKqtC63Wd+KxbCMwk67JTknl50s/Ta7pZkT0eJ+fYMjWNb/ti9kudktNwAHzn540JcYvxpyx9ku9fGCEDMd4fmoQokX06rWYmrL7Or91E3n9FFMxyLoJPq7Ra2DZiSvmLMyzN1O9gUrNjT+XeCfwyr9/+fa7P3Y1CscqXNCCUW7U7QOY6nUjMkVAqCcspAWGyMaCisFnxfh3YMXfrVwVwChK7P+hTmALFFuGIEjB0ehs4qGAVAaIEmHVzPEkmiIQB1NlJBsT7IuvKfHpgndMaoKiU0FJgvU/JouwodfC+bYP5/mux1EiKFZat3etPHp+OlVT8x0g1XIgZ+AE25/G6kPWF90mWNuPcDnTpKqp0 XMgDHXYK Hs1Qr8d9XuuTsMTAG+Py9dkjUlRPpohVP4Y6dEC1aDlYaJrgaDp4pHQJ6JxdLgP8d9Ripwft/qov8+HOkpriZxGB8uvpURnrDGTJsxJSfGnnstYDgOE6hkHavLbQvR7iitKZEgeWdJJDcMUxMokGFbffZPX4ar/dMZ879OPO59jFpkvrTZbhKiSSGkmmmZV3pekUFtatufgN6vZoZVObbDErZA8pu0FbOgYYPeqbtTiCmqNWxkQJMQv5fd2WDcS6xzDBG01xKj5LQpzjUHCpRV+PRV4erhgokeBeJWHRW0zbENbeD7F67tjCpQgXaaItaBKkNQU7Ya40H+oUT6vp4Yo0MjWzCAl+EOxk5524DTMyJgHQA13PrN2CpGU/+XDuAwXXNWWE8cH1etSkdzXwheWOkC7B+i49Iomr9mO6IwBGswb3gxoKCkVDaCyMgO6Syg2O1eiMrFJR4U1lO4rQdwmbW7ht5U98YTBqy25D6lQyYbQ5wEKtbXsplRUeSdnFCM1XkdXcq2tIYlGJLfizRP0wsWgQC5bKHxpPGvB9NgrHKDvDAHglyS80xE2uvyIItDUCNtWGyxWR0asTsfbIa0imfiREBLfsR6Xpb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a node reaches its low watermarks and wakes up kswapd, notify all userspace programs waiting on the workingset page age histogram of the memory pressure, so a userspace agent can read the workingset report in time and make policy decisions, such as logging, oom-killing, or migration. Sysfs interface: /sys/devices/system/node/nodeX/workingset_report/report_threshold time in milliseconds that specifies how often the userspace agent can be notified for node memory pressure. Signed-off-by: Yuanchu Xie Change-Id: Ib1fdb726da921e8d467822a99da8f384c6181951 --- include/linux/workingset_report.h | 4 +++ mm/internal.h | 12 ++++++++ mm/vmscan.c | 46 +++++++++++++++++++++++++++++++ mm/workingset_report.c | 43 ++++++++++++++++++++++++++++- 4 files changed, 104 insertions(+), 1 deletion(-) diff --git a/include/linux/workingset_report.h b/include/linux/workingset_report.h index 8bae6a600410..2ec8b927b200 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -37,7 +37,11 @@ struct wsr_page_age_histo { }; struct wsr_state { + unsigned long report_threshold; unsigned long refresh_interval; + + struct kernfs_node *page_age_sys_file; + /* breakdown of workingset by page age */ struct mutex page_age_lock; struct wsr_page_age_histo *page_age; diff --git a/mm/internal.h b/mm/internal.h index f7d790f5d41c..19af882c506e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -416,6 +416,18 @@ bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, bool can_swap, bool force_scan); void set_task_reclaim_state(struct task_struct *task, struct reclaim_state *rs); +#ifdef CONFIG_WORKINGSET_REPORT +/* + * in mm/wsr.c + */ +void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat); +#else +static inline void notify_workingset(struct mem_cgroup *memcg, + struct pglist_data *pgdat) +{ +} +#endif + /* * in mm/rmap.c: */ diff --git a/mm/vmscan.c b/mm/vmscan.c index 8ab1b456d2cf..b07fd2016c75 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2574,6 +2574,15 @@ static bool can_age_anon_pages(struct pglist_data *pgdat, return can_demote(pgdat->node_id, sc); } +#ifdef CONFIG_WORKINGSET_REPORT +static void try_to_report_workingset(struct pglist_data *pgdat, struct scan_control *sc); +#else +static inline void try_to_report_workingset(struct pglist_data *pgdat, + struct scan_control *sc) +{ +} +#endif + #ifdef CONFIG_LRU_GEN #ifdef CONFIG_LRU_GEN_ENABLED @@ -4000,6 +4009,8 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) set_initial_priority(pgdat, sc); + try_to_report_workingset(pgdat, sc); + memcg = mem_cgroup_iter(NULL, NULL, NULL); do { struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); @@ -5640,6 +5651,38 @@ static int __init init_lru_gen(void) }; late_initcall(init_lru_gen); +#ifdef CONFIG_WORKINGSET_REPORT +static void try_to_report_workingset(struct pglist_data *pgdat, + struct scan_control *sc) +{ + struct mem_cgroup *memcg = sc->target_mem_cgroup; + struct wsr_state *wsr = &mem_cgroup_lruvec(memcg, pgdat)->wsr; + unsigned long threshold = READ_ONCE(wsr->report_threshold); + + if (sc->priority == DEF_PRIORITY) + return; + + if (!threshold) + return; + + if (!mutex_trylock(&wsr->page_age_lock)) + return; + + if (!wsr->page_age) { + mutex_unlock(&wsr->page_age_lock); + return; + } + + if (time_is_after_jiffies(wsr->page_age->timestamp + threshold)) { + mutex_unlock(&wsr->page_age_lock); + return; + } + + mutex_unlock(&wsr->page_age_lock); + notify_workingset(memcg, pgdat); +} +#endif /* CONFIG_WORKINGSET_REPORT */ + #else /* !CONFIG_LRU_GEN */ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) @@ -6191,6 +6234,9 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc) if (zone->zone_pgdat == last_pgdat) continue; last_pgdat = zone->zone_pgdat; + + if (!sc->proactive) + try_to_report_workingset(zone->zone_pgdat, sc); shrink_node(zone->zone_pgdat, sc); } diff --git a/mm/workingset_report.c b/mm/workingset_report.c index fe553c0a653e..801ac8e5c1da 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -311,6 +311,33 @@ static struct wsr_state *kobj_to_wsr(struct kobject *kobj) return &mem_cgroup_lruvec(NULL, kobj_to_pgdat(kobj))->wsr; } +static ssize_t report_threshold_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct wsr_state *wsr = kobj_to_wsr(kobj); + unsigned int threshold = READ_ONCE(wsr->report_threshold); + + return sysfs_emit(buf, "%u\n", jiffies_to_msecs(threshold)); +} + +static ssize_t report_threshold_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t len) +{ + unsigned int threshold; + struct wsr_state *wsr = kobj_to_wsr(kobj); + + if (kstrtouint(buf, 0, &threshold)) + return -EINVAL; + + WRITE_ONCE(wsr->report_threshold, msecs_to_jiffies(threshold)); + + return len; +} + +static struct kobj_attribute report_threshold_attr = + __ATTR_RW(report_threshold); + static ssize_t refresh_interval_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) { @@ -465,6 +492,7 @@ static ssize_t page_age_show(struct kobject *kobj, struct kobj_attribute *attr, static struct kobj_attribute page_age_attr = __ATTR_RO(page_age); static struct attribute *workingset_report_attrs[] = { + &report_threshold_attr.attr, &refresh_interval_attr.attr, &page_age_intervals_attr.attr, &page_age_attr.attr, @@ -486,8 +514,13 @@ void wsr_init_sysfs(struct node *node) wsr = kobj_to_wsr(kobj); - if (sysfs_create_group(kobj, &workingset_report_attr_group)) + if (sysfs_create_group(kobj, &workingset_report_attr_group)) { pr_warn("Workingset report failed to create sysfs files\n"); + return; + } + + wsr->page_age_sys_file = + kernfs_walk_and_get(kobj->sd, "workingset_report/page_age"); } EXPORT_SYMBOL_GPL(wsr_init_sysfs); @@ -500,6 +533,14 @@ void wsr_remove_sysfs(struct node *node) return; wsr = kobj_to_wsr(kobj); + kernfs_put(wsr->page_age_sys_file); sysfs_remove_group(kobj, &workingset_report_attr_group); } EXPORT_SYMBOL_GPL(wsr_remove_sysfs); + +void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat) +{ + struct wsr_state *wsr = &mem_cgroup_lruvec(memcg, pgdat)->wsr; + + kernfs_notify(wsr->page_age_sys_file); +} From patchwork Tue Aug 13 16:56:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13762347 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84F87C52D7C for ; Tue, 13 Aug 2024 16:59:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 86D696B009F; Tue, 13 Aug 2024 12:59:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7F3E76B00A0; Tue, 13 Aug 2024 12:59:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 586516B00A1; Tue, 13 Aug 2024 12:59:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2FD9C6B009F for ; Tue, 13 Aug 2024 12:59:47 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CC32B160909 for ; Tue, 13 Aug 2024 16:59:46 +0000 (UTC) X-FDA: 82447833972.05.219145F Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf12.hostedemail.com (Postfix) with ESMTP id F0ABE40014 for ; Tue, 13 Aug 2024 16:59:44 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=CJmeDm6q; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3_5C7ZgcKCEwC8o1qv8u22uzs.q20zw18B-00y9oqy.25u@flex--yuanchu.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3_5C7ZgcKCEwC8o1qv8u22uzs.q20zw18B-00y9oqy.25u@flex--yuanchu.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723568327; a=rsa-sha256; cv=none; b=zcsJrCiTkqD9DzJOLmgE5fIedsX9EWqLn42o/ScSHMpzPhG5nc5EpsRdzuCcMz1NqrqOlY UFGQMhtc44B6YqIK7AZbTnFMeWRZXQ4v8Vd5hZGFIfsPSPizK4TlPYI/luHI9ckYALCu+f MibT7lnHH45DLQH8iLg6fW/T/CeBaHA= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=CJmeDm6q; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3_5C7ZgcKCEwC8o1qv8u22uzs.q20zw18B-00y9oqy.25u@flex--yuanchu.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3_5C7ZgcKCEwC8o1qv8u22uzs.q20zw18B-00y9oqy.25u@flex--yuanchu.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723568327; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nX5Or29rWGSYnqI8Mal7bT79wmdNbU4VM0EJZ849BDE=; b=8qNoi+X3QM4kGIXE8WBDrc8VFTVOxJ6j0eKmRfbfvPKsKwQEoZQk81soZVnHQQKPiDbsow b+GbYIPg1f71tJstyZFjESJy5p8dgrbuFbPT5YJQU41ldAuQRgZ4ZGhrlkfuqZjXLSdKLG k/V+655cvy3dtAlupUChgVyqVUJTM90= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-672bea19bedso135340877b3.3 for ; Tue, 13 Aug 2024 09:59:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723568384; x=1724173184; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nX5Or29rWGSYnqI8Mal7bT79wmdNbU4VM0EJZ849BDE=; b=CJmeDm6qrr+ILt8/AD9jK/iE3zN5z9NcMmGhsV44IA8v4256p0TQEULKM3TF0uj6hz QF2toYHJCyjYpxdRsxaJldqF1ePLUGr1VpsA9oilzbbPrvrcsLrGoCyU30Q0nZUoGMF5 y7I8vCgk33JcBEfzTwHR8F0hMC9rVyXBqkJfyDXZ7nQyC+8VCmhLSNPhkMGnHCqp2CMV O3hnxcLcP924yjMPRblkN6nliPeylmHELweJfWZXM8bjhaw3X9C0MQoa9Ja+yLBQ64TW oqGveRlxq0j4l7KrN5iWqf5stXwqSDt/BlRsGr73bqNaRvyxhMANnXSA8KlDf46zVz8t B3Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723568384; x=1724173184; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nX5Or29rWGSYnqI8Mal7bT79wmdNbU4VM0EJZ849BDE=; b=OEeaQf1KHwALVZuBScd6TAVyLWd7ZpCfWeoJbYsjHp2nvfvtgpqgFVBgLDY+xpgk2e zkNQdi5ZmrRP2ibV9+JSKN0om/Kl9viVUh3oJ9AFzrr0cD1+9fuvdgdJEK846FcTrV6c TazUgJJt1Q9Oei1UXxLB3Wel08DCW4ajfjQvYxBGoXOD80MIo7mrvR8nKEQgRolXBWPX fn3qS27d7ys1un87Y8zx9gnDk72qKQUqSgvRi9v2QVs4W/xxen7Zrf8jDO7rmb5DF7hg iDxuaAxHrVT4bEruzVKRBDDT0UmRdN1qN2CnJygkkiUq98JOKXRkN4ANwmQc6iP95duW BSdw== X-Forwarded-Encrypted: i=1; AJvYcCV9v9RNjFe1yzyut9/VhwlUFjSBHfxj2oanxMYIvTwetRqMq4hXOKS5JAmW5fwLcp1/JLqCRyWfFA==@kvack.org X-Gm-Message-State: AOJu0Yw7q+7ucprfcKx305lAe9EQYT1mAhdcU0HH6ZivsbKFqxCNR0tA urFEND8Aknkree9q8h8R3JbLeq7b5kRATUToYHIqWhyIMqBifUd8D0VHbahkhoDeNwkU03uJpTI b6+D5UA== X-Google-Smtp-Source: AGHT+IEDV077DvpPZgcSOtEG+50DqdAd9HxWks1Jm8Rh/HY1wmqFnkHTld2tYhmnQaOc9LEXewCUzyWTHQqf X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:b50c:66e8:6532:a371]) (user=yuanchu job=sendgmr) by 2002:a25:a444:0:b0:e05:6e45:84b6 with SMTP id 3f1490d57ef6-e1155b79891mr171276.8.1723568383787; Tue, 13 Aug 2024 09:59:43 -0700 (PDT) Date: Tue, 13 Aug 2024 09:56:15 -0700 In-Reply-To: <20240813165619.748102-1-yuanchu@google.com> Mime-Version: 1.0 References: <20240813165619.748102-1-yuanchu@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813165619.748102-5-yuanchu@google.com> Subject: [PATCH v3 4/7] mm: extend working set reporting to memcgs From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Andrew Morton , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: F0ABE40014 X-Stat-Signature: qgi6yxins5m78j66jwaor5qe3xgxbnh3 X-Rspam-User: X-HE-Tag: 1723568384-688403 X-HE-Meta: U2FsdGVkX18MyOL23rr2jRDmj/UT9XDmkPtw6kgMZGHvzASrhS5B8vj+U2dshE4HPHLQIQAhQtg0hxdkbceXVk4ZLcabWR8XUoBun/1TjlL8UyzjXa6g6X57ZVjQtPML8jy0kGE5Svh5hhAsfYruEcz4v89X9l+MoJejWlp2oanYK8TpEswdh2212uN6ZrK9p4pfSlUlxG5yoFwgAXFTplbYtQEznKidR2oTyHNqUO+avfu7i4Xjsdy2fRqw9vZAWSSmBP+IHvElJH0hM6EWsAngYyxL+OlU+xn4iXRVEzk5oo+G8NIhYPAY+fldBfzrT0c4+jBw2gnbt6Uj3xjnJffP6iUidXFe2XI6hRabmmi41y8nF3W6AeP4B0n2ie227aRZZsPOmwqvR6djj9Yc3WZ2ESDj/mSL+ddrTZtyYyNMVVg7mAI4nrI3TKsu3Q5qO3z2wUsSZouMPid2layAdSVTuq45VK4YQOJLi402gZGr10GH7xli/ADMnLKg07cApxGWwU2WOdsMehwR3hgT4s/mWDhJ9yiInKzsPGe+uJma0HWZSa47RFO0856GgyLTkCdEZgznrne0kI6YgEkrcQ7HB4xL0+jtlHB+vKm6Qe+/lnVstYW6cvQCYhsAzvH7yPVum8+heOAY5g55SljqJ2G6/AHw2pO8KBXoPWqngObtuEEF0U5Swp/ugelafbAG5czJeuNotPvwU7npKR7ZbWWuz73T7zxNHAo5YKm2pFqaCJltu10auO80nJE5DLZ+kyVblsTRhl9dUUE9nm8U8bWh2XVxDVn3Zs9VTvgPP/pKg8wY8cLxx+UJ/+u2o2tIWj2tYpDT6IMOooGMN/X2ZXro/96E+PAhPpboyVUZ+zSYKC0S4zq+H8W74mTr3uqjts/0sGJeAgBcqZ4HdoPWaO+gVLg5N7FZ56mhw2+Rx5+70N4vR/cdm1tCQplldt1vnmqDd1dwNt+dyKvoAn1 RirT31Dj xpaWjTI4JzVERudeidAlmfA0/HP6r0HPr5Xl4pCZk8KP+qBodawqMxnwWh3B9zaY8jZGh05vtja7pTx3JebF9D5vT1gT/V7oaBwiCNI7Wq07KuzYVuWjROPYaqGqSQ9ylxilrkC9m9Ktbg28vgWik43ZDgmoCgSC1MWer6ck3aUJCXHqgfmUqXhWSqNfHRHIUEAGaHu2sXZEATheviDol5bI2kx9aktpwphBRj5Zb0rxFrY/vCbZ/JRWZqplUZWZBn7sAhVn+4C7ibSt3J5AYQ/AeQJWn2P843UBqMzBVUG6frm4gyM3M8ez25cIyp5TrVjErVHTWxHDGG/5LwotLwTuUL700Xuhd10fEaWqMQJfzHUgDaKDjR5gvpHx9p2TBIRAmlRI4Q2QIkIZAJ5PDiqdwDG29++IaEACtNxI12lORvjyVrKVwFbziNU/yayNe3+3dkHa9iah/TlnchnXxnRoAn5y0dOla+LVIeOVijqGsJHa23k/CS7Zt2JFCciUVV06d7vGs72bSKS8JNtCoctRS5lE6vZpsR3mKNZtUTZsIWiHH8ApZIcGpza1AXletYhop40wKmS7WNbgmCamORd9+ng== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Break down the system-wide working set reporting into per-memcg reports, which aggregages its children hierarchically. The per-node working set reporting histograms and refresh/report threshold files are presented as memcg files, showing a report containing all the nodes. The per-node page age interval is configurable in sysfs and not available per-memcg, while the refresh interval and report threshold are configured per-memcg. Memcg interface: /sys/fs/cgroup/.../memory.workingset.page_age The memcg equivalent of the sysfs workingset page age histogram breaks down the workingset of this memcg and its children into page age intervals. Each node is prefixed with a node header and a newline. Non-proactive direct reclaim on this memcg can also wake up userspace agents that are waiting on this file. e.g. N0 1000 anon=0 file=0 2000 anon=0 file=0 3000 anon=0 file=0 4000 anon=0 file=0 5000 anon=0 file=0 18446744073709551615 anon=0 file=0 /sys/fs/cgroup/.../memory.workingset.refresh_interval The memcg equivalent of the sysfs refresh interval. A per-node number of how much time a page age histogram is valid for, in milliseconds. e.g. echo N0=2000 > memory.workingset.refresh_interval /sys/fs/cgroup/.../memory.workingset.report_threshold The memcg equivalent of the sysfs report threshold. A per-node number of how often userspace agent waiting on the page age histogram can be woken up, in milliseconds. e.g. echo N0=1000 > memory.workingset.report_threshold Signed-off-by: Yuanchu Xie Change-Id: I6c8a39cddd248c031c85f144787c9044912f3c21 --- include/linux/memcontrol.h | 21 ++++ include/linux/workingset_report.h | 6 +- mm/internal.h | 2 + mm/memcontrol.c | 178 +++++++++++++++++++++++++++++- mm/workingset_report.c | 12 +- 5 files changed, 214 insertions(+), 5 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 0e5bf25d324f..6d67f1072943 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -315,6 +315,11 @@ struct mem_cgroup { spinlock_t event_list_lock; #endif /* CONFIG_MEMCG_V1 */ +#ifdef CONFIG_WORKINGSET_REPORT + /* memory.workingset.page_age file */ + struct cgroup_file workingset_page_age_file; +#endif + struct mem_cgroup_per_node *nodeinfo[]; }; @@ -1063,6 +1068,16 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm, void split_page_memcg(struct page *head, int old_order, int new_order); +static inline struct cgroup_file * +mem_cgroup_page_age_file(struct mem_cgroup *memcg) +{ +#ifdef CONFIG_WORKINGSET_REPORT + return &memcg->workingset_page_age_file; +#else + return NULL; +#endif +} + #else /* CONFIG_MEMCG */ #define MEM_CGROUP_ID_SHIFT 0 @@ -1470,6 +1485,12 @@ void count_memcg_event_mm(struct mm_struct *mm, enum vm_event_item idx) static inline void split_page_memcg(struct page *head, int old_order, int new_order) { } + +static inline struct cgroup_file * +mem_cgroup_page_age_file(struct mem_cgroup *memcg) +{ + return NULL; +} #endif /* CONFIG_MEMCG */ /* diff --git a/include/linux/workingset_report.h b/include/linux/workingset_report.h index 2ec8b927b200..ae412d408037 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -9,6 +9,7 @@ struct mem_cgroup; struct pglist_data; struct node; struct lruvec; +struct cgroup_file; #ifdef CONFIG_WORKINGSET_REPORT @@ -40,7 +41,10 @@ struct wsr_state { unsigned long report_threshold; unsigned long refresh_interval; - struct kernfs_node *page_age_sys_file; + union { + struct kernfs_node *page_age_sys_file; + struct cgroup_file *page_age_cgroup_file; + }; /* breakdown of workingset by page age */ struct mutex page_age_lock; diff --git a/mm/internal.h b/mm/internal.h index 19af882c506e..cc54358ed582 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -421,6 +421,8 @@ void set_task_reclaim_state(struct task_struct *task, struct reclaim_state *rs); * in mm/wsr.c */ void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat); +int workingset_report_intervals_parse(char *src, + struct wsr_report_bins *bins); #else static inline void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5d16fd9d2c3f..56848dc94d48 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4318,6 +4318,162 @@ static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf, return nbytes; } +#ifdef CONFIG_WORKINGSET_REPORT +static int memory_ws_refresh_interval_show(struct seq_file *m, void *v) +{ + int nid; + struct mem_cgroup *memcg = mem_cgroup_from_seq(m); + + for_each_node_state(nid, N_MEMORY) { + struct wsr_state *wsr = + &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + + seq_printf(m, "N%d=%u ", nid, + jiffies_to_msecs(READ_ONCE(wsr->refresh_interval))); + } + seq_putc(m, '\n'); + + return 0; +} + +static ssize_t memory_wsr_threshold_parse(char *buf, size_t nbytes, + unsigned int *nid_out, + unsigned int *msecs) +{ + char *node, *threshold; + unsigned int nid; + int err; + + buf = strstrip(buf); + threshold = buf; + node = strsep(&threshold, "="); + + if (*node != 'N') + return -EINVAL; + + err = kstrtouint(node + 1, 0, &nid); + if (err) + return err; + + if (nid >= nr_node_ids || !node_state(nid, N_MEMORY)) + return -EINVAL; + + err = kstrtouint(threshold, 0, msecs); + if (err) + return err; + + *nid_out = nid; + + return nbytes; +} + +static ssize_t memory_ws_refresh_interval_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, + loff_t off) +{ + unsigned int nid, msecs; + struct wsr_state *wsr; + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); + ssize_t ret = memory_wsr_threshold_parse(buf, nbytes, &nid, &msecs); + + if (ret < 0) + return ret; + + wsr = &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + + mutex_lock(&wsr->page_age_lock); + if (msecs && !wsr->page_age) { + struct wsr_page_age_histo *page_age = + kzalloc(sizeof(struct wsr_page_age_histo), GFP_KERNEL); + + if (!page_age) { + ret = -ENOMEM; + goto unlock; + } + wsr->page_age = page_age; + } + if (!msecs && wsr->page_age) { + kfree(wsr->page_age); + wsr->page_age = NULL; + } + + WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(msecs)); +unlock: + mutex_unlock(&wsr->page_age_lock); + return ret; +} + +static int memory_ws_report_threshold_show(struct seq_file *m, void *v) +{ + int nid; + struct mem_cgroup *memcg = mem_cgroup_from_seq(m); + + for_each_node_state(nid, N_MEMORY) { + struct wsr_state *wsr = + &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + + seq_printf(m, "N%d=%u ", nid, + jiffies_to_msecs(READ_ONCE(wsr->report_threshold))); + } + seq_putc(m, '\n'); + + return 0; +} + +static ssize_t memory_ws_report_threshold_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, + loff_t off) +{ + unsigned int nid, msecs; + struct wsr_state *wsr; + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); + ssize_t ret = memory_wsr_threshold_parse(buf, nbytes, &nid, &msecs); + + if (ret < 0) + return ret; + + wsr = &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + WRITE_ONCE(wsr->report_threshold, msecs_to_jiffies(msecs)); + return ret; +} + +static int memory_ws_page_age_show(struct seq_file *m, void *v) +{ + int nid; + struct mem_cgroup *memcg = mem_cgroup_from_seq(m); + + for_each_node_state(nid, N_MEMORY) { + struct wsr_state *wsr = + &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + struct wsr_report_bin *bin; + + if (!READ_ONCE(wsr->page_age)) + continue; + + wsr_refresh_report(wsr, memcg, NODE_DATA(nid)); + mutex_lock(&wsr->page_age_lock); + if (!wsr->page_age) + goto unlock; + seq_printf(m, "N%d\n", nid); + for (bin = wsr->page_age->bins; + bin->idle_age != WORKINGSET_INTERVAL_MAX; bin++) + seq_printf(m, "%u anon=%lu file=%lu\n", + jiffies_to_msecs(bin->idle_age), + bin->nr_pages[0] * PAGE_SIZE, + bin->nr_pages[1] * PAGE_SIZE); + + seq_printf(m, "%lu anon=%lu file=%lu\n", WORKINGSET_INTERVAL_MAX, + bin->nr_pages[0] * PAGE_SIZE, + bin->nr_pages[1] * PAGE_SIZE); + +unlock: + mutex_unlock(&wsr->page_age_lock); + } + + return 0; +} +#endif + static struct cftype memory_files[] = { { .name = "current", @@ -4386,7 +4542,27 @@ static struct cftype memory_files[] = { .flags = CFTYPE_NS_DELEGATABLE, .write = memory_reclaim, }, - { } /* terminate */ +#ifdef CONFIG_WORKINGSET_REPORT + { + .name = "workingset.refresh_interval", + .flags = CFTYPE_NOT_ON_ROOT | CFTYPE_NS_DELEGATABLE, + .seq_show = memory_ws_refresh_interval_show, + .write = memory_ws_refresh_interval_write, + }, + { + .name = "workingset.report_threshold", + .flags = CFTYPE_NOT_ON_ROOT | CFTYPE_NS_DELEGATABLE, + .seq_show = memory_ws_report_threshold_show, + .write = memory_ws_report_threshold_write, + }, + { + .name = "workingset.page_age", + .flags = CFTYPE_NOT_ON_ROOT | CFTYPE_NS_DELEGATABLE, + .file_offset = offsetof(struct mem_cgroup, workingset_page_age_file), + .seq_show = memory_ws_page_age_show, + }, +#endif + {} /* terminate */ }; struct cgroup_subsys memory_cgrp_subsys = { diff --git a/mm/workingset_report.c b/mm/workingset_report.c index 801ac8e5c1da..333e51e3ee12 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -37,9 +37,12 @@ void wsr_destroy_pgdat(struct pglist_data *pgdat) void wsr_init_lruvec(struct lruvec *lruvec) { struct wsr_state *wsr = &lruvec->wsr; + struct mem_cgroup *memcg = lruvec_memcg(lruvec); memset(wsr, 0, sizeof(*wsr)); mutex_init(&wsr->page_age_lock); + if (memcg && !mem_cgroup_is_root(memcg)) + wsr->page_age_cgroup_file = mem_cgroup_page_age_file(memcg); } void wsr_destroy_lruvec(struct lruvec *lruvec) @@ -51,8 +54,8 @@ void wsr_destroy_lruvec(struct lruvec *lruvec) memset(wsr, 0, sizeof(*wsr)); } -static int workingset_report_intervals_parse(char *src, - struct wsr_report_bins *bins) +int workingset_report_intervals_parse(char *src, + struct wsr_report_bins *bins) { int err = 0, i = 0; char *cur, *next = strim(src); @@ -542,5 +545,8 @@ void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat) { struct wsr_state *wsr = &mem_cgroup_lruvec(memcg, pgdat)->wsr; - kernfs_notify(wsr->page_age_sys_file); + if (mem_cgroup_is_root(memcg)) + kernfs_notify(wsr->page_age_sys_file); + else + cgroup_file_notify(wsr->page_age_cgroup_file); } From patchwork Tue Aug 13 16:56:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13762348 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E80BC52D7C for ; Tue, 13 Aug 2024 16:59:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 75A406B00A0; Tue, 13 Aug 2024 12:59:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C0546B00A1; Tue, 13 Aug 2024 12:59:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4726D6B00A2; Tue, 13 Aug 2024 12:59:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 247DE6B00A0 for ; Tue, 13 Aug 2024 12:59:49 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D0A7A1209C6 for ; Tue, 13 Aug 2024 16:59:48 +0000 (UTC) X-FDA: 82447834056.08.ABB3E4D Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf14.hostedemail.com (Postfix) with ESMTP id F318A10001C for ; Tue, 13 Aug 2024 16:59:46 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=a74FgDtV; spf=pass (imf14.hostedemail.com: domain of 3AZG7ZgcKCE4EAq3sxAw44w1u.s421y3AD-220Bqs0.47w@flex--yuanchu.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3AZG7ZgcKCE4EAq3sxAw44w1u.s421y3AD-220Bqs0.47w@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723568315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xpAcupGObnfzDmBhsss97bo400nzNISpyaH5VDbdw9g=; b=KVX6ujKXm6HZmsAWPFeN8HR5DcOTOjFVR/5Su1xESe12ZEuBdNZid2tPAgLcIdPDbH/y2k ynJ/UNtgZCdeKKclupGgi4f99+wZfGVl7I6HiBVWQ0cUeBNrJUgLMYePRAKfAdv17xYBeI aA37neZJ6lTjZi7TBa9cpvkW0f9dzf4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723568315; a=rsa-sha256; cv=none; b=fy7Z+B7KZXQM87uASEzjLqYeLoDdeiAuBATpUz++29A62ooYgz7dTjpDxI3t/xBc0l8kVa YLQuSDKB1uke20isWki4alyEU7iR7aBqgd0HBZC7Rq3gpSCLCI379CpBiCif00XFDVRENS 8Eof2SEEcwm8aJFsNmf9SxaXA8azEDk= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=a74FgDtV; spf=pass (imf14.hostedemail.com: domain of 3AZG7ZgcKCE4EAq3sxAw44w1u.s421y3AD-220Bqs0.47w@flex--yuanchu.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3AZG7ZgcKCE4EAq3sxAw44w1u.s421y3AD-220Bqs0.47w@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-664b7a67ad4so130375617b3.2 for ; Tue, 13 Aug 2024 09:59:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723568386; x=1724173186; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xpAcupGObnfzDmBhsss97bo400nzNISpyaH5VDbdw9g=; b=a74FgDtV5G37gItFnoOROBRIS5WkHjgUD1lRsXYB1hFhalXJP3/hFmNegXdU3Ut8Lj DYuTBTGd4R1Mb2nunIIBQ70zvnpRpn6LBEOoBdjD7GAOswWRZwfuhkUdUfVUMLRS3yzm MRj2dZMvPRBjV8naUAQcauD5kMk+keCg/eh7ee6SjbVaqwRVM/FOSl+3F6RQ1BuN2KwG vs//wtXaoYVmnJIYszFAZ3414C7b/F3jhYlYBnchHoOxdRC3g/Wse0oWIpGTuvPSlrCA ldryMtYwWAEK/cPmZmjP+cAzEak4K72YH5j4SxsXKOLsKbr5MrOYl72WA/p9KrZ+XPUe 0blA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723568386; x=1724173186; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xpAcupGObnfzDmBhsss97bo400nzNISpyaH5VDbdw9g=; b=kDaM5WY1qFO0eHH0xll59U3SWnPNn1+NdwLJ/oNd33bc8H+ICSScxLOARIPbZ7HKL1 KRwCqs1Uy9jiatiJ9Z3tBizB4Z73flpBgRsgsx9ryywC3te32yYkzSph9yqzOMBn7uQa ojKEeMSpmCehpvlWMwY3d2CNBqsrRSwgKR6Qc7lGYKMqRtRwqg7fBrRBod7Jx9isiBhR PT0VNbelQoWkTdExC/uA1ZFC+0pTFzcFseaVL8xKsOWX07ZETXe4aCZqTdPLO7aBArqc oZsq+ZDMPcbSX2xQ80TYLIV4l8iOOSSPCC/M2hh8z5c3TycGtieGmuwQvecGmRDujo6E 01+w== X-Forwarded-Encrypted: i=1; AJvYcCXFnCCn5F1tvPq+uPgrQo5ohVrUcA/2vgyk2+p+6xBOiTgtmcusLGuEl/NyEeYP4wciJQhzkqmWaQ==@kvack.org X-Gm-Message-State: AOJu0YwLIHzi+wqXexiki2BTBa8WgfTRLWEkTA+VuTgCOJbTEGh0FkBS l4FUA9HE6poBhLJkj0M0C/souFQuqNDYC2kYlX7jUNNyFnVMK4NORMAV4yoAPBD2/AMQkO9CQIA yuXUrXg== X-Google-Smtp-Source: AGHT+IHuzPFm66066+PM+qeXLjE3MUrZ+OKfquaNrkkMfQw82fR/11BL+BIef/h89u4WvUPnZWHI7Qy55Ias X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:b50c:66e8:6532:a371]) (user=yuanchu job=sendgmr) by 2002:a0d:c607:0:b0:630:e8a:8a15 with SMTP id 00721157ae682-6ac8feff7e5mr7727b3.0.1723568385881; Tue, 13 Aug 2024 09:59:45 -0700 (PDT) Date: Tue, 13 Aug 2024 09:56:16 -0700 In-Reply-To: <20240813165619.748102-1-yuanchu@google.com> Mime-Version: 1.0 References: <20240813165619.748102-1-yuanchu@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813165619.748102-6-yuanchu@google.com> Subject: [PATCH v3 5/7] mm: add kernel aging thread for workingset reporting From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Andrew Morton , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org X-Stat-Signature: y8bjbmt9uunjatqen9xisxewpq361p9x X-Rspamd-Queue-Id: F318A10001C X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1723568386-915517 X-HE-Meta: U2FsdGVkX19az93FHRWrg4FRZ9dx2vY4e04Dtxa5G5Kssl0XEg5yC0q+b/RPRZWhuYalb1y2tJaK2tO4n9sDa9Axgnu6LKwrQV2aem4uRKXr6IVPm6sRFFH5RZLun5ow96Y2UgC6eLyWfJ7fxk/YbRmr0wZjNjXzmevmTaBUm8OmQ8RR0YoehfZ56efwBEiiIlErf0kZXJuv/w60DFr0gUVTi67+Si3XV9pBC3/txK8RteobjZrEvuOUhDIuspkH3jjJTXAKktO5Y6NG1j2AkilEeO50EJ5CK9exWEU8oiPogTS7kXGSZ9tSnBMDe+tMYsKbt87PfpcBjhvWibvism7j41buUgky6cQfd6WrZtShoGGE5hgWcS7i30cYseTOAZp2HSadB0Q1AYfieaUPs7iwgb3cEgmCDt3sP/YDeOqUDhhYLhRX3yqma3kBgZTIKhvEG+Du1ps7UacL2cYca5N1ffDuY1UQz3NezRDNa6ls+p8UbztoU7vwhKlpD2L+9PyYynMu2xsjTGzwv9SrQLfUah4IIawnpKfcq8I8/GncZ9xiXG7fkFwFUdklwXedXFMgTA7m0QlkKHSiOlCGcukx1Od5esvI0SyAuoWchiiZt3Jebyv7+aRmt95O7zQnbwz5ZeoITWfQazjcNTFTJC1Wgi0eup+QOFaKO9ZvPmBb4vWs+sKIc1R4nUdMprKIVN21Q5rsO2b5r8jX9VKegDYO2afypKd3xLSi0Hw3hI+lsPQ5LXjpXfU2EN3B9EgR2c9h29zE6nLs0R4F1BaWqzTMLT1kWHOUIs3oKdb5Cl4ujpDsryZCvzzZpcZdPTkx+LzL+9tV0cg+01UD+QUK0L105zITvPFOaEcd3gwgxj9UIcWvDMgZnZU4ZDmcXDbuo7HDLl9xVIhLQU/zkUIWHP8mU6eC8L67pCAwIBzj3BqA86fuRyIYZ7fmp29EOvSWDruR2ie8qKyRSAyTfq5 S9ve5vTV 8sjG5Ci8cf8opBJZT8FKUGqt8CG0GSAJ9aF+y+axbh9ww4p0IYs2buaOdd7JnpGw36fnLRWjULCqD9Amx5pHsU+KqtpxMHhWZOwE499a6q9+tWzaBjGf4pN7cbLALM/LeLD7K+KS2S99HPMisgo7WmXJMAajzF5Atx2KPgpmDd9zSN5CEA9oagHv4dXy7pooBqxkWO93MQLoqfy2ki0+tP402t1zzCyajUUxfEaq1FER2+MQJwdVPIOrKyQwNnEHsyK8f4+pK2QPIs40MNIoSotL4bb9c7NzEzC+3zpMjG0FUQH8ftBc9eOomng9jXkfeCLrSUeyN9SSwXw5sdmWt85liKvdHUZ/l2kHdM88C85/cmOPuk9hS/8z7O1pnRVdrQbo+pZIwRlFUwqLGjTEpzphCxUZujvuQoO3eauWT2raY1PYEaYFYaSy1RXJswAoR53kkyPANqRcXJUyYm7hUOmDJ2uimbbkREDIH6ngko4NBTrfbh7NBB0KuBzekn22Nfvp/3AxtEFtAsOE5emYOS3ikFRIwfrJc8GDGyROCPWXZKNiH6G2RuIBBarTA0EREx5SA1BR4Hd2cgGXfqmKygb2Ol0AdbWZQbtnF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For reliable and timely aging on memcgs, one has to read the page age histograms on time. A kernel thread makes it easier by aging memcgs with valid refresh_interval when they can be refreshed, and also reduces the latency of any userspace consumers of the page age histogram. The kerne aging thread is gated behind CONFIG_WORKINGSET_REPORT_AGING. Debugging stats may be added in the future for when aging cannot keep up with the configured refresh_interval. Signed-off-by: Yuanchu Xie Change-Id: Ia34a5d6b7eb91e39fab5f2d4ca2f18a9c370890b --- include/linux/workingset_report.h | 11 ++- mm/Kconfig | 6 ++ mm/Makefile | 1 + mm/memcontrol.c | 8 +- mm/workingset_report.c | 15 +++- mm/workingset_report_aging.c | 127 ++++++++++++++++++++++++++++++ 6 files changed, 162 insertions(+), 6 deletions(-) create mode 100644 mm/workingset_report_aging.c diff --git a/include/linux/workingset_report.h b/include/linux/workingset_report.h index ae412d408037..9294023db5a8 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -63,7 +63,16 @@ void wsr_remove_sysfs(struct node *node); * The next refresh time is stored in refresh_time. */ bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, - struct pglist_data *pgdat); + struct pglist_data *pgdat, unsigned long *refresh_time); + +#ifdef CONFIG_WORKINGSET_REPORT_AGING +void wsr_wakeup_aging_thread(void); +#else /* CONFIG_WORKINGSET_REPORT_AGING */ +static inline void wsr_wakeup_aging_thread(void) +{ +} +#endif /* CONFIG_WORKINGSET_REPORT_AGING */ + #else static inline void wsr_init_lruvec(struct lruvec *lruvec) { diff --git a/mm/Kconfig b/mm/Kconfig index 951b4e7300d2..c1dfece5b71c 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1272,6 +1272,12 @@ config WORKINGSET_REPORT This option exports stats and events giving the user more insight into its memory working set. +config WORKINGSET_REPORT_AGING + bool "Workingset report kernel aging thread" + depends on WORKINGSET_REPORT + help + Performs aging on memcgs with their configured refresh intervals. + source "mm/damon/Kconfig" endmenu diff --git a/mm/Makefile b/mm/Makefile index 62cef72ce7f8..4a64b2454b6a 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -99,6 +99,7 @@ obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o obj-$(CONFIG_WORKINGSET_REPORT) += workingset_report.o +obj-$(CONFIG_WORKINGSET_REPORT_AGING) += workingset_report_aging.o ifdef CONFIG_SWAP obj-$(CONFIG_MEMCG) += swap_cgroup.o endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 56848dc94d48..52123b6cc9ce 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4373,12 +4373,12 @@ static ssize_t memory_ws_refresh_interval_write(struct kernfs_open_file *of, { unsigned int nid, msecs; struct wsr_state *wsr; + unsigned long old_interval; struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); ssize_t ret = memory_wsr_threshold_parse(buf, nbytes, &nid, &msecs); if (ret < 0) return ret; - wsr = &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; mutex_lock(&wsr->page_age_lock); @@ -4397,9 +4397,13 @@ static ssize_t memory_ws_refresh_interval_write(struct kernfs_open_file *of, wsr->page_age = NULL; } + old_interval = READ_ONCE(wsr->refresh_interval); WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(msecs)); unlock: mutex_unlock(&wsr->page_age_lock); + if (ret > 0 && msecs && + (!old_interval || jiffies_to_msecs(old_interval) > msecs)) + wsr_wakeup_aging_thread(); return ret; } @@ -4450,7 +4454,7 @@ static int memory_ws_page_age_show(struct seq_file *m, void *v) if (!READ_ONCE(wsr->page_age)) continue; - wsr_refresh_report(wsr, memcg, NODE_DATA(nid)); + wsr_refresh_report(wsr, memcg, NODE_DATA(nid), NULL); mutex_lock(&wsr->page_age_lock); if (!wsr->page_age) goto unlock; diff --git a/mm/workingset_report.c b/mm/workingset_report.c index 333e51e3ee12..6f64d89fe70d 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -274,7 +274,7 @@ static void copy_node_bins(struct pglist_data *pgdat, } bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, - struct pglist_data *pgdat) + struct pglist_data *pgdat, unsigned long *refresh_time) { struct wsr_page_age_histo *page_age; unsigned long refresh_interval = READ_ONCE(wsr->refresh_interval); @@ -291,10 +291,14 @@ bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, goto unlock; if (page_age->timestamp && time_is_after_jiffies(page_age->timestamp + refresh_interval)) - goto unlock; + goto time; refresh_scan(wsr, root, pgdat, refresh_interval); copy_node_bins(pgdat, page_age); refresh_aggregate(page_age, root, pgdat); + +time: + if (refresh_time) + *refresh_time = page_age->timestamp + refresh_interval; unlock: mutex_unlock(&wsr->page_age_lock); return !!page_age; @@ -357,6 +361,7 @@ static ssize_t refresh_interval_store(struct kobject *kobj, unsigned int interval; int err; struct wsr_state *wsr = kobj_to_wsr(kobj); + unsigned long old_interval = 0; err = kstrtouint(buf, 0, &interval); if (err) @@ -378,9 +383,13 @@ static ssize_t refresh_interval_store(struct kobject *kobj, wsr->page_age = NULL; } + old_interval = READ_ONCE(wsr->refresh_interval); WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(interval)); unlock: mutex_unlock(&wsr->page_age_lock); + if (!err && interval && + (!old_interval || jiffies_to_msecs(old_interval) > interval)) + wsr_wakeup_aging_thread(); return err ?: len; } @@ -470,7 +479,7 @@ static ssize_t page_age_show(struct kobject *kobj, struct kobj_attribute *attr, int ret = 0; struct wsr_state *wsr = kobj_to_wsr(kobj); - wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj)); + wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj), NULL); mutex_lock(&wsr->page_age_lock); if (!wsr->page_age) diff --git a/mm/workingset_report_aging.c b/mm/workingset_report_aging.c new file mode 100644 index 000000000000..91ad5020778a --- /dev/null +++ b/mm/workingset_report_aging.c @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Workingset report kernel aging thread + * + * Performs aging on behalf of memcgs with their configured refresh interval. + * While a userspace program can periodically read the page age breakdown + * per-memcg and trigger aging, the kernel performing aging is less overhead, + * more consistent, and more reliable for the use case where every memcg should + * be aged according to their refresh interval. + */ +#define pr_fmt(fmt) "workingset report aging: " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static DECLARE_WAIT_QUEUE_HEAD(aging_wait); +static bool refresh_pending; + +static bool do_aging_node(int nid, unsigned long *next_wake_time) +{ + struct mem_cgroup *memcg; + bool should_wait = true; + struct pglist_data *pgdat = NODE_DATA(nid); + + memcg = mem_cgroup_iter(NULL, NULL, NULL); + do { + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); + struct wsr_state *wsr = &lruvec->wsr; + unsigned long refresh_time; + + /* use returned time to decide when to wake up next */ + if (wsr_refresh_report(wsr, memcg, pgdat, &refresh_time)) { + if (should_wait) { + should_wait = false; + *next_wake_time = refresh_time; + } else if (time_before(refresh_time, *next_wake_time)) { + *next_wake_time = refresh_time; + } + } + + cond_resched(); + } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL))); + + return should_wait; +} + +static int do_aging(void *unused) +{ + while (!kthread_should_stop()) { + int nid; + long timeout_ticks; + unsigned long next_wake_time; + bool should_wait = true; + + WRITE_ONCE(refresh_pending, false); + for_each_node_state(nid, N_MEMORY) { + unsigned long node_next_wake_time; + + if (do_aging_node(nid, &node_next_wake_time)) + continue; + if (should_wait) { + should_wait = false; + next_wake_time = node_next_wake_time; + } else if (time_before(node_next_wake_time, + next_wake_time)) { + next_wake_time = node_next_wake_time; + } + } + + if (should_wait) { + wait_event_interruptible(aging_wait, refresh_pending); + continue; + } + + /* sleep until next aging */ + timeout_ticks = next_wake_time - jiffies; + if (timeout_ticks > 0 && + timeout_ticks != MAX_SCHEDULE_TIMEOUT) { + schedule_timeout_idle(timeout_ticks); + continue; + } + } + return 0; +} + +/* Invoked when refresh_interval shortens or changes to a non-zero value. */ +void wsr_wakeup_aging_thread(void) +{ + WRITE_ONCE(refresh_pending, true); + wake_up_interruptible(&aging_wait); +} + +static struct task_struct *aging_thread; + +static int aging_init(void) +{ + struct task_struct *task; + + task = kthread_run(do_aging, NULL, "kagingd"); + + if (IS_ERR(task)) { + pr_err("Failed to create aging kthread\n"); + return PTR_ERR(task); + } + + aging_thread = task; + pr_info("module loaded\n"); + return 0; +} + +static void aging_exit(void) +{ + kthread_stop(aging_thread); + aging_thread = NULL; + pr_info("module unloaded\n"); +} + +module_init(aging_init); +module_exit(aging_exit); From patchwork Tue Aug 13 16:56:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13762349 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2354C52D7B for ; Tue, 13 Aug 2024 16:59:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 958926B00A2; Tue, 13 Aug 2024 12:59:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DF826B00A3; Tue, 13 Aug 2024 12:59:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 622BF6B00A4; Tue, 13 Aug 2024 12:59:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2C4476B00A2 for ; Tue, 13 Aug 2024 12:59:51 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CF4A6408BF for ; Tue, 13 Aug 2024 16:59:50 +0000 (UTC) X-FDA: 82447834140.27.3FBDE1B Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 04EB9180011 for ; Tue, 13 Aug 2024 16:59:48 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=af63XKbm; spf=pass (imf24.hostedemail.com: domain of 3A5G7ZgcKCFAGCs5uzCy66y3w.u64305CF-442Dsu2.69y@flex--yuanchu.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3A5G7ZgcKCFAGCs5uzCy66y3w.u64305CF-442Dsu2.69y@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723568353; a=rsa-sha256; cv=none; b=VSXDdvI4uIptHoS8/il+9vNBkI1HuAZIHNTu9Pg52sGxkeq7sMvGVI5UpYrmmtmRh76093 SWbLW4L1U7k4gTCjTUqGQvOTi8qKBwuG8VD/B1BhbRpYWfbI8gIEmHFGlNFmneTis+Fjs9 NSQC5Qes4oZnMTRdvr+aw5c44N9CwjE= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=af63XKbm; spf=pass (imf24.hostedemail.com: domain of 3A5G7ZgcKCFAGCs5uzCy66y3w.u64305CF-442Dsu2.69y@flex--yuanchu.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3A5G7ZgcKCFAGCs5uzCy66y3w.u64305CF-442Dsu2.69y@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723568353; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nmNRGoZ1KbCR4Ovo2GA2r3+/VwovN+l6r9v92Jg3Hew=; b=U9MjU9P0OzV3L63ttvoo7BVmV/siNTt6Y4SpGwU906+OfjQBr1FqI8eJeFCIaSmaC4AH89 DqjeHu9RdzjV+Rlc/qiRIxE3Qt5JwcjAGKZKXk0TPLYNdDG3SJkbdJ53GA9ovmI4AnETA9 NWU84CUMKR0TeJ9RDgA9P/i9s8Q+3Jc= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-66a2aee82a0so111848427b3.0 for ; Tue, 13 Aug 2024 09:59:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723568388; x=1724173188; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nmNRGoZ1KbCR4Ovo2GA2r3+/VwovN+l6r9v92Jg3Hew=; b=af63XKbmv8rhtDc75mu9SBSsWYU+TdmINC3E0vVmw/WB+TQgoYDFMECr0g+wS7fN6V urj9wHfxUY9iRa4I8JQZJB9z2f814WMILjgvsGra2HftfNo1PyVfa4UbCOKfUfpF2s8l 0ArfpjI+vdPz2eKcjm2uhvZe7DTEsj5fETP6nVggNBTi2Krklgp9ssu7XZmYY/NCzUxl Xy6wryyXTEGawzgs40ePFYvuAHvaCHB8s0jOtSflXQVewzd/XGhp2Fh1tMfQ18nFZ6lE HKfPtgPVAkpEZmE7Dwhy2z91dsXrumu7f57e8B35v+D1UCuytqpDBivgFVRfWTphElZX 7U5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723568388; x=1724173188; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nmNRGoZ1KbCR4Ovo2GA2r3+/VwovN+l6r9v92Jg3Hew=; b=meDIGfO/u5xqD9ZqtXZI8xyk73SDB+ZvQ+0c7QplzvyPaiVtFSe+P5iMmakfcdG4VZ WzK9wdQfYcAR0S5R9PiG686Rxrx85QbGoxqtjPJBm/rk9pxuWG7UIMKaBM9sxvSwKDy7 awoiMgx/dUe7ee6dQGd/tBtY4MutLdZIij72eTMP7gq7UFlKoVn+towGIz7JZE7dn4Ze qO/XNU75QLqCZru3vScQZCfHY1h0kLn4vk+6jRZzrtkeg0JOEHspWt12w6QyS0W4OrH1 gelHiFhpxGDjbwf8aH9d1Jy9DEzlfTPomcdjrEebmanXpytZd3ba+DUkQQRKl594eoHl dMcw== X-Forwarded-Encrypted: i=1; AJvYcCX4XoPKHgyEZjMqZsvgalzMTXc79IaTeKvJzjTwJzFs9hJStTOU5g63ZZ3qafc39aQPT7d4QTl12zIRlbiarhG3vE4= X-Gm-Message-State: AOJu0Yx4w9XeHpz9z9hlGxSZWKqjmS2mH+1xPd2LrTxNpB8LsTa5dxix 4sU5dX+4wIqklA4qB73+YCBwg7yTiF9S9pYFyE7SIgaD2iK09SPfNaQXSSXFY9BefkYrlLP7OXJ jrfIFrg== X-Google-Smtp-Source: AGHT+IHDb1zAvYhwkH2Nz7owaIopIGytMqAr45V8NRtzWfOlB9K6iqc0dkiowScYN9kSuCEnjpoAnU+/LUcr X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:b50c:66e8:6532:a371]) (user=yuanchu job=sendgmr) by 2002:a0d:e586:0:b0:61b:e103:804d with SMTP id 00721157ae682-6ac91bde66dmr2347b3.0.1723568387967; Tue, 13 Aug 2024 09:59:47 -0700 (PDT) Date: Tue, 13 Aug 2024 09:56:17 -0700 In-Reply-To: <20240813165619.748102-1-yuanchu@google.com> Mime-Version: 1.0 References: <20240813165619.748102-1-yuanchu@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813165619.748102-7-yuanchu@google.com> Subject: [PATCH v3 6/7] selftest: test system-wide workingset reporting From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Andrew Morton , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org X-Stat-Signature: h8fpare78ht4fwk5uum9bfizf55931c9 X-Rspamd-Queue-Id: 04EB9180011 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1723568388-378523 X-HE-Meta: U2FsdGVkX19MMoOKRHVrPFglm+Nd1Shwv88cPRgwQqQiGqmml0MqqMorMj12//al2iRzX+4FSjBUgDUAi5iJ05LCDxJ8MGAd/iou+YmFkji6CwarrD9E71/+A4cNmZK8o9MHW9Em7wSDyWAiY1i00tXgJPz40xt23kGco2NdauK6YmxETW60nbo9HbY1ekT9ID5+j/FE5X7UlhUQcQH6Qvlex7iCM8dsSHPVZgIsBDE+n9HO+kDy6sFnzQdjCtOmtuRquf6bKgoIvAajPOmy6LHLS/wNZvWye//BWZo6f+2+3QnwltojB4OTZYKUdRkISOZc3jXAR9MxCndh4pI78xKgNvR6ZAFqC2PvZhWxZqCAmcytn5QStmbBRbhhd7nGselMlPYBe0Ohru3RgYFIcOikzXa3I/xSMY4aMtAcXZj9eyReLnl7RGrOOcc2rQ14QvW2Em3do5H0aAgBdb+KvUrhkOVziRuzw8hAYdxFB3hVwySBruwH1RVE1rB+RQrA0LuIlP85ts/S+oVzng1lvFgGNPWrVJQpXaSP7zBJBkODqPmQWjD5+2VkiWYuED3Ows2rlF3Cwew6Tl/0eHAv1dZnxwEo83QicIWnCrts76BjZqcrpC5IXRR3SGc2Rp5CNq6Gn6uiyfjQIxjTL/23pz3rOTm89bC5xoB4vWEXv0KVfgIysGaEcZGYIEC5eeTLlnGSafYGHjX6jUPnBG/rLhbpW3GkXms4Lvu4ZWSA8MKp7zb/cErcKCiAwH/xY72OJICkRJ1kg94kAN1v00StPW2V+I6QYtel0gKHiFU7GIEEfm3ERunJzozvPhTD15Dznjjn017RpMvuXrNXG5WPSyazLRaZpe6Jm3UTFFfJgCnREGU5YWdK7R2Zm7HLPj3adV0g1AUZUQEGauLtaWcKmf+SIvU1F92Z1uyF3ccmBvF5u5YHOtbTTRbY9GsEEAaw4RoBj10g2KX5xApHgxX r2oJcdov fW3eTat45TUWFhffeLWJzWO6BYaVITbqF213GqiVewr9wdHxNL7wt+xT+ItZ9MTmyB2HFkAjWBAWFOMQx+1Jm4X5yqjBdVSZwURb82bTbQPNVUShjG50WE53GGou1ahZSKuJ+5FsZdlY+Du8vsPOCHw5+ZiDI5itNA8r7hSpr58eJ/DQC0PY/VEhwVhihTZpl7P7fi9MrfuR0BIBr3q8ySQSiZ7+mu+Xfw8/W5jpvQ32giGDDYDB2A+rs9lqgeRZQZ2zau2MAqLVtYjKACEwmL9mMfuZjuL2+Q1jJ7isPtWx+JOGH4fXOBgtErgYO2yr1qRDGYKW8OOf8l18BAA3RJJPFEBy2+8DNZAaEoIA5rnPwpgHiCUe0bImy0TkOgF5EzEhARzynOXM8QwDhzUydAxjMnLS8ZOW6DwzB46j6fwdp7bBEv+SgD5tdvsbn2Yoy269nfBDcaLRZFQuBMQmas1Kp2P/OFHke9iYniog+bH2ceDxWvrfZYtjtpACGR77RreNY7XG3uyq5OE/0Gi3fcZ2rv5SZgBqXlXVPzSwfDZjU87JeR6goKMi0xpbKCe9UfCEoC9RhJZaZ8iwDkbWo+DzI54L4OiB3Gszg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A basic test that verifies the working set size of a simple memory accessor. It should work with or without the aging thread. When running tests with run_vmtests.sh, file workingset report testing requires an environment variable WORKINGSET_REPORT_TEST_FILE_PATH to store a temporary file, which is passed into the test invocation as a parameter. Signed-off-by: Yuanchu Xie Change-Id: I1a55364164b9a67934f8500aaf77df4372edaa55 --- tools/testing/selftests/mm/.gitignore | 1 + tools/testing/selftests/mm/Makefile | 3 + tools/testing/selftests/mm/run_vmtests.sh | 5 + .../testing/selftests/mm/workingset_report.c | 306 ++++++++++++++++ .../testing/selftests/mm/workingset_report.h | 39 +++ .../selftests/mm/workingset_report_test.c | 330 ++++++++++++++++++ 6 files changed, 684 insertions(+) create mode 100644 tools/testing/selftests/mm/workingset_report.c create mode 100644 tools/testing/selftests/mm/workingset_report.h create mode 100644 tools/testing/selftests/mm/workingset_report_test.c diff --git a/tools/testing/selftests/mm/.gitignore b/tools/testing/selftests/mm/.gitignore index da030b43e43b..e5cd0085ab74 100644 --- a/tools/testing/selftests/mm/.gitignore +++ b/tools/testing/selftests/mm/.gitignore @@ -51,3 +51,4 @@ hugetlb_madv_vs_map mseal_test seal_elf droppable +workingset_report_test diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile index 7b8a5def54a1..8d1a7d24ecd1 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -77,6 +77,7 @@ TEST_GEN_FILES += hugetlb_fault_after_madv TEST_GEN_FILES += hugetlb_madv_vs_map TEST_GEN_FILES += hugetlb_dio TEST_GEN_FILES += droppable +TEST_GEN_FILES += workingset_report_test ifneq ($(ARCH),arm64) TEST_GEN_FILES += soft-dirty @@ -135,6 +136,8 @@ $(TEST_GEN_FILES): vm_util.c thp_settings.c $(OUTPUT)/uffd-stress: uffd-common.c $(OUTPUT)/uffd-unit-tests: uffd-common.c +$(OUTPUT)/workingset_report_test: workingset_report.c + ifeq ($(ARCH),x86_64) BINARIES_32 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_32)) BINARIES_64 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_64)) diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index 03ac4f2e1cce..c4a81f17a3a3 100755 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -75,6 +75,8 @@ separated by spaces: read-only VMAs - mdwe test prctl(PR_SET_MDWE, ...) +- workingset_report + test workingset reporting example: ./run_vmtests.sh -t "hmm mmap ksm" EOF @@ -453,6 +455,9 @@ CATEGORY="mkdirty" run_test ./mkdirty CATEGORY="mdwe" run_test ./mdwe_test +CATEGORY="workingset_report" run_test ./workingset_report_test \ + "${WORKINGSET_REPORT_TEST_FILE_PATH}" + echo "SUMMARY: PASS=${count_pass} SKIP=${count_skip} FAIL=${count_fail}" | tap_prefix echo "1..${count_total}" | tap_output diff --git a/tools/testing/selftests/mm/workingset_report.c b/tools/testing/selftests/mm/workingset_report.c new file mode 100644 index 000000000000..ee4dda5c371d --- /dev/null +++ b/tools/testing/selftests/mm/workingset_report.c @@ -0,0 +1,306 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "workingset_report.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../kselftest.h" + +#define SYSFS_NODE_ONLINE "/sys/devices/system/node/online" +#define PROC_DROP_CACHES "/proc/sys/vm/drop_caches" + +/* Returns read len on success, or -errno on failure. */ +static ssize_t read_text(const char *path, char *buf, size_t max_len) +{ + ssize_t len; + int fd, err; + size_t bytes_read = 0; + + if (!max_len) + return -EINVAL; + + fd = open(path, O_RDONLY); + if (fd < 0) + return -errno; + + while (bytes_read < max_len - 1) { + len = read(fd, buf + bytes_read, max_len - 1 - bytes_read); + + if (len <= 0) + break; + bytes_read += len; + } + + buf[bytes_read] = '\0'; + + err = -errno; + close(fd); + return len < 0 ? err : bytes_read; +} + +/* Returns written len on success, or -errno on failure. */ +static ssize_t write_text(const char *path, const char *buf, ssize_t max_len) +{ + int fd, len, err; + size_t bytes_written = 0; + + fd = open(path, O_WRONLY | O_APPEND); + if (fd < 0) + return -errno; + + while (bytes_written < max_len) { + len = write(fd, buf + bytes_written, max_len - bytes_written); + + if (len < 0) + break; + bytes_written += len; + } + + err = -errno; + close(fd); + return len < 0 ? err : bytes_written; +} + +static long read_num(const char *path) +{ + char buf[21]; + + if (read_text(path, buf, sizeof(buf)) <= 0) + return -1; + return (long)strtoul(buf, NULL, 10); +} + +static int write_num(const char *path, unsigned long n) +{ + char buf[21]; + + sprintf(buf, "%lu", n); + if (write_text(path, buf, strlen(buf)) < 0) + return -1; + return 0; +} + +long sysfs_get_refresh_interval(int nid) +{ + char file[128]; + + snprintf(file, sizeof(file), + "/sys/devices/system/node/node%d/workingset_report/refresh_interval", + nid); + return read_num(file); +} + +int sysfs_set_refresh_interval(int nid, long interval) +{ + char file[128]; + + snprintf(file, sizeof(file), + "/sys/devices/system/node/node%d/workingset_report/refresh_interval", + nid); + return write_num(file, interval); +} + +int sysfs_get_page_age_intervals_str(int nid, char *buf, int len) +{ + char path[128]; + + snprintf(path, sizeof(path), + "/sys/devices/system/node/node%d/workingset_report/page_age_intervals", + nid); + return read_text(path, buf, len); + +} + +int sysfs_set_page_age_intervals_str(int nid, const char *buf, int len) +{ + char path[128]; + + snprintf(path, sizeof(path), + "/sys/devices/system/node/node%d/workingset_report/page_age_intervals", + nid); + return write_text(path, buf, len); +} + +int sysfs_set_page_age_intervals(int nid, const char *const intervals[], + int nr_intervals) +{ + char file[128]; + char buf[1024]; + int i; + int err, len = 0; + + for (i = 0; i < nr_intervals; ++i) { + err = snprintf(buf + len, sizeof(buf) - len, "%s", intervals[i]); + + if (err < 0) + return err; + len += err; + + if (i < nr_intervals - 1) { + err = snprintf(buf + len, sizeof(buf) - len, ","); + if (err < 0) + return err; + len += err; + } + } + + snprintf(file, sizeof(file), + "/sys/devices/system/node/node%d/workingset_report/page_age_intervals", + nid); + return write_text(file, buf, len); +} + +int get_nr_nodes(void) +{ + char buf[22]; + char *found; + + if (read_text(SYSFS_NODE_ONLINE, buf, sizeof(buf)) <= 0) + return -1; + found = strstr(buf, "-"); + if (found) + return (int)strtoul(found + 1, NULL, 10) + 1; + return (long)strtoul(buf, NULL, 10) + 1; +} + +int drop_pagecache(void) +{ + return write_num(PROC_DROP_CACHES, 1); +} + +ssize_t sysfs_page_age_read(int nid, char *buf, size_t len) + +{ + char file[128]; + + snprintf(file, sizeof(file), + "/sys/devices/system/node/node%d/workingset_report/page_age", + nid); + return read_text(file, buf, len); +} + +/* + * Finds the first occurrence of "N\n" + * Modifies buf to terminate before the next occurrence of "N". + * Returns a substring of buf starting after "N\n" + */ +char *page_age_split_node(char *buf, int nid, char **next) +{ + char node_str[5]; + char *found; + int node_str_len; + + node_str_len = snprintf(node_str, sizeof(node_str), "N%u\n", nid); + + /* find the node prefix first */ + found = strstr(buf, node_str); + if (!found) { + ksft_print_msg("cannot find '%s' in page_idle_age", node_str); + return NULL; + } + found += node_str_len; + + *next = strchr(found, 'N'); + if (*next) + *(*next - 1) = '\0'; + + return found; +} + +ssize_t page_age_read(const char *buf, const char *interval, int pagetype) +{ + static const char * const type[ANON_AND_FILE] = { "anon=", "file=" }; + char *found; + + found = strstr(buf, interval); + if (!found) { + ksft_print_msg("cannot find %s in page_age", interval); + return -1; + } + found = strstr(found, type[pagetype]); + if (!found) { + ksft_print_msg("cannot find %s in page_age", type[pagetype]); + return -1; + } + found += strlen(type[pagetype]); + return (long)strtoul(found, NULL, 10); +} + +static const char *TEMP_FILE = "/tmp/workingset_selftest"; +void cleanup_file_workingset(void) +{ + remove(TEMP_FILE); +} + +int alloc_file_workingset(void *arg) +{ + int err = 0; + char *ptr; + int fd; + int ppid; + char *mapped; + size_t size = (size_t)arg; + size_t page_size = getpagesize(); + + ppid = getppid(); + + fd = open(TEMP_FILE, O_RDWR | O_CREAT); + if (fd < 0) { + err = -errno; + ksft_perror("failed to open temp file\n"); + goto cleanup; + } + + if (fallocate(fd, 0, 0, size) < 0) { + err = -errno; + ksft_perror("fallocate"); + goto cleanup; + } + + mapped = (char *)mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, + fd, 0); + if (mapped == NULL) { + err = -errno; + ksft_perror("mmap"); + goto cleanup; + } + + while (getppid() == ppid) { + sync(); + for (ptr = mapped; ptr < mapped + size; ptr += page_size) + *ptr = *ptr ^ 0xFF; + } + +cleanup: + cleanup_file_workingset(); + return err; +} + +int alloc_anon_workingset(void *arg) +{ + char *buf, *ptr; + int ppid = getppid(); + size_t size = (size_t)arg; + size_t page_size = getpagesize(); + + buf = malloc(size); + + if (!buf) { + ksft_print_msg("cannot allocate anon workingset"); + exit(1); + } + + while (getppid() == ppid) { + for (ptr = buf; ptr < buf + size; ptr += page_size) + *ptr = *ptr ^ 0xFF; + } + + free(buf); + return 0; +} diff --git a/tools/testing/selftests/mm/workingset_report.h b/tools/testing/selftests/mm/workingset_report.h new file mode 100644 index 000000000000..c5c281e4069b --- /dev/null +++ b/tools/testing/selftests/mm/workingset_report.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef WORKINGSET_REPORT_H_ +#define WORKINGSET_REPORT_H_ + +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif + +#include +#include +#include +#include +#include + +#define PAGETYPE_ANON 0 +#define PAGETYPE_FILE 1 +#define ANON_AND_FILE 2 + +int get_nr_nodes(void); +int drop_pagecache(void); + +long sysfs_get_refresh_interval(int nid); +int sysfs_set_refresh_interval(int nid, long interval); + +int sysfs_get_page_age_intervals_str(int nid, char *buf, int len); +int sysfs_set_page_age_intervals_str(int nid, const char *buf, int len); + +int sysfs_set_page_age_intervals(int nid, const char *const intervals[], + int nr_intervals); + +char *page_age_split_node(char *buf, int nid, char **next); +ssize_t sysfs_page_age_read(int nid, char *buf, size_t len); +ssize_t page_age_read(const char *buf, const char *interval, int pagetype); + +int alloc_file_workingset(void *arg); +void cleanup_file_workingset(void); +int alloc_anon_workingset(void *arg); + +#endif /* WORKINGSET_REPORT_H_ */ diff --git a/tools/testing/selftests/mm/workingset_report_test.c b/tools/testing/selftests/mm/workingset_report_test.c new file mode 100644 index 000000000000..89ff4e9d746e --- /dev/null +++ b/tools/testing/selftests/mm/workingset_report_test.c @@ -0,0 +1,330 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "workingset_report.h" + +#include +#include +#include +#include + +#include "../clone3/clone3_selftests.h" + +#define REFRESH_INTERVAL 5000 +#define MB(x) (x << 20) + +static void sleep_ms(int milliseconds) +{ + struct timespec ts; + + ts.tv_sec = milliseconds / 1000; + ts.tv_nsec = (milliseconds % 1000) * 1000000; + nanosleep(&ts, NULL); +} + +/* + * Checks if two given values differ by less than err% of their sum. + */ +static inline int values_close(long a, long b, int err) +{ + return labs(a - b) <= (a + b) / 100 * err; +} + +static const char * const PAGE_AGE_INTERVALS[] = { + "6000", "10000", "15000", "18446744073709551615", +}; +#define NR_PAGE_AGE_INTERVALS (ARRAY_SIZE(PAGE_AGE_INTERVALS)) + +static int set_page_age_intervals_all_nodes(const char *intervals, int nr_nodes) +{ + int i; + + for (i = 0; i < nr_nodes; ++i) { + int err = sysfs_set_page_age_intervals_str( + i, &intervals[i * 1024], strlen(&intervals[i * 1024])); + + if (err < 0) + return err; + } + return 0; +} + +static int get_page_age_intervals_all_nodes(char *intervals, int nr_nodes) +{ + int i; + + for (i = 0; i < nr_nodes; ++i) { + int err = sysfs_get_page_age_intervals_str( + i, &intervals[i * 1024], 1024); + + if (err < 0) + return err; + } + return 0; +} + +static int set_refresh_interval_all_nodes(const long *interval, int nr_nodes) +{ + int i; + + for (i = 0; i < nr_nodes; ++i) { + int err = sysfs_set_refresh_interval(i, interval[i]); + + if (err < 0) + return err; + } + return 0; +} + +static int get_refresh_interval_all_nodes(long *interval, int nr_nodes) +{ + int i; + + for (i = 0; i < nr_nodes; ++i) { + long val = sysfs_get_refresh_interval(i); + + if (val < 0) + return val; + interval[i] = val; + } + return 0; +} + +static pid_t clone_and_run(int fn(void *arg), void *arg) +{ + pid_t pid; + + struct __clone_args args = { + .exit_signal = SIGCHLD, + }; + + pid = sys_clone3(&args, sizeof(struct __clone_args)); + + if (pid == 0) + exit(fn(arg)); + + return pid; +} + +static int read_workingset(int pagetype, int nid, + unsigned long page_age[NR_PAGE_AGE_INTERVALS]) +{ + int i, err; + char buf[4096]; + + err = sysfs_page_age_read(nid, buf, sizeof(buf)); + if (err < 0) + return err; + + for (i = 0; i < NR_PAGE_AGE_INTERVALS; ++i) { + err = page_age_read(buf, PAGE_AGE_INTERVALS[i], pagetype); + if (err < 0) + return err; + page_age[i] = err; + } + + return 0; +} + +static ssize_t read_interval_all_nodes(int pagetype, int interval) +{ + int i, err; + unsigned long page_age[NR_PAGE_AGE_INTERVALS]; + ssize_t ret = 0; + int nr_nodes = get_nr_nodes(); + + for (i = 0; i < nr_nodes; ++i) { + err = read_workingset(pagetype, i, page_age); + if (err < 0) + return err; + + ret += page_age[interval]; + } + + return ret; +} + +#define TEST_SIZE MB(500l) + +static int run_test(int f(void)) +{ + int i, err, test_result; + long *old_refresh_intervals; + long *new_refresh_intervals; + char *old_page_age_intervals; + int nr_nodes = get_nr_nodes(); + + if (nr_nodes <= 0) { + ksft_print_msg("failed to get nr_nodes\n"); + return KSFT_FAIL; + } + + old_refresh_intervals = calloc(nr_nodes, sizeof(long)); + new_refresh_intervals = calloc(nr_nodes, sizeof(long)); + old_page_age_intervals = calloc(nr_nodes, 1024); + + if (!(old_refresh_intervals && new_refresh_intervals && + old_page_age_intervals)) { + ksft_print_msg("failed to allocate memory for intervals\n"); + return KSFT_FAIL; + } + + err = get_refresh_interval_all_nodes(old_refresh_intervals, nr_nodes); + if (err < 0) { + ksft_print_msg("failed to read refresh interval\n"); + return KSFT_FAIL; + } + + err = get_page_age_intervals_all_nodes(old_page_age_intervals, nr_nodes); + if (err < 0) { + ksft_print_msg("failed to read page age interval\n"); + return KSFT_FAIL; + } + + for (i = 0; i < nr_nodes; ++i) + new_refresh_intervals[i] = REFRESH_INTERVAL; + + for (i = 0; i < nr_nodes; ++i) { + err = sysfs_set_page_age_intervals(i, PAGE_AGE_INTERVALS, + NR_PAGE_AGE_INTERVALS - 1); + if (err < 0) { + ksft_print_msg("failed to set page age interval\n"); + test_result = KSFT_FAIL; + goto fail; + } + } + + err = set_refresh_interval_all_nodes(new_refresh_intervals, nr_nodes); + if (err < 0) { + ksft_print_msg("failed to set refresh interval\n"); + test_result = KSFT_FAIL; + goto fail; + } + + sync(); + drop_pagecache(); + + test_result = f(); + +fail: + err = set_refresh_interval_all_nodes(old_refresh_intervals, nr_nodes); + if (err < 0) { + ksft_print_msg("failed to restore refresh interval\n"); + test_result = KSFT_FAIL; + } + err = set_page_age_intervals_all_nodes(old_page_age_intervals, nr_nodes); + if (err < 0) { + ksft_print_msg("failed to restore page age interval\n"); + test_result = KSFT_FAIL; + } + return test_result; +} + +static char *file_test_path; +static int test_file(void) +{ + ssize_t ws_size_ref, ws_size_test; + int ret = KSFT_FAIL, i; + pid_t pid = 0; + + if (!file_test_path) { + ksft_print_msg("Set a path to test file workingset\n"); + return KSFT_SKIP; + } + + ws_size_ref = read_interval_all_nodes(PAGETYPE_FILE, 0); + if (ws_size_ref < 0) + goto cleanup; + + pid = clone_and_run(alloc_file_workingset, (void *)TEST_SIZE); + if (pid < 0) + goto cleanup; + + read_interval_all_nodes(PAGETYPE_FILE, 0); + sleep_ms(REFRESH_INTERVAL); + + for (i = 0; i < 3; ++i) { + sleep_ms(REFRESH_INTERVAL); + ws_size_test = read_interval_all_nodes(PAGETYPE_FILE, 0); + ws_size_test += read_interval_all_nodes(PAGETYPE_FILE, 1); + if (ws_size_test < 0) + goto cleanup; + + if (!values_close(ws_size_test - ws_size_ref, TEST_SIZE, 10)) { + ksft_print_msg( + "file working set size difference too large: actual=%ld, expected=%ld\n", + ws_size_test - ws_size_ref, TEST_SIZE); + goto cleanup; + } + } + ret = KSFT_PASS; + +cleanup: + if (pid > 0) + kill(pid, SIGKILL); + cleanup_file_workingset(); + return ret; +} + +static int test_anon(void) +{ + ssize_t ws_size_ref, ws_size_test; + pid_t pid = 0; + int ret = KSFT_FAIL, i; + + ws_size_ref = read_interval_all_nodes(PAGETYPE_ANON, 0); + ws_size_ref += read_interval_all_nodes(PAGETYPE_ANON, 1); + if (ws_size_ref < 0) + goto cleanup; + + pid = clone_and_run(alloc_anon_workingset, (void *)TEST_SIZE); + if (pid < 0) + goto cleanup; + + sleep_ms(REFRESH_INTERVAL); + read_interval_all_nodes(PAGETYPE_ANON, 0); + + for (i = 0; i < 5; ++i) { + sleep_ms(REFRESH_INTERVAL); + ws_size_test = read_interval_all_nodes(PAGETYPE_ANON, 0); + ws_size_test += read_interval_all_nodes(PAGETYPE_ANON, 1); + if (ws_size_test < 0) + goto cleanup; + + if (!values_close(ws_size_test - ws_size_ref, TEST_SIZE, 10)) { + ksft_print_msg( + "anon working set size difference too large: actual=%ld, expected=%ld\n", + ws_size_test - ws_size_ref, TEST_SIZE); + goto cleanup; + } + } + ret = KSFT_PASS; + +cleanup: + if (pid > 0) + kill(pid, SIGKILL); + return ret; +} + + +#define T(x) { x, #x } +struct workingset_test { + int (*fn)(void); + const char *name; +} tests[] = { + T(test_anon), + T(test_file), +}; +#undef T + +int main(int argc, char **argv) +{ + int i, err; + + if (argc > 1) + file_test_path = argv[1]; + + for (i = 0; i < ARRAY_SIZE(tests); i++) { + err = run_test(tests[i].fn); + ksft_test_result_code(err, tests[i].name, NULL); + } + return 0; +} From patchwork Tue Aug 13 16:56:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13762350 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A84F8C52D7C for ; Tue, 13 Aug 2024 16:59:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D06EF6B0082; Tue, 13 Aug 2024 12:59:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C8BAC6B0089; Tue, 13 Aug 2024 12:59:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB50C6B00A4; Tue, 13 Aug 2024 12:59:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 833BC6B0082 for ; Tue, 13 Aug 2024 12:59:53 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3C54C1209B8 for ; Tue, 13 Aug 2024 16:59:53 +0000 (UTC) X-FDA: 82447834266.25.5A23074 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf05.hostedemail.com (Postfix) with ESMTP id 3B33D10000D for ; Tue, 13 Aug 2024 16:59:51 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3W+3OL6J; spf=pass (imf05.hostedemail.com: domain of 3BZG7ZgcKCFIIEu7w1E08805y.w86527EH-664Fuw4.8B0@flex--yuanchu.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3BZG7ZgcKCFIIEu7w1E08805y.w86527EH-664Fuw4.8B0@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723568320; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hOP5SFFIvPyG4OwQTVSroxGri5wlFhobHFeL7lq0KZY=; b=KRXZTNHfwdjxzzp0NEavnc1AjyKse8NmxFx56j7QSJcBDdCLWD3zD/6XqbGTdYtbBiCWvd s7cGE44bPJBR0sMzsHkuYAQ6p3MsdmrV5hNzCe5ZB9cQVgRG5mTc+UiiTcXr2wJdywkGml ezN5IVJzoIonaDLmgLYLm0o7qRhIEbw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723568320; a=rsa-sha256; cv=none; b=4GXedSBSXZ3usyUik6h4Kih4ul2gwt0Dbhkjfz/qn8uyGeFH2P7uXBenNdW/nD5OMjsHbz c9rjxsH0WolO1kSjmKu/kZENOfSaUqorOxpz/Gu0e5ezXne0lbZfbuUUMeGdP9S5k0uD/5 geIMfqZPrNuwu5dhsTtPDPgJ8OqM6a4= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3W+3OL6J; spf=pass (imf05.hostedemail.com: domain of 3BZG7ZgcKCFIIEu7w1E08805y.w86527EH-664Fuw4.8B0@flex--yuanchu.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3BZG7ZgcKCFIIEu7w1E08805y.w86527EH-664Fuw4.8B0@flex--yuanchu.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e05e3938a37so9427382276.2 for ; Tue, 13 Aug 2024 09:59:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723568390; x=1724173190; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hOP5SFFIvPyG4OwQTVSroxGri5wlFhobHFeL7lq0KZY=; b=3W+3OL6JrqQ64Z+r6HcWV+xWwPgzO1WGzI1DzfhbUgrxBSkqOdNSWpknCAE6Idtbmm 7qOlhu5a0eNbAOsdbkJesSdm4Tb2ne08OAbuS9GFGSO3o9GYNhM2i77qvtSj01UgVC6F zs2wMUmvVEZ1Qx4c/cZC6tBQeD2HPJA6tdXa8RvtPYZ4Ncq9pA5NUzgjnm7vYkNLy/aA V0gHlF84cApPzTLFLUAJl1fRReGg2im0BByzOmgy4AromKKnzMKd27LbEvtb8XUuFf+i QJXMqtqgF5GdkhOvxy9FLc1bzE1cBRsAIDqksJUFEQWrK8yfdUmi7WAPMTZNW8b5PJm+ fVDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723568390; x=1724173190; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hOP5SFFIvPyG4OwQTVSroxGri5wlFhobHFeL7lq0KZY=; b=iFTWZpIjUiNsQBv+FG8GzyoVs/ygx2rVrfaIGcuQkwMC1cW6nGUBY7qtWpZpAeyH+U m76B3+e1C31BlBvTnNc022aveFsSAHIkk4GHkrLpGI7IZK+wMjic/oZpOfhvsCeAUwXp S5Lzb7cgerFxK1GV7XP/T/6joaidPzQ/TW54eq2eHLgdXtUZbGGLtjarX0QuR5rS6nvX NCMqrpol2aEY4XkDCkNMj93SQmXypZuLkUNLfl5OarA3oXH2cvr3ogfziwvclgCxjJWv IKfaycChBOgEyldG9bET02emJn0sDSPoCe/OG6MhsUhgW7BAz3rvtApjjcqfc/WhvvaB r7ig== X-Forwarded-Encrypted: i=1; AJvYcCUZaQSybTELdtDKo+H/DhFwb5Rk2RAUotINUkzbd+34tV8/Ux+kRhwm7apSBjcuvPWh+nrEtS2M4JyXmWKYFf6SGmU= X-Gm-Message-State: AOJu0YzaC2j/zdBSZ7OO21oPXw36dGN5oqmntPudrRBdgaUSF/wBqSjI Pft8cHpBQNmMLGswjDve6UzYoQzihEYeq5jRglCAOsB5KG2/YnyEI8zRSVQKz4Qa3pqF3BfNXb5 pFoH9KA== X-Google-Smtp-Source: AGHT+IECAXtBsp42GLKJ5N1BoyX/FjeEZsFBUtplL8hab3FEghl0VDuEX9j5TFSLO+qQ7fOJRfdfV8wWTHGk X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:b50c:66e8:6532:a371]) (user=yuanchu job=sendgmr) by 2002:a25:7450:0:b0:e05:fc91:8935 with SMTP id 3f1490d57ef6-e1155a4ac06mr3055276.3.1723568389892; Tue, 13 Aug 2024 09:59:49 -0700 (PDT) Date: Tue, 13 Aug 2024 09:56:18 -0700 In-Reply-To: <20240813165619.748102-1-yuanchu@google.com> Mime-Version: 1.0 References: <20240813165619.748102-1-yuanchu@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813165619.748102-8-yuanchu@google.com> Subject: [PATCH v3 7/7] Docs/admin-guide/mm/workingset_report: document sysfs and memcg interfaces From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Andrew Morton , Lance Yang , Randy Dunlap , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3B33D10000D X-Stat-Signature: 3wc7buw1cqp4hor4suzshgurcetie5dp X-HE-Tag: 1723568391-146996 X-HE-Meta: U2FsdGVkX1/2SRW5vPS9LmrVo1BL/nRi7Eft5Yy6evbJPXmGYa341WZbb+veTDQCqydPdQizQg0BxoWriZuh3S7FmE6dT6ANdAygGs0pwE7kKS9LluQRva9KQP/6htYoS8zSB/fHH4YxDVtZ/6xa1hfFXVQJIudu+JvU1ew1QkQeCkb1TbfPbwJlhiR33QV8vS99wyjAXryshh+X2PUgSTO70Ab/zp2CqF52KMaS9WJp468EGWyOaOnfV0OBG04Cb5ihiRSzStMiWFaF/M2dYqvw7R1oMDUKk6E97ciiPZoWXR8ztUAOBnATNDC2VmW+Q9v+oSq+rJ8YBSX0gHegnM2dF49T8uTXgtWjlS6kpjpKPPR/7no40Blt3bZmU+hHFo6KJGGPNADYGfM7QxaIM8Bh+q/FIbtDpY2+Kxw2aDDwgFF39+uYgUNPcsoS0OpOFExoStNkaew3qdI8Pvn1kuNfU0vxc6BgZbQWwyzL3E3q+jV0GUj+fW7lNlQAzPYfzfvNTYaAxtfGKEhIzFWcPSE6tvxUGP1sPAewloI3wnnoRLwFrxFEwBDdZnliaMln3a8O+nKIg+6mSJY5pDjcOw4mEjdK03vYJsZJXwp+Iyz8oWTuyqL3vfxqTh6kWr5pIK7R4XanJyewKKezYRnwIKjRND0lQO51ArnNKa9PhKQa/NqzSS3aX4WzDPsIpgXDCt+RZkmzFOGcEWG4sPlmSwp43JCxSToRPwTM4Jth2OafEDKtY5MmTzGCA/6TiDxVY7pW00k2vvRuQx4Gq59Bo4bvlnGDaW4FurydKMcrcoz2/cfAt40Zqpta/JOnAGO/cDd1hWWel4Kr/VRp/xkzuPp0E2+0BorNkFdo7XTMTgd9yZtuO6l1O3WUbhzUX/PKeVaNGrgAbX0+zNjHvBnshhv+I1txyfYxb3AfXX9xsXtjUc1m4FP2v6QhuE4npBetg5mSADEweMx+jDI4323 kM8TEsQ1 Nl/7Ur4Hz2q9592hMyBA4zjLWyd0zsWzVJJRBEPYudTXDbtROFRyzdn9WEdgWb5ofXuTzRUDyjFfGoGj7K+na8nxhN4WcQrCzul2HuGAsTELpIAc+Js/JTFSPtwp7wMk61nsLCR1OGGCXc4afsaEijHVgeViojXqM8s7ajcvUUf5FkUOIpj+Ds8Xt1o90TvHm6b6k9oGdBtJ/vcSFBHgwvh4MU8W8lzLLzJxTh9VXRtc4cJ3GZlDGhlytk+yrpeN984qivcwCkdylfR8JrUNN+vhnWSOxbWp8SuY3bvS7+WBIIfO+C69g7QARsqEGJvN4n8IQn6XvgkLRZ4A/tr8oCZatFsC4G1tXyKU52BK0lBa85kTRBQinMNQOITrNSm03U2bV8pBvQC46SWIuQ+njmLN1yHYPrehuDrdbd5GiEyQUjj9Q6SJDrlUjrVaY91Lre8/aZWqhk0d5FU6Q89oKVMy1W3XG9IbSCVR0/PPz4371rUWVj9oiJrpCfZyfCXpikk7fE2mB2IH9mRMr9D2T8Kens66wN6dVjxTsS/Of50UVX7WXChrcVE9DQqepNcZUewZikofJzDBbLW2CfPkYutaBTVY38tShLfyLI7yAgqJk8E173YkuML9b16GebrMORm5K X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add workingset reporting documentation for better discoverability of its sysfs and memcg interfaces. Also document the required kernel config to enable workingset reporting. Change-Id: Ib9dfc9004473baa6ef26ca7277d220b6199517de Signed-off-by: Yuanchu Xie --- Documentation/admin-guide/mm/index.rst | 1 + .../admin-guide/mm/workingset_report.rst | 105 ++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 Documentation/admin-guide/mm/workingset_report.rst diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-guide/mm/index.rst index 8b35795b664b..61a2a347fc91 100644 --- a/Documentation/admin-guide/mm/index.rst +++ b/Documentation/admin-guide/mm/index.rst @@ -41,4 +41,5 @@ the Linux memory management. swap_numa transhuge userfaultfd + workingset_report zswap diff --git a/Documentation/admin-guide/mm/workingset_report.rst b/Documentation/admin-guide/mm/workingset_report.rst new file mode 100644 index 000000000000..ddcc0c33a8df --- /dev/null +++ b/Documentation/admin-guide/mm/workingset_report.rst @@ -0,0 +1,105 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +Workingset Report +================= +Workingset report provides a view of memory coldness in user-defined +time intervals, i.e. X bytes are Y milliseconds cold. It breaks down +the user pages in the system per-NUMA node, per-memcg, for both +anonymous and file pages into histograms that look like: +:: + + 1000 anon=137368 file=24530 + 20000 anon=34342 file=0 + 30000 anon=353232 file=333608 + 40000 anon=407198 file=206052 + 9223372036854775807 anon=4925624 file=892892 + +The workingset reports can be used to drive proactive reclaim, by +identifying the number of cold bytes in a memcg, then writing to +``memory.reclaim``. + +Quick start +=========== +Build the kernel with the following configurations. The report relies +on Multi-gen LRU for page coldness. + +* ``CONFIG_LRU_GEN=y`` +* ``CONFIG_LRU_GEN_ENABLED=y`` +* ``CONFIG_WORKINGSET_REPORT=y`` + +Optionally, the aging kernel daemon can be enabled with the following +configuration. +* ``CONFIG_WORKINGSET_REPORT_AGING=y`` + +Sysfs interfaces +================ +``/sys/devices/system/node/nodeX/workingset_report/page_age`` provides +a per-node page age histogram, showing an aggregate of the node's lruvecs. +Reading this file causes a hierarchical aging of all lruvecs, scanning +pages and creates a new Multi-gen LRU generation in each lruvec. +For example: +:: + + 1000 anon=0 file=0 + 2000 anon=0 file=0 + 100000 anon=5533696 file=5566464 + 18446744073709551615 anon=0 file=0 + +``/sys/devices/system/node/nodeX/workingset_report/page_age_intervals`` +is a comma separated list of time in milliseconds that configures what +the page age histogram uses for aggregation. For the above histogram, +the intervals are: +:: + 1000,2000,100000 + +``/sys/devices/system/node/nodeX/workingset_report/refresh_interval`` +defines the amount of time the report is valid for in milliseconds. +When a report is still valid, reading the ``page_age`` file shows +the existing valid report, instead of generating a new one. + +``/sys/devices/system/node/nodeX/workingset_report/report_threshold`` +specifies how often the userspace agent can be notified for node +memory pressure, in milliseconds. When a node reaches its low +watermarks and wakes up kswapd, programs waiting on ``page_age`` are +woken up so they can read the histogram and make policy decisions. + +Memcg interface +=============== +While ``page_age_interval`` is defined per-node in sysfs, ``page_age``, +``refresh_interval`` and ``report_threshold`` are available per-memcg. + +``/sys/fs/cgroup/.../memory.workingset.page_age`` +The memcg equivalent of the sysfs workingset page age histogram +breaks down the workingset of this memcg and its children into +page age intervals. Each node is prefixed with a node header and +a newline. Non-proactive direct reclaim on this memcg can also +wake up userspace agents that are waiting on this file. +e.g. +:: + + N0 + 1000 anon=0 file=0 + 2000 anon=0 file=0 + 3000 anon=0 file=0 + 4000 anon=0 file=0 + 5000 anon=0 file=0 + 18446744073709551615 anon=0 file=0 + +``/sys/fs/cgroup/.../memory.workingset.refresh_interval`` +The memcg equivalent of the sysfs refresh interval. A per-node +number of how much time a page age histogram is valid for, in +milliseconds. +e.g. +:: + + echo N0=2000 > memory.workingset.refresh_interval + +``/sys/fs/cgroup/.../memory.workingset.report_threshold`` +The memcg equivalent of the sysfs report threshold. A per-node +number of how often userspace agent waiting on the page age +histogram can be woken up, in milliseconds. +e.g. +:: + + echo N0=1000 > memory.workingset.report_threshold