From patchwork Mon Jul 29 14:37:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Finkel X-Patchwork-Id: 13745824 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF514C3DA61 for ; Mon, 29 Jul 2024 21:23:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1A00B6B0089; Mon, 29 Jul 2024 17:23:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 05DF76B008A; Mon, 29 Jul 2024 17:23:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2E9A6B0093; Mon, 29 Jul 2024 17:23:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8E9456B0089 for ; Mon, 29 Jul 2024 17:23:05 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 4A814A0736 for ; Mon, 29 Jul 2024 21:23:05 +0000 (UTC) X-FDA: 82394065530.27.163F5B4 Received: from m35-116.mailgun.net (m35-116.mailgun.net [69.72.35.116]) by imf03.hostedemail.com (Postfix) with ESMTP id 766C720019 for ; Mon, 29 Jul 2024 21:23:03 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=relay.vimeo.com header.s=mailo header.b=hdlomxcP; spf=pass (imf03.hostedemail.com: domain of "bounce+ea57f2.9d2a1c-linux-mm=kvack.org@relay.vimeo.com" designates 69.72.35.116 as permitted sender) smtp.mailfrom="bounce+ea57f2.9d2a1c-linux-mm=kvack.org@relay.vimeo.com"; dmarc=pass (policy=reject) header.from=vimeo.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722288179; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Z9Ty21bXLKt+aLev+0Zjobm/lX8yDqJvSrMYOndR8XE=; b=rBamAcZ9eYzotpjA2HQn6jozcrOG5a0oRsGK4GvuGhSPreyxgGbpifugkMUlsD+LSCc0Vf MxGwG2j9QQabo3NdqeWLiEJWGPL5KKZ+h4nLO4Bkk9joPdYVRPelBoG94Wtj2trUiwsKzs +6cUKhkna3bDzgU1dQDXMhuxtchfxEo= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=relay.vimeo.com header.s=mailo header.b=hdlomxcP; spf=pass (imf03.hostedemail.com: domain of "bounce+ea57f2.9d2a1c-linux-mm=kvack.org@relay.vimeo.com" designates 69.72.35.116 as permitted sender) smtp.mailfrom="bounce+ea57f2.9d2a1c-linux-mm=kvack.org@relay.vimeo.com"; dmarc=pass (policy=reject) header.from=vimeo.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722288179; a=rsa-sha256; cv=none; b=TUWTJ1y57glPeZ3ZNeMCqR+7Zh1Mqmo6EljUNqSjeY+cnirTBIoLJ/miD3Xrt0xxg6xIip Nd5UvoSn7jjzisrX7lPSQp6oZp8hzU4hPklZtnasJCBXVJ/Wa4bXukQX48tl8pDm9BH2Cf aW0js11ixzbesmDnDKHS16nV3M9Fds0= DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=relay.vimeo.com; q=dns/txt; s=mailo; t=1722288182; x=1722295382; h=Content-Transfer-Encoding: MIME-Version: References: In-Reply-To: Message-Id: Date: Subject: Subject: Cc: To: To: From: From: Sender: Sender; bh=Z9Ty21bXLKt+aLev+0Zjobm/lX8yDqJvSrMYOndR8XE=; b=hdlomxcPl27l3s8B2jumPGzVccwf968oOWkCn2YDHDVcwr1hnhfCBc2xRr4ycTDJIMt/CoGn2clCOAgk0QqgwZx8AMNiFXgvkuqUGZlnAqrn/bcZM3OJJIHsaMZn0VKsgeQRU0A7Pwpv5adU/ijihlA5qi/VPI4uT0C7jZaw/oI= X-Mailgun-Sending-Ip: 69.72.35.116 X-Mailgun-Sid: WyI5NTRmYiIsImxpbnV4LW1tQGt2YWNrLm9yZyIsIjlkMmExYyJd Received: from smtp.vimeo.com (215.71.185.35.bc.googleusercontent.com [35.185.71.215]) by 750663bb2215 with SMTP id 66a808362c8e1673cbb4d90e (version=TLS1.2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Mon, 29 Jul 2024 21:23:02 GMT Received: from nutau (gke-sre-us-east1-main-7f6ba6de-7c30.c.vimeo-core.internal [10.56.27.212]) by smtp.vimeo.com (Postfix) with ESMTP id 263D765CBC; Mon, 29 Jul 2024 21:23:02 +0000 (UTC) Received: by nutau (Postfix, from userid 1001) id 6D459B40A7C; Mon, 29 Jul 2024 10:38:01 -0400 (EDT) From: David Finkel To: Muchun Song , Tejun Heo , Roman Gushchin , Andrew Morton Cc: core-services@vimeo.com, Jonathan Corbet , Michal Hocko , Shakeel Butt , Shuah Khan , Johannes Weiner , Zefan Li , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, =?utf-8?q?Michal_Koutn=C3=BD?= , David Finkel , Waiman Long Subject: [PATCH v6 1/2] mm, memcg: cg2 memory{.swap,}.peak write handlers Date: Mon, 29 Jul 2024 10:37:42 -0400 Message-Id: <20240729143743.34236-2-davidf@vimeo.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240729143743.34236-1-davidf@vimeo.com> References: <20240729143743.34236-1-davidf@vimeo.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: c5m7ne3dywa9md55kzocadqcfhb5taqw X-Rspamd-Queue-Id: 766C720019 X-Rspamd-Server: rspam11 X-HE-Tag: 1722288183-897532 X-HE-Meta: U2FsdGVkX187bHYA45pJ6Qm3s5M749AuWN/eLZt3HYGipqPTlzYZj39ee2T8NIbu30pqLxnc019CFOjp1Ps9Ssn+4+YPAk8JdHfapPcd/zlUCDR+YgboV/qrPgu6KzwWAQsFrcHdoo0YNGG4PxgGlEdlHfP6vSbPmlCe38Z4MWqTvGwn9mtlEcCAOQZP+zu8kVnMk5NOVYvbh6J1aqtFKNQrwX79lpLbHtdJ1HGzHWVA98R8Ig1T6BwOnKiLxbLzZgt2IXZTCBCpb4RiCBOq07zNOYxh3U9lbvzV7LFbpk6G0dSE5pHRaovZJHoChwQQcGWUlEwb38Em9QpHir+3DBf3zLL0R5CyEZo41e8CiuSiucQ5g1qOLOF4Su6/VN9GD8AvjEMTLXZ2lVY/tENzYQNWfUIH36rKiMBcg++fFeuLoSvEeODHsDRfBaZ0WA9gc55OK4MJ8Hh2yGMbYJJsBEBKl8zKjxn2AoQECEimmLHyDvB5JfX2JPJV08hIurLz9QIr2GqDxQ7d4jnLcYfvwpgWYD+X49m68VFYS91Tn83m5SW0cuKYURhNYK1Gcm42BypPI7dnvU3nf3QX0jg2XVECGE09WMKO1n89vIRhzmAO+PNV+YyTCy1tg+yVI7RbYfRM/1M9EDiE116IIAVfscadheF5R/fJeSilGs0efsZooq1YKhYKMMS0V6qonVanIgvZTapfouJLjzxPonYUov8vVHr66Cr3EZQqx5ou90xiEqqsL36gj4I/hvDG1z8OpqnRm+he56pUzT5HiZb72klYGfluXDfHnM6ade+z0Xt4z2tcrsDyjE/9bktzpZgvEoosvgPay5waNJ5HMWR4CESnmM2vSc0a30MZrEU6VuzL39xyjxpr64JP7kCIIOpLcLuQJWFHbNrZGEsScuFcvPvIlQopTHT2EgJo2uLB06absAnB9INsD2V59udaRkY7JOoDkMUURodrclWdjrQ CwAeEn0h d7ScBLI6ZEFiiyeRsicWr7g9Z8Iw7+Dhf9K04lGcDMfLNfnUWC99cp2ZocNpDRl79govlf6GaT0tGtOkNunbLz6FFUmEzKqSYy+G4VbmAu+5jW3DCwh/lah5qee9DlNp9H6VlU5hK6K+ctBjSqhYpz41MRRUO0cVgQ9q7m+vX+DzIRM2DYGp0JvruPyRR0uxIhg4nlbeftZAVnJtUiC2E7DGS2893Cilwv77fRMwXpAbHQUDWSLM9abOv2zrWvVW9jDkBdEgaLnY/6AIUP24hUppVc7q0kAGFx6kJA1iWjG2YDqqRC17CoTyoOmmMnNffZ52h3MIGdvqhPwjoy/JrXxUyqYl6qAxqUB1q4pkuHaCVgOWrF6T9j4q/K/FNbeRVbLwwHnGgYVooRaFdOvJ05snq5GdHi88Gxa3VwIyTXLrnqh/YJEXSCZePIkTKCWFI685Qhm5yqD24Go3HqN4bGPJjESsTJMLcig2b4FsvIgbRDgRyXUs/IWHdHxWu1fpmQ4FwpasCXNDu0hrEbu0L+dYrygh2jTebiMwHoEdWGvN3rnhinqSQ9E1nqgZYTLuHWWf0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Other mechanisms for querying the peak memory usage of either a process or v1 memory cgroup allow for resetting the high watermark. Restore parity with those mechanisms, but with a less racy API. For example: - Any write to memory.max_usage_in_bytes in a cgroup v1 mount resets the high watermark. - writing "5" to the clear_refs pseudo-file in a processes's proc directory resets the peak RSS. This change is an evolution of a previous patch, which mostly copied the cgroup v1 behavior, however, there were concerns about races/ownership issues with a global reset, so instead this change makes the reset filedescriptor-local. Writing any non-empty string to the memory.peak and memory.swap.peak pseudo-files reset the high watermark to the current usage for subsequent reads through that same FD. Notably, following Johannes's suggestion, this implementation moves the O(FDs that have written) behavior onto the FD write(2) path. Instead, on the page-allocation path, we simply add one additional watermark to conditionally bump per-hierarchy level in the page-counter. Additionally, this takes Longman's suggestion of nesting the page-charging-path checks for the two watermarks to reduce the number of common-case comparisons. This behavior is particularly useful for work scheduling systems that need to track memory usage of worker processes/cgroups per-work-item. Since memory can't be squeezed like CPU can (the OOM-killer has opinions), these systems need to track the peak memory usage to compute system/container fullness when binpacking workitems. Most notably, Vimeo's use-case involves a system that's doing global binpacking across many Kubernetes pods/containers, and while we can use PSI for some local decisions about overload, we strive to avoid packing workloads too tightly in the first place. To facilitate this, we track the peak memory usage. However, since we run with long-lived workers (to amortize startup costs) we need a way to track the high watermark while a work-item is executing. Polling runs the risk of missing short spikes that last for timescales below the polling interval, and peak memory tracking at the cgroup level is otherwise perfect for this use-case. As this data is used to ensure that binpacked work ends up with sufficient headroom, this use-case mostly avoids the inaccuracies surrounding reclaimable memory. Suggested-by: Johannes Weiner Suggested-by: Waiman Long Signed-off-by: David Finkel Acked-by: Johannes Weiner Reviewed-by: Michal Koutný --- Documentation/admin-guide/cgroup-v2.rst | 22 +++-- include/linux/cgroup-defs.h | 5 + include/linux/cgroup.h | 3 + include/linux/memcontrol.h | 5 + include/linux/page_counter.h | 11 ++- kernel/cgroup/cgroup-internal.h | 2 + kernel/cgroup/cgroup.c | 7 ++ mm/memcontrol.c | 116 ++++++++++++++++++++++-- mm/page_counter.c | 30 ++++-- 9 files changed, 174 insertions(+), 27 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 86311c2907cd3..f0499884124d2 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -1333,11 +1333,14 @@ The following nested keys are defined. all the existing limitations and potential future extensions. memory.peak - A read-only single value file which exists on non-root - cgroups. + A read-write single value file which exists on non-root cgroups. + + The max memory usage recorded for the cgroup and its descendants since + either the creation of the cgroup or the most recent reset for that FD. - The max memory usage recorded for the cgroup and its - descendants since the creation of the cgroup. + A write of any non-empty string to this file resets it to the + current memory usage for subsequent reads through the same + file descriptor. memory.oom.group A read-write single value file which exists on non-root @@ -1663,11 +1666,14 @@ The following nested keys are defined. Healthy workloads are not expected to reach this limit. memory.swap.peak - A read-only single value file which exists on non-root - cgroups. + A read-write single value file which exists on non-root cgroups. + + The max swap usage recorded for the cgroup and its descendants since + the creation of the cgroup or the most recent reset for that FD. - The max swap usage recorded for the cgroup and its - descendants since the creation of the cgroup. + A write of any non-empty string to this file resets it to the + current memory usage for subsequent reads through the same + file descriptor. memory.swap.max A read-write single value file which exists on non-root diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index ae04035b6cbe5..7fc2d0195f560 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -775,6 +775,11 @@ struct cgroup_subsys { extern struct percpu_rw_semaphore cgroup_threadgroup_rwsem; +struct cgroup_of_peak { + unsigned long value; + struct list_head list; +}; + /** * cgroup_threadgroup_change_begin - threadgroup exclusion for cgroups * @tsk: target task diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index c60ba0ab14627..3e0563753cc3e 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -11,6 +11,7 @@ #include #include +#include #include #include #include @@ -854,4 +855,6 @@ static inline void cgroup_bpf_put(struct cgroup *cgrp) {} struct cgroup *task_get_cgroup1(struct task_struct *tsk, int hierarchy_id); +struct cgroup_of_peak *of_peak(struct kernfs_open_file *of); + #endif /* _LINUX_CGROUP_H */ diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 0e5bf25d324f0..cc74d73d3b065 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -193,6 +193,11 @@ struct mem_cgroup { struct page_counter memsw; /* v1 only */ }; + /* registered local peak watchers */ + struct list_head memory_peaks; + struct list_head swap_peaks; + spinlock_t peaks_lock; + /* Range enforcement for interrupt charges */ struct work_struct high_work; diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h index 904c52f97284f..898f562c0b838 100644 --- a/include/linux/page_counter.h +++ b/include/linux/page_counter.h @@ -26,6 +26,8 @@ struct page_counter { atomic_long_t children_low_usage; unsigned long watermark; + /* Latest cg2 reset watermark */ + unsigned long local_watermark; unsigned long failcnt; /* Keep all the read most fields in a separete cacheline. */ @@ -78,7 +80,14 @@ int page_counter_memparse(const char *buf, const char *max, static inline void page_counter_reset_watermark(struct page_counter *counter) { - counter->watermark = page_counter_read(counter); + unsigned long usage = page_counter_read(counter); + + /* + * Update local_watermark first, so it's always <= watermark + * (modulo CPU/compiler re-ordering) + */ + counter->local_watermark = usage; + counter->watermark = usage; } void page_counter_calculate_protection(struct page_counter *root, diff --git a/kernel/cgroup/cgroup-internal.h b/kernel/cgroup/cgroup-internal.h index 520b90dd97eca..c964dd7ff967a 100644 --- a/kernel/cgroup/cgroup-internal.h +++ b/kernel/cgroup/cgroup-internal.h @@ -81,6 +81,8 @@ struct cgroup_file_ctx { struct { struct cgroup_pidlist *pidlist; } procs1; + + struct cgroup_of_peak peak; }; /* diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index c8e4b62b436a4..0a97cb2ef1245 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -1972,6 +1972,13 @@ static int cgroup2_parse_param(struct fs_context *fc, struct fs_parameter *param return -EINVAL; } +struct cgroup_of_peak *of_peak(struct kernfs_open_file *of) +{ + struct cgroup_file_ctx *ctx = of->priv; + + return &ctx->peak; +} + static void apply_cgroup_root_flags(unsigned int root_flags) { if (current->nsproxy->cgroup_ns == &init_cgroup_ns) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9603717886877..2663e2108cdbe 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -25,6 +25,7 @@ * Copyright (C) 2020 Alibaba, Inc, Alex Shi */ +#include #include #include #include @@ -41,6 +42,7 @@ #include #include #include +#include #include #include #include @@ -3558,6 +3560,9 @@ static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent) INIT_WORK(&memcg->high_work, high_work_func); vmpressure_init(&memcg->vmpressure); + INIT_LIST_HEAD(&memcg->memory_peaks); + INIT_LIST_HEAD(&memcg->swap_peaks); + spin_lock_init(&memcg->peaks_lock); memcg->socket_pressure = jiffies; memcg1_memcg_init(memcg); memcg->kmemcg_id = -1; @@ -3950,14 +3955,91 @@ static u64 memory_current_read(struct cgroup_subsys_state *css, return (u64)page_counter_read(&memcg->memory) * PAGE_SIZE; } -static u64 memory_peak_read(struct cgroup_subsys_state *css, - struct cftype *cft) +#define OFP_PEAK_UNSET (((-1UL))) + +static int peak_show(struct seq_file *sf, void *v, struct page_counter *pc) { - struct mem_cgroup *memcg = mem_cgroup_from_css(css); + struct cgroup_of_peak *ofp = of_peak(sf->private); + u64 fd_peak = READ_ONCE(ofp->value), peak; + + /* User wants global or local peak? */ + if (fd_peak == OFP_PEAK_UNSET) + peak = pc->watermark; + else + peak = max(fd_peak, READ_ONCE(pc->local_watermark)); + + seq_printf(sf, "%llu\n", peak * PAGE_SIZE); + return 0; +} + +static int memory_peak_show(struct seq_file *sf, void *v) +{ + struct mem_cgroup *memcg = mem_cgroup_from_css(seq_css(sf)); + + return peak_show(sf, v, &memcg->memory); +} + +static int peak_open(struct kernfs_open_file *of) +{ + struct cgroup_of_peak *ofp = of_peak(of); + + ofp->value = OFP_PEAK_UNSET; + return 0; +} + +static void peak_release(struct kernfs_open_file *of) +{ + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); + struct cgroup_of_peak *ofp = of_peak(of); + + if (ofp->value == OFP_PEAK_UNSET) { + /* fast path (no writes on this fd) */ + return; + } + spin_lock(&memcg->peaks_lock); + list_del(&ofp->list); + spin_unlock(&memcg->peaks_lock); +} + +static ssize_t peak_write(struct kernfs_open_file *of, char *buf, size_t nbytes, + loff_t off, struct page_counter *pc, + struct list_head *watchers) +{ + unsigned long usage; + struct cgroup_of_peak *peer_ctx; + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); + struct cgroup_of_peak *ofp = of_peak(of); + + spin_lock(&memcg->peaks_lock); + + usage = page_counter_read(pc); + WRITE_ONCE(pc->local_watermark, usage); + + list_for_each_entry(peer_ctx, watchers, list) + if (usage > peer_ctx->value) + WRITE_ONCE(peer_ctx->value, usage); + + /* initial write, register watcher */ + if (ofp->value == -1) + list_add(&ofp->list, watchers); + + WRITE_ONCE(ofp->value, usage); + spin_unlock(&memcg->peaks_lock); + + return nbytes; +} + +static ssize_t memory_peak_write(struct kernfs_open_file *of, char *buf, + size_t nbytes, loff_t off) +{ + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); - return (u64)memcg->memory.watermark * PAGE_SIZE; + return peak_write(of, buf, nbytes, off, &memcg->memory, + &memcg->memory_peaks); } +#undef OFP_PEAK_UNSET + static int memory_min_show(struct seq_file *m, void *v) { return seq_puts_memcg_tunable(m, @@ -4307,7 +4389,10 @@ static struct cftype memory_files[] = { { .name = "peak", .flags = CFTYPE_NOT_ON_ROOT, - .read_u64 = memory_peak_read, + .open = peak_open, + .release = peak_release, + .seq_show = memory_peak_show, + .write = memory_peak_write, }, { .name = "min", @@ -5099,12 +5184,20 @@ static u64 swap_current_read(struct cgroup_subsys_state *css, return (u64)page_counter_read(&memcg->swap) * PAGE_SIZE; } -static u64 swap_peak_read(struct cgroup_subsys_state *css, - struct cftype *cft) +static int swap_peak_show(struct seq_file *sf, void *v) { - struct mem_cgroup *memcg = mem_cgroup_from_css(css); + struct mem_cgroup *memcg = mem_cgroup_from_css(seq_css(sf)); + + return peak_show(sf, v, &memcg->swap); +} + +static ssize_t swap_peak_write(struct kernfs_open_file *of, char *buf, + size_t nbytes, loff_t off) +{ + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); - return (u64)memcg->swap.watermark * PAGE_SIZE; + return peak_write(of, buf, nbytes, off, &memcg->swap, + &memcg->swap_peaks); } static int swap_high_show(struct seq_file *m, void *v) @@ -5188,7 +5281,10 @@ static struct cftype swap_files[] = { { .name = "swap.peak", .flags = CFTYPE_NOT_ON_ROOT, - .read_u64 = swap_peak_read, + .open = peak_open, + .release = peak_release, + .seq_show = swap_peak_show, + .write = swap_peak_write, }, { .name = "swap.events", diff --git a/mm/page_counter.c b/mm/page_counter.c index 0153f5bb31611..ad9bdde5d5d20 100644 --- a/mm/page_counter.c +++ b/mm/page_counter.c @@ -79,9 +79,22 @@ void page_counter_charge(struct page_counter *counter, unsigned long nr_pages) /* * This is indeed racy, but we can live with some * inaccuracy in the watermark. + * + * Notably, we have two watermarks to allow for both a globally + * visible peak and one that can be reset at a smaller scope. + * + * Since we reset both watermarks when the global reset occurs, + * we can guarantee that watermark >= local_watermark, so we + * don't need to do both comparisons every time. + * + * On systems with branch predictors, the inner condition should + * be almost free. */ - if (new > READ_ONCE(c->watermark)) - WRITE_ONCE(c->watermark, new); + if (new > READ_ONCE(c->local_watermark)) { + WRITE_ONCE(c->local_watermark, new); + if (new > READ_ONCE(c->watermark)) + WRITE_ONCE(c->watermark, new); + } } } @@ -129,12 +142,13 @@ bool page_counter_try_charge(struct page_counter *counter, goto failed; } propagate_protected_usage(c, new); - /* - * Just like with failcnt, we can live with some - * inaccuracy in the watermark. - */ - if (new > READ_ONCE(c->watermark)) - WRITE_ONCE(c->watermark, new); + + /* see comment on page_counter_charge */ + if (new > READ_ONCE(c->local_watermark)) { + WRITE_ONCE(c->local_watermark, new); + if (new > READ_ONCE(c->watermark)) + WRITE_ONCE(c->watermark, new); + } } return true; From patchwork Mon Jul 29 14:37:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Finkel X-Patchwork-Id: 13745823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2B82C3DA70 for ; Mon, 29 Jul 2024 21:23:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E24EB6B0088; Mon, 29 Jul 2024 17:23:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DA41D6B008C; Mon, 29 Jul 2024 17:23:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABDC46B008A; Mon, 29 Jul 2024 17:23:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7C3606B0085 for ; Mon, 29 Jul 2024 17:23:05 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0B5DAA4C19 for ; Mon, 29 Jul 2024 21:23:05 +0000 (UTC) X-FDA: 82394065530.13.5C825A1 Received: from m35-116.mailgun.net (m35-116.mailgun.net [69.72.35.116]) by imf21.hostedemail.com (Postfix) with ESMTP id 3B2C61C0015 for ; Mon, 29 Jul 2024 21:23:03 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=relay.vimeo.com header.s=mailo header.b="M/XK7BsX"; dmarc=pass (policy=reject) header.from=vimeo.com; spf=pass (imf21.hostedemail.com: domain of "bounce+ea57f2.9d2a1c-linux-mm=kvack.org@relay.vimeo.com" designates 69.72.35.116 as permitted sender) smtp.mailfrom="bounce+ea57f2.9d2a1c-linux-mm=kvack.org@relay.vimeo.com" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722288179; a=rsa-sha256; cv=none; b=l2ycuN+0qfgxhzQQXh/mPeUUWZT0L7evpFu0+2WQPpwRNUISwD4mtYIZIzBa+RWZrbuOb6 k4Q4haShWgbriF4ISJJ4iTN8NbFrYUOIXrIrWYFDc4w+GVlBaucqHB722C+f0XTVvXSnWP H2u61EWm/gRCzIBTKZ2bBYsxrIEju7U= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=relay.vimeo.com header.s=mailo header.b="M/XK7BsX"; dmarc=pass (policy=reject) header.from=vimeo.com; spf=pass (imf21.hostedemail.com: domain of "bounce+ea57f2.9d2a1c-linux-mm=kvack.org@relay.vimeo.com" designates 69.72.35.116 as permitted sender) smtp.mailfrom="bounce+ea57f2.9d2a1c-linux-mm=kvack.org@relay.vimeo.com" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722288179; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=H3UMxB4M7BrujHBPr+UgcfPfxgI2rlDhcD8iL3igSy4=; b=NVTBXs5vMta+QQXLq3F4T+A/AihVVD/RnlxZ7BGIEhONTo0Kj6p9n/d+ol7k/PtMk66eJR 0zHqW0ZZGxmU5VeDZkIdqChcGCxG/Wwb2USiI098238zXnodRVBtLJM9iJinWLOut0lOxv tPJxkQ4wXB1QQknV3W04crIK5hrQ7KM= DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=relay.vimeo.com; q=dns/txt; s=mailo; t=1722288182; x=1722295382; h=Content-Transfer-Encoding: MIME-Version: References: In-Reply-To: Message-Id: Date: Subject: Subject: Cc: To: To: From: From: Sender: Sender; bh=H3UMxB4M7BrujHBPr+UgcfPfxgI2rlDhcD8iL3igSy4=; b=M/XK7BsXSHOLe84nAHAP6i7AQUWOWpeNieQQpAzlDNlm1zDrk3+8SRY3v7RESQbawpx0wg+D+ZpzBSF6McFuOLRdlvlCbgS8Vra0Bz1A2wz3FqttSqkjGR9z0ovXZiwZmVgRFMVf2QDwPm2kiiYuDzZ7yW2zAEb6ciDNxA2xLJ4= X-Mailgun-Sending-Ip: 69.72.35.116 X-Mailgun-Sid: WyI5NTRmYiIsImxpbnV4LW1tQGt2YWNrLm9yZyIsIjlkMmExYyJd Received: from smtp.vimeo.com (215.71.185.35.bc.googleusercontent.com [35.185.71.215]) by 5a80eff7c455 with SMTP id 66a808364527901af3858a25 (version=TLS1.2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Mon, 29 Jul 2024 21:23:02 GMT Received: from nutau (gke-sre-us-east1-main-7f6ba6de-loec.c.vimeo-core.internal [10.56.27.213]) by smtp.vimeo.com (Postfix) with ESMTP id 17B7C65CBB; Mon, 29 Jul 2024 21:23:02 +0000 (UTC) Received: by nutau (Postfix, from userid 1001) id 0D850B40AC9; Mon, 29 Jul 2024 10:38:11 -0400 (EDT) From: David Finkel To: Muchun Song , Tejun Heo , Roman Gushchin , Andrew Morton Cc: core-services@vimeo.com, Jonathan Corbet , Michal Hocko , Shakeel Butt , Shuah Khan , Johannes Weiner , Zefan Li , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, =?utf-8?q?Michal_Koutn=C3=BD?= , David Finkel Subject: [PATCH v6 2/2] mm, memcg: cg2 memory{.swap,}.peak write tests Date: Mon, 29 Jul 2024 10:37:43 -0400 Message-Id: <20240729143743.34236-3-davidf@vimeo.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240729143743.34236-1-davidf@vimeo.com> References: <20240729143743.34236-1-davidf@vimeo.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 3B2C61C0015 X-Rspamd-Server: rspam01 X-Stat-Signature: 5n1zfa76651y3p4m1uxqec6onokehzb1 X-HE-Tag: 1722288183-285801 X-HE-Meta: U2FsdGVkX18gwSEBZK52Lt7qMqdhTjIBQRhOqgnSj8bIeNgv58drhjbG8uJYAA7rJwc/ENKkUm/277LkklKPGXQoO99q018+kYsy2cXkCdPcq4pX5LOauQEo6h+0JdDEKwg6QRR5uzE7Rv3MtGteI+g0AR+cnJjPZafqTxtzHOXQmDVkWdmjXL98C0htrYbucA2wZKUPAFezMkS5PU30e7y3k9N5s7BdFn0bOyo9Ni3/uO6+ZU8o/4rljqEqwnZfyuVYiBYhl0/KOyMphJ116pbwwIKnfelmsTAWc2u8TOlKAFSvkcMTyxYQvEOrIF4RTqBkVBrQ43Lp0XBGlXYQq/lkSrJOwQS1E+RyXt4Hl/GFMPHk4jRzQ7bHM7uMrnTCK5/MRLKyuDHZVwqbSb04JRPfbQXd07ts32TaZiVHKNmeKfdM1Q3U8GFblp2S3olUTRTTb460vJeZG6ob96M3GL+ZRu5WvKXS8flZbrS1JdaFoGlfmRxnhP8+ue+s+Ak3uLnq4Z/LL7AgXAe5svqDD2APOMmm+nima+Epb6HEQh2w063OA0VmzOhraIoyYza0ApPgwDctmSvETdIlZqqK5zQbca4hIGGcJbejgtbtebjfh3IjZ71AirVKy5g9eqHtm7hvofDVg4i/miFgB/coiLjRLWAZTzfh9nIMJsycXptqxBrpHGiIpK2aXz7sWAx+v5nQt7XbFWel+L90lOGk33lOiINT50ZRiP9ykZMaSSnnUI7Q408bq1KgBy9KJKYU8W5pRvdj0XEf72aytEm0FC8Y1+FH1r4Hy9sE++1hBGgGkEZOnk/BhzuGJUUplWw6xLgJsngi/pOQC0eKJp57z6PMagd8d8O8zjWuroyM/wFCD3RHSbhm9AhH9BFDbhh6fjDyob5WbNd/1kJ5wNvaa2DxhE83ix/mzsqFeLBs0qcfF0IVv2eIGdiRVAqPa1p+t+FuwbS97yM771OeArO k51boGHD ExyV6Peh0ThjH20oovuW6tYoRIrXJVk8kgCpggt0HHbbomYDjyoY6Il8cBSzxTxYMS8JAn7Uo3337xU829FEkRResaXYTvBP/8NJfTJFj2tJzlkWA/0xXiaAqH0qFE4u9xyK5sNw8b+x4gtXZ5O+zAjMrS9vWoHUtdfy3DvQSBUPrhWeolBA1ByIWoGII7SVkeaCyTQ4HtQu3Nbdv0HyZvAoApuSbn6LboVnAKetoCUP8a9zFsDr74L50pIGvUfn42dcBjj4KyP5u5P3xkDED7ejYeNf+7rZsDZlnQo5uogYLIrEuYcD80pkdL18SDp3psVd46im3bBv2aVP+bOZlkyw1XTIpeAgEdD7BkhcGaTunNkADYVSx2HOEyZQzf1lftIknakUetB5JvISUTJd/ulJoOKvTbwETrbcgsznp1tfmHt2XiiBOkv6dZCMrOLcKsL6fmE6OekKoALNmCTCT83upeurrr5H8hRoLj2BE/wC7qEc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.001874, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Extend two existing tests to cover extracting memory usage through the newly mutable memory.peak and memory.swap.peak handlers. In particular, make sure to exercise adding and removing watchers with overlapping lifetimes so the less-trivial logic gets tested. Signed-off-by: David Finkel --- tools/testing/selftests/cgroup/cgroup_util.c | 22 ++ tools/testing/selftests/cgroup/cgroup_util.h | 2 + .../selftests/cgroup/test_memcontrol.c | 229 +++++++++++++++++- 3 files changed, 245 insertions(+), 8 deletions(-) diff --git a/tools/testing/selftests/cgroup/cgroup_util.c b/tools/testing/selftests/cgroup/cgroup_util.c index 432db923bced0..1e2d46636a0ca 100644 --- a/tools/testing/selftests/cgroup/cgroup_util.c +++ b/tools/testing/selftests/cgroup/cgroup_util.c @@ -141,6 +141,16 @@ long cg_read_long(const char *cgroup, const char *control) return atol(buf); } +long cg_read_long_fd(int fd) +{ + char buf[128]; + + if (pread(fd, buf, sizeof(buf), 0) <= 0) + return -1; + + return atol(buf); +} + long cg_read_key_long(const char *cgroup, const char *control, const char *key) { char buf[PAGE_SIZE]; @@ -183,6 +193,18 @@ int cg_write(const char *cgroup, const char *control, char *buf) return ret == len ? 0 : ret; } +/* + * Returns fd on success, or -1 on failure. + * (fd should be closed with close() as usual) + */ +int cg_open(const char *cgroup, const char *control, int flags) +{ + char path[PATH_MAX]; + + snprintf(path, sizeof(path), "%s/%s", cgroup, control); + return open(path, flags); +} + int cg_write_numeric(const char *cgroup, const char *control, long value) { char buf[64]; diff --git a/tools/testing/selftests/cgroup/cgroup_util.h b/tools/testing/selftests/cgroup/cgroup_util.h index e8d04ac9e3d23..19b131ee77072 100644 --- a/tools/testing/selftests/cgroup/cgroup_util.h +++ b/tools/testing/selftests/cgroup/cgroup_util.h @@ -34,9 +34,11 @@ extern int cg_read_strcmp(const char *cgroup, const char *control, extern int cg_read_strstr(const char *cgroup, const char *control, const char *needle); extern long cg_read_long(const char *cgroup, const char *control); +extern long cg_read_long_fd(int fd); long cg_read_key_long(const char *cgroup, const char *control, const char *key); extern long cg_read_lc(const char *cgroup, const char *control); extern int cg_write(const char *cgroup, const char *control, char *buf); +extern int cg_open(const char *cgroup, const char *control, int flags); int cg_write_numeric(const char *cgroup, const char *control, long value); extern int cg_run(const char *cgroup, int (*fn)(const char *cgroup, void *arg), diff --git a/tools/testing/selftests/cgroup/test_memcontrol.c b/tools/testing/selftests/cgroup/test_memcontrol.c index 41ae8047b8895..f54c1f75b6da7 100644 --- a/tools/testing/selftests/cgroup/test_memcontrol.c +++ b/tools/testing/selftests/cgroup/test_memcontrol.c @@ -161,13 +161,15 @@ static int alloc_pagecache_50M_check(const char *cgroup, void *arg) /* * This test create a memory cgroup, allocates * some anonymous memory and some pagecache - * and check memory.current and some memory.stat values. + * and checks memory.current, memory.peak, and some memory.stat values. */ -static int test_memcg_current(const char *root) +static int test_memcg_current_peak(const char *root) { int ret = KSFT_FAIL; - long current; + long current, peak, peak_reset; char *memcg; + bool fd2_closed = false, fd3_closed = false, fd4_closed = false; + int peak_fd = -1, peak_fd2 = -1, peak_fd3 = -1, peak_fd4 = -1; memcg = cg_name(root, "memcg_test"); if (!memcg) @@ -180,15 +182,108 @@ static int test_memcg_current(const char *root) if (current != 0) goto cleanup; + peak = cg_read_long(memcg, "memory.peak"); + if (peak != 0) + goto cleanup; + if (cg_run(memcg, alloc_anon_50M_check, NULL)) goto cleanup; + peak = cg_read_long(memcg, "memory.peak"); + if (peak < MB(50)) + goto cleanup; + + /* + * We'll open a few FDs for the same memory.peak file to exercise the free-path + * We need at least three to be closed in a different order than writes occurred to test + * the linked-list handling. + */ + peak_fd = cg_open(memcg, "memory.peak", O_RDWR | O_APPEND | O_CLOEXEC); + + if (peak_fd == -1) + goto cleanup; + + peak_fd2 = cg_open(memcg, "memory.peak", O_RDWR | O_APPEND | O_CLOEXEC); + + if (peak_fd2 == -1) + goto cleanup; + + peak_fd3 = cg_open(memcg, "memory.peak", O_RDWR | O_APPEND | O_CLOEXEC); + + if (peak_fd3 == -1) + goto cleanup; + + /* any non-empty string resets, but make it clear */ + static const char reset_string[] = "reset\n"; + + peak_reset = write(peak_fd, reset_string, sizeof(reset_string)); + if (peak_reset != sizeof(reset_string)) + goto cleanup; + + peak_reset = write(peak_fd2, reset_string, sizeof(reset_string)); + if (peak_reset != sizeof(reset_string)) + goto cleanup; + + peak_reset = write(peak_fd3, reset_string, sizeof(reset_string)); + if (peak_reset != sizeof(reset_string)) + goto cleanup; + + /* Make sure a completely independent read isn't affected by our FD-local reset above*/ + peak = cg_read_long(memcg, "memory.peak"); + if (peak < MB(50)) + goto cleanup; + + fd2_closed = true; + if (close(peak_fd2)) + goto cleanup; + + peak_fd4 = cg_open(memcg, "memory.peak", O_RDWR | O_APPEND | O_CLOEXEC); + + if (peak_fd4 == -1) + goto cleanup; + + peak_reset = write(peak_fd4, reset_string, sizeof(reset_string)); + if (peak_reset != sizeof(reset_string)) + goto cleanup; + + peak = cg_read_long_fd(peak_fd); + if (peak > MB(30) || peak < 0) + goto cleanup; + if (cg_run(memcg, alloc_pagecache_50M_check, NULL)) goto cleanup; + peak = cg_read_long(memcg, "memory.peak"); + if (peak < MB(50)) + goto cleanup; + + /* Make sure everything is back to normal */ + peak = cg_read_long_fd(peak_fd); + if (peak < MB(50)) + goto cleanup; + + peak = cg_read_long_fd(peak_fd4); + if (peak < MB(50)) + goto cleanup; + + fd3_closed = true; + if (close(peak_fd3)) + goto cleanup; + + fd4_closed = true; + if (close(peak_fd4)) + goto cleanup; + ret = KSFT_PASS; cleanup: + close(peak_fd); + if (!fd2_closed) + close(peak_fd2); + if (!fd3_closed) + close(peak_fd3); + if (!fd4_closed) + close(peak_fd4); cg_destroy(memcg); free(memcg); @@ -817,13 +912,17 @@ static int alloc_anon_50M_check_swap(const char *cgroup, void *arg) /* * This test checks that memory.swap.max limits the amount of - * anonymous memory which can be swapped out. + * anonymous memory which can be swapped out. Additionally, it verifies that + * memory.swap.peak reflects the high watermark and can be reset. */ -static int test_memcg_swap_max(const char *root) +static int test_memcg_swap_max_peak(const char *root) { int ret = KSFT_FAIL; char *memcg; - long max; + long max, peak; + + /* any non-empty string resets */ + static const char reset_string[] = "foobarbaz"; if (!is_swap_enabled()) return KSFT_SKIP; @@ -840,6 +939,45 @@ static int test_memcg_swap_max(const char *root) goto cleanup; } + int swap_peak_fd = cg_open(memcg, "memory.swap.peak", + O_RDWR | O_APPEND | O_CLOEXEC); + + if (swap_peak_fd == -1) + goto cleanup; + + int mem_peak_fd = cg_open(memcg, "memory.peak", O_RDWR | O_APPEND | O_CLOEXEC); + + if (mem_peak_fd == -1) + goto cleanup; + + if (cg_read_long(memcg, "memory.swap.peak")) + goto cleanup; + + if (cg_read_long_fd(swap_peak_fd)) + goto cleanup; + + /* switch the swap and mem fds into local-peak tracking mode*/ + int peak_reset = write(swap_peak_fd, reset_string, sizeof(reset_string)); + + if (peak_reset != sizeof(reset_string)) + goto cleanup; + + if (cg_read_long_fd(swap_peak_fd)) + goto cleanup; + + if (cg_read_long(memcg, "memory.peak")) + goto cleanup; + + if (cg_read_long_fd(mem_peak_fd)) + goto cleanup; + + peak_reset = write(mem_peak_fd, reset_string, sizeof(reset_string)); + if (peak_reset != sizeof(reset_string)) + goto cleanup; + + if (cg_read_long_fd(mem_peak_fd)) + goto cleanup; + if (cg_read_strcmp(memcg, "memory.max", "max\n")) goto cleanup; @@ -862,6 +1000,61 @@ static int test_memcg_swap_max(const char *root) if (cg_read_key_long(memcg, "memory.events", "oom_kill ") != 1) goto cleanup; + peak = cg_read_long(memcg, "memory.peak"); + if (peak < MB(29)) + goto cleanup; + + peak = cg_read_long(memcg, "memory.swap.peak"); + if (peak < MB(29)) + goto cleanup; + + peak = cg_read_long_fd(mem_peak_fd); + if (peak < MB(29)) + goto cleanup; + + peak = cg_read_long_fd(swap_peak_fd); + if (peak < MB(29)) + goto cleanup; + + /* + * open, reset and close the peak swap on another FD to make sure + * multiple extant fds don't corrupt the linked-list + */ + peak_reset = cg_write(memcg, "memory.swap.peak", (char *)reset_string); + if (peak_reset) + goto cleanup; + + peak_reset = cg_write(memcg, "memory.peak", (char *)reset_string); + if (peak_reset) + goto cleanup; + + /* actually reset on the fds */ + peak_reset = write(swap_peak_fd, reset_string, sizeof(reset_string)); + if (peak_reset != sizeof(reset_string)) + goto cleanup; + + peak_reset = write(mem_peak_fd, reset_string, sizeof(reset_string)); + if (peak_reset != sizeof(reset_string)) + goto cleanup; + + peak = cg_read_long_fd(swap_peak_fd); + if (peak > MB(10)) + goto cleanup; + + /* + * The cgroup is now empty, but there may be a page or two associated + * with the open FD accounted to it. + */ + peak = cg_read_long_fd(mem_peak_fd); + if (peak > MB(1)) + goto cleanup; + + if (cg_read_long(memcg, "memory.peak") < MB(29)) + goto cleanup; + + if (cg_read_long(memcg, "memory.swap.peak") < MB(29)) + goto cleanup; + if (cg_run(memcg, alloc_anon_50M_check_swap, (void *)MB(30))) goto cleanup; @@ -869,9 +1062,29 @@ static int test_memcg_swap_max(const char *root) if (max <= 0) goto cleanup; + peak = cg_read_long(memcg, "memory.peak"); + if (peak < MB(29)) + goto cleanup; + + peak = cg_read_long(memcg, "memory.swap.peak"); + if (peak < MB(29)) + goto cleanup; + + peak = cg_read_long_fd(mem_peak_fd); + if (peak < MB(29)) + goto cleanup; + + peak = cg_read_long_fd(swap_peak_fd); + if (peak < MB(19)) + goto cleanup; + ret = KSFT_PASS; cleanup: + if (close(mem_peak_fd)) + ret = KSFT_FAIL; + if (close(swap_peak_fd)) + ret = KSFT_FAIL; cg_destroy(memcg); free(memcg); @@ -1295,7 +1508,7 @@ struct memcg_test { const char *name; } tests[] = { T(test_memcg_subtree_control), - T(test_memcg_current), + T(test_memcg_current_peak), T(test_memcg_min), T(test_memcg_low), T(test_memcg_high), @@ -1303,7 +1516,7 @@ struct memcg_test { T(test_memcg_max), T(test_memcg_reclaim), T(test_memcg_oom_events), - T(test_memcg_swap_max), + T(test_memcg_swap_max_peak), T(test_memcg_sock), T(test_memcg_oom_group_leaf_events), T(test_memcg_oom_group_parent_events),