From patchwork Fri Jul 26 20:31:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 13743176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DF9AC3DA4A for ; Fri, 26 Jul 2024 20:31:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E6466B0089; Fri, 26 Jul 2024 16:31:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3962A6B008A; Fri, 26 Jul 2024 16:31:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 25D216B0092; Fri, 26 Jul 2024 16:31:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0961D6B0089 for ; Fri, 26 Jul 2024 16:31:27 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A2EC4C0134 for ; Fri, 26 Jul 2024 20:31:26 +0000 (UTC) X-FDA: 82383048972.29.E0887E3 Received: from out-174.mta1.migadu.com (out-174.mta1.migadu.com [95.215.58.174]) by imf12.hostedemail.com (Postfix) with ESMTP id BD5BA40015 for ; Fri, 26 Jul 2024 20:31:24 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=q5tpDC4X; spf=pass (imf12.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.174 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722025834; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1Ma39JIt9UMBhtRRWP/3mWelLcidI6/lxwwSADCSCQQ=; b=fXbOxQDJ4Rt9Rtn9t3j6nKqoR0d5LXXgeZN7Fu2VUAKWkQ/DOQcF+Eg1VWr+Rn/aCBLGiD Rktiy0DlH1RpAEaxEgkW+3YEwf2Xl55s478Y6g7Si5NxEyNFYOf1iPhe7VzKqhphK0gHVK mVmqN16NRoKIdEGtBLezMRcoxZXf0cU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722025834; a=rsa-sha256; cv=none; b=gANjCQUhyRpGI7jgV/dwV5IxtzMVPB9be+iHzlcqvCBEg3eDgHJO3X4enPB0qOfptNB6/o mcPao1yl2mqgw5fEnQcu7XQ2ZPhuFC2Nsi6+Nkubxy4xNdMHpZx8geaSjy5CWx28ZvXKMu dgu5f8LNi9BpQzytrmCd3rEOdnirCt0= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=q5tpDC4X; spf=pass (imf12.hostedemail.com: domain of roman.gushchin@linux.dev designates 95.215.58.174 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1722025883; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1Ma39JIt9UMBhtRRWP/3mWelLcidI6/lxwwSADCSCQQ=; b=q5tpDC4XomuJOhtLaY+kadmUPS+OMFMVzClcXxawG2eHzIaiy1oyqfDg30EKaEDyW0wRmu dabktA+KuLySiabaO8AcCcwBpKFTqII+nklD1yNK2RRCgK3p+XewC720u1iMe6urpV+RIO UPsNGo4NoXLOxUsSqeFHwf1yIm7PtOw= From: Roman Gushchin To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Roman Gushchin Subject: [PATCH v3 1/3] mm: memcg: don't call propagate_protected_usage() needlessly Date: Fri, 26 Jul 2024 20:31:08 +0000 Message-ID: <20240726203110.1577216-2-roman.gushchin@linux.dev> In-Reply-To: <20240726203110.1577216-1-roman.gushchin@linux.dev> References: <20240726203110.1577216-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Stat-Signature: nderzdaxjjguqzmskpornpmt3g1xfcus X-Rspamd-Queue-Id: BD5BA40015 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1722025884-93614 X-HE-Meta: U2FsdGVkX19DPW8Bs3YcsAhqM7D9BHvnEEjmCabbdtfJkWxDwZPgbm48r2KDk7csslhe+1TvCQnaiExzD1yCdz12R9ZMRQRJgBNu9I8IjbHndxqgzJfheV1v6w0AuG2VyTamPkUAhcNQoz9Foiz+C4cKSwPl5qQBkqE7U0Nyy0HXePMOaVcyQTGEWnXuEtzTMsqfefIdUOqSvvjMsyQEL5lE9Px7BurjNSwRvAY+DO08Fpl2LKHa8EuJHk5XdmXTsuFNLO1RgXkixtwQfYm5pv0n8afOgo+AE71zIIcqdbmUUgdoP1+BoougizAbZXIHwaFPxhpG6vwiWHwICbRrkqmVkbKvtIrwaVKM9unUyr/tc3i1p+mE4eefvg81RlyyUslIGipnDCdS1i0P9IxLeEA0DixzbIIZVZcuyHFv/aAL1xscHHAsEnbujTaTrtVScV0TEX4x4eUAe8paVy45FisI9LwlUs0pSWDJi4TX1TnvoN18WiQKmYwLdsC840rVc3gkLE6ZOjHHZwdDYLMqFsF0Pqz2kACj1xUn9sfXI/0C2rp+GLdVdNc74VfSJ2iNsZeNiDkQMP/3bLIU5cN2k34BEdm6LeOnpa2ZbL+1YJoWp5xge21atMd8pCu+u7h2u7c1AiGkB5GJRZsdPc6ohUsfI/ktmhmS3rBRXrjyYr6h/f3iuf8b+BXXnLIakyODz/3ya/ihRlBMUFVmLaUP8G+3Rm4QHiyQM25b9guiazm7FXQCDtxIIhYZO/7iEl5s95CyV8zmPWbR7ne7dETnAUud+RZuHw8ObCZxOIHmcbv6Nimk6yYQNGWiC4d2l55fIHtnfH8UXJ+p334LEGfDtkeMeuWqd6mOAUQBDxLx8byVhzapy5llOXav+40RyjDY7VY3l4Ky7cs+lhh7/U07i9G9Nd5QdRG+qt1hICb8hRuy6GZC7EQwdAciVB9F4HU+UY0mar5APLiZaRyZ+bN bzJCWfgB ckKnJPVVekNNOOBMvtDbSqP6LrkawHh0atKuw2rESpQPAURAfTj3IQ0G6rhgN7o5fQ7g1kWEayNpqElQJjdJjZwEnB63tJLlqeJzdC6bWyETr1g63V9x+qADU01HiHCQACgtSRMFOUsapL8CXI3uGEaZ3bQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Memory protection (min/low) requires a constant tracking of protected memory usage. propagate_protected_usage() is called on each page counters update and does a number of operations even in cases when the actual memory protection functionality is not supported (e.g. hugetlb cgroups or memcg swap counters). It's obviously inefficient and leads to a waste of CPU cycles. It can be addressed by calling propagate_protected_usage() only for the counters which do support memory guarantees. As of now it's only memcg->memory - the unified memory memcg counter. Signed-off-by: Roman Gushchin Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- include/linux/page_counter.h | 8 +++++++- mm/hugetlb_cgroup.c | 4 ++-- mm/memcontrol.c | 16 ++++++++-------- mm/page_counter.c | 16 +++++++++++++--- 4 files changed, 30 insertions(+), 14 deletions(-) diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h index e1295e3e1f19..aadd42f5ab7b 100644 --- a/include/linux/page_counter.h +++ b/include/linux/page_counter.h @@ -32,6 +32,7 @@ struct page_counter { /* Keep all the read most fields in a separete cacheline. */ CACHELINE_PADDING(_pad2_); + bool protection_support; unsigned long min; unsigned long low; unsigned long high; @@ -45,12 +46,17 @@ struct page_counter { #define PAGE_COUNTER_MAX (LONG_MAX / PAGE_SIZE) #endif +/* + * Protection is supported only for the first counter (with id 0). + */ static inline void page_counter_init(struct page_counter *counter, - struct page_counter *parent) + struct page_counter *parent, + bool protection_support) { atomic_long_set(&counter->usage, 0); counter->max = PAGE_COUNTER_MAX; counter->parent = parent; + counter->protection_support = protection_support; } static inline unsigned long page_counter_read(struct page_counter *counter) diff --git a/mm/hugetlb_cgroup.c b/mm/hugetlb_cgroup.c index f443a56409a9..d8d0e665caed 100644 --- a/mm/hugetlb_cgroup.c +++ b/mm/hugetlb_cgroup.c @@ -114,10 +114,10 @@ static void hugetlb_cgroup_init(struct hugetlb_cgroup *h_cgroup, } page_counter_init(hugetlb_cgroup_counter_from_cgroup(h_cgroup, idx), - fault_parent); + fault_parent, false); page_counter_init( hugetlb_cgroup_counter_from_cgroup_rsvd(h_cgroup, idx), - rsvd_parent); + rsvd_parent, false); limit = round_down(PAGE_COUNTER_MAX, pages_per_huge_page(&hstates[idx])); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index eb92c21615eb..58169c34ae55 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3585,21 +3585,21 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) if (parent) { WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent)); - page_counter_init(&memcg->memory, &parent->memory); - page_counter_init(&memcg->swap, &parent->swap); + page_counter_init(&memcg->memory, &parent->memory, true); + page_counter_init(&memcg->swap, &parent->swap, false); #ifdef CONFIG_MEMCG_V1 WRITE_ONCE(memcg->oom_kill_disable, READ_ONCE(parent->oom_kill_disable)); - page_counter_init(&memcg->kmem, &parent->kmem); - page_counter_init(&memcg->tcpmem, &parent->tcpmem); + page_counter_init(&memcg->kmem, &parent->kmem, false); + page_counter_init(&memcg->tcpmem, &parent->tcpmem, false); #endif } else { init_memcg_stats(); init_memcg_events(); - page_counter_init(&memcg->memory, NULL); - page_counter_init(&memcg->swap, NULL); + page_counter_init(&memcg->memory, NULL, true); + page_counter_init(&memcg->swap, NULL, false); #ifdef CONFIG_MEMCG_V1 - page_counter_init(&memcg->kmem, NULL); - page_counter_init(&memcg->tcpmem, NULL); + page_counter_init(&memcg->kmem, NULL, false); + page_counter_init(&memcg->tcpmem, NULL, false); #endif root_mem_cgroup = memcg; return &memcg->css; diff --git a/mm/page_counter.c b/mm/page_counter.c index ad9bdde5d5d2..a54382a58ace 100644 --- a/mm/page_counter.c +++ b/mm/page_counter.c @@ -13,6 +13,11 @@ #include #include +static bool track_protection(struct page_counter *c) +{ + return c->protection_support; +} + static void propagate_protected_usage(struct page_counter *c, unsigned long usage) { @@ -57,7 +62,8 @@ void page_counter_cancel(struct page_counter *counter, unsigned long nr_pages) new = 0; atomic_long_set(&counter->usage, new); } - propagate_protected_usage(counter, new); + if (track_protection(counter)) + propagate_protected_usage(counter, new); } /** @@ -70,12 +76,14 @@ void page_counter_cancel(struct page_counter *counter, unsigned long nr_pages) void page_counter_charge(struct page_counter *counter, unsigned long nr_pages) { struct page_counter *c; + bool protection = track_protection(counter); for (c = counter; c; c = c->parent) { long new; new = atomic_long_add_return(nr_pages, &c->usage); - propagate_protected_usage(c, new); + if (protection) + propagate_protected_usage(c, new); /* * This is indeed racy, but we can live with some * inaccuracy in the watermark. @@ -112,6 +120,7 @@ bool page_counter_try_charge(struct page_counter *counter, struct page_counter **fail) { struct page_counter *c; + bool protection = track_protection(counter); for (c = counter; c; c = c->parent) { long new; @@ -141,7 +150,8 @@ bool page_counter_try_charge(struct page_counter *counter, *fail = c; goto failed; } - propagate_protected_usage(c, new); + if (protection) + propagate_protected_usage(c, new); /* see comment on page_counter_charge */ if (new > READ_ONCE(c->local_watermark)) {