From patchwork Mon Aug 28 23:33:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13368408 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AC67C83F12 for ; Mon, 28 Aug 2023 23:33:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ADBDE280028; Mon, 28 Aug 2023 19:33:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A63118E001E; Mon, 28 Aug 2023 19:33:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88FB5280028; Mon, 28 Aug 2023 19:33:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6C9DB8E001E for ; Mon, 28 Aug 2023 19:33:27 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1AA351C9512 for ; Mon, 28 Aug 2023 23:33:27 +0000 (UTC) X-FDA: 81175117254.30.D82CF8A Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf02.hostedemail.com (Postfix) with ESMTP id 5E47880026 for ; Mon, 28 Aug 2023 23:33:25 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Y50mBPgL; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3xC7tZAoKCAU3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3xC7tZAoKCAU3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693265605; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9qA6GTuqMP/rKfRIqhAnRSpZvROd0PfSA6OAmdp3iwc=; b=xygkgbpIMuDcCVNCiQtK5x+NOp6PRd7E575KlbEUGNHmU90r4bbd1d80rxY5bA6MvLf1m6 IXW7umvAPixIRKSBj9f8TQFMxmfiK9OCid0yXh+pENFyYKOBsQfMjTyLbfN4PkV39asAVn pQQP2RpYOwfU2CBVsyND1fctvbGwOFM= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Y50mBPgL; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3xC7tZAoKCAU3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3xC7tZAoKCAU3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693265605; a=rsa-sha256; cv=none; b=W0im5uzpEoQaCtAKGs3WkVQIv6d0MrZvyCPRSSPtZiS/73MRxwgVcgFcA2pmmuETHDGTFq 4mNvwsHILeozxUD96fCD/eWAJfDV3GKdOtlHp+HNzu8XMI27eKfqrHXJYxwwGYK7TlbyrW YzKq6HnfU0R48It+fZ0YVG+zEVNjzXU= Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-55afcc54d55so4719600a12.0 for ; Mon, 28 Aug 2023 16:33:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693265604; x=1693870404; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9qA6GTuqMP/rKfRIqhAnRSpZvROd0PfSA6OAmdp3iwc=; b=Y50mBPgL4ZnkSNZPluDfxqKm78sl1PPL8DiBaO19bDXgbRQJe1evxnOtKTQpEKaJt9 q6TFZ3ydpr0Iw287D1OlWUwjUcpdUuXymoPCstZstIPkQOj45oKQlFtnKtjN/mbLEvhM o9GCuUO1ZEUXUUx28qzHrrmiTuYE2TYf6adAH5P9xxUWLusmQEIv/Oe70pHaiYbwOVfm oMqTZe5koJnHF0raIu3Xd3TXHK4UNZwUPvI/t+PGnq6Qn7V4/toAyK3UAB+IEgfCPW4e xQTid6or7KJCLcX0Z+QlSvZSxNkrkenRrCxd7dBhnnRCQKzILS7s/2q/I8bBvCq1Ofti cLVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693265604; x=1693870404; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9qA6GTuqMP/rKfRIqhAnRSpZvROd0PfSA6OAmdp3iwc=; b=EJzqtShhxzdQ1sMLIp0qe4Y+vAwPQ6p0uYAvkT92b/8FfmcT9iBjXDOCTe/K4wfm0V FA27Qey+oro63dEM9S8clyjCGINi2Lwb7cLbKrHKieqkkQXG1ilfkPKAwtnSPKZb0lC3 dm0+fDsCjwdIyG1WhXpBjpzsobeTtVXYlS4VdOKLlk74TGHmEDBRtZBkVTXU1Rftks2h 9GfyFIpGCsxxBVHMMOOhFagmTxtYlXds1Ud2rsSiMNfYJpRq+lCg4lbEKRY/gzsOXR8C yIrf4Ia1QPDcad5u+hXfPujx/miJ1zAF+61Sp943U2LIptkGhNSM4SuAUI6aps/l+jkP L/Dg== X-Gm-Message-State: AOJu0YwTZCeihTpsp5v69ZPtvBVDV4truwbk6z82wkEAB1po7M8/Fr9T JPwCQl/KgPqgUIjwau0nWfd1s2z8wc9rb2Tr X-Google-Smtp-Source: AGHT+IFAK9Y6dgVDrmbxac+NiYlaeGIY5ZOe1MFU0Kqd9mAZdB6yEnhkHBiWrdmzYBIlnQNCLJdURVSriZmVyWHy X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:408a:b0:26d:5ce:b77f with SMTP id l10-20020a17090a408a00b0026d05ceb77fmr272628pjg.1.1693265604159; Mon, 28 Aug 2023 16:33:24 -0700 (PDT) Date: Mon, 28 Aug 2023 23:33:15 +0000 In-Reply-To: <20230828233319.340712-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230828233319.340712-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828233319.340712-2-yosryahmed@google.com> Subject: [PATCH v2 1/4] mm: memcg: properly name and document unified stats flushing From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 5E47880026 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: fhuap394hccjgyqfecop7c1hn4h3h4yx X-HE-Tag: 1693265605-593877 X-HE-Meta: U2FsdGVkX18NubBIagrD6tGVU0mUnYZpM79Jq5W/DNhNmkQlWRH4Jsca5eCRaefJTRKp/BacI4aDM210RnnBAhtZgLpQcK1wJMte5N4TpdAmViE97zk2Ki8jwJlYl3LuGWZr0MS4ghIfGO2Z1Sh+pMR1fzeJWMYWR2/nEkqoRl0m8czkAiK91t8M+ncHggatSsbylg1wNGL/kF4I+b/I0pvGu7vvnf+kfJA0nzqqVcTfqdpv2tirIQvss8tcX/DmaBR7n3ufHGcCvLTlHn+DkiOx1wmbz0C/9wWXz2i7t4ChQJTUW2vKVyLZOzWbVFK89i5g9sKRbrTVroRjoBmTsa7/JZ2wgAGpSbqNNcFB7OV7smD5TSUpd7a+FPnyFxsSiXwE9qz1LEvtyCjFqtOSeuegI/w5wogcvvGzAUpstnLf29pV6fb9mNtamXa6Q4bhCl21gRVk4vf55EDlya+iTsSSprVmCi1LvakeKCYEPzjccDguDsmbw3ZN9spvPAls3IarI4kIpvinzJ5w2QZnWV0s2VJhcy0ur4CR3iHEEReSM2PfnBR8S4LavK0f4g/pVbvvyFv9MCoIc79iL8EnbgdIhyHYTY+oRy/0khJx8DRIo8bUEIaFcZqe4iNb1xJqBHDtauzM6RilQo4C/WBXTqt7yKGZTNFCmAi8MVKaswGEtKLzAVoqPG0MacqvJZyKM05yaKf2bEk/MQ+6ZS7bPxAiBO50aNJgAq5YCoaP1nk1rKFp7MXNtMeKYXXcwCZk/czd7fbz2slTXiAiSAeIe+PAwnp84aPXD2o7+9RPa3/oCuX5O1M+oqwfzhTvcwxygy0qFSmcYRh0gQKIjV1VSQEAq9LvdgQqNV1zAZOzM1jUaBPegverIaVtjj0MkB3bzTwMgaMREKieyHd4byYKIq8KtDrEl/vjDZVz67MubdRUeAn+t/Wn5/nQ8wwnQ34SEb2QeTQwNrr13MrdNig CLjMledx riPsYHZ/dDg2S7NWIxhh69X+uPl1056AvKhWUW0dFI823WVnbEMM6iKUEGxAc5RH39ysaPeYOfodIVEkMpbEbVKHxzH/xPVsLK7798oke+ax7cC1LVfIa0NPO/eRUaL0Vlz2Pfx06MzEhUAlYQ92oU7YTEgha0mT5ODy1DSRbkH5Fzpo7jViUFgdRLAUNlD++Xrx01XoEL1pMd+C9VLkmQhFM1TFr3y29vmfTJSCPKXoprpTKo6IukC1OI1Me9EDsmD/n90kDpTg59Tm+kgOxT+RZOonjkaPiAV5Oa9HEZFRde4tTqH3cvRAkfqXssAH5R2qolHOVn5YNmfO6ftM6szcOQvXTtQyjHbUDWdtRDezh3s3lmP7ZNi/hjflf3rWzvBVpW60wHVxtGGPON62ypCH6U5Y54I1VD9zUTNpfcujveqWYbwjfhPwP71r64kIHq5wBv4aESz/vhLrL7KiUThuBAu3pcW0NCn3Q9+nn+aFzRMyGp5UwsnB/6EHPY9XtchzRRfF946k/6tSFYTOtzwynCtq3JM/Im++bg2/dLU5DcbxBmBhNrW3pHnYWxyPuduqTwBEEg06HFQegEkSWEOXD2lwx8SUQzBrZETWkZs0VF2XcyOIkHZC8/2lFy6jv41GQe1b+lRd5mKA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Most contexts that flush memcg stats use "unified" flushing, where basically all flushers attempt to flush the entire hierarchy, but only one flusher is allowed at a time, others skip flushing. This is needed because we need to flush the stats from paths such as reclaim or refaults, which may have high concurrency, especially on large systems. Serializing such performance-sensitive paths can introduce regressions, hence, unified flushing offers a tradeoff between stats staleness and the performance impact of flushing stats. Document this properly and explicitly by renaming the common flushing helper from do_flush_stats() to do_unified_stats_flush(), and adding documentation to describe unified flushing. Additionally, rename flushing APIs to add "try" in the name, which implies that flushing will not always happen. Also add proper documentation. No functional change intended. Signed-off-by: Yosry Ahmed --- include/linux/memcontrol.h | 8 +++--- mm/memcontrol.c | 53 ++++++++++++++++++++++++++------------ mm/vmscan.c | 2 +- mm/workingset.c | 4 +-- 4 files changed, 43 insertions(+), 24 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 11810a2cfd2d..d517b0cc5221 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1034,8 +1034,8 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, return x; } -void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_ratelimited(void); +void mem_cgroup_try_flush_stats(void); +void mem_cgroup_try_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1519,11 +1519,11 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); } -static inline void mem_cgroup_flush_stats(void) +static inline void mem_cgroup_try_flush_stats(void) { } -static inline void mem_cgroup_flush_stats_ratelimited(void) +static inline void mem_cgroup_try_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index cf57fe9318d5..c6150ea54d48 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -630,7 +630,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) /* * If stats_flush_threshold exceeds the threshold * (>num_online_cpus()), cgroup stats update will be triggered - * in __mem_cgroup_flush_stats(). Increasing this var further + * in mem_cgroup_try_flush_stats(). Increasing this var further * is redundant and simply adds overhead in atomic update. */ if (atomic_read(&stats_flush_threshold) <= num_online_cpus()) @@ -639,13 +639,17 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } -static void do_flush_stats(void) +/* + * do_unified_stats_flush - do a unified flush of memory cgroup statistics + * + * A unified flush tries to flush the entire hierarchy, but skips if there is + * another ongoing flush. This is meant for flushers that may have a lot of + * concurrency (e.g. reclaim, refault, etc), and should not be serialized to + * avoid slowing down performance-sensitive paths. A unified flush may skip, and + * hence may yield stale stats. + */ +static void do_unified_stats_flush(void) { - /* - * We always flush the entire tree, so concurrent flushers can just - * skip. This avoids a thundering herd problem on the rstat global lock - * from memcg flushers (e.g. reclaim, refault, etc). - */ if (atomic_read(&stats_flush_ongoing) || atomic_xchg(&stats_flush_ongoing, 1)) return; @@ -658,16 +662,31 @@ static void do_flush_stats(void) atomic_set(&stats_flush_ongoing, 0); } -void mem_cgroup_flush_stats(void) +/* + * mem_cgroup_try_flush_stats - try to flush memory cgroup statistics + * + * Try to flush the stats of all memcgs that have stat updates since the last + * flush. We do not flush the stats if: + * - The magnitude of the pending updates is below a certain threshold. + * - There is another ongoing unified flush (see do_unified_stats_flush()). + * + * Hence, the stats may be stale, but ideally by less than FLUSH_TIME due to + * periodic flushing. + */ +void mem_cgroup_try_flush_stats(void) { if (atomic_read(&stats_flush_threshold) > num_online_cpus()) - do_flush_stats(); + do_unified_stats_flush(); } -void mem_cgroup_flush_stats_ratelimited(void) +/* + * Like mem_cgroup_try_flush_stats(), but only flushes if the periodic flusher + * is late. + */ +void mem_cgroup_try_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); } static void flush_memcg_stats_dwork(struct work_struct *w) @@ -676,7 +695,7 @@ static void flush_memcg_stats_dwork(struct work_struct *w) * Always flush here so that flushing in latency-sensitive paths is * as cheap as possible. */ - do_flush_stats(); + do_unified_stats_flush(); queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } @@ -1576,7 +1595,7 @@ static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) * * Current memory state: */ - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { u64 size; @@ -4018,7 +4037,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) int nid; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { seq_printf(m, "%s=%lu", stat->name, @@ -4093,7 +4112,7 @@ static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats)); - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { unsigned long nr; @@ -4595,7 +4614,7 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); @@ -6610,7 +6629,7 @@ static int memory_numa_stat_show(struct seq_file *m, void *v) int i; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { int nid; diff --git a/mm/vmscan.c b/mm/vmscan.c index c7c149cb8d66..457a18921fda 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2923,7 +2923,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); /* * Determine the scan balance between anon and file LRUs. diff --git a/mm/workingset.c b/mm/workingset.c index da58a26d0d4d..affb8699e58d 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -520,7 +520,7 @@ void workingset_refault(struct folio *folio, void *shadow) } /* Flush stats (and potentially sleep) before holding RCU read lock */ - mem_cgroup_flush_stats_ratelimited(); + mem_cgroup_try_flush_stats_ratelimited(); rcu_read_lock(); @@ -664,7 +664,7 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker, struct lruvec *lruvec; int i; - mem_cgroup_flush_stats(); + mem_cgroup_try_flush_stats(); lruvec = mem_cgroup_lruvec(sc->memcg, NODE_DATA(sc->nid)); for (pages = 0, i = 0; i < NR_LRU_LISTS; i++) pages += lruvec_page_state_local(lruvec, From patchwork Mon Aug 28 23:33:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13368409 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDE5CC83F11 for ; Mon, 28 Aug 2023 23:33:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E5CAB28002B; Mon, 28 Aug 2023 19:33:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E0BC38E001E; Mon, 28 Aug 2023 19:33:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C876728002B; Mon, 28 Aug 2023 19:33:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B223A8E001E for ; Mon, 28 Aug 2023 19:33:29 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8860B14066C for ; Mon, 28 Aug 2023 23:33:28 +0000 (UTC) X-FDA: 81175117296.19.27CA5A5 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf03.hostedemail.com (Postfix) with ESMTP id CD50F2000C for ; Mon, 28 Aug 2023 23:33:26 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=sSpYPt7r; spf=pass (imf03.hostedemail.com: domain of 3xS7tZAoKCAY4uyx4gnskjmuumrk.iusrot03-ssq1giq.uxm@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3xS7tZAoKCAY4uyx4gnskjmuumrk.iusrot03-ssq1giq.uxm@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693265606; a=rsa-sha256; cv=none; b=aich4zunBQjjUgICmyFn4wCV3/8p5S4M9TBp+UZyTn6koid1I/oWFdORm+WpLW1WoR59M8 RPKq3K5fFuA4TIn47hx8XWQ9+3Gfmc0JL0G4EygitXrr0l0OvH/q56Kqo6sqcM+VR28Bzb ceUctMhiujbBjpLO49sr7iRXDJW1lDY= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=sSpYPt7r; spf=pass (imf03.hostedemail.com: domain of 3xS7tZAoKCAY4uyx4gnskjmuumrk.iusrot03-ssq1giq.uxm@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3xS7tZAoKCAY4uyx4gnskjmuumrk.iusrot03-ssq1giq.uxm@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693265606; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I5n9hS9q/cqFV2qh34A2AZEA4DGYELaonjfoFoCAI8g=; b=T39vURDEs2ZRIu/zlF/fV5YWxqEE+q1rjVeIINijNcLWY6kCSMC4/wxeFMCAEOjPiT7R0R +k9SQbdihP/bZvvUas+iRDB2YQwf8t+9bNIgwwEK2Pqz03ZOH3yAHas3N52++PUAxGJWxp NibG9Gb/9npv4bChl+F29Nrjm3tZdq0= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-c8f360a07a2so4461482276.2 for ; Mon, 28 Aug 2023 16:33:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693265606; x=1693870406; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=I5n9hS9q/cqFV2qh34A2AZEA4DGYELaonjfoFoCAI8g=; b=sSpYPt7rKW3DtUM15JcLQJplUoRrPMx95M/aXCWUkUGG4gZ450QgwVE4yFUm7CogQH zFv9UI4D9YSxvW6bodhDxj3Ik66YmhEKFtQN01JiGkk9R6pRwoVFGBpQEhzaG5mXyhg+ 7vKrj39/+K2fsqeCBVG9cqcEQsLe/O8dA6loSupqE+p8mxgQd1Y/Cch38q9mTgfWAdOU Mi/PT8DoWdqlQunEGCPiAM+0xArU4vTZKSIUUhJFt8vF/ApSZdUHhIi/YFoCfVUArao7 1xkMsp1FV1LDKVh9Dm5/i7gS4Zp0X1DnugiYstil6izVooJ18sBWRLBaLBmogvidXImj ZLUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693265606; x=1693870406; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=I5n9hS9q/cqFV2qh34A2AZEA4DGYELaonjfoFoCAI8g=; b=KCYMqJ9vP3xfHttpyBHE/enhQTAJlCyzieuOw11YjluxcylbGJy6b3qhYzVjmlP6q3 yW7EIeVamVwEuoLcc8kyAy9K1t90RcmrJ4meCnoXeusXe0yZNnJgoBjjJjuFrzVfo7e+ 40xqAo3UIFfqEJ6/W8sysjOhI6+Na0lQRPm/zPKNqK4pVPKUxr4cegsz+Q5DSmF+MADk pSYv35gM93+9lhQKZRwKXGPUOXATAD0WW2jqbYVD/s0feqtdDiBvxkox91qN3aZ82mvC UM7SJevkvuoqLt2ScAmmlvsxdd4+RF8eJ4LFbGLH7fxWdjRChiPew5Kt3p/BtgmOIQ5x PPWQ== X-Gm-Message-State: AOJu0YzdMBKn3+rAhvz5GFs+ZQNMu12lLnjRev+4TJMUSVK7NSQisZaH p+8YW+3mYDHVE1jPsBbzQelWLBWKIhlRLolf X-Google-Smtp-Source: AGHT+IHgw1Dv81YEOafHari2+haF5L0KUBFcH2n6KPrSQNlL17+4o9VM2wgivkwVdbEOE7or4T1k1wZFPbeokEu2 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:1788:b0:d78:3a29:214f with SMTP id ca8-20020a056902178800b00d783a29214fmr506633ybb.10.1693265605965; Mon, 28 Aug 2023 16:33:25 -0700 (PDT) Date: Mon, 28 Aug 2023 23:33:16 +0000 In-Reply-To: <20230828233319.340712-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230828233319.340712-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828233319.340712-3-yosryahmed@google.com> Subject: [PATCH v2 2/4] mm: memcg: add a helper for non-unified stats flushing From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: CD50F2000C X-Stat-Signature: gm3gwi1a5byxhk85znu95ybmiumj1dab X-Rspam-User: X-HE-Tag: 1693265606-839644 X-HE-Meta: U2FsdGVkX19FbEDCXEH00GvsR3K789Gmv+plp8L9TU/0pbplr4A4VwAcsIQvnswUxhmNB1DK6/0/G4ItQsbVtF1Gc/HpXe/3lNfCOPPltIEsgrmlX1HvNnzbZ4OW2ocS8GSLNGKhXwORzTpIeKoKjoME6wSXKqB1tgLP4ykeaEX3+sYabzM4D8ZF4ra31ZMUkcnrsaD5cwTjAEwSFrGKCJ5luWKbS8FmCWjdCMGK+b9CuodMAVMoqk2VBSyQ/+tBjCoPRxleb1g6W3oP9MEu3Qh54uONXfa0yfhtpIwS5cRyB8Q/zUPdl0iwoFcXpE3Kw7PctA31NCzin7hjCYXfWsG2ePiAj6zokDnT7JAbzKlQggIW7dD9zpWXVgOdAPMb+21wXbF/PHxsK3p2p1NcTWDx9ABe/iWsDi3KHnvWFLMXJmrd3akZ8d+Vi/2rZ+q8fNbeeo3YNkT6yCmVIUzpUziZZI5G2ChbIJf8QLe31HabATiRMZyXmkXewXCTwQdG1lDL0z59wio/C/ZyUBW+gih070JBMh7WER3YdjkwBvywHZUOZH4lZ7dkQPSmTwn3J2RiGWbuubFtzKAFuYy10TyOIR5zehic6XO9VwsUK9dOKNMHkuWyVRxwI4kWTOsEupkR39ga4pC7xLLY3B27V+167OvMr1UBZ9ZNVVu+3BTFhiMNnIMvA3qZ68NOjppSAH8lOnSrRWEEleUZOMZUBuOtKcpasVLmrCnkQRUl55AFB4U5thEJGlgTPVKT2ze50sbARYr4vk4Ey8e53xNWSEcnDE66Bi/qjXFglKUr+wg708FnbqKLz6kaHGkr9JKag+qVqJvkUhEZTWHHURgygiNQ3OgXEseYqN310Ccc3MkQ6TvYaOyUQ8JuxoiIIaVJDfUSp4yRei8Gd5noh5KWr+CG4DNhshTnIKlIGVRwxWV8+JItjjdwyY9xzy/UtL6rAVWi0EkqNga/Y/iYhQR +DNvgX61 321WYmv67jxAWmLfIXpbAO4ZSJBtuZEcY0qb5XsDj7/qTW7wrvWzZ89+KWOcty18uZ5HEYbkmvpgfUi3w/nUY2SaZVHHy9qVdNF4rz6zpGBhXgMoNewx+aGNKcmfTufoHuRsyt3okVXTe3K4S+evDRl7YMZo1i8fDLA7l3tmUFZakMGXEnQ8fijUkKfqwIaLd+L7FOFyc2FRtCxV4xhTLmnH/U42O9XJYOAcSjm4mqdyBuyxYlbrMpsbgJfI2N5TGrsHRhcUzzbNK7qlSnVxFNH5kDdJ9sM2fC8OQwtqt5OEcyZ8lEVfRR2HAki5DuhbyGt4zX5JPV32mKPsdmQs88bi5bW6tCw7pDFJBZMmgul85R2+zkGd/0d0yLcRxn+vQ1O0agJqw1gbOjnaMf0ZNPU5UmRqSsptUvVtR5zUCga/SWubENKD29AwkdXYSymJcEPvdPMfheYEu1furBUrM/2dCpO0FNrWWHXxKp4Mco/NEBmyIR/H4v0kwCFyliph04ADu7E2DphoIzSCBuLbuM/0UePu9dfOIlgp9EZCl5cEcj9xeK77azkdRvXWHfB0stYa4ZsqHz3UQZuoDnHzQnF3DwHbzPbA7/R4o9C5wkXdhoa/IsWGow7SRxDGFwklcGL7AAJg4GKB6sLQi7VLZTwF9JSx4xiYvo5ZM/8XB9eRvYZmC2G1PNV1T1Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some contexts flush memcg stats outside of unified flushing, directly using cgroup_rstat_flush(). Add a helper for non-unified flushing, a counterpart for do_unified_stats_flush(), and use it in those contexts, as well as in do_unified_stats_flush() itself. This abstracts the rstat API and makes it easy to introduce modifications to either unified or non-unified flushing functions without changing callers. No functional change intended. Signed-off-by: Yosry Ahmed --- mm/memcontrol.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c6150ea54d48..90f08b35fa77 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -639,6 +639,17 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } +/* + * do_stats_flush - do a flush of the memory cgroup statistics + * @memcg: memory cgroup to flush + * + * Only flushes the subtree of @memcg, does not skip under any conditions. + */ +static void do_stats_flush(struct mem_cgroup *memcg) +{ + cgroup_rstat_flush(memcg->css.cgroup); +} + /* * do_unified_stats_flush - do a unified flush of memory cgroup statistics * @@ -656,7 +667,7 @@ static void do_unified_stats_flush(void) WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); - cgroup_rstat_flush(root_mem_cgroup->css.cgroup); + do_stats_flush(root_mem_cgroup); atomic_set(&stats_flush_threshold, 0); atomic_set(&stats_flush_ongoing, 0); @@ -7790,7 +7801,7 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) break; } - cgroup_rstat_flush(memcg->css.cgroup); + do_stats_flush(memcg); pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE; if (pages < max) continue; @@ -7855,8 +7866,10 @@ void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size) static u64 zswap_current_read(struct cgroup_subsys_state *css, struct cftype *cft) { - cgroup_rstat_flush(css->cgroup); - return memcg_page_state(mem_cgroup_from_css(css), MEMCG_ZSWAP_B); + struct mem_cgroup *memcg = mem_cgroup_from_css(css); + + do_stats_flush(memcg); + return memcg_page_state(memcg, MEMCG_ZSWAP_B); } static int zswap_max_show(struct seq_file *m, void *v) From patchwork Mon Aug 28 23:33:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13368410 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72062C83F14 for ; Mon, 28 Aug 2023 23:33:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1644328002C; Mon, 28 Aug 2023 19:33:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 117298E001E; Mon, 28 Aug 2023 19:33:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED53428002C; Mon, 28 Aug 2023 19:33:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D46C98E001E for ; Mon, 28 Aug 2023 19:33:30 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A416C1606A0 for ; Mon, 28 Aug 2023 23:33:30 +0000 (UTC) X-FDA: 81175117380.06.70B8A59 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf13.hostedemail.com (Postfix) with ESMTP id DFE1520008 for ; Mon, 28 Aug 2023 23:33:28 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=BaJDLeW2; spf=pass (imf13.hostedemail.com: domain of 3xy7tZAoKCAg6w0z6ipumlowwotm.kwutqv25-uus3iks.wzo@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3xy7tZAoKCAg6w0z6ipumlowwotm.kwutqv25-uus3iks.wzo@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693265608; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cdnModp3UHPesHrMiYbasA77T6X4L8s34myJUUsNDqw=; b=Q8+p+q5GppDzkTjavQVaUqpm2BiAURfbVV2WVf1Az4I6kw82aiVIFCnKXdjTcpp3rgKHtI bnj1P5CN7ncwX9HyO1E5zoZJ3kOmjZKpnyBw1k9RbZDCRJX7y7b2fTxbqY4JtXNvfk18zT L6OCVSn9ywjy/iXxI3AZB0qXdpAwWic= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693265608; a=rsa-sha256; cv=none; b=YYNjEhGAZjBNJzZb3E3WenaJyW5qFR9Cte1eXLewJqn5VAlX+0JQuOtHmk/MQz01kc/y+i Bv0pXmfLr9gtR3JG79gXBRNn4+2I/CDlU9K+Ne8gioGi9nhJZn86eN8P6uD6GPj3WQ2KVk /RQVL0c+ZYL3FX5J7XzcmO+M3Y77/JY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=BaJDLeW2; spf=pass (imf13.hostedemail.com: domain of 3xy7tZAoKCAg6w0z6ipumlowwotm.kwutqv25-uus3iks.wzo@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3xy7tZAoKCAg6w0z6ipumlowwotm.kwutqv25-uus3iks.wzo@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-68bf02547dbso4650472b3a.1 for ; Mon, 28 Aug 2023 16:33:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693265608; x=1693870408; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=cdnModp3UHPesHrMiYbasA77T6X4L8s34myJUUsNDqw=; b=BaJDLeW20NuEFec2oAO68NEPY8XNBrqkpeisxk5dZbxgnWHQ0r2PhoENHFifb29a85 wkcQ36f5pboCfp5upZdXeouVQ/0c7EuGiOlftOTQ7QED4HRVXJaE8f3Wet7A9n2Rj856 vSFPpwUufWhdTYEK9e0YxtG9T7V5niVLg1OGtXwve+hU15y1awhfiS+ph4sxssMScLfY 3FyWbOnA+EFg7ZsT4U1Ke97tXPWCjqO0X2CsZjdqw14iDY0ncsKKN+1F4C/05uDjI0Yu Vcck/XfFVgXP+e2uByfABv9Rt2GwqKStdZUxd2y8AJciGF9zoxxWBpNJ74POirqQMS0t +/dQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693265608; x=1693870408; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=cdnModp3UHPesHrMiYbasA77T6X4L8s34myJUUsNDqw=; b=cbpp+XToWyT/6bQC2gvfJCyQa6y3arJHHGi+nHfErHHfmaH442a9dki97rgXgLnMCc SFVLxH190Zmq3yo9G5C2TKrsnm9/UIuUmWj2TwPuW6ckvOgW6a5RXB+Mi0eekh1ydFDt j9UvKCxZhY0d4CClleQ0GHgbB5QZg8Lnx0fsTbMyjrh10PEg2EG3+5xl79Szofji8SiK hu0tZEs7cWarO1lcKgJw68+jNXpp1nU8amV3RXZmgmoOVYYruEVD+xheRno8bVoYTm4x YcD0pAVM8fmhzcyUlFjgzXhr49b+8uK4HumrCLI8wRfN/gBMW67OUlu205Da4OSVbGzu umDw== X-Gm-Message-State: AOJu0Yz6zssv4GTHHR31S3o+YloXQ6LVS8UPhEdoPSpu9Ai4VJbxOWpP YWQ4wFmQwT6/wYNaL1yS1Vsgh3xY/RsZHZUE X-Google-Smtp-Source: AGHT+IHAq4VgXKo8P6aYPET0HHb7Rp2LuZeLVocuW54MrcmyDai0zFV6jlRJW28aJMih5PpmlindTt5puuPdOpLx X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:13a7:b0:68a:5467:9974 with SMTP id t39-20020a056a0013a700b0068a54679974mr8979137pfg.0.1693265607909; Mon, 28 Aug 2023 16:33:27 -0700 (PDT) Date: Mon, 28 Aug 2023 23:33:17 +0000 In-Reply-To: <20230828233319.340712-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230828233319.340712-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828233319.340712-4-yosryahmed@google.com> Subject: [PATCH v2 3/4] mm: memcg: let non-unified root stats flushes help unified flushes From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: DFE1520008 X-Rspam-User: X-Stat-Signature: hbks9pgu37dxdfxg9ftpbztmnuh5pxau X-Rspamd-Server: rspam03 X-HE-Tag: 1693265608-391743 X-HE-Meta: U2FsdGVkX19eL/TOR560ENK8pzXBiojRF73zZjvmtY7XoOA4uQk20Ox3bxCPy9WaskxweHzhkKwazS03NYRYYFy3xIrhDp6gh4sTmsCV4ZafFoyCq0x83EJp3aPHn9CLU5kIq6UPiW2pAnUmD3CEi4tG1Fph/Cq0HtVymAUDg/671Qoo20isD9Uspul4fgSiXT+2hiWLgdKuapP1ylY3fyzoPrr1AzakDzzGu2BO/Uu391i+Z1gHA/tQIxSD4kTr3ir7YmgQfjfPChQ6hHOiIzeVf/dgS5jClTjrMpBvY39Wl/7KyFI291MgCeKNuPii12rr30I1PiE1JKTfK8OVtyBHahWAe6kKsfN58q3mWajusCtBFOSVCmWIU5xSnXxTcon6Tsr0tj5zi9hvVdRfQaSq6Y/QhQBqpL28QLpus4sPVQufFps5ISQXXVVaQIooXOCb2pqX//QwwoLTRwbEIgmDutAgcLsTNvK6UdqmLO25ZRuBdAju5Xp7x5PNiBFgvTHCwsWnzGJDrKz3DCqxWxoO2t5AN7AlRSyt/W4OCKkiPi1DHuK43RBUsK/0A2+gcCGhoTm9US1HcLt55BOfjjtiWnG28tmZmhZj5H93wnTfJppbVM66LUFinkkXUDFPMKjT33ewZyvdozWjGFMxa9vw5wKCzccFLfWiyuIb2tT+qh4gnffItrq9E4+R/Y9M/8Mvq9ysh0NLkkWuO/iFrLuNquI1CvRHGvbUCXFM83wMm6l80GnZyeiSqnYaP0eWqzKysLgbotrkHYOTJvE50BSLu+wW3F9V+RV/RF8QGViOrJqx5XmpY7rKC1tt7cxHCwapoQBvfzJgzFBemU26X4+vRkWVOfiMfsWnjsFaRMvSnLwtplVO0w5UYPSjEI6iTSKWInV0kimwGtdjGYCrL27+05a1rirAnRKBtUzBAqYtK4K3TejoIjPar6hrjZ7rsD0Apql1cDr3XfxFkRB EWv1coFy OsjEP/+jMjtfrRHBDfcDgNc/XEjy+0sGyTNsWGKj2YMeJCN7SV7gDzBBXDWHh4kBiPwjMVlbzb0e5pwAecP0+h7MHQxYALorMVFyokyWLC7vpHihbGei0/TvILowwWvOx7goyFIkldYFhD665rTEZdhftYe5SMRagNRzi0sb2mIexUGoj5rP5lMVUTdtwWjCwtqzmth6kB9enWkM1LrZf4OOYOUokeiEmqb0MSeCdBGA4WzOJZFqKY4fpKN3rFpt0J7kZkiONsjI77cWFaj8IgY2+IKP0Q5GpvhJoLJ7q+tN2pBCsezkaOsNXbEQpBxhlF0Xhcsc2vt5j3NZy8If7qwO5j673m0VeaTGyi5njhoz6HokAmg/a7eKCborBRUPtiEKPWi4seWKpJrEaYWMQG14O3JP2cDO9iIIBuq74QfPf+a/SxPAU4xdvknsG53ZzD4q9VAfN6uiy4dJ23cJZWku9/ziX93AVi9lCF2kUnqenRwjtnKIzxj2d3bM5/gg81cHH0xun+ZpvCyz9bvaN/7gtPyWuB6sVX7JVkDycVfEbiWBsm0RAP6iNwtC4iy4vWW1+GMZ9tdqE/sn85anCoxq/70S9HLH2XLN8bhTxNatO58oXwlNWzU3SwnEIv2coSgZ5B6LrYBh/VnT1NZ03ur0afRd4MAYzwXJj4V8S0Q3pzoI9fld0RguQDIzW0yFlA9i6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Unified flushing of memcg stats keeps track of the magnitude of pending updates, and only allows a flush if that magnitude exceeds a threshold. It also keeps track of the time at which ratelimited flushing should be allowed as flush_next_time. A non-unified flush on the root memcg has the same effect as a unified flush, so let it help unified flushing by resetting pending updates and kicking flush_next_time forward. Move the logic into the common do_stats_flush() helper, and do it for all root flushes, unified or not. There is a subtle change here, we reset stats_flush_threshold before a flush rather than after a flush. This probably okay because: (a) For flushers: only unified flushers check stats_flush_threshold, and those flushers skip anyway if there is another unified flush ongoing. Having them also skip if there is an ongoing non-unified root flush is actually more consistent. (b) For updaters: Resetting stats_flush_threshold early may lead to more atomic updates of stats_flush_threshold, as we start updating it earlier. This should not be significant in practice because we stop updating stats_flush_threshold when it reaches the threshold anyway. If we start early and stop early, the number of atomic updates remain the same. The only difference is the scenario where we reset stats_flush_threshold early, start doing atomic updates early, and then the periodic flusher kicks in before we reach the threshold. In this case, we will have done more atomic updates. However, since the threshold wasn't reached, then we did not do a lot of updates anyway. Suggested-by: Michal Koutný Signed-off-by: Yosry Ahmed --- mm/memcontrol.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 90f08b35fa77..f3716478bf4e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -647,6 +647,11 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) */ static void do_stats_flush(struct mem_cgroup *memcg) { + /* for unified flushing, root non-unified flushing can help as well */ + if (mem_cgroup_is_root(memcg)) { + WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); + atomic_set(&stats_flush_threshold, 0); + } cgroup_rstat_flush(memcg->css.cgroup); } @@ -665,11 +670,7 @@ static void do_unified_stats_flush(void) atomic_xchg(&stats_flush_ongoing, 1)) return; - WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); - do_stats_flush(root_mem_cgroup); - - atomic_set(&stats_flush_threshold, 0); atomic_set(&stats_flush_ongoing, 0); } From patchwork Mon Aug 28 23:33:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13368411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F4C7C83F11 for ; Mon, 28 Aug 2023 23:33:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CFFD828002D; Mon, 28 Aug 2023 19:33:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BEA4C8E001E; Mon, 28 Aug 2023 19:33:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A198128002D; Mon, 28 Aug 2023 19:33:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 886328E001E for ; Mon, 28 Aug 2023 19:33:32 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5B2ABC0671 for ; Mon, 28 Aug 2023 23:33:32 +0000 (UTC) X-FDA: 81175117464.27.E9DDC45 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf23.hostedemail.com (Postfix) with ESMTP id 912D3140017 for ; Mon, 28 Aug 2023 23:33:30 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=eoDxnQJF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3yS7tZAoKCAo8y218krwonqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yosryahmed.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3yS7tZAoKCAo8y218krwonqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693265610; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wRGpIX3lyKI055RI8cMjafWOHR4EuDVDQeE9aWNLVUM=; b=zajZ5rytp5+tSKXtfihv2eeWDHMc+1r4RT+6H7EbuJnLFNhF4PZ3MZfeeDv1vvIh16/oNc G3uudxa76GJ94jvhaxLVKx1tsj3GqOdVneod0P2+GjGU29Vk6Xk3DtKNx9kF/p0wKDHT0w a4GNCbvjP+Z8aekgXeyc0QudKhL8i4k= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=eoDxnQJF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3yS7tZAoKCAo8y218krwonqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yosryahmed.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3yS7tZAoKCAo8y218krwonqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693265610; a=rsa-sha256; cv=none; b=BBMQ7GvUAy+QxodE2kjl5qlDR70T7HZlTERUoGUfAx1MKtFhaZgFa+76Ac+KIqq0dXG9qy rvWx8GZ43wPhQokNdEs2BVaetOJzkwkWv8vFVcS0O0LZm/D/ENLAvbZhudaCnzLjEmgOiC O0c38k0GwFVzgpuB1/dq/vlvtkS+bwQ= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-1c0d58f127fso43974985ad.2 for ; Mon, 28 Aug 2023 16:33:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693265609; x=1693870409; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wRGpIX3lyKI055RI8cMjafWOHR4EuDVDQeE9aWNLVUM=; b=eoDxnQJF3X52LCWoBIxxsewZGAjILMgMJpEG4EXQ2YAh0jGR93SKaYdX81UglrX6fo 1wKEaixnEVtkrTCbB/CHL7SnwHFabZVbsYdY6keYxwCeXgS6yY/2Kto8tSYSMEJ/Y2YG pN89ky8l7MzmQ1FBTbeb5i+s04d3wUfDyK8F758aH26chFWvDGxLJn0ispz/NJwci9YX zZyG7Q8Xdlaanlqu4MxEXr+HClTY0BRCmUuRbsTIW6YTWap0g/zatE8xcF8lie3HxjPZ icLK7HqerVKwHQ7cBYh7IgKJ5U1e/mmoItt4cDNTSXRVs/dV4Xbtf/Jcf41tWtLG9YIA UabQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693265609; x=1693870409; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wRGpIX3lyKI055RI8cMjafWOHR4EuDVDQeE9aWNLVUM=; b=gZ1eOPDNZXEqUxfoon4heBsNeaebUMYjQ1lsT+ECcupxUXEAMmz3nkDlcb2D3vyjZy 8IEMNnaUu9l7ccKVBUy/qYmeHPy7wWrLlTc6pCOMZh4ioldMLtlhz3ueDjv9MClKkVcm 9pT0tKGwsdeZJM+smkw2bUSsiNcDeYfIu350ZlhFC0GNcTZO4mH9SnEnR2LJu9ijP2sS rshiuqwsXbceoTp6YRvSxqS0QQD167fl/PvU9Be7m9sDTmaGNexekqKYyW18hHYSh2SG QfFY+YOnT9BlCxERg38cgwmW/uFm9WJeSo/iDpDCwraJ9qbIR7fJZXRtyPqIPmhmlZnN cXlA== X-Gm-Message-State: AOJu0Yzg7StO10pic8hC2sSXRXOELEHYbZkkSwh6J7FaYwQdf0LbSvUq 1EZSMqKaHvNN43NtPbyo1bFkP4QALN7iNECO X-Google-Smtp-Source: AGHT+IHG/TRKUOO5sYh5EIId++RdUCNl9wMlTbyfze6/NjsSKmPfNT57qBN5bgwEcgW3z1bCs66pOUtMUV59CqzY X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:902:dacd:b0:1bc:1866:fd0f with SMTP id q13-20020a170902dacd00b001bc1866fd0fmr9420007plx.9.1693265609521; Mon, 28 Aug 2023 16:33:29 -0700 (PDT) Date: Mon, 28 Aug 2023 23:33:18 +0000 In-Reply-To: <20230828233319.340712-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230828233319.340712-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828233319.340712-5-yosryahmed@google.com> Subject: [PATCH v2 4/4] mm: memcg: use non-unified stats flushing for userspace reads From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 912D3140017 X-Stat-Signature: z7kix8of91h9z8bm7meioniho6y4n7f6 X-HE-Tag: 1693265610-775833 X-HE-Meta: U2FsdGVkX1+Z9bHZ1c8YMsoQCIYC3Bl+eUOIV5lnx2Y5SqtOyD08oizQcpdvzDvaV5/M6Mdhv5oKiBESimJRta4ilCHVLRx6mVZRvO9rauHVdIisd8VpNQ3eBl7Y86c8vREsSot5BWOaurRNCrZrDffzJQVmUDXPqoQ8nVKFWj+BZjWzP9brFL57mZSAz64sVD9DftWvFMfJNr8nvpriT7bPizGWdSsu9TfnegwqsYWTQOUeKI0cTqeSY/MtEbfvImBqc1mp9JMxRiiGd1sVbwMYni+Qzk65pLCHKMZVXRWLYZoqWMG3yg3lI/lvGcNU+Rvfy5YTlpuVk9XJFLi9yd9lr7IbChD4Smn/6Npxz1126B3VC3WsEcCxTOqk2EVriy8UQE3CzDiEvNonCu4m/EoaklbDbYjI+geDLYl8Qrgp7hiFpsCcSI4VaxeenqjSkxg0wiaPR7OWTrjRWzYoyPHSssIBClzo3MzzNeVxlqYXliTvhz49ScTzE8tDEXlUVYs9dfpHIqp0NpVCOzxexVq6MADv6QmC/2w5/HjLITvfPA+jeqx4V7DaQdr7PvedxpOtecRmeWtADtcrJcT3vGRss28vI0Of0M5R1XWkFTbW1CRRSP9bqjO2IN1GEkmCllJiQu7vCFwAUsRWZNjkBbHJWKPi1Cfq47WtORIYWdIZ+am6woRNkAL/Mf8XmFA9jwsOKI97sOO4kFX21ja4rKjLc9yVzdi9RZTNR2h0FPzmpPE7Fr4/s2TSfxuarOwlXirWYtJyz0WQegGIVj1uyZnxn/Far1FihnxufzMiT72s82wlczRkTTu9fjJDcZXnK5pgGixpuLY4jkX4KdBYMh7LI8V999WdkKrKyI9Ft8vHc9MNrsgyfdG0bAyV2drHJVY+taHyzRUdYtb0HG8fDvbFbAL32JpSfxb+Wn51CN+CfJeOQLXRdnDV9iA8OzRjCfMU0sL8C6uNwUVIWQC xzoAQxam Joi20rG4JOdlFLQFhC802V4Q23zsdFqseEmU4ayIHl9OWxlVq5zdQTQS8mbyoeChEz73qMZwu6mSKJHFKtDNu4ekBNrjfxm6ms/H9cnP2CZ0MehIxVhDrs91uMTzhLJTKNb0PBkNHgCeAOIhp7IW9JuBpgwo+qZ/d2U5GVgsawJp1DQe0R25v9uces4DQPu0mtlAXkxqAfQFkzgjOcH8a269DKFClZsIfd3IMwuUHpRqx7LaQ1LeoLf/v/H0xMNdxDW3vYfSfFmkfx5qRCG2Oyqcx8iaCOHBseVaySx25ojF3KcWWbmQe4ElawGbT89Kml842NKcjeX+UPYY4QhouAiec4tUlYLwKfx51jWgL/gCg9+aYc3y3Jf2ovrEclnBv/w2/jUE9NAkiAz0+2T5O7LLNoHYFT62tM7Od7uizXfYsqX1bV+5DjYpq/SyTEOD4VUcVovv1R42D9dplMWVAaiFtEj3Pf8/SkdYyKy0PCYMxU28Naz74nMZtpWjyMLTroZkXEbAXyYkRCRK5NK9CpRRg+8MVPzrRSEvLianz/ObeqFBtj8/TthGEB1tCsuidsGISKmOlb2UrQf81DDzMxSyAs/mHrRXq1aGD1eXUPSt1idUn1pkcWwOP9YSqy2mLHOOfuqBk9ylyqeXdW5O+jVKJn4NmhM8tkz0QEbhNrsGRupSEC1F7JuumsIULxfSh+1ecRyjo861cz7VyUDlRohZhkTIzFlT4KOtK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Unified flushing allows for great concurrency for paths that attempt to flush the stats, at the expense of potential staleness and a single flusher paying the extra cost of flushing the full tree. This tradeoff makes sense for in-kernel flushers that may observe high concurrency (e.g. reclaim, refault). For userspace readers, stale stats may be unexpected and problematic, especially when such stats are used for critical paths such as userspace OOM handling. Additionally, a userspace reader will occasionally pay the cost of flushing the entire hierarchy, which also causes problems in some cases [1]. Opt userspace reads out of unified flushing. This makes the cost of reading the stats more predictable (proportional to the size of the subtree), as well as the freshness of the stats. Since userspace readers are not expected to have similar concurrency to in-kernel flushers, serializing them among themselves and among in-kernel flushers should be okay. Note that this may make the worst case latency for reading stats worse. Flushers may give up cgroup_rstat_lock and sleep, causing the number of waiters for the spinlock theoritically unbounded. A reader may grab the lock and do some work, give it up, sleep, and then wait for a long time before acquiring it again and continuing. This is only possible if there is high concurrency among processes reading stats of different parts of the hierarchy, such that they are not helping each other out. Other stats interfaces such as cpu.stat have the same theoritical problem, so this is unlikely to be a problem in practice. If it is, we can introduce a mutex in the stats reading path to guard against concurrent readers competing for the lock. We have similar protection for unified flushing, except that concurrent flushers skip instead of waiting. An alternative is to remove flushing from the stats reading path completely, and rely on the periodic flusher. This should be accompanied by making the periodic flushing period tunable, and providing an interface for userspace to force a flush, following a similar model to /proc/vmstat. However, such a change will be hard to reverse if the implementation needs to be changed because: - The cost of reading stats will be very cheap and we won't be able to take that back easily. - There are user-visible interfaces involved. Hence, let's go with the change that's most reversible first. If problems arise, we can add a mutex in the stats reading path as described above, or follow the more user-visible approach. This was tested on a machine with 256 cpus by running a synthetic test The script that creates 50 top-level cgroups, each with 5 children (250 leaf cgroups). Each leaf cgroup has 10 processes running that allocate memory beyond the cgroup limit, invoking reclaim (which is an in-kernel unified flusher). Concurrently, one thread is spawned per-cgroup to read the stats every second (including root, top-level, and leaf cgroups -- so total 251 threads). No regressions were observed in the total running time; which means that non-unified userspace readers are not slowing down in-kernel unified flushers: Base (mm-unstable): real 0m18.228s user 0m9.463s sys 60m15.879s real 0m20.828s user 0m8.535s sys 70m12.364s real 0m19.789s user 0m9.177s sys 66m10.798s With this patch: real 0m19.632s user 0m8.608s sys 64m23.483s real 0m18.463s user 0m7.465s sys 60m34.089s real 0m20.309s user 0m7.754s sys 68m2.392s Additionally, the average latency for reading stats went down up to 8 times when reading stats of leaf cgroups in the script, as we only have to flush the cgroup(s) being read. [1]https://lore.kernel.org/lkml/CABWYdi0c6__rh-K7dcM_pkf9BJdTRtAU08M43KO9ME4-dsgfoQ@mail.gmail.com/ Signed-off-by: Yosry Ahmed --- mm/memcontrol.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f3716478bf4e..8bfb0e3395ce 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1607,7 +1607,7 @@ static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) * * Current memory state: */ - mem_cgroup_try_flush_stats(); + do_stats_flush(memcg); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { u64 size; @@ -4049,7 +4049,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) int nid; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_try_flush_stats(); + do_stats_flush(memcg); for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { seq_printf(m, "%s=%lu", stat->name, @@ -4124,7 +4124,7 @@ static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats)); - mem_cgroup_try_flush_stats(); + do_stats_flush(memcg); for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { unsigned long nr; @@ -4626,7 +4626,7 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - mem_cgroup_try_flush_stats(); + do_stats_flush(memcg); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); @@ -6641,7 +6641,7 @@ static int memory_numa_stat_show(struct seq_file *m, void *v) int i; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_try_flush_stats(); + do_stats_flush(memcg); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { int nid;