From patchwork Wed Jul 26 15:32:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13328301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E463C001B0 for ; Wed, 26 Jul 2023 15:32:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD22E6B0072; Wed, 26 Jul 2023 11:32:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B822F6B0074; Wed, 26 Jul 2023 11:32:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A4B076B0078; Wed, 26 Jul 2023 11:32:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8AA0E6B0072 for ; Wed, 26 Jul 2023 11:32:33 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 34886120F66 for ; Wed, 26 Jul 2023 15:32:33 +0000 (UTC) X-FDA: 81054154986.09.8EA9A17 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf27.hostedemail.com (Postfix) with ESMTP id E815A4000C for ; Wed, 26 Jul 2023 15:32:29 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=lGrx2OlM; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 3jDzBZAoKCDcrhlkrTafXWZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--yosryahmed.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3jDzBZAoKCDcrhlkrTafXWZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690385550; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IS3cSpJD3j6ulMLRBZ9gZy0A5tXWGEsHEODupeAki5g=; b=Fmy4+8vHu0/KmFJDflDdNuL/fzqw9by17Xh9Nn3snnQlAtXKPtlbvECGLIIlqCrtC7GnzL mqYhleT6b8sU68OZKV1Q7DwTCrGfhK+gF3jkgc+yjkLA7ulsgrcLp1uAgAwMbmLdAmBscn 7yKASiT3xoOF6JX2s1eMevaBb2AB540= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=lGrx2OlM; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 3jDzBZAoKCDcrhlkrTafXWZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--yosryahmed.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3jDzBZAoKCDcrhlkrTafXWZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690385550; a=rsa-sha256; cv=none; b=S+5OZ2l7XT8y6cpW1dmZgeHFNRCeuC1p+HBYxHHAJ7tJ852mjO4y77OxuQOlsNsz7S2pS1 ttk7k9NZ4ol/AHm28UqOSjuHo/Bxb1VjSlQ/0zPOGUGtBqdpnbXFjn8pA2mQv/HUiLLD0j 7lFXS87Pnv6wvLHK1FryYvMmjJj1C3o= Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-55c79a5565aso3298523a12.3 for ; Wed, 26 Jul 2023 08:32:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690385548; x=1690990348; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IS3cSpJD3j6ulMLRBZ9gZy0A5tXWGEsHEODupeAki5g=; b=lGrx2OlMN5uaelIDW6jt45bRnkk4+1lMaY5s8DWR98+qigtB0H1XUg9D48/l1wsWoa MHZluRtLsrga2YkZCNL0ea+2D/F8Y7FqLK8R1Frs5T/mT+lB3CFmO+1ZKoRPpjP+AETu qfJo/HRl4Bl0bJmqTPUPTtFy4ol0tVMMRqBMGu2CU7GooPCWatnwAbljFZrOd/nR1FGi fRmrdrBVgJamIwFAHoFgYexvqgB/4Ah3pV0MkLiW/24P5gzffngDNJWFvR0yadlbHsYY XS4OSZxKJZYHJMr+7CfJu06VdJuIe4UsMCJjmR/hbVyYlIN17HU/t898Mjwajet8zBVp oTMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690385548; x=1690990348; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IS3cSpJD3j6ulMLRBZ9gZy0A5tXWGEsHEODupeAki5g=; b=RQZ4S3Jh0dgRW9na5jMDrMLALwyqeqxIhe202OZDA7qSH5eK0AZO800ih7avKZAuMy ysaRvjTPzkRyB8VPGYhLmD1PGmDuhrWD7XFy75XssIU9g9tb5K08vGXWSkKp0PoDLpJd t3iAnXSq5f5aRZZ9ulLH9JAFo1mFmSNbW0NLFtelhGJdTIzUQKfOjOgPKLsY2wPB21Md +Lvh5pNzHEsryepuzMpFLpLv0jPZbNvrGUGc280FFdPN218KveVEMFyw4I2bVwpbP8F4 XlqRtnJNacvH23emwLGcQCB7J3AaXMU0J821WHMYo+F8jM6dgFKgV8hWknnHfFZAElIy 7kcg== X-Gm-Message-State: ABy/qLYU3rehZPIB6C8lPJ3V9TtFDzwmIicpINOtJhpB6bOOyGpztLlv gr0/vUICEOGmqMrmnOfb8tG3a06Um1hK1Yut X-Google-Smtp-Source: APBJJlEvcyRPqrsqMfMFVG6L7B2adf5xvtqnrPJBmZDGeMXGimjLr5H3BE/fx3eBaIUlKjI38E2NZv8M6BdgFF6r X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:7d1a:0:b0:55b:4887:9e99 with SMTP id y26-20020a637d1a000000b0055b48879e99mr10753pgc.1.1690385548324; Wed, 26 Jul 2023 08:32:28 -0700 (PDT) Date: Wed, 26 Jul 2023 15:32:23 +0000 In-Reply-To: <20230726153223.821757-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230726153223.821757-1-yosryahmed@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230726153223.821757-2-yosryahmed@google.com> Subject: [PATCH v3] mm: memcg: use rstat for non-hierarchical stats From: Yosry Ahmed To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: E815A4000C X-Stat-Signature: 43ybq9bzxby6swwn5eiuf95spxokgm6n X-Rspam-User: X-HE-Tag: 1690385549-607868 X-HE-Meta: U2FsdGVkX18DqvQxrX37oMJkBDAdaZbD+LMK8THQC98uo74fpW+5tmpTJpdAqT1sVKjKW99z9CHQ8cJcdFHxf+u1nvHBAVSjIwOJpCs+AM8sZxOqw7ThJsWwNE81l59UkbMKzntkKL5KRaA/8LP6ynh013N2SXpB++JnYAA3i3ZTe2T6zHHZxtIDPmgLQVK4DnVWD0LrgNKIH1KQIP2Vey+KO1N3QPFsz4Pz2D6yYi8OtzIVZWVsDdNJdAs0G2SIWrMeCR+PXlzTUgCd2QaoEInyNzBnG+XTBOMSjNe66zirZ5A1LRGVcLoyOfJJLGMJH6VGhOMNJJQZfcK98XNzAE/0jmgdxE83ER1H8/lm6RVi4vNcbDmE7cCL+wEobeOcIVmyn9fpsByTWine0XxkMoEuNM7NWoduHC1WZ6OZILKtyKZOElCG5uh84X/6wLWcE6C8UkdhQdYEFHOKXmMaD6DulVt3SZGz3C/F+A+JoWDt8gf5e54QIMvBfRLsRU2BrKzECZwDDd/01mIEmFPNinSgz7hAERNqJx1+ZiW1Hpoz/jEAqYGRkiDyMtv69Oa2xDvFAX/7E4y0muFXDl7NE/TV7u0YC5RlnlHc0sKnIZRRLgdZ2nivWliN+/K+XdKBpE/xDmMXfM2vFPR/8QQXgJrXHzHIHqr7l/B9KfFWgf6Rt33yCjnzCr6ZBA0YFxKuProarLezUgSMmnnnm4A+oG15Rax+HKA8MQZV1YHAzPWe/Y+QYEGrPp2co88c7KG8IECM+xw0n4SL60jH6kwTa4BaJ4b8dZxZH6VYJePAZUqkCmNtiXGuYE+H0RD10ZHS82JhDTe8Xh6sghXwdJ2IoJDwIanaAN4RuCxB1x5yMFUsrvWQolYEsJWDTJHPGA6g50kcqMOTkCSwgRfOKiIyiCLLEu+auGsVPiAN440ooFAXHZ2dBYAZloZjNw5B0j4ikg+AzUKDfNV63NICSJ+ cGWPk7Yi XJOpxCKNgQa0AzAsqXiCzfq5H4uzASRCbskl5uPm6023JHjNQ0HsATmawjZCvuW6WJ3geKZ13HbngTzejUYHVHPbnfNS6jj7IEsnEQrDOK9T6oxOnhKnIfCeONpnOaRlqY9GfbAG4TuKZhg4a3ntaPHRa5LVepCof64a9iGYCQ6kqCnNPb0mzo03TVuDTG34YdgXQJHYrEIo+qImrU4CUwK/E7N8awWT6JOi5OaeNDtJiq4oVr/rmMW70U9PbYwnhx1uf/O7U3DXWvM7jAi6coRtgZnkCbmiPcJ0EAtwkbLishvfqZaE2c4A/Gk1jTqYhM7i/sxVJsacCG29xHD1nfS98mz8Dgbtj0GcRGEWhbp2/NTHE7kAj1uKB8g+OmfPAQNtwjM/45c3kaiAwzNC48ncco7t2ZnNhPH+XQ20hr4pyd6+EPU7SArkX17IWXZD6nZcSdM3D++BTzSVmGULVSOAmhhszoLnt3DqXneaGj0DkjYP1LrNWDzqAiuCmyqX55LVak7gJs0mEc5l0Nd2iU5vEzxsgK+IXYn5pvvXRPQQQRTd82kAhJT5md0ChL/vJwMxeapIy9EXOBRoI3QkSWVLuMWEpnmiepl7XsHmpAH8Q39PLIwpTG+zSECRYDH+qgqxAE42H9lc9H/nh365qYm//jddFS3Hl9k8EiV47Bom35s0d1Uf7QQIPWH+fsOmnZRnUCu5WrnUMA3hbxr9p5mzfeejWt+GZnD8HYxlLKXogqZf+MlWRhU4oJRfKMeq29UAp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, memcg uses rstat to maintain aggregated hierarchical stats. Counters are maintained for hierarchical stats at each memcg. Rstat tracks which cgroups have updates on which cpus to keep those counters fresh on the read-side. Non-hierarchical stats are currently not covered by rstat. Their per-cpu counters are summed up on every read, which is expensive. The original implementation did the same. At some point before rstat, non-hierarchical aggregated counters were introduced by commit a983b5ebee57 ("mm: memcontrol: fix excessive complexity in memory.stat reporting"). However, those counters were updated on the performance critical write-side, which caused regressions, so they were later removed by commit 815744d75152 ("mm: memcontrol: don't batch updates of local VM stats and events"). See [1] for more detailed history. Kernel versions in between a983b5ebee57 & 815744d75152 (a year and a half) enjoyed cheap reads of non-hierarchical stats, specifically on cgroup v1. When moving to more recent kernels, a performance regression for reading non-hierarchical stats is observed. Now that we have rstat, we know exactly which percpu counters have updates for each stat. We can maintain non-hierarchical counters again, making reads much more efficient, without affecting the performance critical write-side. Hence, add non-hierarchical (i.e local) counters for the stats, and extend rstat flushing to keep those up-to-date. A caveat is that we now need a stats flush before reading local/non-hierarchical stats through {memcg/lruvec}_page_state_local() or memcg_events_local(), where we previously only needed a flush to read hierarchical stats. Most contexts reading non-hierarchical stats are already doing a flush, add a flush to the only missing context in count_shadow_nodes(). With this patch, reading memory.stat from 1000 memcgs is 3x faster on a machine with 256 cpus on cgroup v1: # for i in $(seq 1000); do mkdir /sys/fs/cgroup/memory/cg$i; done # time cat /dev/cgroup/memory/cg*/memory.stat > /dev/null real 0m0.125s user 0m0.005s sys 0m0.120s After: real 0m0.032s user 0m0.005s sys 0m0.027s [1]https://lore.kernel.org/lkml/20230725201811.GA1231514@cmpxchg.org/ Signed-off-by: Yosry Ahmed Acked-by: Johannes Weiner Acked-by: Roman Gushchin Acked-by: Michal Hocko --- v2 -> v3: - Commit log fixes (thanks Roman & Johannes). v2: https://lore.kernel.org/lkml/20230726002904.655377-2-yosryahmed@google.com/ --- include/linux/memcontrol.h | 7 ++-- mm/memcontrol.c | 67 +++++++++++++++++++++----------------- mm/workingset.c | 1 + 3 files changed, 43 insertions(+), 32 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 5818af8eca5a..a9f2861a57a5 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -112,6 +112,9 @@ struct lruvec_stats { /* Aggregated (CPU and subtree) state */ long state[NR_VM_NODE_STAT_ITEMS]; + /* Non-hierarchical (CPU aggregated) state */ + long state_local[NR_VM_NODE_STAT_ITEMS]; + /* Pending child counts during tree propagation */ long state_pending[NR_VM_NODE_STAT_ITEMS]; }; @@ -1020,14 +1023,12 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, { struct mem_cgroup_per_node *pn; long x = 0; - int cpu; if (mem_cgroup_disabled()) return node_page_state(lruvec_pgdat(lruvec), idx); pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - for_each_possible_cpu(cpu) - x += per_cpu(pn->lruvec_stats_percpu->state[idx], cpu); + x = READ_ONCE(pn->lruvec_stats.state_local[idx]); #ifdef CONFIG_SMP if (x < 0) x = 0; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e8ca4bdcb03c..50f8035e998a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -742,6 +742,10 @@ struct memcg_vmstats { long state[MEMCG_NR_STAT]; unsigned long events[NR_MEMCG_EVENTS]; + /* Non-hierarchical (CPU aggregated) page state & events */ + long state_local[MEMCG_NR_STAT]; + unsigned long events_local[NR_MEMCG_EVENTS]; + /* Pending child counts during tree propagation */ long state_pending[MEMCG_NR_STAT]; unsigned long events_pending[NR_MEMCG_EVENTS]; @@ -775,11 +779,8 @@ void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val) /* idx can be of type enum memcg_stat_item or node_stat_item. */ static unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) { - long x = 0; - int cpu; + long x = READ_ONCE(memcg->vmstats->state_local[idx]); - for_each_possible_cpu(cpu) - x += per_cpu(memcg->vmstats_percpu->state[idx], cpu); #ifdef CONFIG_SMP if (x < 0) x = 0; @@ -926,16 +927,12 @@ static unsigned long memcg_events(struct mem_cgroup *memcg, int event) static unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) { - long x = 0; - int cpu; int index = memcg_events_index(event); if (index < 0) return 0; - for_each_possible_cpu(cpu) - x += per_cpu(memcg->vmstats_percpu->events[index], cpu); - return x; + return READ_ONCE(memcg->vmstats->events_local[index]); } static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, @@ -5526,7 +5523,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) struct mem_cgroup *memcg = mem_cgroup_from_css(css); struct mem_cgroup *parent = parent_mem_cgroup(memcg); struct memcg_vmstats_percpu *statc; - long delta, v; + long delta, delta_cpu, v; int i, nid; statc = per_cpu_ptr(memcg->vmstats_percpu, cpu); @@ -5542,19 +5539,23 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) memcg->vmstats->state_pending[i] = 0; /* Add CPU changes on this level since the last flush */ + delta_cpu = 0; v = READ_ONCE(statc->state[i]); if (v != statc->state_prev[i]) { - delta += v - statc->state_prev[i]; + delta_cpu = v - statc->state_prev[i]; + delta += delta_cpu; statc->state_prev[i] = v; } - if (!delta) - continue; - /* Aggregate counts on this level and propagate upwards */ - memcg->vmstats->state[i] += delta; - if (parent) - parent->vmstats->state_pending[i] += delta; + if (delta_cpu) + memcg->vmstats->state_local[i] += delta_cpu; + + if (delta) { + memcg->vmstats->state[i] += delta; + if (parent) + parent->vmstats->state_pending[i] += delta; + } } for (i = 0; i < NR_MEMCG_EVENTS; i++) { @@ -5562,18 +5563,22 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) if (delta) memcg->vmstats->events_pending[i] = 0; + delta_cpu = 0; v = READ_ONCE(statc->events[i]); if (v != statc->events_prev[i]) { - delta += v - statc->events_prev[i]; + delta_cpu = v - statc->events_prev[i]; + delta += delta_cpu; statc->events_prev[i] = v; } - if (!delta) - continue; + if (delta_cpu) + memcg->vmstats->events_local[i] += delta_cpu; - memcg->vmstats->events[i] += delta; - if (parent) - parent->vmstats->events_pending[i] += delta; + if (delta) { + memcg->vmstats->events[i] += delta; + if (parent) + parent->vmstats->events_pending[i] += delta; + } } for_each_node_state(nid, N_MEMORY) { @@ -5591,18 +5596,22 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) if (delta) pn->lruvec_stats.state_pending[i] = 0; + delta_cpu = 0; v = READ_ONCE(lstatc->state[i]); if (v != lstatc->state_prev[i]) { - delta += v - lstatc->state_prev[i]; + delta_cpu = v - lstatc->state_prev[i]; + delta += delta_cpu; lstatc->state_prev[i] = v; } - if (!delta) - continue; + if (delta_cpu) + pn->lruvec_stats.state_local[i] += delta_cpu; - pn->lruvec_stats.state[i] += delta; - if (ppn) - ppn->lruvec_stats.state_pending[i] += delta; + if (delta) { + pn->lruvec_stats.state[i] += delta; + if (ppn) + ppn->lruvec_stats.state_pending[i] += delta; + } } } } diff --git a/mm/workingset.c b/mm/workingset.c index 4686ae363000..da58a26d0d4d 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -664,6 +664,7 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker, struct lruvec *lruvec; int i; + mem_cgroup_flush_stats(); lruvec = mem_cgroup_lruvec(sc->memcg, NODE_DATA(sc->nid)); for (pages = 0, i = 0; i < NR_LRU_LISTS; i++) pages += lruvec_page_state_local(lruvec,