From patchwork Thu Aug 3 18:50:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13340776 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1367CEB64DD for ; Thu, 3 Aug 2023 18:50:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C617280294; Thu, 3 Aug 2023 14:50:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 675E728022C; Thu, 3 Aug 2023 14:50:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53DED280294; Thu, 3 Aug 2023 14:50:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 454A328022C for ; Thu, 3 Aug 2023 14:50:52 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id F00C7C071B for ; Thu, 3 Aug 2023 18:50:51 +0000 (UTC) X-FDA: 81083685102.23.A60A9F2 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf18.hostedemail.com (Postfix) with ESMTP id 3A8B01C001B for ; Thu, 3 Aug 2023 18:50:49 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=5lrHqege; spf=pass (imf18.hostedemail.com: domain of 3CffLZAoKCFQKAEDKw380z2AA270.yA8749GJ-886Hwy6.AD2@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3CffLZAoKCFQKAEDKw380z2AA270.yA8749GJ-886Hwy6.AD2@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691088650; a=rsa-sha256; cv=none; b=1dq13YESysZ3bCK5Aa2bhbTrgfNPSiKUuiWIgwhmTx73XBPB1Y092NWZeTgmn0Zfj5GOZS vbpsdfFP3zdPQlysQbauvDddd3mhXqwoqDZNNFMnrWfQ7SNuBN3zCLgdI8qEVIUZV9SP58 GvcAexxQ/b9Rz3xN3tgFPqm76z2bPhY= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=5lrHqege; spf=pass (imf18.hostedemail.com: domain of 3CffLZAoKCFQKAEDKw380z2AA270.yA8749GJ-886Hwy6.AD2@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3CffLZAoKCFQKAEDKw380z2AA270.yA8749GJ-886Hwy6.AD2@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691088650; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=1XyQ/ZegLbwguWS1UpeIGzNx959kwuq92WVqqIghPaA=; b=wghKe2YLe5veE+woj3c8NmqlUerXwFbyt76cHy6N3mtapVX2aXzGuhT9866rCNjKgcW0Za PnqKp7zLGnFStB7bbExyqfajwf8b3oXC36DaPUgwdmAoxKr9DwGkCYHi/4+u7fosWj2cxT 70Ftox0tMwVDRHQRz//3n+fnE8lGX3U= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5840614b13cso21299017b3.0 for ; Thu, 03 Aug 2023 11:50:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691088649; x=1691693449; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=1XyQ/ZegLbwguWS1UpeIGzNx959kwuq92WVqqIghPaA=; b=5lrHqege0J6Q+nTQjju/1hs3TARjmexNgoAE2A9oG9GF2NC2KD1hUJ7HaPG/gRMhkQ rRQefydgm0kytCsXqbAgd1S2i3d+MPxOQDcxsSH4Af3Cte4QEiM6ZJlmNkwNNVN7IIiH 1XEc6we68gQMUoCp/2JetDLXcbXQIHYLfl1LdUl+vosg84WXC8GWVXbgc6uyLZldxyeT PTOSMPGWfrM0L9cioRb7+Dil5J5+VfpN94tUq/Josdqo5hq3xuHtMIernY6tKkN8d8wr Z7EfeopWF6FEzvu6p1JCiglQluqndliVsPsd1kTtqooCDKQNUF1UFsMM0F3tAsVDZqyA ZR3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691088649; x=1691693449; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=1XyQ/ZegLbwguWS1UpeIGzNx959kwuq92WVqqIghPaA=; b=gWX2p6pI3j3M1D+DgSFeUZcTfFs2gKMzxRk73E5nDacTX9rvFF40t1epAZK44lRqJ8 3W818qZL1YZSZlacQlf4TtHLBfosjFe47vpRPMMUkOqCNMkDek2BOyaSeEHlVManYr8/ aBwtx/rrkL2CUuHdd0Ygm1OiKrxOrzMS1nZwb+WsmA2ktm+3tk5IW6DZmtRwEPjvZ4sW sh8FHHAigcauJVS++eCIniZCw3H3cHBsIO6mFNbKFtviM86Rt61iZ0nBuz2S12FAFFae 9VthiHWWB8gCmrGhiIQVBBcsJMB11VjyV1fP2A9XwlKlQiEAA6Qepw6uB0jtMr7ALUjU UDUQ== X-Gm-Message-State: ABy/qLbfcCB2JJtT8bbt8P++TazMzB0IYpbXY7SFUuyQzbq7gPenRvj5 mB2LWtsheZ8juqg9TDGtDEH+kTL51WuyedLX X-Google-Smtp-Source: APBJJlHI2nVKiol9et2npfTT5QKoKbQD2ArkaR13nsAGLsR8kw/O9V61X2Ty/DNHrqH+I1hNa2NJ+dRN6ofI4XIw X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a81:788e:0:b0:57a:6019:62aa with SMTP id t136-20020a81788e000000b0057a601962aamr263668ywc.5.1691088649209; Thu, 03 Aug 2023 11:50:49 -0700 (PDT) Date: Thu, 3 Aug 2023 18:50:46 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.585.gd2178a4bd4-goog Message-ID: <20230803185046.1385770-1-yosryahmed@google.com> Subject: [PATCH v4] mm: memcg: use rstat for non-hierarchical stats From: Yosry Ahmed To: Andrew Morton Cc: Michal Hocko , Johannes Weiner , Roman Gushchin , Shakeel Butt , Muchun Song , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 3A8B01C001B X-Stat-Signature: fdq7hr99g9jzdzi1gtydu8fqaqp336jp X-Rspam-User: X-HE-Tag: 1691088649-449567 X-HE-Meta: U2FsdGVkX1+Klcg2XFLtU9vSaGbJxhx1JDmmvqNd9SNSSh1QABziRbHg0NatN8NGn2HSNgs0WWrEX0WW2zM4O44EMQFydSAIPzt6xkEgqoDlh+ZmB+P/CIJdJB88TqzLV+gj9CrLHrxcYh1WzOKLfP9Qj96MZrWtcBFHwLRqn4osgL5rY42tunTRPGQGUeEnsN2vXMVoZd4vNx0kTQrhvJ4sn+K/DRXft84Bfl4cfTZdCCRrpeFdncJbYxW2OxXDiplZZB1zMXXV2ISoY53U1bh9m/ZhQM3BB3tqX4QQlt9Xzrzp4M8ML6dxNWHxKUl3STesBUeWrK8kltFfnZMmBBCY+atUOWuLjYkvMLr1kCPvlrpNZvA20F+cpUeNueHbPFhi/bYiVJ/mCaXzHkwvuDRqiIdEZK3RCCHV9JP4OD/HLDZAwNRk3wQRborCt1bbN/o4LCPp6piVFg8CjH/PYb0BR1VRpZhqrTdF4eBYwFUaVjYkFjC8QYnpudRIqurrsiwm5GUskd5zSgtq5UF1QkWY8pb+jr2vJHPBFVGJCyLPmxBXKM/TPJnGGMFBbv4OSn9w4y1p8BoY+NAUyRywdwMop01qSEAH8Etc9V5omnX6dbY+WpNejRFAjbUyDpB6FMnULQ61jxmuFeYE6B2fy9CLGyVWxjjOP8Ul9zF+tJwQBT6YeHNb+6OlbVNtUXfT8rUkzayY7YRZdAC2Yn4oPuBYE58URWvPCRI+fDftu0PHqHwI6FCDjdyrkO/Wy7RQ1zB2DpLOWMyomVg1V1aCRyBtk0reohPH9796QR+oNF2TePsb7ND5+ikUHAIcsJ/5OgkbpjWLGNncSqnnIj/64H6sdc8rCP/iouMR7jwCoDTre7IG/fPFwbbp0apl92VgxNeNHlSOSpNmx0Bnm2FRagtGDmAAyJsaviafHiUGQLiYqOIkEoMMzpZXltcNUaSKE7QF/LhxWPnReyZdY4l wkEcQ0Q9 yedJkZjq/ydjcBNpdhTl7CzRJv45BLrddC4pE1hQoUOJfyjMtNFNxb1GTD7z5A9UfNABI7ZV1kP3KbHP8pttVxdfffO5DyxUgcu8gl8OAj1KY/0YmDA67EKCuqSXF+cLfVNrDTwOv/H5Von+PV2EIb2dGxhu38CcpW1b3+q5lnnpPLDHx4P02m1dzPPgXsa/fh6Xtt64DtBQWbUAe47eXz+OaXZQPMs5+U5L/ROE7AVaaJ9Mrm7SKfFNG74+e1bUmbjFVjO6e0R58eAUgPzh/Z2wUclNU8w4QXZ2k5HSW7O1OnIaBUQPECCyTSJsqQFdk91ADiqvazF75HemNmnMGkjLBg0jBoc/MGIbrtORy/dMlbLf8/94g1dkXvkDi4d7WjavqgzFAw4CGBhN5ukA/Gh79A/OJTknFAbVCOF8m/cFVsDfPlCSGhglgMw4Efs7J86m1uNXDltXfrRfRPxrodPhP8dwBN3hCM6Y9H0/hGbet8x1RQOSg4+0xxdBZ1pOcUB0bzZw8wr+1HJyfUZ5NncNtLffe85uLECSi2m1fg1hZZstPRImE4SK4+3o77VtTC1ZqxeolVA5jj82DdLFLZltJxCv4dylXCt95X598TNX5P7Buul9qBI7oP+g1nk9nL0ZaeOgAC8jQGiyPeora1rdMmhSpPTHWaMtlx/mpRl2dpPxieGyKSOMzV7JWkzQKyPmImXF0e+J+mN1HK6kERhlWipqpK0/0whOrd1PF4eqmBNC0+T0cO57mC2UiTpPGrDN4iMoldc9ZziFXS7PfF6UKXe/TgU8mU6o/x7gf3Rdsq/7LdztHtQ4l8X4E4JbL1k8LSTmBESL1O+w= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, memcg uses rstat to maintain aggregated hierarchical stats. Counters are maintained for hierarchical stats at each memcg. Rstat tracks which cgroups have updates on which cpus to keep those counters fresh on the read-side. Non-hierarchical stats are currently not covered by rstat. Their per-cpu counters are summed up on every read, which is expensive. The original implementation did the same. At some point before rstat, non-hierarchical aggregated counters were introduced by commit a983b5ebee57 ("mm: memcontrol: fix excessive complexity in memory.stat reporting"). However, those counters were updated on the performance critical write-side, which caused regressions, so they were later removed by commit 815744d75152 ("mm: memcontrol: don't batch updates of local VM stats and events"). See [1] for more detailed history. Kernel versions in between a983b5ebee57 & 815744d75152 (a year and a half) enjoyed cheap reads of non-hierarchical stats, specifically on cgroup v1. When moving to more recent kernels, a performance regression for reading non-hierarchical stats is observed. Now that we have rstat, we know exactly which percpu counters have updates for each stat. We can maintain non-hierarchical counters again, making reads much more efficient, without affecting the performance critical write-side. Hence, add non-hierarchical (i.e local) counters for the stats, and extend rstat flushing to keep those up-to-date. A caveat is that we now need a stats flush before reading local/non-hierarchical stats through {memcg/lruvec}_page_state_local() or memcg_events_local(), where we previously only needed a flush to read hierarchical stats. Most contexts reading non-hierarchical stats are already doing a flush, add a flush to the only missing context in count_shadow_nodes(). With this patch, reading memory.stat from 1000 memcgs is 3x faster on a machine with 256 cpus on cgroup v1: # for i in $(seq 1000); do mkdir /sys/fs/cgroup/memory/cg$i; done # time cat /sys/fs/cgroup/memory/cg*/memory.stat > /dev/null real 0m0.125s user 0m0.005s sys 0m0.120s After: real 0m0.032s user 0m0.005s sys 0m0.027s To make sure there are no regressions on cgroup v2, I ran an artificial reclaim/refault stress test [2] that creates (NR_CPUS * 2) cgroups, assigns them limits, runs a worker process in each cgroup that allocates tmpfs memory equal to quadruple the limit (to invoke reclaim continuously), and then reads back the entire file (to invoke refaults). All workers are run in parallel, and zram is used as a swapping backend. Both reclaim and refault have conditional stats flushing. I ran this on a machine with 112 cpus, once on mm-unstable, and once on mm-unstable with this patch reverted. (1) A few runs without this patch: # time ./stress_reclaim_refault.sh real 0m9.949s user 0m0.496s sys 14m44.974s # time ./stress_reclaim_refault.sh real 0m10.049s user 0m0.486s sys 14m55.791s # time ./stress_reclaim_refault.sh real 0m9.984s user 0m0.481s sys 14m53.841s (2) A few runs with this patch: # time ./stress_reclaim_refault.sh real 0m9.885s user 0m0.486s sys 14m48.753s # time ./stress_reclaim_refault.sh real 0m9.903s user 0m0.495s sys 14m48.339s # time ./stress_reclaim_refault.sh real 0m9.861s user 0m0.507s sys 14m49.317s No regressions are observed with this patch. There is actually a very slight improvement. If I have to guess, maybe it's because we avoid the percpu loop in count_shadow_nodes() when calling lruvec_page_state_local(), but I could not prove this using perf, it's probably in the noise. [1] https://lore.kernel.org/lkml/20230725201811.GA1231514@cmpxchg.org/ [2] https://lore.kernel.org/lkml/CAJD7tkb17x=qwoO37uxyYXLEUVp15BQKR+Xfh7Sg9Hx-wTQ_=w@mail.gmail.com/ Link: https://lkml.kernel.org/r/20230726153223.821757-2-yosryahmed@google.com Signed-off-by: Yosry Ahmed Acked-by: Johannes Weiner Acked-by: Roman Gushchin Acked-by: Michal Hocko --- v3 -> v4: - Added A-b from Michal Hocko (Thanks!). - Amended the changelog with additional testing for regressions (as requested by Michal Hocko). v3: https://lore.kernel.org/lkml/20230726153223.821757-2-yosryahmed@google.com/ --- include/linux/memcontrol.h | 7 ++-- mm/memcontrol.c | 67 +++++++++++++++++++++----------------- mm/workingset.c | 1 + 3 files changed, 43 insertions(+), 32 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 058fb748e128..e765d1ff9cbb 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -111,6 +111,9 @@ struct lruvec_stats { /* Aggregated (CPU and subtree) state */ long state[NR_VM_NODE_STAT_ITEMS]; + /* Non-hierarchical (CPU aggregated) state */ + long state_local[NR_VM_NODE_STAT_ITEMS]; + /* Pending child counts during tree propagation */ long state_pending[NR_VM_NODE_STAT_ITEMS]; }; @@ -1019,14 +1022,12 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, { struct mem_cgroup_per_node *pn; long x = 0; - int cpu; if (mem_cgroup_disabled()) return node_page_state(lruvec_pgdat(lruvec), idx); pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - for_each_possible_cpu(cpu) - x += per_cpu(pn->lruvec_stats_percpu->state[idx], cpu); + x = READ_ONCE(pn->lruvec_stats.state_local[idx]); #ifdef CONFIG_SMP if (x < 0) x = 0; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 062d925336cb..35d7e66ab032 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -742,6 +742,10 @@ struct memcg_vmstats { long state[MEMCG_NR_STAT]; unsigned long events[NR_MEMCG_EVENTS]; + /* Non-hierarchical (CPU aggregated) page state & events */ + long state_local[MEMCG_NR_STAT]; + unsigned long events_local[NR_MEMCG_EVENTS]; + /* Pending child counts during tree propagation */ long state_pending[MEMCG_NR_STAT]; unsigned long events_pending[NR_MEMCG_EVENTS]; @@ -775,11 +779,8 @@ void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val) /* idx can be of type enum memcg_stat_item or node_stat_item. */ static unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) { - long x = 0; - int cpu; + long x = READ_ONCE(memcg->vmstats->state_local[idx]); - for_each_possible_cpu(cpu) - x += per_cpu(memcg->vmstats_percpu->state[idx], cpu); #ifdef CONFIG_SMP if (x < 0) x = 0; @@ -926,16 +927,12 @@ static unsigned long memcg_events(struct mem_cgroup *memcg, int event) static unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) { - long x = 0; - int cpu; int index = memcg_events_index(event); if (index < 0) return 0; - for_each_possible_cpu(cpu) - x += per_cpu(memcg->vmstats_percpu->events[index], cpu); - return x; + return READ_ONCE(memcg->vmstats->events_local[index]); } static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, @@ -5517,7 +5514,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) struct mem_cgroup *memcg = mem_cgroup_from_css(css); struct mem_cgroup *parent = parent_mem_cgroup(memcg); struct memcg_vmstats_percpu *statc; - long delta, v; + long delta, delta_cpu, v; int i, nid; statc = per_cpu_ptr(memcg->vmstats_percpu, cpu); @@ -5533,19 +5530,23 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) memcg->vmstats->state_pending[i] = 0; /* Add CPU changes on this level since the last flush */ + delta_cpu = 0; v = READ_ONCE(statc->state[i]); if (v != statc->state_prev[i]) { - delta += v - statc->state_prev[i]; + delta_cpu = v - statc->state_prev[i]; + delta += delta_cpu; statc->state_prev[i] = v; } - if (!delta) - continue; - /* Aggregate counts on this level and propagate upwards */ - memcg->vmstats->state[i] += delta; - if (parent) - parent->vmstats->state_pending[i] += delta; + if (delta_cpu) + memcg->vmstats->state_local[i] += delta_cpu; + + if (delta) { + memcg->vmstats->state[i] += delta; + if (parent) + parent->vmstats->state_pending[i] += delta; + } } for (i = 0; i < NR_MEMCG_EVENTS; i++) { @@ -5553,18 +5554,22 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) if (delta) memcg->vmstats->events_pending[i] = 0; + delta_cpu = 0; v = READ_ONCE(statc->events[i]); if (v != statc->events_prev[i]) { - delta += v - statc->events_prev[i]; + delta_cpu = v - statc->events_prev[i]; + delta += delta_cpu; statc->events_prev[i] = v; } - if (!delta) - continue; + if (delta_cpu) + memcg->vmstats->events_local[i] += delta_cpu; - memcg->vmstats->events[i] += delta; - if (parent) - parent->vmstats->events_pending[i] += delta; + if (delta) { + memcg->vmstats->events[i] += delta; + if (parent) + parent->vmstats->events_pending[i] += delta; + } } for_each_node_state(nid, N_MEMORY) { @@ -5582,18 +5587,22 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) if (delta) pn->lruvec_stats.state_pending[i] = 0; + delta_cpu = 0; v = READ_ONCE(lstatc->state[i]); if (v != lstatc->state_prev[i]) { - delta += v - lstatc->state_prev[i]; + delta_cpu = v - lstatc->state_prev[i]; + delta += delta_cpu; lstatc->state_prev[i] = v; } - if (!delta) - continue; + if (delta_cpu) + pn->lruvec_stats.state_local[i] += delta_cpu; - pn->lruvec_stats.state[i] += delta; - if (ppn) - ppn->lruvec_stats.state_pending[i] += delta; + if (delta) { + pn->lruvec_stats.state[i] += delta; + if (ppn) + ppn->lruvec_stats.state_pending[i] += delta; + } } } } diff --git a/mm/workingset.c b/mm/workingset.c index 4686ae363000..da58a26d0d4d 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -664,6 +664,7 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker, struct lruvec *lruvec; int i; + mem_cgroup_flush_stats(); lruvec = mem_cgroup_lruvec(sc->memcg, NODE_DATA(sc->nid)); for (pages = 0, i = 0; i < NR_LRU_LISTS; i++) pages += lruvec_page_state_local(lruvec,