From patchwork Tue Oct 10 03:21:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13414777 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F238CCD68EB for ; Tue, 10 Oct 2023 03:21:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 355396B00BF; Mon, 9 Oct 2023 23:21:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2DA536B00C5; Mon, 9 Oct 2023 23:21:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17B8D6B00CC; Mon, 9 Oct 2023 23:21:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id EFF906B00BF for ; Mon, 9 Oct 2023 23:21:24 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C2FE280525 for ; Tue, 10 Oct 2023 03:21:24 +0000 (UTC) X-FDA: 81328101288.07.AD2430A Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf22.hostedemail.com (Postfix) with ESMTP id 0C4EAC0022 for ; Tue, 10 Oct 2023 03:21:22 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=l4bpwvQM; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3MsMkZQoKCHsxnrqxZgldcfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3MsMkZQoKCHsxnrqxZgldcfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696908083; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Txx2Z/24+3F8Cj+NRtOMU7JL27MtRQwWcbI2dA3u7/E=; b=3vlbwFYonAdNu34ei8R6OCMiXGQuiLFDdr8ZpPJ8UHYIkwN/LIsj6S3Mr19OB8US0NvZ4N l1/VUm2fgFbmz8dfNnHqc5y38ezYdmJyoFm2qDceIw7hK0Vt+gP9Q2xypWXAcz7DX3YDqw R9apCYS9Sj+ChsXDDrb1TakMPug/6ZA= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=l4bpwvQM; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3MsMkZQoKCHsxnrqxZgldcfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3MsMkZQoKCHsxnrqxZgldcfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696908083; a=rsa-sha256; cv=none; b=JdMPZR9PDKNrbYHk3k9kg2fLk81nKUKh7RQOu05osgsh++aqmusK5DspDsT3Oujds8+oN3 0t5FTMb8hoH+ladfgN8+NF5uiO1ekXGFDQUmUCwN1R12BtOtjFTJhf5cw55ZJZSfj1rrJZ s79dnVc70CVFHKybzmK6eZyaG5mYqaM= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-c647150c254so4608765276.1 for ; Mon, 09 Oct 2023 20:21:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696908082; x=1697512882; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Txx2Z/24+3F8Cj+NRtOMU7JL27MtRQwWcbI2dA3u7/E=; b=l4bpwvQM7b0PWBjCQ+KrqegPePcY2JHTL66TNjGJDpXbQF25Q6kxVxQFuyJBXQNn5R xKNdIeb6O2ijJGO54NSkdqwNGFm5bC0NEY9Lghi85KzssKLi/My2mukPiFYPLFKBt12J 21rfzLNkibFGqPopX0cMyw0cX/vAX7i+FR8e0DVeQ/lQMZ18kSm6i9k/bCNS9m3On4GO je0/hkjtYXpoF6tRoT32UIIVG34XWpdyVKKkjYJ0lEBCXU5KctzhJ5fLPUwKCbjrlvmy jWYhvJphw7yMLyIpyCCVXq3cwZpQ55Pg4j1AF22jqjyRkCuk69NDAofq6k3zAmy5iXAm cL4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696908082; x=1697512882; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Txx2Z/24+3F8Cj+NRtOMU7JL27MtRQwWcbI2dA3u7/E=; b=qm1N7KUggLHf6weFjgOn0Pc59qXTYKD5ubgGHntJWQZv4Iyn7aifS7vEm8OYwuiG0/ NgQjqhNZvRAbHZTGpv2h7xx4xhfvutgYR0ZgXEWPp8T4ccIP9wGUAw9TC4FhuBTK8bQY 6cNDpf/qYs5p4xkOvYnuEyR8+Slzey+fTz35PVDCVgg/4JtV7TkO4CmsCJm3ofWAd9Il m7iswxhyUtrsmW589eyyvM/Jyme+Zxzk4JZHaQuPV0Rv/vRWfRERg3xK1TcLuq4ZOCWm Xwpzy2QE7g3RRYDHatOwR4YmM7n7vUKrdruZFqKN63Pz1fyl99nOsE7ITMR0e8/lZz9U fwxw== X-Gm-Message-State: AOJu0Yw6+ydo+e6ks81dMpK+B7G6/+v+K7K+99Ta/oFCB+H4HMiui8iL aOAL0AyPmjkfLZA/gUy7FLi7+s4nMAMr2ErM X-Google-Smtp-Source: AGHT+IHSbb76MuDQgDE4hYiCssfBhZ3erWto8iacOIEWAeUiJ3Vxcha6qRANQmltbAbW17YqbUuNXt2O9PobRH8R X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a25:824f:0:b0:d9a:66eb:b516 with SMTP id d15-20020a25824f000000b00d9a66ebb516mr1411ybn.6.1696908082104; Mon, 09 Oct 2023 20:21:22 -0700 (PDT) Date: Tue, 10 Oct 2023 03:21:12 +0000 In-Reply-To: <20231010032117.1577496-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20231010032117.1577496-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.609.gbb76f46606-goog Message-ID: <20231010032117.1577496-2-yosryahmed@google.com> Subject: [PATCH v2 1/5] mm: memcg: change flush_next_time to flush_last_time From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , kernel-team@cloudflare.com, Wei Xu , Greg Thelen , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 0C4EAC0022 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: u36g4cprcatirxc31oyf5tjoh64paecj X-HE-Tag: 1696908082-924051 X-HE-Meta: U2FsdGVkX19RuhDSU9HtXzSSuxmWg/ZDTOFQK5LRx+vg+9lPxe8CH7g2wj2wTIc5JCV0UIPGL1N06NqjBYLm9Jsje6JPth8beVMYpCCJXwCzXlWNzXYnjMge7um70YW2u7eO+PYNkruijDfdL5tyUz9Rzu4Tc545OAdaoD1SopFXBLcHP9E6q6Okkrek18rfBXoiAj5sNYehe9lDi4n68UTRNWkGeOOPeSPh0CVBigWQnGf09+bGN/nIKvlHvLOJ8bLYFMuiQvS6NnGcGHiSBpnWFlzl3GPq1PgB68wqWiuU2jhz3P0HIf+DYho4I321KLiPJl8uQOQ1Si9p8kGCg+bOmd5+K2aAiJdCBNTWvbzAtV8svnpJpF48bpMXGXDTbsmfk7QokKt3B1gHWM8/A0QUyYjEDGzh3TDlSonj7eon4FE2OuvH/qbFIAL1O3iYE4bRCZEbSz3TZhGbYoHmSGA60uGnguGY+AjDtYMmRlqUXV46zACq53x8IBrr8Etqgd2tMhSUikYlUQgqjqrAHrkN9mS9UuOAVxNttSNiyyvYgvjZmBoNqsy2dGDqhKfaDngktQ9pp0/tFcTYJDS490rd3Jw900bbB4i/QdTl1j4UVIN7BxnwkqWyLB1z8TbLO3Y1WSuQAlNlfTBrZF0gdKkCcjtzB3Ak//aPAsgjUFKop3VFVjlo6zvULpgKZQh3ZbGGByb5eyxdlW1phGlwy5SlnU0mw5MxjGCo9hfiPabjN/hU0aYRJXRKc7hJBnxnH6qzzPQzDPAa70FQGtnm3rm4og0VWv49H0RCQypym+NOQf+eX6mvXhjbHggJaJttzOmQkO7QfEyhe69S65FiyZtXAdm6j+G0MTpqVjLUmwA53izYmzZmHAwjw1oj567noXpS+4NS8hvBrEGrTNRhqUj5J5/xjr9Tm8YKUv0pe6p9i83PWOVklO23tAhK53K0t6JtFm1KNzzn3z62q2V 9lHN0H7g En7C9eMZReKxL44wzZMu+QP468oBF87tUqlneODlHBGLX4c9KPZ2oGUpNdkXL9S72NYCz5W+fFY59+ekbVlEdPi0fha9iWgSKPM0rEiH4RaB3JVKAfdA+IV5IJjkuOtXEnSMRTlBX6aZopCXyF3iYHSBR7pIxCTxpV14ATNkbNxMnO+pRW3juCQAXcbq7a5aj27L0aOtSiyndtpC8BpQfaa7b8hPDHdEjhBcxjx1n9MZozcP09iKzm+v6DYDIaqgMhyaXGNISdjCtzU6RJcih0a9vPzsBylGXNfe7SfxE70nJ+fG3PcPvJ6kQwbBrxz8Krh6wk2XpMhNQYyM96io+2eMTzEIO1LaWBWeAEfRNWW7Mu8o7NBsOVsI0JWCNM9mbelKl/Lz+PCgMV5DfKlUExZ4wIRa+wbVPcS0CQPDx/HwV+/uITLn0/ms7L0tBGRn0YPLyV0Hczz8QuvxMjw4+MsLOOv9QBLxMYTYQJ3BcERabk9ExjRmUbrY8C16mOOHeq+UJTVdOqbhK3IKy3vZaxO0yykFwW0UXpOPWQ0UjJ0V6GItcbZiwMLd0n7YtrP/Hr10KwqRHCXJezmLn9Ec3ds/mmV1te1cGaPrTHWjqMrsM+EN8COCy6fnANDeGEtmvd61ReLMDUatbV0koWQvrBdINUe6j116uzbwbMckp39E2QW4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: flush_next_time is an inaccurate name. It's not the next time that periodic flushing will happen, it's rather the next time that ratelimited flushing can happen if the periodic flusher is late. Simplify its semantics by just storing the timestamp of the last flush instead, flush_last_time. Move the 2*FLUSH_TIME addition to mem_cgroup_flush_stats_ratelimited(), and add a comment explaining it. This way, all the ratelimiting semantics live in one place. No functional change intended. Signed-off-by: Yosry Ahmed --- mm/memcontrol.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2fb30abaf267..4a194fcc9533 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -590,7 +590,7 @@ static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); static DEFINE_PER_CPU(unsigned int, stats_updates); static atomic_t stats_flush_ongoing = ATOMIC_INIT(0); static atomic_t stats_flush_threshold = ATOMIC_INIT(0); -static u64 flush_next_time; +static u64 flush_last_time; #define FLUSH_TIME (2UL*HZ) @@ -650,7 +650,7 @@ static void do_flush_stats(void) atomic_xchg(&stats_flush_ongoing, 1)) return; - WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); + WRITE_ONCE(flush_last_time, jiffies_64); cgroup_rstat_flush(root_mem_cgroup->css.cgroup); @@ -666,7 +666,8 @@ void mem_cgroup_flush_stats(void) void mem_cgroup_flush_stats_ratelimited(void) { - if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) + /* Only flush if the periodic flusher is one full cycle late */ + if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME)) mem_cgroup_flush_stats(); } From patchwork Tue Oct 10 03:21:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13414778 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66234CD68EC for ; Tue, 10 Oct 2023 03:21:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28F366B01FF; Mon, 9 Oct 2023 23:21:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 229146B0204; Mon, 9 Oct 2023 23:21:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED0516B0200; Mon, 9 Oct 2023 23:21:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D2C826B00E5 for ; Mon, 9 Oct 2023 23:21:26 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8DF6BC0559 for ; Tue, 10 Oct 2023 03:21:26 +0000 (UTC) X-FDA: 81328101372.30.94E286D Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf28.hostedemail.com (Postfix) with ESMTP id D2195C000F for ; Tue, 10 Oct 2023 03:21:24 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=09diwHIJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 3M8MkZQoKCHw3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3M8MkZQoKCHw3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696908084; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hC1pWunCMBCtxVf3YGG9j/q6Xl3rVbI2nQig93EJD1c=; b=Tm8879KvWIiWw6zc0M3MwIqkF8mtzoGoP+cE3S89qIgBhueG08+//VSNxYLTiSCt6BHMT9 uRh8eNNFtvinjHEOZTAs6yHJ22hwHhp2p5HTBaecNWQwj2RwdcL8cYZ/s4OeubD2l/i4Ir nI7er+W6iuWXCGYi6b14Y5TcrSyipY4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=09diwHIJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 3M8MkZQoKCHw3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3M8MkZQoKCHw3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696908084; a=rsa-sha256; cv=none; b=p1t9LFk+enDaG+K3/1iNNe+hYaeSru1IlsHPq98194YXXKn21H1DxAw+UROCwMHr9Qg3IX xe+FSeEMsoLgmsQRhTfkmbTIQBSpxCe6Tq0yU3/8g7TcWkKc+lhTt5jCiMe6OoEzPfEx+S 7qmu4FC6nNI5Ph0VImO6DgGUjggT69c= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a7be940fe1so4404517b3.2 for ; Mon, 09 Oct 2023 20:21:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696908084; x=1697512884; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hC1pWunCMBCtxVf3YGG9j/q6Xl3rVbI2nQig93EJD1c=; b=09diwHIJ5/Mh66XOvAYpG2sOJBMr22FYzXEaYVYljupuXh5Knl8bbQHWrsg+zkz0Al B0a86f75fPJHJ+FQkirBzyPiCaq7SI2cV93OHSUumFRQw4RTpjvgwYzDUNPSdJD209ZM 3ymbnVH4MwhQyO0ZJMyeSGxonDMM5NK3CJVNTQ+ALK/vxrzrTWpPgxBLSiYJdpa5kSx6 YyboXjc2QpGe09CSIrX+aeRBTuwIKvzfZCTCaW2IhIU5hwJ+EUAiUcqNiSz8J8LMasW/ +ga7JPQKsRspXCltu8fdrtE/TUYHtE1JdeB/JTLPLtJXQzRUkOxoxJTkXar+WbE9cSmy qjsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696908084; x=1697512884; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hC1pWunCMBCtxVf3YGG9j/q6Xl3rVbI2nQig93EJD1c=; b=bKWwXZNPqv7aFxvg1xLaQFDRngTavy2viJ0LfwShLWx8SrXnhwAGWkZUAh7Xt+HFR6 Gw1x61r3Cnus27Lz5B78c/XwdqRxSbu/cndCO+h3ilHFtBqKbbJlEmcu3JGD4yvQ91oi GRGPIX+tZFo80aVcsax9Gv30V3M62JuNcD6K84z9NrP8IzxyGpThmIUHLa+jD7XPSx8Q uQAmmn84QdY8vKrb+jN982yZPm1X/Kj9X5z3T5Jm66lurLDpKFvdnWlVgc2guU8EvDEI ELB73nhO7jsxV6ezQuaVvL/qwskLFMx1iqJaN5Eb1M6hY8K4B21mGqGUSpn/FZogFM9R Gd1w== X-Gm-Message-State: AOJu0YyGyDegWUrgbF/nmX7yjEMPaxpObUgPCyBBrtflUybga9VmAB3u oK+NpHy5NiF+yG1K1n3H2Rkun66ZkbtckQT3 X-Google-Smtp-Source: AGHT+IFIrbr9jnt8AKgjFXbiV/tkMci7vpdggI/qLBKz9UQMIl0xyavjRIGPF8xVVyjEs532DFKybC2OPjpal40i X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a5b:b50:0:b0:d80:904d:c211 with SMTP id b16-20020a5b0b50000000b00d80904dc211mr250724ybr.7.1696908083837; Mon, 09 Oct 2023 20:21:23 -0700 (PDT) Date: Tue, 10 Oct 2023 03:21:13 +0000 In-Reply-To: <20231010032117.1577496-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20231010032117.1577496-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.609.gbb76f46606-goog Message-ID: <20231010032117.1577496-3-yosryahmed@google.com> Subject: [PATCH v2 2/5] mm: memcg: move vmstats structs definition above flushing code From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , kernel-team@cloudflare.com, Wei Xu , Greg Thelen , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: D2195C000F X-Stat-Signature: 7bywebwm67of6tepb18koo6fy3gnujft X-Rspam-User: X-HE-Tag: 1696908084-662514 X-HE-Meta: U2FsdGVkX18FO0qmIVJhpvRm5Kx0wG9bFcfRhRNBZbCXMjpNSShGJW81DRf8JqWukHiwr85C5QXtUZs7yV8DRf71LSs56OSMCUGrV6zoR9iZqsAbeNk1kq9ZgjJW9kXlZovY06LY3dcturC14PbDPP8hoacowjYmjiHBHlFqYYYjifsa46pqUmeLhw0+iuPuAoM7U1uJg65mMIn4HxGqPs9ixkxFiYtJEbyjLGQuPz72wrV9kG1u6/Z4i68Zsln7YGpCIlmumgloiOHV5yyqT2kIupqLg3yFTSK6F++92TexFyGlSiLhPHgzSR3r7vrCndC/JWh9WpUIR2XTizgJFvv3uMpHJxOSFR50GXPXPZGEgCosQoJvacYQMbWzgmTVo8pCUE6mFPgMUzeJsvzAIfHCRD9Ml+kZEZi/DYiaDtLxfyAGlf4oE3q7yIUWi8qyDFFW6n8sfHm/UI8HM/HAsfb3e9EHgMnxgo4bHADGSlR+YjAQ4LYhb6DYY4kUnbSx5Cj+Z7tH8FZzF/+9vW3zYDsBLSTYALm7sXnNdBXJVSPxNlEmBEbXykAw6EK2IBOoWje00UQl7hZLY5aFgEprzyfd5MypJiO0DOH5qss5EDpTcLdW+JmmK/PtT1XJVvVflBDgtOhVA/zZmOx64X4aC2wG7Btgx4Srft5/IWwTkNTneu7G5S4iBknwN+byB2viowjq0xLIW6vI1BopdoWT8UX7+B3fGQbIdNB2GjvxDpr9TobyoI4fvVUkh42Pn6U41qBFmSIRkuRGNEH4o3AxRVbFzncAKrWM4mJRTVTKaGmL7ViG7427U5Wkuf8o2u+bkppeECVMBSgKL+9fok9+h16c0THhTYffwPcvDSxsmMmuRxE4eh3SnL+f18lkvaDdmFYPN0WAlkhwJtDreM9Msb20Qd5SpCB8QJBmJpCELReMdRyTyBkTbRC1A27/sf1peO1GPY2b4gkvqlY7EOk zis7hAoS f/9JKPyQ2pI8BQ+c6GtSRmsE1yU29Z0iHmnycTjHmWiiz+a/KFFX8JrKy7/tpruBG+FzJrpXNk1lV6Ur2Eyk1b5idlUGxPYTRM1xLWxCPuZ46mDh9ae3M50jp/RpVAQVAPnq1pdt3HzO4B47s4AMhhOJJmAgbseMGJfJRz9bCeknG21V0eeOO7qNHvgTI0Hi6EyuSmhY2DY+/bJpuUI5fBrDKoch7fj7NJzqDsdMuNAFya2P08HelXcAFd1dTZqIjfZ0pqZ4AtiHQ8dCdUwpz4nxpxZAeFgWZ/YpgPstjCND3HlFCfrDNDIOl7YxDNLmmPifp3m3YqKCJGC17pHb5/VbckLVERZUyJtIqK0mGq7CUPBPWqUNx4++hx6vkavIUO1IhWjDYSURuFAQJmYAEhV2rvCU9hRfbgvAiWtC+t8b4kQq5kWxOX0/z5JbTcpRHd8HsEcg+XqidNKR0BGlQKE/Uq1lseVtHsPLdJ4SBgOyIJDf41dxoACGVdNEcOpktmzh10x5PvK6fuQz1SpmM3yGNuXhsB7/ZI33phmHn3Hl0PdsuqKlrhWiHSHBNgrtDtHLTStoZD1DjsQTQ59huGg+xNf5PAkohFP8juptr+c4+De9SEP5GsTIditY0Cbodb9bM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The following patch will make use of those structs in the flushing code, so move their definitions (and a few other dependencies) a little bit up to reduce the diff noise in the following patch. No functional change intended. Signed-off-by: Yosry Ahmed --- mm/memcontrol.c | 146 ++++++++++++++++++++++++------------------------ 1 file changed, 73 insertions(+), 73 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4a194fcc9533..a393f1399a2b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -570,6 +570,79 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) return mz; } +/* Subset of vm_event_item to report for memcg event stats */ +static const unsigned int memcg_vm_event_stat[] = { + PGPGIN, + PGPGOUT, + PGSCAN_KSWAPD, + PGSCAN_DIRECT, + PGSCAN_KHUGEPAGED, + PGSTEAL_KSWAPD, + PGSTEAL_DIRECT, + PGSTEAL_KHUGEPAGED, + PGFAULT, + PGMAJFAULT, + PGREFILL, + PGACTIVATE, + PGDEACTIVATE, + PGLAZYFREE, + PGLAZYFREED, +#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP) + ZSWPIN, + ZSWPOUT, +#endif +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + THP_FAULT_ALLOC, + THP_COLLAPSE_ALLOC, + THP_SWPOUT, + THP_SWPOUT_FALLBACK, +#endif +}; + +#define NR_MEMCG_EVENTS ARRAY_SIZE(memcg_vm_event_stat) +static int mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly; + +static void init_memcg_events(void) +{ + int i; + + for (i = 0; i < NR_MEMCG_EVENTS; ++i) + mem_cgroup_events_index[memcg_vm_event_stat[i]] = i + 1; +} + +static inline int memcg_events_index(enum vm_event_item idx) +{ + return mem_cgroup_events_index[idx] - 1; +} + +struct memcg_vmstats_percpu { + /* Local (CPU and cgroup) page state & events */ + long state[MEMCG_NR_STAT]; + unsigned long events[NR_MEMCG_EVENTS]; + + /* Delta calculation for lockless upward propagation */ + long state_prev[MEMCG_NR_STAT]; + unsigned long events_prev[NR_MEMCG_EVENTS]; + + /* Cgroup1: threshold notifications & softlimit tree updates */ + unsigned long nr_page_events; + unsigned long targets[MEM_CGROUP_NTARGETS]; +}; + +struct memcg_vmstats { + /* Aggregated (CPU and subtree) page state & events */ + long state[MEMCG_NR_STAT]; + unsigned long events[NR_MEMCG_EVENTS]; + + /* Non-hierarchical (CPU aggregated) page state & events */ + long state_local[MEMCG_NR_STAT]; + unsigned long events_local[NR_MEMCG_EVENTS]; + + /* Pending child counts during tree propagation */ + long state_pending[MEMCG_NR_STAT]; + unsigned long events_pending[NR_MEMCG_EVENTS]; +}; + /* * memcg and lruvec stats flushing * @@ -681,79 +754,6 @@ static void flush_memcg_stats_dwork(struct work_struct *w) queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } -/* Subset of vm_event_item to report for memcg event stats */ -static const unsigned int memcg_vm_event_stat[] = { - PGPGIN, - PGPGOUT, - PGSCAN_KSWAPD, - PGSCAN_DIRECT, - PGSCAN_KHUGEPAGED, - PGSTEAL_KSWAPD, - PGSTEAL_DIRECT, - PGSTEAL_KHUGEPAGED, - PGFAULT, - PGMAJFAULT, - PGREFILL, - PGACTIVATE, - PGDEACTIVATE, - PGLAZYFREE, - PGLAZYFREED, -#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP) - ZSWPIN, - ZSWPOUT, -#endif -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - THP_FAULT_ALLOC, - THP_COLLAPSE_ALLOC, - THP_SWPOUT, - THP_SWPOUT_FALLBACK, -#endif -}; - -#define NR_MEMCG_EVENTS ARRAY_SIZE(memcg_vm_event_stat) -static int mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly; - -static void init_memcg_events(void) -{ - int i; - - for (i = 0; i < NR_MEMCG_EVENTS; ++i) - mem_cgroup_events_index[memcg_vm_event_stat[i]] = i + 1; -} - -static inline int memcg_events_index(enum vm_event_item idx) -{ - return mem_cgroup_events_index[idx] - 1; -} - -struct memcg_vmstats_percpu { - /* Local (CPU and cgroup) page state & events */ - long state[MEMCG_NR_STAT]; - unsigned long events[NR_MEMCG_EVENTS]; - - /* Delta calculation for lockless upward propagation */ - long state_prev[MEMCG_NR_STAT]; - unsigned long events_prev[NR_MEMCG_EVENTS]; - - /* Cgroup1: threshold notifications & softlimit tree updates */ - unsigned long nr_page_events; - unsigned long targets[MEM_CGROUP_NTARGETS]; -}; - -struct memcg_vmstats { - /* Aggregated (CPU and subtree) page state & events */ - long state[MEMCG_NR_STAT]; - unsigned long events[NR_MEMCG_EVENTS]; - - /* Non-hierarchical (CPU aggregated) page state & events */ - long state_local[MEMCG_NR_STAT]; - unsigned long events_local[NR_MEMCG_EVENTS]; - - /* Pending child counts during tree propagation */ - long state_pending[MEMCG_NR_STAT]; - unsigned long events_pending[NR_MEMCG_EVENTS]; -}; - unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) { long x = READ_ONCE(memcg->vmstats->state[idx]); From patchwork Tue Oct 10 03:21:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13414779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66A0ACD68E3 for ; Tue, 10 Oct 2023 03:21:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA73F6B020C; Mon, 9 Oct 2023 23:21:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A375A6B020B; Mon, 9 Oct 2023 23:21:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 83A4F6B020C; Mon, 9 Oct 2023 23:21:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 647AF6B020A for ; Mon, 9 Oct 2023 23:21:28 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 44AAA1A04DF for ; Tue, 10 Oct 2023 03:21:28 +0000 (UTC) X-FDA: 81328101456.18.49F2472 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf05.hostedemail.com (Postfix) with ESMTP id 77956100013 for ; Tue, 10 Oct 2023 03:21:26 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=yeLso6F2; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3NcMkZQoKCH40qut0cjogfiqqing.eqonkpwz-oomxcem.qti@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3NcMkZQoKCH40qut0cjogfiqqing.eqonkpwz-oomxcem.qti@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696908086; a=rsa-sha256; cv=none; b=SGJsqWZT32yJxxhRLi/8fhZ5WoEAxsobu2uQGnQbe4b1wMz9ZKEBBcxPqQ0SUaVt0GfZJX mHMJArS0Z027voKKcjvUHkvRKrgNsFim/bQr4PQSpDddkmBAPHQmk49EmBHGT2GV+YWUeG c8pQacV47v9jAyJ8ZixwBCMBIJFN1go= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=yeLso6F2; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3NcMkZQoKCH40qut0cjogfiqqing.eqonkpwz-oomxcem.qti@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3NcMkZQoKCH40qut0cjogfiqqing.eqonkpwz-oomxcem.qti@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696908086; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XEDO979juocfA6e748Gdg1ojA/11BVhtMp+K0OvWH0c=; b=Vu9kz+yKP1R3eYfv8rc7kppOLohGRgX338Wh3CzwlNbxxDJSwd3TOrokIy1leIfkyELVCK ZtMPwCkd1L2iGO2j9n9iIhP9zmUB3mcHnCwCBCSXc6EQGhtjOKqGPYNyV3SyrTdn1y3lPU 6GY5H9zxqLSPeTw+QhzImBqmOpsASPk= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a234ffeb90so79944917b3.3 for ; Mon, 09 Oct 2023 20:21:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696908085; x=1697512885; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XEDO979juocfA6e748Gdg1ojA/11BVhtMp+K0OvWH0c=; b=yeLso6F2F+bIjxyp9jXXPBDDUC1rGD5eBa/dLXuEADqeSV+4JMq46Phl3YfIbDBF3R IuJBPJCgQwKtWLNaKVieZwEEf+q+qqAOBCbM2Po20qzQgupQnP72+TxSgPAh84x4a/A7 YdpPcdPJn2WqMUyACSImDyOARY4uYpGqdM5p53KJ56eWcXBHaoj9pdGxsFkMw6jsY2ex DOrApquxy4AJpEGtSVzUwlPn058538CmkH5vdAU2ZvqUxUwbZweIO/HtNXnQYjSu6lcb b+pjw7QDptRaUhp8uMp/bllSklcbNFBBjo63vDpbvVvsM1x7JOp5x8Xx4TcF0Dc+VJHm aX4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696908085; x=1697512885; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XEDO979juocfA6e748Gdg1ojA/11BVhtMp+K0OvWH0c=; b=oO9pppfm6Qjm6vsZjjSWPZ7wds9v9jiYQNKBlO91J/VaKFC0AHFfRj+XSyAFigJBwq 83oNVYNhc8GnlQedqbwVErp9ahUnuIB2EJHTDPCu9ESS695tD5zXbR8capxsSxeTOIdK e9ltpTxbcxj8gFARTA4UlcPAUawSqanm3M2+fMaslDTeOxF+Hc6nhgvjbQRJ+KoOeyaQ ghFcx9cwmwsb/kc4rOc5btL7RbwwULn3dkcznTe50FQqP+RUpafUHHRv6BFxkdeeYtGB uEx5VIXVnX7gcJ8yZpuLDZSxLgu+L4lMz1INAmJyPFF0P+AybH7mcVHjs+Am8PWUc7gw +7tw== X-Gm-Message-State: AOJu0YzNTaM7aPyvD8lC0ua3SLTW+mQxI+U+6cz2FhUmYGRIwKk+/MtZ MFX3gunxu2OzPs4j4igUMHVHsRusfc39VJeD X-Google-Smtp-Source: AGHT+IG8kcJ/dn9Ahg31ZEvjLGaGuCajxb48R9RWsECmkth4fjZtYHFlyQOjLuGiUbhEeB/vJfn9WBvb6ywt7IeI X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a81:b209:0:b0:59b:e1db:562c with SMTP id q9-20020a81b209000000b0059be1db562cmr319164ywh.5.1696908085640; Mon, 09 Oct 2023 20:21:25 -0700 (PDT) Date: Tue, 10 Oct 2023 03:21:14 +0000 In-Reply-To: <20231010032117.1577496-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20231010032117.1577496-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.609.gbb76f46606-goog Message-ID: <20231010032117.1577496-4-yosryahmed@google.com> Subject: [PATCH v2 3/5] mm: memcg: make stats flushing threshold per-memcg From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , kernel-team@cloudflare.com, Wei Xu , Greg Thelen , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 77956100013 X-Stat-Signature: mc5pami3se3qogxfjmqd84gkoza8nibw X-HE-Tag: 1696908086-449451 X-HE-Meta: U2FsdGVkX19Dxz61bvvfmxj1t5EbHWaeEErvoaWJfEYCtSCd2iwEq22tF1J9F3t655EOhA+rzh3urUSZsOsnVL3N4X1hIgKRmbz/jl0WnspKiaJrflvEYASlIKj1loQ1zJtxihzP5zdtP4F9YGGo1L+F/SewjTBoYv7alsi3LMA96UA2CKflc0Zt5Rmz24kGc1mnjxSLS8LfZSagekJt5HvsQA1jQIYkKpPImYnthvGSAVrCZvlr4p+ia4AbvAd7yQdpgWw45tXivipWI2LxNUl/OYiQMvEa/ehnicsZ2NT98jdwsC3BFKDfODKsQqfQhhlDPs2w2HwLVEqon6vfWo6wtWvPtrgHPzPiZ/dKjk7KehZUAcd3D18Q2C7hIkPA6sA3PPS67yrChT7BS0o4B4nM9JhCfTqIgyLmQ05d4HflWHvWe1z9/zxuF3hP3/sBz7WTI+MEKmg3odJL1iqBLn0XrTtQqp5rRCUHynUWRjcUy0vuTydRz1dYyxNdVgbDX6P5rWSBA/khUz8kzcPrtJD9acWrjkkBx1alSdASWLNAug9JfoFNZsXYWt+8VBLK7wgrLkcV74JcmcXrz+MNTTPtl8QOob37+bAx6RwiiVH9C68k7FWo3Gvpx9SrzDaBzp4mqs/6bpaBgScFRjrrjjd/eMUAtfbLaTveezw0mcL0dnsm6O2ybRudZgYwbkO+oBeeZRZUERPjm60OvSaGZcIn/J00bYCNKgV/ZRMBnO+hBboNhipAoamW32MQbYUiCuy4jfgfcu5d7eULCqXtMlJ+CuJRX7EApKEFMxStlbjZKD5sxwECySyA4avecNeaiYT/jqvshl1k8HCVJk62hdAmsfdAf+Y0BDpXSc8+u7ACEdKL3r4tmNO3D8zS+ztD1IhWz0pytTjZyK4gRK9oNCIZl46SzEd9/HRTUoqRldb5QKO/XMfhY4/PR8CtVc3B0eJnHgjD/TnQfOhmRYa ED2ltPC8 cjniVnqwg/DSQRGW87+E98cGoLEy2UorU05/8gMauWQ2DNNtyugbhUGlXk87tb9vv8D9a1vMOqED1qyOd+/6LiHkhDykTFT7tbbmxy7wJxIjT36AKpnOtVoRvjKOeeD5kl3JslfnRyUHS1uonyzzaplWUotpg+26kQSFwlfr+SXnCnm9CSnOcGJd5jHFuQ3wi+bUdVNYYSgC6IlZWD5027sElelNg8Q8HNApa3ymqN5Ov129fJrsdUl0ubDzJFBH7gggeRIPIbI9SbSsfhwqtUBsg3hDXBcm8Zx0rgR5k0FUaaqwSPd12uOQ+NpF6CQTCZEe+tQWgJ9m6uW2WMQQ051LZXmcM/b2dzLhpf29PnKk6SrvnTh3Rw68uRIxFLIzl9u49+JfrpBSObcGRmY856q7e5B8LI8zpRPtcqym6psteh3l5DT+6H0PybaFv8sNJYvAYj5WN1BBkgg07dDhVapGB/9PT/qLwYqs/6Z27Nuucs9X4EgW4WSrs19TbpGmbayC7/N2u2fWlX+zUpHv8oPcNSRgI/5QR8G5vWNgBQ7CUlwuZeXFWsVCvAp8b7q6U3KL1Vm+5YVmKTk/OCirl/H2cG0yNMlFUc7FmTZhI0kf35gt3VoDlWQcnpz5v+icxBmRNc7OTbuQFQa5gxTpwSL6ovU0FcZfC3hzsGjT1KG4XOotxxwFruCTPEpfhr9fJgsvphs3PTVRijHUUpXDFtL5YuCs/4tnYXhoP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A global counter for the magnitude of memcg stats update is maintained on the memcg side to avoid invoking rstat flushes when the pending updates are not significant. This avoids unnecessary flushes, which are not very cheap even if there isn't a lot of stats to flush. It also avoids unnecessary lock contention on the underlying global rstat lock. Make this threshold per-memcg. The scheme is followed where percpu (now also per-memcg) counters are incremented in the update path, and only propagated to per-memcg atomics when they exceed a certain threshold. This provides two benefits: (a) On large machines with a lot of memcgs, the global threshold can be reached relatively fast, so guarding the underlying lock becomes less effective. Making the threshold per-memcg avoids this. (b) Having a global threshold makes it hard to do subtree flushes, as we cannot reset the global counter except for a full flush. Per-memcg counters removes this as a blocker from doing subtree flushes, which helps avoid unnecessary work when the stats of a small subtree are needed. Nothing is free, of course. This comes at a cost: (a) A new per-cpu counter per memcg, consuming NR_CPUS * NR_MEMCGS * 4 bytes. The extra memory usage is insigificant. (b) More work on the update side, although in the common case it will only be percpu counter updates. The amount of work scales with the number of ancestors (i.e. tree depth). This is not a new concept, adding a cgroup to the rstat tree involves a parent loop, so is charging. Testing results below show no significant regressions. (c) The error margin in the stats for the system as a whole increases from NR_CPUS * MEMCG_CHARGE_BATCH to NR_CPUS * MEMCG_CHARGE_BATCH * NR_MEMCGS. This is probably fine because we have a similar per-memcg error in charges coming from percpu stocks, and we have a periodic flusher that makes sure we always flush all the stats every 2s anyway. This patch was tested to make sure no significant regressions are introduced on the update path as follows. The following benchmarks were ran in a cgroup that is 4 levels deep (/sys/fs/cgroup/a/b/c/d), which is deeper than a usual setup: (a) neper [1] with 1000 flows and 100 threads (single machine). The values in the table are the average of server and client throughputs in mbps after 30 iterations, each running for 30s: tcp_rr tcp_stream Base 9504218.56 357366.84 Patched 9656205.68 356978.39 Delta +1.6% -0.1% Standard Deviation 0.95% 1.03% An increase in the performance of tcp_rr doesn't really make sense, but it's probably in the noise. The same tests were ran with 1 flow and 1 thread but the throughput was too noisy to make any conclusions (the averages did not show regressions nonetheless). Looking at perf for one iteration of the above test, __mod_memcg_state() (which is where memcg_rstat_updated() is called) does not show up at all without this patch, but it shows up with this patch as 1.06% for tcp_rr and 0.36% for tcp_stream. (b) "stress-ng --vm 0 -t 1m --times --perf". I don't understand stress-ng very well, so I am not sure that's the best way to test this, but it spawns 384 workers and spits a lot of metrics which looks nice :) I picked a few ones that seem to be relevant to the stats update path. I also included cache misses as this patch introduce more atomics that may bounce between cpu caches: Metric Base Patched Delta Cache Misses 3.394 B/sec 3.433 B/sec +1.14% Cache L1D Read 0.148 T/sec 0.154 T/sec +4.05% Cache L1D Read Miss 20.430 B/sec 21.820 B/sec +6.8% Page Faults Total 4.304 M/sec 4.535 M/sec +5.4% Page Faults Minor 4.304 M/sec 4.535 M/sec +5.4% Page Faults Major 18.794 /sec 0.000 /sec Kmalloc 0.153 M/sec 0.152 M/sec -0.65% Kfree 0.152 M/sec 0.153 M/sec +0.65% MM Page Alloc 4.640 M/sec 4.898 M/sec +5.56% MM Page Free 4.639 M/sec 4.897 M/sec +5.56% Lock Contention Begin 0.362 M/sec 0.479 M/sec +32.32% Lock Contention End 0.362 M/sec 0.479 M/sec +32.32% page-cache add 238.057 /sec 0.000 /sec page-cache del 6.265 /sec 6.267 /sec -0.03% This is only using a single run in each case. I am not sure what to make out of most of these numbers, but they mostly seem in the noise (some better, some worse). The lock contention numbers are interesting. I am not sure if higher is better or worse here. No new locks or lock sections are introduced by this patch either way. Looking at perf, __mod_memcg_state() shows up as 0.00% with and without this patch. This is suspicious, but I verified while stress-ng is running that all the threads are in the right cgroup. (3) will-it-scale page_fault tests. These tests (specifically per_process_ops in page_fault3 test) detected a 25.9% regression before for a change in the stats update path [2]. These are the numbers from 30 runs (+ is good): LABEL | MEAN | MEDIAN | STDDEV | ------------------------------+-------------+-------------+------------- page_fault1_per_process_ops | | | | (A) base | 265207.738 | 262941.000 | 12112.379 | (B) patched | 249249.191 | 248781.000 | 8767.457 | | -6.02% | -5.39% | | page_fault1_per_thread_ops | | | | (A) base | 241618.484 | 240209.000 | 10162.207 | (B) patched | 229820.671 | 229108.000 | 7506.582 | | -4.88% | -4.62% | | page_fault1_scalability | | | (A) base | 0.03545 | 0.035705 | 0.0015837 | (B) patched | 0.029952 | 0.029957 | 0.0013551 | | -9.29% | -9.35% | | page_fault2_per_process_ops | | | (A) base | 203916.148 | 203496.000 | 2908.331 | (B) patched | 186975.419 | 187023.000 | 1991.100 | | -6.85% | -6.90% | | page_fault2_per_thread_ops | | | (A) base | 170604.972 | 170532.000 | 1624.834 | (B) patched | 163100.260 | 163263.000 | 1517.967 | | -4.40% | -4.26% | | page_fault2_scalability | | | (A) base | 0.054603 | 0.054693 | 0.00080196 | (B) patched | 0.044882 | 0.044957 | 0.0011766 | | -0.05% | +0.33% | | page_fault3_per_process_ops | | | (A) base | 1299821.099 | 1297918.000 | 9882.872 | (B) patched | 1248700.839 | 1247168.000 | 8454.891 | | -3.93% | -3.91% | | page_fault3_per_thread_ops | | | (A) base | 387216.963 | 387115.000 | 1605.760 | (B) patched | 368538.213 | 368826.000 | 1852.594 | | -4.82% | -4.72% | | page_fault3_scalability | | | (A) base | 0.59909 | 0.59367 | 0.01256 | (B) patched | 0.59995 | 0.59769 | 0.010088 | | +0.14% | +0.68% | | There is some microbenchmarks regressions (and some minute improvements), but nothing outside the normal variance of this benchmark between kernel versions. The fix for [2] assumes that 3% is noise -- and there were no further practical complaints), so hopefully this means that such variations in these microbenchmarks do not reflect on practical workloads. [1]https://github.com/google/neper [2]https://lore.kernel.org/all/20190520063534.GB19312@shao2-debian/ Signed-off-by: Yosry Ahmed Signed-off-by: Yosry Ahmed --- mm/memcontrol.c | 49 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a393f1399a2b..9a586893bd3e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -627,6 +627,9 @@ struct memcg_vmstats_percpu { /* Cgroup1: threshold notifications & softlimit tree updates */ unsigned long nr_page_events; unsigned long targets[MEM_CGROUP_NTARGETS]; + + /* Stats updates since the last flush */ + unsigned int stats_updates; }; struct memcg_vmstats { @@ -641,6 +644,9 @@ struct memcg_vmstats { /* Pending child counts during tree propagation */ long state_pending[MEMCG_NR_STAT]; unsigned long events_pending[NR_MEMCG_EVENTS]; + + /* Stats updates since the last flush */ + atomic64_t stats_updates; }; /* @@ -660,9 +666,7 @@ struct memcg_vmstats { */ static void flush_memcg_stats_dwork(struct work_struct *w); static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); -static DEFINE_PER_CPU(unsigned int, stats_updates); static atomic_t stats_flush_ongoing = ATOMIC_INIT(0); -static atomic_t stats_flush_threshold = ATOMIC_INIT(0); static u64 flush_last_time; #define FLUSH_TIME (2UL*HZ) @@ -689,26 +693,37 @@ static void memcg_stats_unlock(void) preempt_enable_nested(); } + +static bool memcg_should_flush_stats(struct mem_cgroup *memcg) +{ + return atomic64_read(&memcg->vmstats->stats_updates) > + MEMCG_CHARGE_BATCH * num_online_cpus(); +} + static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) { + int cpu = smp_processor_id(); unsigned int x; if (!val) return; - cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); + cgroup_rstat_updated(memcg->css.cgroup, cpu); + + for (; memcg; memcg = parent_mem_cgroup(memcg)) { + x = __this_cpu_add_return(memcg->vmstats_percpu->stats_updates, + abs(val)); + + if (x < MEMCG_CHARGE_BATCH) + continue; - x = __this_cpu_add_return(stats_updates, abs(val)); - if (x > MEMCG_CHARGE_BATCH) { /* - * If stats_flush_threshold exceeds the threshold - * (>num_online_cpus()), cgroup stats update will be triggered - * in __mem_cgroup_flush_stats(). Increasing this var further - * is redundant and simply adds overhead in atomic update. + * If @memcg is already flush-able, increasing stats_updates is + * redundant. Avoid the overhead of the atomic update. */ - if (atomic_read(&stats_flush_threshold) <= num_online_cpus()) - atomic_add(x / MEMCG_CHARGE_BATCH, &stats_flush_threshold); - __this_cpu_write(stats_updates, 0); + if (!memcg_should_flush_stats(memcg)) + atomic64_add(x, &memcg->vmstats->stats_updates); + __this_cpu_write(memcg->vmstats_percpu->stats_updates, 0); } } @@ -727,13 +742,12 @@ static void do_flush_stats(void) cgroup_rstat_flush(root_mem_cgroup->css.cgroup); - atomic_set(&stats_flush_threshold, 0); atomic_set(&stats_flush_ongoing, 0); } void mem_cgroup_flush_stats(void) { - if (atomic_read(&stats_flush_threshold) > num_online_cpus()) + if (memcg_should_flush_stats(root_mem_cgroup)) do_flush_stats(); } @@ -747,8 +761,8 @@ void mem_cgroup_flush_stats_ratelimited(void) static void flush_memcg_stats_dwork(struct work_struct *w) { /* - * Always flush here so that flushing in latency-sensitive paths is - * as cheap as possible. + * Deliberately ignore memcg_should_flush_stats() here so that flushing + * in latency-sensitive paths is as cheap as possible. */ do_flush_stats(); queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); @@ -5803,6 +5817,9 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) } } } + /* We are in a per-cpu loop here, only do the atomic write once */ + if (atomic64_read(&memcg->vmstats->stats_updates)) + atomic64_set(&memcg->vmstats->stats_updates, 0); } #ifdef CONFIG_MMU From patchwork Tue Oct 10 03:21:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13414780 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6851CD68E4 for ; Tue, 10 Oct 2023 03:21:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D6526B020B; Mon, 9 Oct 2023 23:21:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6630B6B020D; Mon, 9 Oct 2023 23:21:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B2F66B020E; Mon, 9 Oct 2023 23:21:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 330576B020B for ; Mon, 9 Oct 2023 23:21:30 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0B729B4CE7 for ; Tue, 10 Oct 2023 03:21:30 +0000 (UTC) X-FDA: 81328101540.05.ABADC68 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf11.hostedemail.com (Postfix) with ESMTP id 3DC8240015 for ; Tue, 10 Oct 2023 03:21:28 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=F0dkyk3y; spf=pass (imf11.hostedemail.com: domain of 3N8MkZQoKCIA2swv2elqihksskpi.gsqpmry1-qqozego.svk@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3N8MkZQoKCIA2swv2elqihksskpi.gsqpmry1-qqozego.svk@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696908088; a=rsa-sha256; cv=none; b=hut5uoMpdwxrLmUWxQZtnPUlz8UIgcCvaDlyAUqQzaKJ3GHaZQDiCgXqRVuxln/bY0rPfl /rgcLC1HkK1kIp4MfSnJZ1rVi1voQ9NZJTbNhyRh5XO/AV+FBSAJYTMFo7asN7YPq1DjzE D/gtIeJlPfGojh62LHETuR+KEFAoEOk= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=F0dkyk3y; spf=pass (imf11.hostedemail.com: domain of 3N8MkZQoKCIA2swv2elqihksskpi.gsqpmry1-qqozego.svk@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3N8MkZQoKCIA2swv2elqihksskpi.gsqpmry1-qqozego.svk@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696908088; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2uIMTmtGdqUKunPgdDdu073V0kaqxVUtvtbHday0wrc=; b=nF5BLllLAPtVty9QcmlQ1nYe2swmAZEbXtgFhnDQGq2FhgsCtZKhu7ivNRG2icwkKX+o8b 2CpMM9Omluic5TdLQ1Gajqpxlo/R/nnwjISrz0srLllZbfmBRLDTKpfkZlxs0lRBxgQtKc rxl8kyIH1ssFg9q8b+xz0h0cASf57Xs= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a7b53a48e5so10550827b3.1 for ; Mon, 09 Oct 2023 20:21:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696908087; x=1697512887; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2uIMTmtGdqUKunPgdDdu073V0kaqxVUtvtbHday0wrc=; b=F0dkyk3ygiXSbRY8kq1sG+ZR6Ik5RNwrFMNXMRtUbGSYzBbXivlr63jxKyz9OJXWlA E90kaZsJYtpUYe2Ba8huxh43npX9C4o9x2svbhdTSt2d0AUp8Ty0cZ3eeLmFBifMikrY PR4ZeAeL8yKrlu4sKjFgd2+8VddmR050yVkBWvgGSX2lA4c+JnriNcYhZX8kqMgwohkP 0QvoBDGTHeItdHsR7rRxkPMS2CVTWf3W2dTA6gqOrLR9SDQ5WGfKAmci7ekcGXSGLSsi mDi7o5N6r339fOOmk4mTV/hbdS+Ol0PoNaPLE0VBYdzGIv0RL3bDZ0EgdRoTE+YT6fgI 5bEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696908087; x=1697512887; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2uIMTmtGdqUKunPgdDdu073V0kaqxVUtvtbHday0wrc=; b=DN5SaCyfCfR9F3Lg/VH8ZcORfRneMGtR5blMS8FhIVztcsgtGAwIJY+CUxzKqPo3HE JIOENhQROq/eTI2X45dEja0t0gOMc8APFt6LOmj/3M1yogAUwrllxZbH1QAoxdDBCWo+ W4erlRSLPGvjFjGBVdQ8HEqd6GnLaPrkg8pFHqcmQcJJh4xkpJ1HKjj/bok7F7+aR5d9 hN4JV+3bFcn59hIP3txz4F84PsfSmy8ffDi+3y8JoEs8n1TZ7PNbFGWFsWqinEX+FwX8 wZBCaawkVdrXd+9I9Uyxnu9wYP7w8OoKCXwD4ubFQJ0uSRcjfTfX4erWFW9oltMHzmkm f8VQ== X-Gm-Message-State: AOJu0YwGab71RmN5PpuuIkyby0M4uCQIpe0JDH2TBlEy4zIhDkDGCH3E Ne8BFOad+vOhodsMKw+4Q4Nh/gyJ6CYmDlMS X-Google-Smtp-Source: AGHT+IH27StA7m2woHqh/tcnCp3uYnoxUCBEP5s+DN8GVWI1rrXovlh8AYsn+eA2pINcXrMN7noHvFZw+UL9M22y X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a81:c44c:0:b0:586:50cf:e13f with SMTP id s12-20020a81c44c000000b0058650cfe13fmr286393ywj.1.1696908087335; Mon, 09 Oct 2023 20:21:27 -0700 (PDT) Date: Tue, 10 Oct 2023 03:21:15 +0000 In-Reply-To: <20231010032117.1577496-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20231010032117.1577496-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.609.gbb76f46606-goog Message-ID: <20231010032117.1577496-5-yosryahmed@google.com> Subject: [PATCH v2 4/5] mm: workingset: move the stats flush into workingset_test_recent() From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , kernel-team@cloudflare.com, Wei Xu , Greg Thelen , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 3DC8240015 X-Stat-Signature: hxbsjxeso7sjuex9q7cgo641ioe5915s X-Rspam-User: X-HE-Tag: 1696908088-848472 X-HE-Meta: U2FsdGVkX18Q3C91slTCb6PXtcUjsNDJfUMNRHQDUXhrd6nhf+Wsz7tTtB/XcsVrcFwdAafUfwSInHTKCUqQlQ0SWuyvYQ4f2zUs7JoSoO6MkQICwnVSdKKiiz01hCLTEATV/rdmWNUQTM5c7+zz57QPjUF5lhCqErbp+uo7vDNTTcYsmbCj4HWKBZo7Oy3kCgwc2FZX0DNvo87oCT/EiP9eTyhXJgFI3AIfh2cpOPIWVvVofmIx+MRKKBwkIid629K3Q0Ygbx0L4nsgHGNKnSUS0HZr4ABhk0fhpoXcpQTOIXdzV/5vd5aJdg1qjQ3wiYfawbbbtr/p1+gGnez0+Rj1177NI9tLJG0GrtJH6zjx2gvc1NokoOUBd8ua4OXZyuGcQfB6dUy9o5yz0ZBR8t1aQJNgCWpYg8oYAC1OBo8f+DWhSLZzGkyEFYCPbwWgwYpjiCKfowJ/jBO+AYOrXOJQlDWxZ9Rx+lvseSn/QTwlBGuCnPifQy8Ea4vv5xepPaMqBYLaWFc2SF6BJ5ljCUGwU+Xn9EFXobpA7W0MiEfg5GG1RgySQER+7xSPcaz+mGx7z9sTcpYLsDqj3H2NNDEP/JFOvy8UsaNz9PsiZPYopncoKNmVXnq52sM1j2l0T/2Nf6TeZiRZbAGtVPnMkyfjCCo+PdGnsQ2Ggzy6kvdwoV1N+bSaQI/30gOJX2vV9ldZWyYMm9veSjgHJ538KvCFy+fVtbxvFDtHVjWFEnIbZJLgJtz4u3Ksc4q4nl0xng4ebW70CXhuXIb9Dz5DnfP7R6xUj9CCFtERIWvg0QIh7h515reyvLTHURa15rOFbz3WhcekcrcU7nNcHEBVTPOauTB9JPuYzpfHMmWpMvBKneyA32+ysEZsvFZrKf59r1f1N3BNn1sKHZJO4KtzHrXpyXRhM7or/wUafZHbPmr8Nferd1tJn7RDePU2x5Lalb61343h29Xvig3AvSR imHYRG3s s/eVKnMUvls2WhKUJ0XlH5O/XFzPiBJ1UiJMap3DI366nQkrjUuxzgxc37T5r/qGU7/YXAEwH2dml4bF29YuwZ3bSzsRjlBLCseqsMQ3MXGi7hpfaYp9gRade01x78dCvNIhgpnU6JX6qpprVxPnSauFQcy0h65GfmcBV31HKAcTqyrPjqJJdZ3vN8M9+Zur9kBpMDIkBuKRqymyDNQQb3qpdDdesh2F8GpDKTlqWxd4GMUVUZJ0iE3lel+fcpXAT+SWeHD42Mx4PzmfQWYl2EbxcoQExv2QoRxkbfCzTtY17d+aPbFsPugS/a/YLThjggNrs07B7FUg02vdu9aECIkS53K1vYrmjFVfGSbBRqlia1dw0+6K5TITaZlkeQlrPqbOE6z0FQZG05Gw3BvY4UgX3LiKP0GKhFniQ4YSubDjBj6JjQaYUU41sdNx42sEHDVzIMHgi4WVly+dGHEgazhqy5dVon+Jn9qfMU+gXt4dgILoJB7mJkLcLYziWMBiOOEuL+mVDcE/h5aYVcNHw5U5ditaJ2G/ROBSH2swXWMNjzrGJZPJ9Vv2DgjzXDcGgDwoGBNLm5oG1EN6E/fdz7rMRAMFZR3AzFRAou0IV2CAOe09vz2JOQvMUIAWfP/fe9LTMTv2h9sTO0nfq9ASvDLNCph8r/3psou1nzT8d6xzCu6H6W8M2U25TIQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000068, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The workingset code flushes the stats in workingset_refault() to get accurate stats of the eviction memcg. In preparation for more scoped flushed and passing the eviction memcg to the flush call, move the call to workingset_test_recent() where we have a pointer to the eviction memcg. The flush call is sleepable, and cannot be made in an rcu read section. Hence, minimize the rcu read section by also moving it into workingset_test_recent(). Furthermore, instead of holding the rcu read lock throughout workingset_test_recent(), only hold it briefly to get a ref on the eviction memcg. This allows us to make the flush call after we get the eviction memcg. As for workingset_refault(), nothing else there appears to be protected by rcu. The memcg of the faulted folio (which is not necessarily the same as the eviction memcg) is protected by the folio lock, which is held from all callsites. Add a VM_BUG_ON() to make sure this doesn't change from under us. No functional change intended. Signed-off-by: Yosry Ahmed --- mm/workingset.c | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index b192e44a0e7c..a573be6c59fd 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -425,8 +425,16 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) struct pglist_data *pgdat; unsigned long eviction; - if (lru_gen_enabled()) - return lru_gen_test_recent(shadow, file, &eviction_lruvec, &eviction, workingset); + rcu_read_lock(); + + if (lru_gen_enabled()) { + bool recent = lru_gen_test_recent(shadow, file, + &eviction_lruvec, &eviction, workingset); + + rcu_read_unlock(); + return recent; + } + unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset); eviction <<= bucket_order; @@ -448,8 +456,16 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) * configurations instead. */ eviction_memcg = mem_cgroup_from_id(memcgid); - if (!mem_cgroup_disabled() && !eviction_memcg) + if (!mem_cgroup_disabled() && + (!eviction_memcg || !mem_cgroup_tryget(eviction_memcg))) { + rcu_read_unlock(); return false; + } + + rcu_read_unlock(); + + /* Flush stats (and potentially sleep) outside the RCU read section */ + mem_cgroup_flush_stats_ratelimited(); eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat); refault = atomic_long_read(&eviction_lruvec->nonresident_age); @@ -493,6 +509,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) } } + mem_cgroup_put(eviction_memcg); return refault_distance <= workingset_size; } @@ -519,19 +536,16 @@ void workingset_refault(struct folio *folio, void *shadow) return; } - /* Flush stats (and potentially sleep) before holding RCU read lock */ - mem_cgroup_flush_stats_ratelimited(); - - rcu_read_lock(); - /* * The activation decision for this folio is made at the level * where the eviction occurred, as that is where the LRU order * during folio reclaim is being determined. * * However, the cgroup that will own the folio is the one that - * is actually experiencing the refault event. + * is actually experiencing the refault event. Make sure the folio is + * locked to guarantee folio_memcg() stability throughout. */ + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); nr = folio_nr_pages(folio); memcg = folio_memcg(folio); pgdat = folio_pgdat(folio); @@ -540,7 +554,7 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); if (!workingset_test_recent(shadow, file, &workingset)) - goto out; + return; folio_set_active(folio); workingset_age_nonresident(lruvec, nr); @@ -556,8 +570,6 @@ void workingset_refault(struct folio *folio, void *shadow) lru_note_cost_refault(folio); mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr); } -out: - rcu_read_unlock(); } /** From patchwork Tue Oct 10 03:21:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13414781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC6D7CD68EC for ; Tue, 10 Oct 2023 03:21:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C3EB6B024F; Mon, 9 Oct 2023 23:21:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F37E16B0251; Mon, 9 Oct 2023 23:21:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D3F406B0253; Mon, 9 Oct 2023 23:21:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B7AAB6B024F for ; Mon, 9 Oct 2023 23:21:32 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8113E12048C for ; Tue, 10 Oct 2023 03:21:32 +0000 (UTC) X-FDA: 81328101624.24.AAAC120 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf14.hostedemail.com (Postfix) with ESMTP id A82B1100024 for ; Tue, 10 Oct 2023 03:21:29 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=H2CHIpRR; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3OMMkZQoKCIE3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3OMMkZQoKCIE3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696908089; a=rsa-sha256; cv=none; b=FBsixETKn/J55EI9OtFwKtzDeIj0SGoY4MfL3kh2wxo0o/g8kPqYm7DO3y7JwJJmMnOytj 1ZqNvzvdIvztOFxdUjJbgORjqcCgX5eNRa9QuLHHR+aUlx4qxmdpdzqdsYELwObjwSmHZ/ 6a5w3S4BHGM8hNK0s7Kjwv5TxjBVG84= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=H2CHIpRR; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3OMMkZQoKCIE3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3OMMkZQoKCIE3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696908089; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/7pubW/PIkdb48xbcDVraC66arnzXQVqiP5CbuKjtvs=; b=SCQNO7rSpU6xTB1I4EaYV3XzE0yLNA+p6ZP/vyn+TirXHpjvX9TvyRwtwaefRce08plEhq z9XEZBI8NKAspLn85gSJMG7upOO+CQVDiagSY0Q3j8AipAoJmSqbaQihdWjnmIkj89/N3M FYLvPtLfW92DvP9ymHHPaCcPco/Txcs= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a7b9e83b70so4138317b3.0 for ; Mon, 09 Oct 2023 20:21:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696908089; x=1697512889; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/7pubW/PIkdb48xbcDVraC66arnzXQVqiP5CbuKjtvs=; b=H2CHIpRRzPYFYSoG3iTNYrilsFVTXGWm/k4fdAceAU79HSYYaU2uevpGeFnCEv6WC0 XO4SjE9TLdNb0c5lgJxyMbFvmz0AlUSiEMPjEpjiUiZi3DODJPeGqD1iUNhZHqJwfSjG U3mB6b6v0UgDvAKmaBsf2TkXabwR0NGOZNFx7QDqh+oPN01sq+lutsMYfkTz1Gyza7A5 mUubcOtO298+czyXcvKpHtjQen96g5h+iy0NZSDxnbJ3Sb00oP/rZHTzW66qVEr64s8V AlBm2EeSCYj9dvm8cqEU5Upz3jfhpyP5DyRow4EcRon1ubReqpxLcuyga0nY7OfvF0mi 1Lcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696908089; x=1697512889; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/7pubW/PIkdb48xbcDVraC66arnzXQVqiP5CbuKjtvs=; b=xFT7Oc9eaRquc4bKeuZmo47zgJToSy5PdeA05Aoqle2ct3axnko7CRfj89KK/KQMOF HnhIYGTeCzLIixvI9GIoP3MXeLXp3kTq5xblqpPOYDkkGaruFWT2kDrQAcROG4sqiX27 PXrODnF5/Z7rcmfpdp1+TsWBpp/wHeL9tK/ROZvJara1Ez7HVkYvsHjwDVw5MzY/nGG0 cElxA+3XKLZEYeyRNlnb+hWrSvcDr58mue7011p+vMIKpK7ibH1fIK/TUKjQEn1KuKQ7 Gk26IgrjLp3kz+uo/vQzJmvzBIdLP80nbAEgWUIbTglbQFQbO2VTfw7lrCZVCrmoI12I lEiw== X-Gm-Message-State: AOJu0Yz/zPDvVibKQjPV12vedcJ4SzkiZUeSmXm8urXHvOzHWC9wFvK/ J6SKi8Zzs+p5VM4k3K6P7IEK9MCp/VeX9qd9 X-Google-Smtp-Source: AGHT+IGZZZH3XMFz9Y1SUioLqXwa9WuvTjduHQeZqULDOWCJSsAn3ByEEvhnBldRhKwmLPtnt2IlTigbq67SFmDv X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a81:760d:0:b0:595:8166:7be with SMTP id r13-20020a81760d000000b00595816607bemr329711ywc.0.1696908088781; Mon, 09 Oct 2023 20:21:28 -0700 (PDT) Date: Tue, 10 Oct 2023 03:21:16 +0000 In-Reply-To: <20231010032117.1577496-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20231010032117.1577496-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.609.gbb76f46606-goog Message-ID: <20231010032117.1577496-6-yosryahmed@google.com> Subject: [PATCH v2 5/5] mm: memcg: restore subtree stats flushing From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , kernel-team@cloudflare.com, Wei Xu , Greg Thelen , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A82B1100024 X-Stat-Signature: oh8jcgdzjqo7mri6hfgiuph88ikqo1rr X-HE-Tag: 1696908089-848073 X-HE-Meta: U2FsdGVkX18NzNF69/YZNa/FawJxjMNBW9C9nQg27j1QxqvpJNoEMKDeS3EgEZHXrZEVPiL4dwf67K25Z4+tP8fBxojDz/x7lkUCDNXRHC7OFILZ1C/49twB4UDFXe8pHOKFMMECVxc2vt6Sy9Yz4ZJzs2mZgEZvy9C+FsvsM5j0NT2Rslx0ium9F0jIJ6MvTJZedr34+mHOC3tncJeM+IPbijH++Lmi5E3Ee79KwVl0Lbe07n+jdDrwwTY8PQSGlHzmoI5vXyVIIO+LFZmmH8CUbWCx63QFHuaHh7HitTyqLJOxlmobxrAJ8FAKNsjNc8b7k5fvN+Sc8h/IekR2WPDNBpSfdS1dmrHacBqBTHtOnyL6F/jy6QWCrR3wJLlg+XHc4nbZD6Kq0mS9360+TA/cAfQqPa18gGbI/MA9yi/X5tvb8NMbpL86LtNv6lF2Xt5rMUylY53c7jcayn3CtXTEHhoLyQxVYjM4WhAxC+JJgcon/hc1SlDBamgJlPqIUrFLt1eDvM+n6oMYzrOL2vhVKixD20XbTYoRCNcQUccocfEp+X0rJa+J+q701iPVYq7SNUkZ8vGov0ftxOZ2R+nB8zc3/U+75WfCOcsB5joeUaO0WBBij5dJO8KWDGevuOAXI7quHA8xUt2wx9fKNMRex+vAxXPGRpXINZ2rOt1TOK8PET1eIGjjoWWCcpHBY90TRbNpHA4HWBOH6JR8VNmf2iIuyvvM9DdZ2p2+u4r974Zi7S9jWYdJu0kozmizAE0JO4TpWmUgPE+9/p8pk+HZpFw3WZGqwf+6zlTEJZH3sSLQOPf0KRL2mcq/1vnTHsVWVeVMI/v3ZN3Y40PJA2AsimrPtNdDySyDUV8LzMrQEpR/4Z93XdDuLiKMUCD0RmLT9EP1TKw7NplwQIpyMC67v/5R9AVvQjLoYj2UAX/JhG4pYKQNZnVJ1oHCSTNNwR2aWa7q+DYG0EKhLXD KC9GspnZ bOEp6ZfiQMVsSLySsU8SuA5v2InIsYwsiGJhHrszlvXUYvaElLBsxal3533yzO3+GLAtWAhMI1cByvMix/BtWfTM8Jq52oQsobXLMf6wUVBVKBP+nqbUafWmIBQPT1K43A6ghDC6V+JZHN1/V+QN+w0FvZjiK7rOawKjvUIY7kyNdD/Gf2aH4e6Yze5aI55ACC8ewzK310jMRJkspZ8lvYbHIHfrs1CXGUv30Efpibijg7G1jFLLBW9q14+GrpHcMgSx6AO0thE7Yp4rYdVFLxqrOqWem1Me34aBtZnSTu4Bke5hUTnUcqJQZhhT98i5ZlKG+BJsKo3U6hctffaTFmWyWAjtcyPt1ufGzTec0C/xIvtyJVPm+bKTre7T9Cw8eYtrpDxt4URIia0W0eys3NoY+/se4+lLRorJi7C5Oz2XTTP+CxYGf2oXu2t8ov08HbD648DkDWfvAVcmU+RhYf0UR73HJoHXeRxrMH2X7QBPPsawahjAXah6jTYivNVo4TdNbz3xzIbNTojYW2CEV6vFC3aMRL1EHZQBy4/ENHavMZMtPVEB2slyIoVCSF2pKvEvMoOH7A9lshMChv9XXtAH2ePob85IWwD6iFJOAxzT9VggLCG1zzNGGynzG1+haArQRkoyGqH3gO/yx69+iBCjnm0Nn4YeLo1MA2P0jxY2djEZTgksO0xuNKyUzXVgqbmouAgOqBy+/IRYKkYvZNVxm5jHWOIVfbm68 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Stats flushing for memcg currently follows the following rules: - Always flush the entire memcg hierarchy (i.e. flush the root). - Only one flusher is allowed at a time. If someone else tries to flush concurrently, they skip and return immediately. - A periodic flusher flushes all the stats every 2 seconds. The reason this approach is followed is because all flushes are serialized by a global rstat spinlock. On the memcg side, flushing is invoked from userspace reads as well as in-kernel flushers (e.g. reclaim, refault, etc). This approach aims to avoid serializing all flushers on the global lock, which can cause a significant performance hit under high concurrency. This approach has the following problems: - Occasionally a userspace read of the stats of a non-root cgroup will be too expensive as it has to flush the entire hierarchy [1]. - Sometimes the stats accuracy are compromised if there is an ongoing flush, and we skip and return before the subtree of interest is actually flushed, yielding stale stats (by up to 2s due to periodic flushing). This is more visible when reading stats from userspace, but can also affect in-kernel flushers. The latter problem is particulary a concern when userspace reads stats after an event occurs, but gets stats from before the event. Examples: - When memory usage / pressure spikes, a userspace OOM handler may look at the stats of different memcgs to select a victim based on various heuristics (e.g. how much private memory will be freed by killing this). Reading stale stats from before the usage spike in this case may cause a wrongful OOM kill. - A proactive reclaimer may read the stats after writing to memory.reclaim to measure the success of the reclaim operation. Stale stats from before reclaim may give a false negative. - Reading the stats of a parent and a child memcg may be inconsistent (child larger than parent), if the flush doesn't happen when the parent is read, but happens when the child is read. As for in-kernel flushers, they will occasionally get stale stats. No regressions are currently known from this, but if there are regressions, they would be very difficult to debug and link to the source of the problem. This patch aims to fix these problems by restoring subtree flushing, and removing the unified/coalesced flushing logic that skips flushing if there is an ongoing flush. This change would introduce a significant regression with global stats flushing thresholds. With per-memcg stats flushing thresholds, this seems to perform really well. The thresholds protect the underlying lock from unnecessary contention. Add a mutex to protect the underlying rstat lock from excessive memcg flushing. The thresholds are re-checked after the mutex is grabbed to make sure that a concurrent flush did not already get the subtree we are trying to flush. A call to cgroup_rstat_flush() is not cheap, even if there are no pending updates. This patch was tested in two ways to ensure the latency of flushing is up to bar, on a machine with 384 cpus: - A synthetic test with 5000 concurrent workers in 500 cgroups doing allocations and reclaim, as well as 1000 readers for memory.stat (variation of [2]). No regressions were noticed in the total runtime. Note that significant regressions in this test are observed with global stats thresholds, but not with per-memcg thresholds. - A synthetic stress test for concurrently reading memcg stats while memory allocation/freeing workers are running in the background, provided by Wei Xu [3]. With 250k threads reading the stats every 100ms in 50k cgroups, 99.9% of reads take <= 50us. Less than 0.01% of reads take more than 1ms, and no reads take more than 100ms. [1] https://lore.kernel.org/lkml/CABWYdi0c6__rh-K7dcM_pkf9BJdTRtAU08M43KO9ME4-dsgfoQ@mail.gmail.com/ [2] https://lore.kernel.org/lkml/CAJD7tka13M-zVZTyQJYL1iUAYvuQ1fcHbCjcOBZcz6POYTV-4g@mail.gmail.com/ [3] https://lore.kernel.org/lkml/CAAPL-u9D2b=iF5Lf_cRnKxUfkiEe0AMDTu6yhrUAzX0b6a6rDg@mail.gmail.com/ Signed-off-by: Yosry Ahmed --- include/linux/memcontrol.h | 8 ++--- mm/memcontrol.c | 73 +++++++++++++++++++++++--------------- mm/vmscan.c | 2 +- mm/workingset.c | 10 ++++-- 4 files changed, 56 insertions(+), 37 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 41790e18bf3b..f64ac140083e 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1057,8 +1057,8 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, return x; } -void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_ratelimited(void); +void mem_cgroup_flush_stats(struct mem_cgroup *memcg); +void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1573,11 +1573,11 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); } -static inline void mem_cgroup_flush_stats(void) +static inline void mem_cgroup_flush_stats(struct mem_cgroup *memcg) { } -static inline void mem_cgroup_flush_stats_ratelimited(void) +static inline void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9a586893bd3e..182b4f215fc6 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -666,7 +666,6 @@ struct memcg_vmstats { */ static void flush_memcg_stats_dwork(struct work_struct *w); static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); -static atomic_t stats_flush_ongoing = ATOMIC_INIT(0); static u64 flush_last_time; #define FLUSH_TIME (2UL*HZ) @@ -727,35 +726,45 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } -static void do_flush_stats(void) +static void do_flush_stats(struct mem_cgroup *memcg) { - /* - * We always flush the entire tree, so concurrent flushers can just - * skip. This avoids a thundering herd problem on the rstat global lock - * from memcg flushers (e.g. reclaim, refault, etc). - */ - if (atomic_read(&stats_flush_ongoing) || - atomic_xchg(&stats_flush_ongoing, 1)) - return; - - WRITE_ONCE(flush_last_time, jiffies_64); - - cgroup_rstat_flush(root_mem_cgroup->css.cgroup); + if (mem_cgroup_is_root(memcg)) + WRITE_ONCE(flush_last_time, jiffies_64); - atomic_set(&stats_flush_ongoing, 0); + cgroup_rstat_flush(memcg->css.cgroup); } -void mem_cgroup_flush_stats(void) +/* + * mem_cgroup_flush_stats - flush the stats of a memory cgroup subtree + * @memcg: root of the subtree to flush + * + * Flushing is serialized by the underlying global rstat lock. There is also a + * minimum amount of work to be done even if there are no stat updates to flush. + * Hence, we only flush the stats if the updates delta exceeds a threshold. This + * avoids unnecessary work and contention on the underlying lock. + */ +void mem_cgroup_flush_stats(struct mem_cgroup *memcg) { - if (memcg_should_flush_stats(root_mem_cgroup)) - do_flush_stats(); + static DEFINE_MUTEX(memcg_stats_flush_mutex); + + if (!memcg) + memcg = root_mem_cgroup; + + if (!memcg_should_flush_stats(memcg)) + return; + + mutex_lock(&memcg_stats_flush_mutex); + /* An overlapping flush may have occurred, check again after locking */ + if (memcg_should_flush_stats(memcg)) + do_flush_stats(memcg); + mutex_unlock(&memcg_stats_flush_mutex); } -void mem_cgroup_flush_stats_ratelimited(void) +void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg) { /* Only flush if the periodic flusher is one full cycle late */ if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME)) - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(memcg); } static void flush_memcg_stats_dwork(struct work_struct *w) @@ -764,7 +773,7 @@ static void flush_memcg_stats_dwork(struct work_struct *w) * Deliberately ignore memcg_should_flush_stats() here so that flushing * in latency-sensitive paths is as cheap as possible. */ - do_flush_stats(); + do_flush_stats(root_mem_cgroup); queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } @@ -1639,7 +1648,7 @@ static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) * * Current memory state: */ - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(memcg); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { u64 size; @@ -4208,7 +4217,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) int nid; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(memcg); for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { seq_printf(m, "%s=%lu", stat->name, @@ -4289,7 +4298,7 @@ static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats)); - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(memcg); for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { unsigned long nr; @@ -4785,7 +4794,7 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(memcg); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); @@ -6861,7 +6870,7 @@ static int memory_numa_stat_show(struct seq_file *m, void *v) int i; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(memcg); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { int nid; @@ -8100,7 +8109,11 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) break; } - cgroup_rstat_flush(memcg->css.cgroup); + /* + * mem_cgroup_flush_stats() ignores small changes. Use + * do_flush_stats() directly to get accurate stats for charging. + */ + do_flush_stats(memcg); pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE; if (pages < max) continue; @@ -8165,8 +8178,10 @@ void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size) static u64 zswap_current_read(struct cgroup_subsys_state *css, struct cftype *cft) { - cgroup_rstat_flush(css->cgroup); - return memcg_page_state(mem_cgroup_from_css(css), MEMCG_ZSWAP_B); + struct mem_cgroup *memcg = mem_cgroup_from_css(css); + + mem_cgroup_flush_stats(memcg); + return memcg_page_state(memcg, MEMCG_ZSWAP_B); } static int zswap_max_show(struct seq_file *m, void *v) diff --git a/mm/vmscan.c b/mm/vmscan.c index c16e2b1ea8ae..2cc0cb41fb32 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2246,7 +2246,7 @@ static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(sc->target_mem_cgroup); /* * Determine the scan balance between anon and file LRUs. diff --git a/mm/workingset.c b/mm/workingset.c index a573be6c59fd..11045febc383 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -464,8 +464,12 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) rcu_read_unlock(); - /* Flush stats (and potentially sleep) outside the RCU read section */ - mem_cgroup_flush_stats_ratelimited(); + /* + * Flush stats (and potentially sleep) outside the RCU read section. + * XXX: With per-memcg flushing and thresholding, is ratelimiting + * still needed here? + */ + mem_cgroup_flush_stats_ratelimited(eviction_memcg); eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat); refault = atomic_long_read(&eviction_lruvec->nonresident_age); @@ -676,7 +680,7 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker, struct lruvec *lruvec; int i; - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(sc->memcg); lruvec = mem_cgroup_lruvec(sc->memcg, NODE_DATA(sc->nid)); for (pages = 0, i = 0; i < NR_LRU_LISTS; i++) pages += lruvec_page_state_local(lruvec,