From patchwork Wed Apr 9 23:49:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 14045655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9957AC369A2 for ; Wed, 9 Apr 2025 23:49:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37F322800B4; Wed, 9 Apr 2025 19:49:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 32E212800AF; Wed, 9 Apr 2025 19:49:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F4492800B4; Wed, 9 Apr 2025 19:49:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 00EDC2800AF for ; Wed, 9 Apr 2025 19:49:21 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6A17D589C8 for ; Wed, 9 Apr 2025 23:49:23 +0000 (UTC) X-FDA: 83316149406.29.6E98A95 Received: from out-178.mta0.migadu.com (out-178.mta0.migadu.com [91.218.175.178]) by imf20.hostedemail.com (Postfix) with ESMTP id E1B421C0008 for ; Wed, 9 Apr 2025 23:49:21 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=tW4eYrBv; spf=pass (imf20.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.178 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744242562; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=79KmuShFT0OWZoCqYUxoC00IFXDPi718JpVr84N2Wa4=; b=q19aRUXjmrdR8zjD+ZZIwfYO1l3awzwAk1s0SjMKUgbmJVVWiksPiR0DnzB98GBEeOPVNN HGeyeGhjKqz3c0WXgjAN5Xx5r687OSF5ERHIes19WH3WWYwL8lc/7VCnF07b7DCN01FTdp svWyT4dKQHDmYPK4hzFl74PVqtCK4x4= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=tW4eYrBv; spf=pass (imf20.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.178 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744242562; a=rsa-sha256; cv=none; b=aZ5n4jP3HmG9ysgCobFqAm/rBlhKCalWEgPs1mNij5KiWok882ph5S9qVE/EEunsm5Qb3E zjyy8hJJUda8pzuHNdZiAJwRkl8PpUTiUGFbQJJLP7TQPiS0kVihrXG/TilDNjkheUb64y 1fQPnYKC4ChI5IcFbKXiNAb72PZwsjo= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1744242559; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=79KmuShFT0OWZoCqYUxoC00IFXDPi718JpVr84N2Wa4=; b=tW4eYrBvwAA03RrYgu0wtSylOO4sNUg37R9ECDivbgJG+hNF8BRnqpppDnR/mqr6pDTKQv SaaGCRb3fNeJtPc6gTMEmC2LBKx9ga+ANM0nwDv+MoY1BFmebvIxlsX+doz9nZCqfkFVdk ogjQbR5VbSEyNrOTa9PHKKt80bTWVGU= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Yosry Ahmed , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH] memcg: optimize memcg_rstat_updated Date: Wed, 9 Apr 2025 16:49:08 -0700 Message-ID: <20250409234908.940121-1-shakeel.butt@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: E1B421C0008 X-Stat-Signature: zu6h6ixmdi9einwmizprzmrcwr975p4g X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1744242561-906651 X-HE-Meta: U2FsdGVkX19zCYS5oRDkQ9jurSU0QW19XaAXHar9fbRtoqFSp92RFCJq1nVNj8JYv1AZ5GsIXlyE2AnVrcJ04ybBKONksZPo4raufvIJIaUBslNVhDiE33RJV9RMWpixapxYlcUL9nYdZDvF7B/PgKmQ2+XeqgZG/73SSupsCMz98DYw93N3tnZrOtvg1NbfR2kHDwWO0iOtDxfaJJZrRZ3N0JQb9yhiOsV2GgEiE45Dn24iiUIcs0JqeIuggiy1XPaFzX6OP19c0sHPhxXNxLXYLT9AX7HffRFx+NQ7BT9UvvBnqrtYV9/8TAd4JO6/L59UUgDLHgE/JIoX9y7uRpW1/jDUlf7Zf4u/1eCGbAD8HI3+YpVlyF1BnqBDlmwc1JVMX392Ys1BH90plCpA7/0luLLW3bnLN6N4OQ0PFtgpqKQzCZNXLXiBRYMOIBmuF4bvC4zDclGLMjSwj/uamRrvnz2p2xGGNIuiigPujElPpYd4F7uGJ7rMXSIREiRiiY/KHm8plnrVEwFPUaF/d2c4kvUz/42jtmEuc9Lt+3Wfzx8IKSvVoy0REcblmLRwXNvOgpNcjH0lT4ZwcVDgw6hPB1o3XMUZRR5q8MDbyZFcxX2oDlCJ0OiiMPGm17/174ns8lcj+n4b+tjsbe9P20Jcq+YDVJmB+QbYfr1v+Y8Z/W36ZoOJFQ7bfwxOMvFeI57J3SoED8k3dYLv5dnIJD0CErGhRYyqMHdY9qxa+e0DHn2XdqNCsYotOVzkKP418lSzQwV01oYeeJMVf6/A8isljjq2bCbKMk0Y+1pql50fEo1GSOwWV3Wrgq6zm5Ri2UrvXi1zOQ8tIvQxwIV7O3aatqIq7FJ7c35gDB7L+yBFNd2M/bumYtJ0uYEvf2x0KyiFcHjW3NHCu3dA6aHM6gnu8MaIUxR9hQkvnA/sl68O8QRMioAd8u0u+dwh1XVMdtKJC39tAGY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently the kernel maintains the stats updates per-memcg which is needed to implement stats flushing threshold. On the update side, the update is added to the per-cpu per-memcg update of the given memcg and all of its ancestors. However when the given memcg has passed the flushing threshold, all of its ancestors should have passed the threshold as well. There is no need to traverse up the memcg tree to maintain the stats updates. Perf profile collected from our fleet shows that memcg_rstat_updated is one of the most expensive memcg function i.e. a lot of cumulative CPU is being spent on it. So, even small micro optimizations matter a lot. Signed-off-by: Shakeel Butt --- mm/memcontrol.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 421740f1bcdc..ea3e40e589df 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -585,18 +585,20 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) cgroup_rstat_updated(memcg->css.cgroup, cpu); statc = this_cpu_ptr(memcg->vmstats_percpu); for (; statc; statc = statc->parent) { + /* + * If @memcg is already flushable then all its ancestors are + * flushable as well and also there is no need to increase + * stats_updates. + */ + if (!memcg_vmstats_needs_flush(statc->vmstats)) + break; + stats_updates = READ_ONCE(statc->stats_updates) + abs(val); WRITE_ONCE(statc->stats_updates, stats_updates); if (stats_updates < MEMCG_CHARGE_BATCH) continue; - /* - * If @memcg is already flush-able, increasing stats_updates is - * redundant. Avoid the overhead of the atomic update. - */ - if (!memcg_vmstats_needs_flush(statc->vmstats)) - atomic64_add(stats_updates, - &statc->vmstats->stats_updates); + atomic64_add(stats_updates, &statc->vmstats->stats_updates); WRITE_ONCE(statc->stats_updates, 0); } }