From patchwork Sat Mar 15 17:49:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 14018164 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CAA0C28B2F for ; Sat, 15 Mar 2025 17:49:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 07315280003; Sat, 15 Mar 2025 13:49:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 02266280001; Sat, 15 Mar 2025 13:49:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E55F0280003; Sat, 15 Mar 2025 13:49:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C53F5280001 for ; Sat, 15 Mar 2025 13:49:46 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 19E36161F4D for ; Sat, 15 Mar 2025 17:49:48 +0000 (UTC) X-FDA: 83224523256.13.31A8733 Received: from out-186.mta1.migadu.com (out-186.mta1.migadu.com [95.215.58.186]) by imf04.hostedemail.com (Postfix) with ESMTP id 68E0240009 for ; Sat, 15 Mar 2025 17:49:46 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=LVtOFiDf; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf04.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.186 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742060986; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cmS5hv6xwjGOhhAkz0IaR50zSoZuuoSBRO/S5eqp4mA=; b=hSPwZix/WRvawoicfXLObbBurf6fqQk2Q7RrUG2ZBbF/kcnhnUW2mANAb0HUlDRZcB0GgG qCFZbJlPE8Q/MR9ORbQJG39jBYA9rjfwM3Ehv2HQj3EQ7nJWLUqCRZd1bGIlu+5+ez6Pp2 bR4J/ROo82CUPG2icLQzf3G1F3Vd2EA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742060986; a=rsa-sha256; cv=none; b=8G+n/OZkux/FRluOEcpV/MSyQgo8Cg4OcIVDsF8q8jB5SMQSIDl8LyQlMP/XEr+Q9DM0tY vz9EIVYi5S5LII2rdx287z+Ew6ot2mPSPwZH+355eyC9h0dGQI4E8XrWwqc/L+gQNyc0dN xzz3HprjWvGiKzxHhaJzoRt95KAiTHA= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=LVtOFiDf; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf04.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.186 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1742060984; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cmS5hv6xwjGOhhAkz0IaR50zSoZuuoSBRO/S5eqp4mA=; b=LVtOFiDfJTfbraclra86x+fCaQarFRpDuvgbtcbJPyfEGkmeTvyYet4jd9Nwe7MYVclJnn uuJLQQVD62W9ZBqiSKhDpcCrLRrD//QYT9PyfmzobpzgN56PeZJewgqDV+tc22Dfa0cBft 3TEADM+PDuzpeOAVrq4zauWky81Tgxs= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Sebastian Andrzej Siewior , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH 1/9] memcg: remove root memcg check from refill_stock Date: Sat, 15 Mar 2025 10:49:22 -0700 Message-ID: <20250315174930.1769599-2-shakeel.butt@linux.dev> In-Reply-To: <20250315174930.1769599-1-shakeel.butt@linux.dev> References: <20250315174930.1769599-1-shakeel.butt@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 68E0240009 X-Stat-Signature: 3ez8w8wy8awz3ip1y564appok4umry69 X-HE-Tag: 1742060986-257301 X-HE-Meta: U2FsdGVkX19wSSN0C6ZKMLHVE79x6wqwBr1VjPHCJ/JqOKg08b2x+AG0TdgB/nHlPgypMDBs+BfJmxNYbN1YJD4UVM287Y1A6RS5sQMF4ViTh5qrwwIrh+LLWGcnVaNsNBnsBU5VbmxJ8BkjykUP3EXlB2mqtJT/J50TfIgtB30YK3dGAUorwaGSmOIgtCe9MTNUWI/2bTIiFZ+tcx+mKEg3P4URZC/D5S69Aw68IBXVyfiy2i5jMFSwALj0ZsdKbrIeVgjiwJSd/x08gphsw+Lt5N/xkZvG/wBGMqqRRgBU5rHJQIJ5NIg2G/P/RAspDjLKw7MX+3ufBAikrM5xNWywaDtwtxlAYDHH0rCxyWOqJuyzoTVaw4FimZ93AggAr3CE1QzAeRQYu7AvoyHdcFFr6Vp0k/wNxxRY/gwfzvLZo0LAH+V1iUcGyAAnlSDHxrEnqrePATFmkO6Ge45/bzXTpKrTjJq0eP8fMW8lpE5v3KtSva8CXL3d2NjJbjMOJI+AX56hAjVYK3gaA0cvGl4W8u9Z1qVpnCFrcFuzIeekDaEZWMGrYsehXeytK41uJxMCHyZWVD3Py/uK3EAOBEl2Q8pF4//eBaewCxL2HAtW3fz/j7G6sLs1d2sRPN08t1mR+d/fWFOtUe2xZjFzaaSPIOtB8QP/i08B/Ffe0FzwEP43tAEwSp9jsVxw6TaD+CUKeWzm5P1djv8vBC5Q6yclJEGYi11knRxERvkrUzd0TyjoJY+Xk0BKvYn/5WEZRkWJUWFnMiHQjrjXseETxEzoCVgeA89KJd1u1rfP1yR40fVlNL2q+gdEuhe9hnZcUtHmZPzkWUMHNtuOc8UKY4kNiGQTFjzVMqLTcfhIWn3v9JDupB40lOpaDk1g96KYDU0F1MG0VLhBwsLuFmJEwsHWmckqv6V2KkhR3TZ0YFySxU/BvdTy8iILXC4EYgq6y+ZK5xeoQQIPPXAavAz zY9mV+uO +uTlMxRZCu7twB4vwmSiEU++cnKT+1O3zxt0US8o40FqnOkD/O5NbrvWor1VFy4J/IMUbCMS1g1QgNryQDLSAUSCZu068HHIA6TQRt/j6C1dBI3g3Jf4b9texis01ApP63WrrzMHiMy70n1xF4CcTL65cHYz0YOw1xnC51yU3Vc8pzvFSjg+9ujZIXcNI0iwB5pqpcwL9ot6ZJiVihsgIhkYhWw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: refill_stock can not be called with root memcg, so there is no need to check it. Signed-off-by: Shakeel Butt --- mm/memcontrol.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b29433eb17fa..c09a32e93d39 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1883,6 +1883,7 @@ static void __refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) drain_stock(stock); } +/* Should never be called with root_mem_cgroup. */ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; @@ -1892,8 +1893,6 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) * In case of unlikely failure to lock percpu stock_lock * uncharge memcg directly. */ - if (mem_cgroup_is_root(memcg)) - return; page_counter_uncharge(&memcg->memory, nr_pages); if (do_memsw_account()) page_counter_uncharge(&memcg->memsw, nr_pages); From patchwork Sat Mar 15 17:49:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 14018165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE09AC28B30 for ; Sat, 15 Mar 2025 17:49:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CDCB280004; Sat, 15 Mar 2025 13:49:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 62CC3280001; Sat, 15 Mar 2025 13:49:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A97D280004; Sat, 15 Mar 2025 13:49:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 22FB7280001 for ; Sat, 15 Mar 2025 13:49:49 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 987E3AB063 for ; Sat, 15 Mar 2025 17:49:50 +0000 (UTC) X-FDA: 83224523340.10.3EC4B1A Received: from out-179.mta0.migadu.com (out-179.mta0.migadu.com [91.218.175.179]) by imf09.hostedemail.com (Postfix) with ESMTP id E1B8C140007 for ; Sat, 15 Mar 2025 17:49:48 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Ejy9vqLu; spf=pass (imf09.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.179 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742060989; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oUGjl5IUOeohrue8CflY3qReHtC2T034o0a04sP/4ac=; b=q+bqT6FsrNPtEFv+diI2HWE0F0Oa5BZVJzCYyHKSV0zvMOotfA7PROLfPUowvi0UJm4/EU 9y39GOxWvhwQMZTNb9ziVmGo4JS/iK/8GCP6rOsZKuskn0RrK3nRyefZu1fG/Eb8cKtolE RElrhrD6cdjc2jTVEt5fNeOk7vZlZ7s= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Ejy9vqLu; spf=pass (imf09.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.179 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742060989; a=rsa-sha256; cv=none; b=cInw8npEBpVxjOSt+RmO2f8iSRQqMmFQbvKoyuNlfPydMirrQRT+yfTOi7AAB+HXslT/3u PeLOluYcTxQs36JRDNEPRCblJ1jiH2rdYli5RpEweABZwAC/aCz8C9+q2t7LRjtRQMHWs4 Jyfzay1i5FuUFjjTcxNg6cptWVghi/8= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1742060986; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oUGjl5IUOeohrue8CflY3qReHtC2T034o0a04sP/4ac=; b=Ejy9vqLuhMtkzwYa4BWZ2IiChp1s5sstyd4OAgBMSkax1T0pVT20KV+VkvWvY5R0mCrrCA CWNQcrKeNlCZoz9SnQEPbL3cjNDbBmqiM/2TtzFlraiYrRnTqDXLQBiBdgjMSa88A7UtW7 qGO2/7k9JcdwLvf0v1aCNfGdPeArjgs= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Sebastian Andrzej Siewior , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH 2/9] memcg: decouple drain_obj_stock from local stock Date: Sat, 15 Mar 2025 10:49:23 -0700 Message-ID: <20250315174930.1769599-3-shakeel.butt@linux.dev> In-Reply-To: <20250315174930.1769599-1-shakeel.butt@linux.dev> References: <20250315174930.1769599-1-shakeel.butt@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Stat-Signature: wmr45uy49thwaij6n6pnw7insb9peacf X-Rspamd-Queue-Id: E1B8C140007 X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1742060988-951903 X-HE-Meta: U2FsdGVkX18Ef+iOYSaGTNqZmrsO4pAw2WNS1rO8LhR54Nf4EJsm3yjU5zHEmL64IsZhDhLLEjwNV1I6xH+9+1JgxE0Oxl0D+TnI8tUSzaTegNIakfgeuNTJCR0V4lBPgEeZbqMUGqti3f++cSl7VE62QgblDI/LwIRscoIg05wCv/akExRjqloAJ9Ab8+bo5stfMvbm1X+vyK8JULFZkVyXEcjBRUPQMMdKXEmco/H+eS7j3q8YBC0b53d2DIH9AGW8vTf3v3dLxtIcCihMXosnUvwXH3iDEuzxQABeujo68H17G5ajdMWu5zja7RkQeU3HRNhoBwXW54gVoS1lp3HbjFEJg8KQUAaL3skAm4DCI5dUXYtLOHXEWfoplOnrBsVOXVTz+mr76JJQ9UjrscrGEvslU8fe3v5SBqDpGHSNRC71mbgL5XY/DvTuQDOuNKdq8XnuEDnYXZzERrhlt/Zk6nyUHqmqwt7I1eK44ABnrqn38UBQanqRG9PDf/fNpEPvKSIx6BhLbBvxEwpJLYrmBF/6D11GvRMru/65q6YctqlaLMAPus13OMLV4CRmOdjSJH4gHrgdO22jk9nwjtm+nkCKVjyVUWBNYe/toUhUNDvbOHXvhgudTWPEbzCmGtK3KCQc+QaOj9N7Q1MTxiQnOiu1Ylwm9BBsDojvxpxII6HAhSmq+9q+fNh+bM54DMv0hrW4VoidvjJ4b0tBkGI9wzSzrb/tOSUPKtkAbi6Eco9aZeqYHGu40dJS9AefUVFPY7WThMnoimWFPCOY1yDxJFh5JP844IK8ymzlXKhhLMFDBWBoc1xyNMVWK2jXVVjkFbRJ3u+Jc3t1oKQ5E0YaoChvUHtu+aIKRJagH5YyZN00ADlKxF8TvWRK9vMrAbPa+60sUvJlPnqEPWblLJNRddRCCfo2uK7dPSgPUF07q99z7H5MRLsvc/Fv9thESl2FG0nLNzpwHJuO1Th t9xsZ/nR w9D2tYzoPGKeBNpXboz0HimeIj/aX7qgxJGamqQ5CDbpEoZHMtTRHNc+CLjIgoS/Xw2LiqVcFcxtXuW4/Gu0POBgeOJu+ydS5Z9zGCuQW1bj3zy9CjxZL0hzHr0v4if4GFlDHxbgixkQnb9BFYP5oiaNAsQ+vfI2+kF3qRMaXS1jYqsm/0tzdjX8/Ur7uTFh599Ntfg1aqU2FnXuypuGrLol+1w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently drain_obj_stock() can potentially call __refill_stock which accesses local cpu stock and thus requires memcg stock's local_lock. However if we look at the code paths leading to drain_obj_stock(), there is never a good reason to refill the memcg stock at all from it. At the moment, drain_obj_stock can be called from reclaim, hotplug cpu teardown, mod_objcg_state() and refill_obj_stock(). For reclaim and hotplug there is no need to refill. For the other two paths, most probably the newly switched objcg would be used in near future and thus no need to refill stock with the older objcg. In addition, __refill_stock() from drain_obj_stock() happens on rare cases, so performance is not really an issue. Let's just uncharge directly instead of refill which will also decouple drain_obj_stock from local cpu stock and local_lock requirements. Signed-off-by: Shakeel Butt Acked-by: Vlastimil Babka --- mm/memcontrol.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c09a32e93d39..28cb75b5bc66 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2855,7 +2855,12 @@ static struct obj_cgroup *drain_obj_stock(struct memcg_stock_pcp *stock) mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages); memcg1_account_kmem(memcg, -nr_pages); - __refill_stock(memcg, nr_pages); + if (!mem_cgroup_is_root(memcg)) { + page_counter_uncharge(&memcg->memory, nr_pages); + if (do_memsw_account()) + page_counter_uncharge(&memcg->memsw, + nr_pages); + } css_put(&memcg->css); } From patchwork Sat Mar 15 17:49:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 14018166 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A75ADC28B2F for ; Sat, 15 Mar 2025 17:49:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30FBD280005; Sat, 15 Mar 2025 13:49:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 29700280001; Sat, 15 Mar 2025 13:49:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 111B7280005; Sat, 15 Mar 2025 13:49:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E364E280001 for ; Sat, 15 Mar 2025 13:49:51 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 57927C1E7D for ; Sat, 15 Mar 2025 17:49:53 +0000 (UTC) X-FDA: 83224523466.23.032B1AB Received: from out-189.mta0.migadu.com (out-189.mta0.migadu.com [91.218.175.189]) by imf02.hostedemail.com (Postfix) with ESMTP id A2DA080006 for ; Sat, 15 Mar 2025 17:49:51 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=EsH21tpo; spf=pass (imf02.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.189 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742060991; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D9SGS5ZGVTJYUs7bi8rBcO4GG6XSRL+h8cbbja/hnd4=; b=gwvLOL0LKs4bl6jmAOex//YhlAw8EnXBZt12MpEtxJ200Z9W1Uqn8GCSUKRunbF+dKY/uK vx6aKz4m1BdaYNS6UHdv0O7fC9jV5TwrAUDMTIc4kleXWp0j/g4w5wjA0tLK7DJHgkx2WY E688V4Ic3cEyhYwAO1XaxUyc6E5UJ+4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742060991; a=rsa-sha256; cv=none; b=X4qlRSSnYKiunzxSuu+73zqbrPEVjJCBvi1wiR/mLCplY1eu7zzlYOk+6CVb6nPKZKTEsj WgLMhO30zh+aGR4TPFrLyL6NfWXYu8D11ScPguPHKzq14RAcZJiPu3KR/ToODn1nrrL1AX Lfhi6iRfaPjXIpzfv8o+5JymlibSwL0= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=EsH21tpo; spf=pass (imf02.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.189 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1742060990; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=D9SGS5ZGVTJYUs7bi8rBcO4GG6XSRL+h8cbbja/hnd4=; b=EsH21tpoGivxemQBYrsxMd+ru7bHcE8LYPnGChI+hOtYbRl8sO0Czs0Q7Br3PmMb9k9bpa gce6M48m5W+CSQy02N7oSlHbmeiv0YDYjFi2vfK6zdqUHBTGYUDd4M1RiDkREBX83mGRH4 YD4a7IZYjimlw7tBXcpiVnolqcmcIyI= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Sebastian Andrzej Siewior , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH 3/9] memcg: introduce memcg_uncharge Date: Sat, 15 Mar 2025 10:49:24 -0700 Message-ID: <20250315174930.1769599-4-shakeel.butt@linux.dev> In-Reply-To: <20250315174930.1769599-1-shakeel.butt@linux.dev> References: <20250315174930.1769599-1-shakeel.butt@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: A2DA080006 X-Stat-Signature: 74btm3oar3xjahqxphto9dfba8ixyrcq X-HE-Tag: 1742060991-866007 X-HE-Meta: U2FsdGVkX18ooYvSt/jpSNdvrX81qC4v38sFytfcdr4Sv9GwSs8KT/qGbhdv0yWQUMRA4lmqNSbybISz8ntMkaqs0rhA7JXMPz4voiEpgKHdCtHxN3GV9HPcLjMoAvptEzPL9jBMn2RNZya0NzOIhTj9LMRc5EoGpahE2311vZ2m/5CtHDNe4Pd8DuWaPwKkHDhTqmLGmjjFtWVssf9IKqxPKe8+Wa8fd8N6Hqq7Wzbu/6ADU7awKisY0bj30MWKk1STQxJxjimTDJNs2U0hYlPYuoUYtmUM7QAVa+lteMtb2eVd8jrgNjkpdZE1wFkNRuKvt5UbhlMx0T+0IbkVKVSLFJBHKfpqzG14puuADaGvMjQtY/eRhCEPa794kz1VLm2tAZXuzuN/05lnjv0k3XeswnfP+TOeoTFiss7vqf2+RXzw+/qA/cXYIDHtiSgv644qZJ+MCy4uwCZWG0KtTXc0OtILY25HaOba8RIbsyVPYKOuRE+eva6GYDD3+vd1b4wGq707grN/JVFWe61v8j2JKohX9GcvOJNgDv4nfFg9B78pmcaTitueP/4ZswHG4wVlfcZmxWe5L8qfAcQOhXXA4vK/+AcX8MD6s13ddDFt8V2X9vzwiP0F6GHNI3TTtQVxDmwuxSbYH/+qMearnp6Dx+DzHMmiLHzihorRc0ty3CN3Jn+CChXBzVGXFxAu70XiMc6Oymh78MUf94sOv7iW+ArIYCKDU1ZyYzvNrphEto6N5n1QNaR9YOWBGFEQP9ZEZZ4zS2qFHI6+DUEroHQw6BmUtkxGFS6W9cZFcrsUAKtChvianW6ltctJ/aevcTRVkO+A0pbeORFNTFGKioRmXd7CVXd8+p5kOFiS3n/Sz/rfzgcc59hMtXNttCIyujb/TmkmCMOwwRKllQ3L9CJ1FaKNitVMmuAVmO8CdS8mzDA/LYvCWWS42nY80vuuLglbvcZl7R+ncaOWhj2 RNTthS3P 3Z+pDURWm8mx1H9Wu+FwivslZeiTQz79hzK0sayA5MlJUfPkdGi+hDeC4air4eqkcY3G7WvqjpH1LDjHDm59rnCzgZXRt2/z/E4a3nDAyXIx5H+Ztd8lvZiDKy3gHquS6WrwvDfAOiLxiMOn+w0o0w5ku50TdMFq+MyujJysRQd8h/OIblU4Zi2ZG/efcnC0hW3D6zHhnlshvtoWqDqEqMqlJIrVUXaDJWOZL X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: At multiple places in memcontrol.c, the memory and memsw page counters are being uncharged. This is error-prone. Let's move the functionality to a newly introduced memcg_uncharge and call it from all those places. Signed-off-by: Shakeel Butt Acked-by: Vlastimil Babka --- mm/memcontrol.c | 28 ++++++++++++---------------- 1 file changed, 12 insertions(+), 16 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 28cb75b5bc66..b54e3a1d23bd 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1816,6 +1816,13 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, return ret; } +static void memcg_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages) +{ + page_counter_uncharge(&memcg->memory, nr_pages); + if (do_memsw_account()) + page_counter_uncharge(&memcg->memsw, nr_pages); +} + /* * Returns stocks cached in percpu and reset cached information. */ @@ -1828,10 +1835,7 @@ static void drain_stock(struct memcg_stock_pcp *stock) return; if (stock_pages) { - page_counter_uncharge(&old->memory, stock_pages); - if (do_memsw_account()) - page_counter_uncharge(&old->memsw, stock_pages); - + memcg_uncharge(old, stock_pages); WRITE_ONCE(stock->nr_pages, 0); } @@ -1893,9 +1897,7 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) * In case of unlikely failure to lock percpu stock_lock * uncharge memcg directly. */ - page_counter_uncharge(&memcg->memory, nr_pages); - if (do_memsw_account()) - page_counter_uncharge(&memcg->memsw, nr_pages); + memcg_uncharge(memcg, nr_pages); return; } __refill_stock(memcg, nr_pages); @@ -2855,12 +2857,8 @@ static struct obj_cgroup *drain_obj_stock(struct memcg_stock_pcp *stock) mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages); memcg1_account_kmem(memcg, -nr_pages); - if (!mem_cgroup_is_root(memcg)) { - page_counter_uncharge(&memcg->memory, nr_pages); - if (do_memsw_account()) - page_counter_uncharge(&memcg->memsw, - nr_pages); - } + if (!mem_cgroup_is_root(memcg)) + memcg_uncharge(memcg, nr_pages); css_put(&memcg->css); } @@ -4689,9 +4687,7 @@ static inline void uncharge_gather_clear(struct uncharge_gather *ug) static void uncharge_batch(const struct uncharge_gather *ug) { if (ug->nr_memory) { - page_counter_uncharge(&ug->memcg->memory, ug->nr_memory); - if (do_memsw_account()) - page_counter_uncharge(&ug->memcg->memsw, ug->nr_memory); + memcg_uncharge(ug->memcg, ug->nr_memory); if (ug->nr_kmem) { mod_memcg_state(ug->memcg, MEMCG_KMEM, -ug->nr_kmem); memcg1_account_kmem(ug->memcg, -ug->nr_kmem); From patchwork Sat Mar 15 17:49:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 14018167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 268AFC28B28 for ; Sat, 15 Mar 2025 17:49:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E60EA280006; Sat, 15 Mar 2025 13:49:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E1045280001; Sat, 15 Mar 2025 13:49:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD86C280006; Sat, 15 Mar 2025 13:49:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id AF6E1280001 for ; Sat, 15 Mar 2025 13:49:56 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E989A121E91 for ; Sat, 15 Mar 2025 17:49:57 +0000 (UTC) X-FDA: 83224523634.15.F1E33AD Received: from out-182.mta0.migadu.com (out-182.mta0.migadu.com [91.218.175.182]) by imf09.hostedemail.com (Postfix) with ESMTP id 43575140009 for ; Sat, 15 Mar 2025 17:49:55 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=UCQl96Wa; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf09.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742060996; a=rsa-sha256; cv=none; b=lHlWcJJ8pYXpSrcFDWEXkwQU12DbHx1wA3g//AY5zDAhJwS8Xk7oZ2gaodofDHICxSCm3v 6JlJJPEvpRiuSEVwZIId/H/zeyTJ8ab7h5XM623+Xj7/Vk0mfhPqdFGH7eX95+Y8A+cRmK x7nEHK3Do/TZXMLMG6BfPc5lvvUM37s= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=UCQl96Wa; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf09.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742060996; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Bc6pQ9m45TBV5JfKg0PdI6TDpBclbZtCDqtLw9XzNuU=; b=4CogCOtDIIKqqTDTbBne8z5Oc1zDtp5fiD1iNeS64Hooj9xu+IM+p9oBhRO+mxXDmtqFHK 8vlY95E3U1F+io2Gcr/dxeIXET/nCL9A5nre48joN5xrvTTBXR6YntS+PMMVWiM9D6UCI+ dI55y/M0qyn0O4tqEp1PpP+JQ5ZXk8I= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1742060994; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Bc6pQ9m45TBV5JfKg0PdI6TDpBclbZtCDqtLw9XzNuU=; b=UCQl96Wa1/yEqZlafoPIygsX+2cWELzvlzoYrINa1G0wvMDEnFDPIjjN7q4Jw2b+9PzjIQ IdojMmemgziZOSXNmvSCv6u3VhSKT8RUTTHb5RfhZ0lRiieq8JAC75DNFjhgeKcfJA0DEC /WgjCitdjgMpbaZKjihNY1nTs+Dk+Tc= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Sebastian Andrzej Siewior , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH 4/9] memcg: manually inline __refill_stock Date: Sat, 15 Mar 2025 10:49:25 -0700 Message-ID: <20250315174930.1769599-5-shakeel.butt@linux.dev> In-Reply-To: <20250315174930.1769599-1-shakeel.butt@linux.dev> References: <20250315174930.1769599-1-shakeel.butt@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Queue-Id: 43575140009 X-Stat-Signature: 1rxk1wtbe1yq6c19ffs4u9s6wf56fhm8 X-Rspamd-Server: rspam06 X-HE-Tag: 1742060995-730710 X-HE-Meta: U2FsdGVkX180vmWl8hNdKFro4YKdUH8d3e6/MIzBQppriuGrvAj5Q7GLkLbS5jKjiIgXWLGHC7dgvGSzqPA0HItxrB3kHOxXP8tD2UJwBKW3Yd0hLPo/PoxhLxS5YMjtC2XhUcRe+GrxkRJdNLDhyk6tqscUoCfLRsN9qk6Gy0u69Yvl3kw1nlEFwjDJOT8cw3brewwd0JtpYjz2u2TmY5GUz5Owz3vkag9hJK+dPwISEN8o4racJrlPoIZlMMSTK315PesoKCWF+KD3WqQn/4S3/ZszIT09t1Geajjxh2HSVis2K+i2iA1g3X/riL/eU/zk/7TDUyAdX/gOVo/DP5NpYh5//gNqCGOXHqIJeWtM68N1WQUS6FwetnnJpU1PyWFIF2LxA+wuzMxiariWVO7vQnWbxQDHyanq0mYUFhvi2leLmZA4HJGSyIiLgQyH0lywRxX+A5SmoMDU3zIzX9y/dPMV6ajO5z/VQvDAy8X2kKQqTG6JWYEjvCniHsk10WJ/GocKa3SXh0gWLsG33RW/wxvDhOMIA1N07qq2tWa0qpU6n/43vul60xH83QV1RueFL0+FUdFKIaUKcxtm0fytKxsaYbqShwB/uhYzOfV6T+7FA+ph9vkqFWDaPYx14wso0/lA9Roton0ODI9wKHBFZnrFpZwz6qdHc357vbmNSOX+Shp+Lx0N7h1Bt478U/Se4aGBaN8Z3+1SMTmfujF4kEYJ32ps69QiF/qt3kvAMlKug8w5yn2gMOyHtaRVlAYaeJ7z2yC1IFAXEt4rd1TKWG9TTfKW392OPtOasHD2k/2euhGejk08OPnuf5aMQZDeT3ejlO2knYEPVEm3hbVaWd4oXigFqUwRe6O/yr4oIlB6Q+rs5F5/b9q5y1fQyG2YcIRuN8sDSzFlFyLYHHC6b8um/m5J6HCFBWmUtYbSdgazJNAimR0DLyR+0Rs4gxx85glNQs0DGVR/IgO vBvqwy3e sJGCMxSny5TsLRRe/sqJWXc8dW3pGZyJNgo0VelNWauFXNP+GN9rdv0MECCyJ9sn4hlX4z21etdVf/naJx9xWb/eiyKhCVqTZ0/xxpD80BeO2LplZCzPZMq04zwQ0Y8tcu/kkyKWI4B0v0Z57jHt+UodhTFd2rpn9GYCQesXtMrsoFca0F3Og+mXst9bLOSHIfZn6EeM3itimzuz3UnHWsD8bNlSDyjIpOG6N X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are no more multiple callers of __refill_stock(), so simply inline it to refill_stock(). Signed-off-by: Shakeel Butt Acked-by: Vlastimil Babka --- mm/memcontrol.c | 32 ++++++++++++-------------------- 1 file changed, 12 insertions(+), 20 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b54e3a1d23bd..7054b0ebd207 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1865,14 +1865,21 @@ static void drain_local_stock(struct work_struct *dummy) obj_cgroup_put(old); } -/* - * Cache charges(val) to local per_cpu area. - * This will be consumed by consume_stock() function, later. - */ -static void __refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) +/* Should never be called with root_mem_cgroup. */ +static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { struct memcg_stock_pcp *stock; unsigned int stock_pages; + unsigned long flags; + + if (!localtry_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + /* + * In case of unlikely failure to lock percpu stock_lock + * uncharge memcg directly. + */ + memcg_uncharge(memcg, nr_pages); + return; + } stock = this_cpu_ptr(&memcg_stock); if (READ_ONCE(stock->cached) != memcg) { /* reset if necessary */ @@ -1885,22 +1892,7 @@ static void __refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (stock_pages > MEMCG_CHARGE_BATCH) drain_stock(stock); -} -/* Should never be called with root_mem_cgroup. */ -static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) -{ - unsigned long flags; - - if (!localtry_trylock_irqsave(&memcg_stock.stock_lock, flags)) { - /* - * In case of unlikely failure to lock percpu stock_lock - * uncharge memcg directly. - */ - memcg_uncharge(memcg, nr_pages); - return; - } - __refill_stock(memcg, nr_pages); localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); } From patchwork Sat Mar 15 17:49:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 14018168 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73369C28B2F for ; Sat, 15 Mar 2025 17:50:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A4B8280007; Sat, 15 Mar 2025 13:49:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 32D11280001; Sat, 15 Mar 2025 13:49:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CF2F280007; Sat, 15 Mar 2025 13:49:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id F0896280001 for ; Sat, 15 Mar 2025 13:49:58 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 66022141E48 for ; Sat, 15 Mar 2025 17:50:00 +0000 (UTC) X-FDA: 83224523760.19.1545B73 Received: from out-180.mta0.migadu.com (out-180.mta0.migadu.com [91.218.175.180]) by imf24.hostedemail.com (Postfix) with ESMTP id B59CB180004 for ; Sat, 15 Mar 2025 17:49:58 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="e/E/nuyl"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf24.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.180 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742060998; a=rsa-sha256; cv=none; b=gPQA4CjlxpYX3MaccQmrhwS3tcZw+kfJewSCwJwREpDqMKeUy9iYyc6lUqQ+0EMgOjFppr mwCImPeQCVq8A9Bb3bBwjuOqu9HF81lOH1ueqofCKF/rU/Mpl/G1GmS+KnGhoXaDjspf2p RGzoIMSppH9d1+VOZSvY52k48cMdqFs= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="e/E/nuyl"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf24.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.180 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742060998; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6mWjPqT680ajW18j1m0PlnxhJw6Nvx3pVB+w31Bj7so=; b=r2IKtj3MnM3EO+w+09YO2GZK7SqAIuKot6Ekj8+KEoWXZ1o2591V5JC7ksOl1z+Cg08xKj +toI06ivrqBSp0iJ1FpXX/MJAzgernhtl85AuiT/Os3tv9lU96FDSxqsvLSUaIkGKdqHQv SGMBym6CjMHGFz+Pk2gZhulN/C2iUAA= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1742060997; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6mWjPqT680ajW18j1m0PlnxhJw6Nvx3pVB+w31Bj7so=; b=e/E/nuylkpGXEhqHZNEobS6G+bqcssykgUckBO75NlFgK87SEgIBgQphktZTteBqNnHTbd x6QCYzHUjbLCNKPRZeaMdVcSSLOj6sGoyZmm+1mDUtiEh8IruX1fojGN/FiLHsrwdIn2s5 f2gone2vSuXSW8gupsMmoVL5SC2w8ho= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Sebastian Andrzej Siewior , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH 5/9] memcg: no refilling stock from obj_cgroup_release Date: Sat, 15 Mar 2025 10:49:26 -0700 Message-ID: <20250315174930.1769599-6-shakeel.butt@linux.dev> In-Reply-To: <20250315174930.1769599-1-shakeel.butt@linux.dev> References: <20250315174930.1769599-1-shakeel.butt@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Queue-Id: B59CB180004 X-Rspamd-Server: rspam05 X-Stat-Signature: pmjnf6ux8ei5ikrtr54xm6a4ob8j9whc X-HE-Tag: 1742060998-895501 X-HE-Meta: U2FsdGVkX1+wG0le2DRfXBODBceLuwwewu0V4nLveQG/+cxifLFzk1pS+I8E5pvcIZO7CVNXUvf5MRgZMlqHdxQbnxvGQRqev2Fi9Dq4Be6GJEdOrAaAVL3etenahfGK8NgCokE0MoMgaHA9sMcROEZemKc0OYwkM4PPKfsODWwiI+dkQapoBw+zC0iPzhhQMq3vksiza59og/MZEdBMVbePiGi5iNQehcdBYKAuNxqz/PRjzDjj4zWA4k5vUtqchBP+L9ArWg5HFZO+DhJuCj3RFPjmSkcJvl9RoiVzLj/nb/jO7gMPoSGbgYWd4eSkawbf8iY37kIhA3kX382LQZSJKnQTVs826cc3jdheDVt52r7Wc/t6zzlctRkqpFY5MeFSJRVQ70COJyxnGZ/rPemXZ3Tkb7J9CVlIoKrt1wSt5Y8ysHXFQI8CrWNwIxCSYOwySbfkw665WpjpnHQ+RLZbnaXeqEhY1sTPpfX0/SLtJZ5kIGKQBnwIp9r0vPRQRrrGO/0WuOIYB/PCCwGOo/TeGQU1Q8lcIo2qCCWvo8ZwG1DHTfsaGcwM412W7liw26OcFyoF3gtOQrVGVHBnq3x7fVURMj0t2RJGSjcS20mqCKRYwgI3zCkDOeHCTXe9mefNydRTZuKulnw3fZfj+FcLsZwX0So80r7joQAVoK2dNWvWFxiPUmRwKQ/kNOHPCAle57qQlaaSBTn1wN+GfeGd6Rnzj3Cxk/9uMQofZ+/AkGkJou8vGXD33+VO5jmTm/XcdhQXEJ0gN8HW49hcbhdsZK0eGpVBuzgAviTFKw8BMTFjhSvxCyEuCzztfFsK67Puo7BO8P54sB2FpkjzCO+ubeaNPAYSZSFjt08vuUXOpUu2ijm9k5kn/LmMQIEMlqnVdGunSQvASE8FUzpq/E8ALoKis1s7vUg3xuBiTr80Gmb6KpM6+Khgkqm9s6vFW17eqzEkF+rY2vqRZVN 0xVr4VUz U50mGN014L4fzXeRG7OuNVkZGpzqsaE5yHwNezQDiiMEEIXs2c8TL1Q5P0evshQ+/cz5KNzshfOQ8/TGmhGWZn3IET6eN7VZTmHteXOtXGz6qIxRxo5VO8TGFLML8js2wpFV5/V4Q4yp59/XfKRAqfIy5A3/oBrrPjsxLSWO2OYdABIuyQDu3d4nkNvlKz2LqiH34it6pnNJsMXE3aMaavkYXFe5+Vz+WMEX5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: obj_cgroup_release is called when all the references to the objcg have been released i.e. no more memory objects are pointing to it. Most probably objcg->memcg will be pointing to some ancestor memcg. In obj_cgroup_release(), the kernel calls obj_cgroup_uncharge_pages() which refills the local stock. There is no need to refill the local stock with some ancestor memcg and flush the local stock. Let's decouple obj_cgroup_release() from the local stock by uncharging instead of refilling. One additional benefit of this change is that it removes the requirement to only call obj_cgroup_put() outside of local_lock. Signed-off-by: Shakeel Butt Acked-by: Vlastimil Babka --- mm/memcontrol.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7054b0ebd207..83db180455a1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -129,8 +129,7 @@ bool mem_cgroup_kmem_disabled(void) return cgroup_memory_nokmem; } -static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, - unsigned int nr_pages); +static void memcg_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages); static void obj_cgroup_release(struct percpu_ref *ref) { @@ -163,8 +162,16 @@ static void obj_cgroup_release(struct percpu_ref *ref) WARN_ON_ONCE(nr_bytes & (PAGE_SIZE - 1)); nr_pages = nr_bytes >> PAGE_SHIFT; - if (nr_pages) - obj_cgroup_uncharge_pages(objcg, nr_pages); + if (nr_pages) { + struct mem_cgroup *memcg; + + memcg = get_mem_cgroup_from_objcg(objcg); + mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages); + memcg1_account_kmem(memcg, -nr_pages); + if (!mem_cgroup_is_root(memcg)) + memcg_uncharge(memcg, nr_pages); + css_put(&memcg->css); + } spin_lock_irqsave(&objcg_lock, flags); list_del(&objcg->list); From patchwork Sat Mar 15 17:49:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 14018169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12E16C28B28 for ; Sat, 15 Mar 2025 17:50:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB49B280008; Sat, 15 Mar 2025 13:50:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C64C9280001; Sat, 15 Mar 2025 13:50:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2D1F280008; Sat, 15 Mar 2025 13:50:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 924C0280001 for ; Sat, 15 Mar 2025 13:50:07 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E385A1CB138 for ; Sat, 15 Mar 2025 17:50:08 +0000 (UTC) X-FDA: 83224524096.21.AB489EB Received: from out-183.mta1.migadu.com (out-183.mta1.migadu.com [95.215.58.183]) by imf01.hostedemail.com (Postfix) with ESMTP id 0B32440003 for ; Sat, 15 Mar 2025 17:50:06 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=kZK6wnzZ; spf=pass (imf01.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.183 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742061007; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tHvct7NUO2otKVtcPkikbwH9IxeqmMsC6tyfV+hMsCI=; b=UOUrqBMdbXPhRytFYq7k2ChDVzLp2dO0ovkJb1bdsBqPE9qylruhhI0NGvarKa2ZUgqvT9 bnLwuVaA+o44Ye/V76ad8Rb4DRwI7oifQkDL1/vw9yJCbu5uhQYdYJz++wEde1j8BeTGwz SjRbaxQIVebnTBDBLtahVyF2yAci4d0= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=kZK6wnzZ; spf=pass (imf01.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.183 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742061007; a=rsa-sha256; cv=none; b=s2VH7L7wWnL8fidWyFOagWIFGGmqJiElbhXX2GXpuD5iTWMy0O/1wNdE0SIp/D5XSWjrkW eHcu3p1rK3P/4KbamD/hmSc2d6eUldYTwJamI7YtKzKKH+7CtYevK7uI0RruFoycpXhM+G wy49R1NsD4bNX946/tdGl7pWwaqscVg= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1742061003; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tHvct7NUO2otKVtcPkikbwH9IxeqmMsC6tyfV+hMsCI=; b=kZK6wnzZBdB6KoWZYB2gbmXAmEglH6MpMFhomC5Kcno08qRkhhrz6/f/yhMsfRdCd8x3BY ZJHkxF4tenCzo+3OAxNHXlFxu7B6frqfODK2qZdld+O7uCsw0kDPPXntdcw72Ksl54uqxd 5fjRV2fVuE5qN1Q9kOdIaE6i/nHy4Gc= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Sebastian Andrzej Siewior , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH 6/9] memcg: do obj_cgroup_put inside drain_obj_stock Date: Sat, 15 Mar 2025 10:49:27 -0700 Message-ID: <20250315174930.1769599-7-shakeel.butt@linux.dev> In-Reply-To: <20250315174930.1769599-1-shakeel.butt@linux.dev> References: <20250315174930.1769599-1-shakeel.butt@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Queue-Id: 0B32440003 X-Rspamd-Server: rspam08 X-Stat-Signature: jk5xrw97m8i36891r3xtxpcwcbt8f55z X-HE-Tag: 1742061006-572731 X-HE-Meta: U2FsdGVkX18+qkhcM0vY7i+lqxZYkgAvYbvajAJwFyvdE2XTjfInC9hP1RGKM8yy5c9yxX2rtHpdsk2b0L+12IYaeP6BSSKTClQF5xyTCBebI+U5V1E6ZssFZVxAPNPwDOnM74F1x1bF9v7Pl6dZhtatB9d058M41TU/Q3+2cHaGkTQPULU2jGyw8k2aZpfvFYzdPq0uERXgqsERWkXCKykPepRrNnpVQwQNfwmGDEmpJEtkpExErLecrXGsM/wYTykMzy4wLCN1DgsVEwhigJBAOZb0e7JdeULHITn7FR+0QR9hT1QyRAR2nTBQjI0a7ixHNcX3ZmLm/++nLrypIuTImciIgART/3wWzEUhECU5mo4hfU3phrez/BQcsc90wn3tdGIYxxocGoaLj6NyN1NeiIMghccFwEYT178VGeiJfBiAnymPVwnkOeezqG9E3pwOETrud0XpKIcLEBy9Jl3yREQqDn6VqX4d47guJwi06bR3jadqOGcnXvcSxlp6reL7TjW7BG4fzXxGCd8ToEOKA/ryFd+6OD0osSkf+TDEbP8B/nZQiCgp/QkubsuMmujx6V23xYUIFMGl8JNEbofsnUsQ+B1RC/6lSix/S7zXWpXABVpsc2nlbxNRAEvgUKPMsidohLwE1WS5tHzfdKVW0XwVRCfJcc+lQAzNwdBuLv0Vvn93daTXwHdLABx01MPzhRp6Qt2e+UIDV59R0AMUsPiB5p/C4vBFJABUfLWBrBR1AkGvSvsMZ/OFSpsiRAolKBvT9LGF62koKXxFK0dF1TmNd9ML+5yCsCZ5Knc/q3FP0HxdbW8F5rGiYsGNxyhg5uTQ70zeWfyhXzvtDagiI8TPvRMKcMBP/x+6MWKR7PPN6s+S0y2Oa0wGUcrn3348As1AIpQOaEq0SzTHRCt1uBauq6D4jFCyiXrTrJSQnVk2WjfQS77MGCY+hqTq1hHeXSA5+EuLpsWAXan 51Bi+hJ+ OkxRxenoM5S+X6Vu5M7coSsK7GLestcyd7Dqyym0Lh03qZ3JlgWvBzDN8hVraYDgZCgKCIc4Hh1wvc8Vjun/8k/IjrzynttRlxAiXCsg0AQJCU/5v7OQZlFQ5Sso2U68ETHeFhqX+3Qi8swXydeNIMmdcIHChMeqOp5zph0IxPmfyfKb09i2UrPUwyswnnxWpU3NauIe6Ejr+TVjjO59Fqj6IrQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Previously we could not call obj_cgroup_put() inside the local lock because on the put on the last reference, the release function obj_cgroup_release() may try to re-acquire the local lock. However that chain has been broken. Now simply do obj_cgroup_put() inside drain_obj_stock() instead of returning the old objcg. Signed-off-by: Shakeel Butt --- mm/memcontrol.c | 37 +++++++++++-------------------------- 1 file changed, 11 insertions(+), 26 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 83db180455a1..3c4de384b5a0 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1778,7 +1778,7 @@ static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock) = { }; static DEFINE_MUTEX(percpu_charge_mutex); -static struct obj_cgroup *drain_obj_stock(struct memcg_stock_pcp *stock); +static void drain_obj_stock(struct memcg_stock_pcp *stock); static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, struct mem_cgroup *root_memcg); @@ -1853,7 +1853,6 @@ static void drain_stock(struct memcg_stock_pcp *stock) static void drain_local_stock(struct work_struct *dummy) { struct memcg_stock_pcp *stock; - struct obj_cgroup *old = NULL; unsigned long flags; /* @@ -1864,12 +1863,11 @@ static void drain_local_stock(struct work_struct *dummy) localtry_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock); - old = drain_obj_stock(stock); + drain_obj_stock(stock); drain_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); - obj_cgroup_put(old); } /* Should never be called with root_mem_cgroup. */ @@ -1951,18 +1949,16 @@ void drain_all_stock(struct mem_cgroup *root_memcg) static int memcg_hotplug_cpu_dead(unsigned int cpu) { struct memcg_stock_pcp *stock; - struct obj_cgroup *old; unsigned long flags; stock = &per_cpu(memcg_stock, cpu); /* drain_obj_stock requires stock_lock */ localtry_lock_irqsave(&memcg_stock.stock_lock, flags); - old = drain_obj_stock(stock); + drain_obj_stock(stock); localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); drain_stock(stock); - obj_cgroup_put(old); return 0; } @@ -2745,24 +2741,20 @@ void __memcg_kmem_uncharge_page(struct page *page, int order) } /* Replace the stock objcg with objcg, return the old objcg */ -static struct obj_cgroup *replace_stock_objcg(struct memcg_stock_pcp *stock, - struct obj_cgroup *objcg) +static void replace_stock_objcg(struct memcg_stock_pcp *stock, + struct obj_cgroup *objcg) { - struct obj_cgroup *old = NULL; - - old = drain_obj_stock(stock); + drain_obj_stock(stock); obj_cgroup_get(objcg); stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes) ? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0; WRITE_ONCE(stock->cached_objcg, objcg); - return old; } static void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, enum node_stat_item idx, int nr) { struct memcg_stock_pcp *stock; - struct obj_cgroup *old = NULL; unsigned long flags; int *bytes; @@ -2775,7 +2767,7 @@ static void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, * changes. */ if (READ_ONCE(stock->cached_objcg) != objcg) { - old = replace_stock_objcg(stock, objcg); + replace_stock_objcg(stock, objcg); stock->cached_pgdat = pgdat; } else if (stock->cached_pgdat != pgdat) { /* Flush the existing cached vmstat data */ @@ -2816,7 +2808,6 @@ static void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, __mod_objcg_mlstate(objcg, pgdat, idx, nr); localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); - obj_cgroup_put(old); } static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) @@ -2838,12 +2829,12 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) return ret; } -static struct obj_cgroup *drain_obj_stock(struct memcg_stock_pcp *stock) +static void drain_obj_stock(struct memcg_stock_pcp *stock) { struct obj_cgroup *old = READ_ONCE(stock->cached_objcg); if (!old) - return NULL; + return; if (stock->nr_bytes) { unsigned int nr_pages = stock->nr_bytes >> PAGE_SHIFT; @@ -2896,11 +2887,7 @@ static struct obj_cgroup *drain_obj_stock(struct memcg_stock_pcp *stock) } WRITE_ONCE(stock->cached_objcg, NULL); - /* - * The `old' objects needs to be released by the caller via - * obj_cgroup_put() outside of memcg_stock_pcp::stock_lock. - */ - return old; + obj_cgroup_put(old); } static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, @@ -2922,7 +2909,6 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, bool allow_uncharge) { struct memcg_stock_pcp *stock; - struct obj_cgroup *old = NULL; unsigned long flags; unsigned int nr_pages = 0; @@ -2930,7 +2916,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, stock = this_cpu_ptr(&memcg_stock); if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */ - old = replace_stock_objcg(stock, objcg); + replace_stock_objcg(stock, objcg); allow_uncharge = true; /* Allow uncharge when objcg changes */ } stock->nr_bytes += nr_bytes; @@ -2941,7 +2927,6 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, } localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); - obj_cgroup_put(old); if (nr_pages) obj_cgroup_uncharge_pages(objcg, nr_pages); From patchwork Sat Mar 15 17:49:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 14018170 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BAF5C28B30 for ; Sat, 15 Mar 2025 17:50:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1DA42280009; Sat, 15 Mar 2025 13:50:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 113AC280001; Sat, 15 Mar 2025 13:50:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E330F280009; Sat, 15 Mar 2025 13:50:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C8801280001 for ; Sat, 15 Mar 2025 13:50:08 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4BB37121E61 for ; Sat, 15 Mar 2025 17:50:10 +0000 (UTC) X-FDA: 83224524180.07.9E694AD Received: from out-183.mta0.migadu.com (out-183.mta0.migadu.com [91.218.175.183]) by imf06.hostedemail.com (Postfix) with ESMTP id A675A180003 for ; Sat, 15 Mar 2025 17:50:08 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="YP6Oq/yv"; spf=pass (imf06.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.183 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742061008; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sgD6RlJd90bDD1Zbd+KYFHxVJHSovRtS79ksTm6yv/M=; b=HPOmyWuUMUluNCp/Ud9GP8gNaK171JgaRi2GLo85fyLvRqNOTve+8M5CMMKmIMc+1WBGuQ zjfmxso0KX+iMMJ6HcpotgzAYjo1QSsOeXoM9xHeCoNRbPsNhSkB2+QrWNkdlX5TewLKca FnPOXqRtZ36Xur6KlU7G38DudSNLg+E= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="YP6Oq/yv"; spf=pass (imf06.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.183 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742061008; a=rsa-sha256; cv=none; b=gxW7SYzzWwjq04T+ytxqQQUL0vJhqITsGOho8UMoZN7EUbFekehvydXapX4PJP7+O9Kt2Z fsgztHmaT0Dpj7B8kf/fXzv0y3bkJKHd14JocY/KMSixTN+Ml9wePPdAf7KvNn6oth8rDX 0zKs1kITrXGGtAyzJk4LhzLpNnc59Sc= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1742061007; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sgD6RlJd90bDD1Zbd+KYFHxVJHSovRtS79ksTm6yv/M=; b=YP6Oq/yvNHnoJl3NkSZbo35do6cddRVpTfrN2VvwcssF29KZdMDWRXmLsbJ2Qk2XdDfhR7 d+GAaMirhcUzjsJW5CNYi9dJnKGqdyXAP23/6ynvJ0i7VDeFib1ziRJApLhc7gYJQcJN+k orO43WdiKhpvPg1EWV48jRPfM/a8IVk= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Sebastian Andrzej Siewior , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH 7/9] memcg: use __mod_memcg_state in drain_obj_stock Date: Sat, 15 Mar 2025 10:49:28 -0700 Message-ID: <20250315174930.1769599-8-shakeel.butt@linux.dev> In-Reply-To: <20250315174930.1769599-1-shakeel.butt@linux.dev> References: <20250315174930.1769599-1-shakeel.butt@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Queue-Id: A675A180003 X-Rspamd-Server: rspam03 X-Stat-Signature: szamfyefp8objc69ngwwoirtx18cqcmt X-HE-Tag: 1742061008-14453 X-HE-Meta: U2FsdGVkX1/cLDc36ymAu8WHYi5RQMYGViryBUcrjLPpNbn4UydYpkFafi2vlOwjqvOS3u6aO0XirXlkJns9Z+bhuN2XbUEduEWaco3ftQXVAhiORind0ygj60HH6OMY7E7OlUUX4RyWlNMmh+2T6n26gDEJ0o1i0y/TotHtNB3nibb8BOEyFzoGGgz5puiBXDK+JFLF0O9CWU1TalQew1sQ4mfsTNjQrxJL19qq8D/CcJsZn5flr1/c2YcEbs9Ahoh65QmUoBDtNK8DyU5XhTml6Xlt6DMUT3t3OFX9TetsUBBsp+4I2ZKgeG+JZ5L0pUSQgpWw4ZZwNU9fAkbcieAirfgZr/Xq2160JUMoobxLuhMITJ6UMpXpeLYd36gpCVzm9R2GYREVn/d6VHf6pqN6j2eIc/kEJ5C2aqHN0VKS81hdkF2lIOzlbvaNRJt/8Yl7xdq4i9XcbMTsNS9AN8HDMqIXfg7iFsWe/+LPapGAAt4bdixx9gGf39dRxF102MoCpYJ/R1Ps32zsJQOr49daOdnIQiCShWEPoQA1pSfIGVKiJ3nsV7be851AfObQxLfYFj9qK7nCgXhI9fq61sKIS37llzT9p/IiE3eqxNlHG0/4RdYs4tCjrwbVryV2IWSGlpXkTTJtE7TaN+w9OUHN5AwiLNWxIRTGokWGBNJ5UXsQ/ue4HefKox8oUJhnEso5cznBHzp6dUGtDvZcbhfzXWklFzoN95Wqr9mgLx1XEALaA5J5FJbSX/GuywzVZuvuB95a1RvqWdvzIschQhdV6fao2sMUb2PD+8cKJM1egUH7O3LGs1bInQXU3oFb74Q9dShIuMW5xJvTjDdl59iBVtBvc/EqtkUt6LAhKfEYG5ZXS74+dbghvRF2I0Prp0UH3bW7qoo1HqbckaAE6fb/AK6yT4kmwUvtYQmC2OGGXFQLcO1kVk6IQz7FIHtIUOrxedPDEIg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For non-PREEMPT_RT kernels, drain_obj_stock() is always called with irq disabled, so we can use __mod_memcg_state() instead of mod_memcg_state(). For PREEMPT_RT, we need to add memcg_stats_[un]lock in __mod_memcg_state(). Signed-off-by: Shakeel Butt Reviewed-by: Sebastian Andrzej Siewior --- mm/memcontrol.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 3c4de384b5a0..dfe9c2eb7816 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -707,10 +707,12 @@ void __mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx, if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return; + memcg_stats_lock(); __this_cpu_add(memcg->vmstats_percpu->state[i], val); val = memcg_state_val_in_pages(idx, val); memcg_rstat_updated(memcg, val); trace_mod_memcg_state(memcg, idx, val); + memcg_stats_unlock(); } #ifdef CONFIG_MEMCG_V1 @@ -2845,7 +2847,7 @@ static void drain_obj_stock(struct memcg_stock_pcp *stock) memcg = get_mem_cgroup_from_objcg(old); - mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages); + __mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages); memcg1_account_kmem(memcg, -nr_pages); if (!mem_cgroup_is_root(memcg)) memcg_uncharge(memcg, nr_pages); From patchwork Sat Mar 15 17:49:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 14018171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86A6DC28B2F for ; Sat, 15 Mar 2025 17:50:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 45EF828000A; Sat, 15 Mar 2025 13:50:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E44B280001; Sat, 15 Mar 2025 13:50:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2ABAE28000A; Sat, 15 Mar 2025 13:50:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0C362280001 for ; Sat, 15 Mar 2025 13:50:12 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 581AC1C955E for ; Sat, 15 Mar 2025 17:50:13 +0000 (UTC) X-FDA: 83224524306.12.2536E2B Received: from out-177.mta0.migadu.com (out-177.mta0.migadu.com [91.218.175.177]) by imf24.hostedemail.com (Postfix) with ESMTP id AA437180009 for ; Sat, 15 Mar 2025 17:50:11 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=eqqFYRay; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf24.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.177 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742061011; a=rsa-sha256; cv=none; b=S2fS70r6bPtLhvs5dxhIDuZJTOZZv+RIg8otuTIzyXYH/8bgRDLKKZeqd4aboB1bc5R5Bb EWPmrlEHqcH6RCkiq25Ylr24kYmKM/0A86sVq4UT6kzLhOG3j86Zxy9pppZfqll5gJPt2n VKlcQxm+Yw3ZUd7bd9Ydtzf8nhYpHRM= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=eqqFYRay; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf24.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.177 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742061011; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o4xUih43AY3K2la6xaLoSlR0RMvoWQOKhCt2j9dxoEk=; b=Q6vkn9Kvpjnin1CBjvZfiEbeucuBuWHzVJTnPX68hSLK26PbrnKFj4kwL/yAes676vTQ0B qNY3A4WDX2hHoljOLbxkBfJ1T+g8YfsmhuOurJZ26oVzUz0pTK0tcX1huvN444pfP5VAzt WSxjeKkz6JaW3k36I3XroCsbIyKpTBI= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1742061010; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=o4xUih43AY3K2la6xaLoSlR0RMvoWQOKhCt2j9dxoEk=; b=eqqFYRay+fQyoV6n5jswA6nCcb5FmqTbDPLFlZ2zp0kXlAMkjJwG98vDbE3BTw/x9vWnlN AIIUxGky5BNUCIqwWFkVJ8TuF2dvZuQgv+zD3kE9FReetGUYiptNLv9fON3ko4pdWauiGw 9o7QOfY/CwY1Uuiipxm1wber65uJISc= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Sebastian Andrzej Siewior , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH 8/9] memcg: combine slab obj stock charging and accounting Date: Sat, 15 Mar 2025 10:49:29 -0700 Message-ID: <20250315174930.1769599-9-shakeel.butt@linux.dev> In-Reply-To: <20250315174930.1769599-1-shakeel.butt@linux.dev> References: <20250315174930.1769599-1-shakeel.butt@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam07 X-Rspam-User: X-Stat-Signature: jxmz6caiwzgqwnxcoth5e9enac95b8y7 X-Rspamd-Queue-Id: AA437180009 X-HE-Tag: 1742061011-939254 X-HE-Meta: U2FsdGVkX190xi0HhREGd/bMFooxLYdPaNEcx7Ml4C/yWETbc1iskCa1Q7g/fsRGu9ziKBf015/6wI5YEMBmlNsbxEci0ATVONUctP6FCmEls+gWLhO7mz1D44UFvoDVI4W75GbSPTFcECceUjZw3x1xteXLW3Dx6PreAPC1mMhwDcHNKag6lMuvZCo1QKM0U+yivwLpOWZt78V4hlh6FQRXjU3EEYgznIzV5si3jYCpQWlPWbGuGwo1cPo+rbh/UU/KTbDpRqGqVSBu2Nq8/wF+FB5dTIBJ9IGniL7eU13Yw76DBGYjAcqXIZjzgaz5SFZOBpT2YAA+Jo8F651gE5SuhKdDVghiIebnIxVUiM+CseVntTCngQbT9SnG+Oh5yXqRqitsRIZKBaXX9puZJHlDbHGwo6OTIIF6lpcaArt1Z9Yi9dwcb9Q09wk5/jvSwZ+0MNd82wUvU19OHvX0pMhInUdP4zZZmOwYVUXQA8akO3R9g6CAtZnE2CSsDrnUcid6YazM8qauJc+bEXvlQOTQwAkQgT0eu8M++npmRKvMDaCI5LTGnX26LJrC8fEK20LoZT7Jl/u4fijl3P80LUjCNNDEColbtl1Vkjgzk2WCQ1+3d2N/SqL/+PRjwhIHCLQ3PyyOanqG7XfR4RoE5Xvbvt/yBVo4wjzpWvRNQXWej7XZXrR5VTUo+5ivcDtycVCUxV47/Tj1Wm8AnD9mRzLr5ZKC3XvalOVcpC1pY4x8ZC5hOLlmW2xqDYnRW14HAi99a2nsDFQsi275bZ/WXb4/RsEuzquFPstm3X3mtYaEoasbVSBzBUggTPMSRP5PRprK7EctDY4Zqa8ixtf2+eAXtpHZjW2du8CgR32xl2xYaWvfn3asG5hfZmQvi7EQdya8J+wEDuVDWtWuda1dtf36HtZUUH00kziDJTq59IhbnDCIXuWOBufLCBUpFcjUeGLPRQZhcDdYgRhyiBS oDqi3pFZ BjHbg4jnhMOphBdg6EgZHCp12rteDjOzZjVdqQGhoDU7QtuZzRzTyjGbeZJzfvJB4cshY9shjUSzba7a6S1LnzZraySI6JsT6otng6VuQlx7AFB+gSXdDKklhl/d4xAK53c6MYK5jY2VJNgXNm8r1IsoBkRyWYxM3hLH1QSqVdwJ/reM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Vlastimil Babka When handing slab objects, we use obj_cgroup_[un]charge() for (un)charging and mod_objcg_state() to account NR_SLAB_[UN]RECLAIMABLE_B. All these operations use the percpu stock for performance. However with the calls being separate, the stock_lock is taken twice in each case. By refactoring the code, we can turn mod_objcg_state() into __account_obj_stock() which is called on a stock that's already locked and validated. On the charging side we can call this function from consume_obj_stock() when it succeeds, and refill_obj_stock() in the fallback. We just expand parameters of these functions as necessary. The uncharge side from __memcg_slab_free_hook() is just the call to refill_obj_stock(). Other callers of obj_cgroup_[un]charge() (i.e. not slab) simply pass the extra parameters as NULL/zeroes to skip the __account_obj_stock() operation. In __memcg_slab_post_alloc_hook() we now charge each object separately, but that's not a problem as we did call mod_objcg_state() for each object separately, and most allocations are non-bulk anyway. This could be improved by batching all operations until slab_pgdat(slab) changes. Some preliminary benchmarking with a kfree(kmalloc()) loop of 10M iterations with/without __GFP_ACCOUNT: Before the patch: kmalloc/kfree !memcg: 581390144 cycles kmalloc/kfree memcg: 783689984 cycles After the patch: kmalloc/kfree memcg: 658723808 cycles More than half of the overhead of __GFP_ACCOUNT relative to non-accounted case seems eliminated. Signed-off-by: Vlastimil Babka Signed-off-by: Shakeel Butt --- mm/memcontrol.c | 77 +++++++++++++++++++++++++++++-------------------- 1 file changed, 46 insertions(+), 31 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index dfe9c2eb7816..553eb1d7250a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2753,25 +2753,17 @@ static void replace_stock_objcg(struct memcg_stock_pcp *stock, WRITE_ONCE(stock->cached_objcg, objcg); } -static void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, - enum node_stat_item idx, int nr) +static void __account_obj_stock(struct obj_cgroup *objcg, + struct memcg_stock_pcp *stock, int nr, + struct pglist_data *pgdat, enum node_stat_item idx) { - struct memcg_stock_pcp *stock; - unsigned long flags; int *bytes; - localtry_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); - /* * Save vmstat data in stock and skip vmstat array update unless - * accumulating over a page of vmstat data or when pgdat or idx - * changes. + * accumulating over a page of vmstat data or when pgdat changes. */ - if (READ_ONCE(stock->cached_objcg) != objcg) { - replace_stock_objcg(stock, objcg); - stock->cached_pgdat = pgdat; - } else if (stock->cached_pgdat != pgdat) { + if (stock->cached_pgdat != pgdat) { /* Flush the existing cached vmstat data */ struct pglist_data *oldpg = stock->cached_pgdat; @@ -2808,11 +2800,10 @@ static void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, } if (nr) __mod_objcg_mlstate(objcg, pgdat, idx, nr); - - localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); } -static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) +static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, + struct pglist_data *pgdat, enum node_stat_item idx) { struct memcg_stock_pcp *stock; unsigned long flags; @@ -2824,6 +2815,9 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) if (objcg == READ_ONCE(stock->cached_objcg) && stock->nr_bytes >= nr_bytes) { stock->nr_bytes -= nr_bytes; ret = true; + + if (pgdat) + __account_obj_stock(objcg, stock, nr_bytes, pgdat, idx); } localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags); @@ -2908,7 +2902,8 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, } static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, - bool allow_uncharge) + bool allow_uncharge, int nr_acct, struct pglist_data *pgdat, + enum node_stat_item idx) { struct memcg_stock_pcp *stock; unsigned long flags; @@ -2923,6 +2918,9 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, } stock->nr_bytes += nr_bytes; + if (pgdat) + __account_obj_stock(objcg, stock, nr_acct, pgdat, idx); + if (allow_uncharge && (stock->nr_bytes > PAGE_SIZE)) { nr_pages = stock->nr_bytes >> PAGE_SHIFT; stock->nr_bytes &= (PAGE_SIZE - 1); @@ -2934,12 +2932,13 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, obj_cgroup_uncharge_pages(objcg, nr_pages); } -int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size) +static int obj_cgroup_charge_account(struct obj_cgroup *objcg, gfp_t gfp, size_t size, + struct pglist_data *pgdat, enum node_stat_item idx) { unsigned int nr_pages, nr_bytes; int ret; - if (consume_obj_stock(objcg, size)) + if (likely(consume_obj_stock(objcg, size, pgdat, idx))) return 0; /* @@ -2972,15 +2971,21 @@ int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size) nr_pages += 1; ret = obj_cgroup_charge_pages(objcg, gfp, nr_pages); - if (!ret && nr_bytes) - refill_obj_stock(objcg, PAGE_SIZE - nr_bytes, false); + if (!ret && (nr_bytes || pgdat)) + refill_obj_stock(objcg, nr_bytes ? PAGE_SIZE - nr_bytes : 0, + false, size, pgdat, idx); return ret; } +int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size) +{ + return obj_cgroup_charge_account(objcg, gfp, size, NULL, 0); +} + void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) { - refill_obj_stock(objcg, size, true); + refill_obj_stock(objcg, size, true, 0, NULL, 0); } static inline size_t obj_full_size(struct kmem_cache *s) @@ -3032,23 +3037,32 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, return false; } - if (obj_cgroup_charge(objcg, flags, size * obj_full_size(s))) - return false; - for (i = 0; i < size; i++) { slab = virt_to_slab(p[i]); if (!slab_obj_exts(slab) && alloc_slab_obj_exts(slab, s, flags, false)) { - obj_cgroup_uncharge(objcg, obj_full_size(s)); continue; } + /* + * if we fail and size is 1, memcg_alloc_abort_single() will + * just free the object, which is ok as we have not assigned + * objcg to its obj_ext yet + * + * for larger sizes, kmem_cache_free_bulk() will uncharge + * any objects that were already charged and obj_ext assigned + * + * TODO: we could batch this until slab_pgdat(slab) changes + * between iterations, with a more complicated undo + */ + if (obj_cgroup_charge_account(objcg, flags, obj_full_size(s), + slab_pgdat(slab), cache_vmstat_idx(s))) + return false; + off = obj_to_index(s, slab, p[i]); obj_cgroup_get(objcg); slab_obj_exts(slab)[off].objcg = objcg; - mod_objcg_state(objcg, slab_pgdat(slab), - cache_vmstat_idx(s), obj_full_size(s)); } return true; @@ -3057,6 +3071,8 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects, struct slabobj_ext *obj_exts) { + size_t obj_size = obj_full_size(s); + for (int i = 0; i < objects; i++) { struct obj_cgroup *objcg; unsigned int off; @@ -3067,9 +3083,8 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, continue; obj_exts[off].objcg = NULL; - obj_cgroup_uncharge(objcg, obj_full_size(s)); - mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), - -obj_full_size(s)); + refill_obj_stock(objcg, obj_size, true, -obj_size, + slab_pgdat(slab), cache_vmstat_idx(s)); obj_cgroup_put(objcg); } } From patchwork Sat Mar 15 17:49:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 14018172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F749C28B28 for ; Sat, 15 Mar 2025 17:50:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E6C428000B; Sat, 15 Mar 2025 13:50:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 27B24280001; Sat, 15 Mar 2025 13:50:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0792528000B; Sat, 15 Mar 2025 13:50:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D45D6280001 for ; Sat, 15 Mar 2025 13:50:14 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 319B31A1E95 for ; Sat, 15 Mar 2025 17:50:16 +0000 (UTC) X-FDA: 83224524432.15.9C76C2C Received: from out-186.mta0.migadu.com (out-186.mta0.migadu.com [91.218.175.186]) by imf13.hostedemail.com (Postfix) with ESMTP id 6FB412000C for ; Sat, 15 Mar 2025 17:50:14 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=LHDQm3SA; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf13.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.186 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742061014; a=rsa-sha256; cv=none; b=pNOZLE5+mOD9fmGw3oJ0Oa7GuAqV10qPSwugntQAJiKEsJjCJ2aTvwDRX7Z0WHfLPbg8g0 97mYk8Iw6c4vUchDT0IVSwnvQw7toQOAJaCf4Nia87MykjF1vETWEVtrOPRwOrrjGGRs1c VJ6Oepvnfel+L9lOtXyjL/hpV3CzskU= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=LHDQm3SA; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf13.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.186 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742061014; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TYGBaqcauUTYU40tZGtRDDBbg7pHPWXhJB6/pipNZSg=; b=ljtAKnUB8yOCXyyHoFnO4iFQ0i258Q5SQ0KlSvDq8ekUwmLkzWkcipFRGt5ZTYiYUvHYW7 Q6oPfQr9vObL1TaJtH36MC1uJPVj4sXBl5vCBD0j1H0ZtgC3iJ7Zi2iTA7Zhi3LUbhBXKf uHNkSGBZDX0ElSb4UF0LieUVz3dQVlI= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1742061013; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TYGBaqcauUTYU40tZGtRDDBbg7pHPWXhJB6/pipNZSg=; b=LHDQm3SAQHbOdOc/lx7xNRctT2V94M44B935v1mp0JKaFZD4ihZxYWRfQk1mXWZlM0YnV8 mUJPtgGxHSicxTjlkyDNWob2m9rowYzZQ0N2FbX6cTsr0uOiJu8tvTqevoPkB6VEYHtP9L MnOPP/rrbzJKtED4KZRq6GaquCbHc40= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Sebastian Andrzej Siewior , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH 9/9] memcg: manually inline replace_stock_objcg Date: Sat, 15 Mar 2025 10:49:30 -0700 Message-ID: <20250315174930.1769599-10-shakeel.butt@linux.dev> In-Reply-To: <20250315174930.1769599-1-shakeel.butt@linux.dev> References: <20250315174930.1769599-1-shakeel.butt@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Stat-Signature: bi1osot9poig8og3wtt918dnfnhdjqjf X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 6FB412000C X-Rspam-User: X-HE-Tag: 1742061014-547132 X-HE-Meta: U2FsdGVkX1+9m/d9Q7VU2JMZumMHN1ITZmuHYB9C5mxduepg6fV3EX29b/8FUbwZ6aaQjT1f+83jASznC9y2oMK4cWFWi8ePdbYlM7CkjaHSrnziLR5PPEzZF5xzpzJt/tx/i5++pUIbRHuT4fR85yVp9MYgj26/kVBAZ7IpBUcKuAlSCRwpFNK2GX6JY825l18EkfHchzLBdYGSFCUmL8SLFU57L6fpzeFNhi//WG7OfBpf3YqTrxgaljYG9Y3B5cgy6rwB+sk3BWeQYhCWbq2SAXyrwOVY8mcez15ONdwrAwgE6hLLfK7jdipnP1eOAtcBykZqgbgUUd/UnCu4JTpbAlNlKLD3NaoCGCRvFlFVwJsoieWJCOIrrysem7DX8dZgYagupQ7yMX0Ai+SImWmQmo1sHySJ7PzBKBGpPlhLPX+Ez8Cloc7FXiSNwBQ8DPDDd7ir8/NYe92ajSDNA99cyXrcrQzhkFDjq7uIuQqT46jh4Mes2AqGrySCR+C/3WGD4Y7/owHcVOYImIvATzBJu7ImrSeSAHLNdI362DGxgeuLmR2sVpRWusmjZvOkYYrpW0tVVX4TB0kYPQq/mJ3uDwF8xsKaKOCeVos5rdh77UphVnMlEFSful7Oh85aAJ2SwWJyvVz8SLRJ43h7jWjRDaSsIyiJ0SCiVWiBA9jnuDpK+2bOrk35K58PPps0k6H+36G4Q1a+xx85UO7UNyoGRCBWKYFizWI2ItTKVp+LhLixmRqEK/zYpRCur2oUgArlowIZytVCMt5vGjF2oxE8R86zhlBqPSuacZsQfq7ZmUj9gAiGTI0/zOukondtKBwrwzZxKnNg3zgQH4KO4OtqdjXzAHlPOIodmnLeFY+nu3j9m/crR/3Dbi78sXJggofxdDfUCK2ycizDaEorM5K70p627nxLIO8TdritIfCpugRTDRPg5zARaG7qvn/e/XuGGEIMl6z/obRpExJ kjSqxDnd freZguLwX6fmBfGcbqmh9zPfBZksHif4rvzaqOrXLd9UwwpafsBzmWz2GCUuLzfQT52a3vm4/O3P/BUcA3vpn/NY0K13qH4BuY5PLPJEDy97F7dB/nvfxwrMlYp+PfM0WakzftVsOpQUHxz/iICVK/2zTDabKSuZ7UQ3ZtWsUikiCsnHdwhubJmk3ziP5m295/s7z8QOxx4GZ2XAbqUHt6QQgx/VjAD43sMPD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The replace_stock_objcg() is being called by only refill_obj_stock, so manually inline it. Signed-off-by: Shakeel Butt --- mm/memcontrol.c | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 553eb1d7250a..f6e3fc418866 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2742,17 +2742,6 @@ void __memcg_kmem_uncharge_page(struct page *page, int order) obj_cgroup_put(objcg); } -/* Replace the stock objcg with objcg, return the old objcg */ -static void replace_stock_objcg(struct memcg_stock_pcp *stock, - struct obj_cgroup *objcg) -{ - drain_obj_stock(stock); - obj_cgroup_get(objcg); - stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes) - ? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0; - WRITE_ONCE(stock->cached_objcg, objcg); -} - static void __account_obj_stock(struct obj_cgroup *objcg, struct memcg_stock_pcp *stock, int nr, struct pglist_data *pgdat, enum node_stat_item idx) @@ -2913,7 +2902,12 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, stock = this_cpu_ptr(&memcg_stock); if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */ - replace_stock_objcg(stock, objcg); + drain_obj_stock(stock); + obj_cgroup_get(objcg); + stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes) + ? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0; + WRITE_ONCE(stock->cached_objcg, objcg); + allow_uncharge = true; /* Allow uncharge when objcg changes */ } stock->nr_bytes += nr_bytes;