From patchwork Thu Mar 18 11:06:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12147915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9D29C433E6 for ; Thu, 18 Mar 2021 11:08:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 20C0564DBA for ; Thu, 18 Mar 2021 11:08:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 20C0564DBA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9D9206B0071; Thu, 18 Mar 2021 07:08:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9AF046B0072; Thu, 18 Mar 2021 07:08:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 84F9B6B0073; Thu, 18 Mar 2021 07:08:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id 65F546B0071 for ; Thu, 18 Mar 2021 07:08:32 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 2A4E88157 for ; Thu, 18 Mar 2021 11:08:32 +0000 (UTC) X-FDA: 77932721664.28.2851DE7 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf17.hostedemail.com (Postfix) with ESMTP id 870294080F44 for ; Thu, 18 Mar 2021 11:08:31 +0000 (UTC) Received: by mail-pj1-f53.google.com with SMTP id q6-20020a17090a4306b02900c42a012202so2923598pjg.5 for ; Thu, 18 Mar 2021 04:08:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TztOCzATA+pN2UcYbo781lidO/cJKiaIDTOpooZzWOI=; b=RtHjb3BEb634rw2cRzq5x6Cpc7DPO5fSxKMhj7AjrplO2Bz+w0XXQkEd0hqkubYAVS ITNb907kk8Gybnptayw8r+iyL0/WLT/dBCbZ29VazpJweqObDgC8KJ1+z8owlzrHicDm rgs7L226KXoRR+diND4hyWRgPtkABJvkia/mq6fvYUqPVmEaXKfst1bUPCT2Y74BrkF3 U93aOvrxUcsf1F/jyHEu99pJW3LO6/v8ekeFGITRfy0DwTsY38d8X2WyLtRiR+/nB3Ao ZZxYTYm70kbQJw7KMfCZ5JllTpaH4+CKBow6PzCuobo70034iMKzMNjSQLzBr57NxBgQ xotQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TztOCzATA+pN2UcYbo781lidO/cJKiaIDTOpooZzWOI=; b=SHO0TnoGwpFtGr3sWssBXC2K0WbCmHJ2oVJxhLhIkBTnz7UIZx6vg3qPRknhEgDFOl /KsGUXEawuMkiLzh64A6ZgKnwaRKqBCd8I4akIf2A7/AyXs92fVm1e9NuQmHWPAoj1Hm V1swCBA5FTuCOiQetNHAGc7cQTqdnTREOdkPmgzHiyWQN0jgb5M4qSbsQmXnh3v3HA9k yuggfb8koA6JVNCLp44Ot7S8cBylHKrbohqPSeWHMfPJVQlqVRzuZ/9ol6uf6DRUTpfA BmTXjZ0qHeY1s0/YdaJpH9TUibfBHVhAu0Z1lfTVNxlA/joLoS+f0wklHpyxhx6wRAJU O9UA== X-Gm-Message-State: AOAM531oreoDyC/zgwktc4AhSBxzkR5M6EOczTY1eSwVhnocPV2zuWzn EY+Wk20hw7swBXYAiO1EZdA5BA== X-Google-Smtp-Source: ABdhPJyZog8X8qd5W2MX+ZfV1meTNzZkWX9hR1wGICdHzPhctAPATP7EEG9rfk75V5HElX1wtb3fhA== X-Received: by 2002:a17:90a:a403:: with SMTP id y3mr3740453pjp.227.1616065710773; Thu, 18 Mar 2021 04:08:30 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.231]) by smtp.gmail.com with ESMTPSA id e21sm1779509pgv.74.2021.03.18.04.08.26 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 18 Mar 2021 04:08:30 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH v4 1/5] mm: memcontrol: introduce obj_cgroup_{un}charge_pages Date: Thu, 18 Mar 2021 19:06:54 +0800 Message-Id: <20210318110658.60892-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210318110658.60892-1-songmuchun@bytedance.com> References: <20210318110658.60892-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: uzpnijn8x833naianrd7h7zsuboqhmd9 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 870294080F44 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf17; identity=mailfrom; envelope-from=""; helo=mail-pj1-f53.google.com; client-ip=209.85.216.53 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616065711-28154 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We know that the unit of slab object charging is bytes, the unit of kmem page charging is PAGE_SIZE. If we want to reuse obj_cgroup APIs to charge the kmem pages, we should pass PAGE_SIZE (as third parameter) to obj_cgroup_charge(). Because the size is already PAGE_SIZE, we can skip touch the objcg stock. And obj_cgroup_{un}charge_pages() are introduced to charge in units of page level. In the later patch, we also can reuse those two helpers to charge or uncharge a number of kernel pages to a object cgroup. This is just a code movement without any functional changes. Signed-off-by: Muchun Song Acked-by: Roman Gushchin Acked-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/memcontrol.c | 46 +++++++++++++++++++++++++++++++--------------- 1 file changed, 31 insertions(+), 15 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 845eec01ef9d..fc22da9805fb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3056,6 +3056,34 @@ static void memcg_free_cache_id(int id) ida_simple_remove(&memcg_cache_ida, id); } +static inline void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, + unsigned int nr_pages) +{ + rcu_read_lock(); + __memcg_kmem_uncharge(obj_cgroup_memcg(objcg), nr_pages); + rcu_read_unlock(); +} + +static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, + unsigned int nr_pages) +{ + struct mem_cgroup *memcg; + int ret; + + rcu_read_lock(); +retry: + memcg = obj_cgroup_memcg(objcg); + if (unlikely(!css_tryget(&memcg->css))) + goto retry; + rcu_read_unlock(); + + ret = __memcg_kmem_charge(memcg, gfp, nr_pages); + + css_put(&memcg->css); + + return ret; +} + /** * __memcg_kmem_charge: charge a number of kernel pages to a memcg * @memcg: memory cgroup to charge @@ -3180,11 +3208,8 @@ static void drain_obj_stock(struct memcg_stock_pcp *stock) unsigned int nr_pages = stock->nr_bytes >> PAGE_SHIFT; unsigned int nr_bytes = stock->nr_bytes & (PAGE_SIZE - 1); - if (nr_pages) { - rcu_read_lock(); - __memcg_kmem_uncharge(obj_cgroup_memcg(old), nr_pages); - rcu_read_unlock(); - } + if (nr_pages) + obj_cgroup_uncharge_pages(old, nr_pages); /* * The leftover is flushed to the centralized per-memcg value. @@ -3242,7 +3267,6 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size) { - struct mem_cgroup *memcg; unsigned int nr_pages, nr_bytes; int ret; @@ -3259,24 +3283,16 @@ int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size) * refill_obj_stock(), called from this function or * independently later. */ - rcu_read_lock(); -retry: - memcg = obj_cgroup_memcg(objcg); - if (unlikely(!css_tryget(&memcg->css))) - goto retry; - rcu_read_unlock(); - nr_pages = size >> PAGE_SHIFT; nr_bytes = size & (PAGE_SIZE - 1); if (nr_bytes) nr_pages += 1; - ret = __memcg_kmem_charge(memcg, gfp, nr_pages); + ret = obj_cgroup_charge_pages(objcg, gfp, nr_pages); if (!ret && nr_bytes) refill_obj_stock(objcg, PAGE_SIZE - nr_bytes); - css_put(&memcg->css); return ret; } From patchwork Thu Mar 18 11:06:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12147917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CF03C433E0 for ; Thu, 18 Mar 2021 11:08:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7653A64DBA for ; Thu, 18 Mar 2021 11:08:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7653A64DBA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 062AE6B0072; Thu, 18 Mar 2021 07:08:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 039426B0073; Thu, 18 Mar 2021 07:08:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E1B066B0074; Thu, 18 Mar 2021 07:08:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0183.hostedemail.com [216.40.44.183]) by kanga.kvack.org (Postfix) with ESMTP id CAEC16B0072 for ; Thu, 18 Mar 2021 07:08:36 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 935A5180222AB for ; Thu, 18 Mar 2021 11:08:36 +0000 (UTC) X-FDA: 77932721832.16.802EE0D Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf24.hostedemail.com (Postfix) with ESMTP id 00456A0000FD for ; Thu, 18 Mar 2021 11:08:35 +0000 (UTC) Received: by mail-pl1-f175.google.com with SMTP id v23so1120372ple.9 for ; Thu, 18 Mar 2021 04:08:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6xtw62qD+a/Q8lM+edqL90eMuRS6Phfvme83OFdVlXw=; b=rB7UVX6Cw0ILixk1ob5vWfS1HkVmsmbTSSwfxC7TE5WQ3fh+AvmkDsV1Qx1cqFM2A6 FItEFb7z4z34js3YLg3dXSGQiLrS0rQEDhX6OsyLtz4LHOm6A6WuR/2KiGdFE+/neqvy +wWHWlp8OCl290Ds7S6MKE4TT19nkgoNHkdkC1d8bWuH13kfs6sjYwDKLJG14NWGRMaX rCVfNanTnlMuhyxKyLqY4VIlGGeXZkV4uCDnyQTRukB7VNm9ZgXfQPLsWpn9QTkJdxAo PaC52X38X5Cs2cI/CQVmw1MbkMKivRczESS0Y9WReaIG8x7/Yp/6O/GTVQk3B0GDQvzP HeFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6xtw62qD+a/Q8lM+edqL90eMuRS6Phfvme83OFdVlXw=; b=EcbKs8UMLbX/TZ4aL8WsP1szpwXbeRP0dBUdQRMESBbVePKWy5cT7+iAFpy5stkdbJ Pa1mA9cMiRsxm5F1/fqpPNH+2mdVR5BU+0duD1ejiOx0KXnEzAcrLV9xvNXniWoOY+JD qJamOBA7U5fftHS9hVkBQkUO3OXf4XKVvyzcAY/PvJQ2rp3Tmlj50kiECutK2GYAA4lh vfiaFTiCTlhA+yFfuBjva6hhFBikvVjknvTbxNa7OFut9AOu1HL6EhZ1i4bkXpK/B6aX eB9wGkufO6KKSMV2JAgSTlq6LSxsv2nFzA6qXO9BRRsohoEjBDxCwNJK+yuk9ibBNuXg RJ0g== X-Gm-Message-State: AOAM533vefgQhcy0DepZbe+EqYvofHKug/n+XQSPOXmrPqAapkho5eFd cvvsze0cnTMbg7JNfO+QsFXdQQ== X-Google-Smtp-Source: ABdhPJzIa469RR5z4zKgOSL2tgWTJOS7BOpGioj765KvTrpGIKZhprUgxCm/4fFELrfQOwe7585A4w== X-Received: by 2002:a17:90a:bc06:: with SMTP id w6mr3707116pjr.44.1616065715161; Thu, 18 Mar 2021 04:08:35 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.231]) by smtp.gmail.com with ESMTPSA id e21sm1779509pgv.74.2021.03.18.04.08.31 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 18 Mar 2021 04:08:34 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH v4 2/5] mm: memcontrol: directly access page->memcg_data in mm/page_alloc.c Date: Thu, 18 Mar 2021 19:06:55 +0800 Message-Id: <20210318110658.60892-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210318110658.60892-1-songmuchun@bytedance.com> References: <20210318110658.60892-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: jpwaey8wrk7g6cmyg66xqmmss6t11j4p X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 00456A0000FD Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf24; identity=mailfrom; envelope-from=""; helo=mail-pl1-f175.google.com; client-ip=209.85.214.175 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616065715-861255 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page_memcg() is not suitable for use by page_expected_state() and page_bad_reason(). Because it can BUG_ON() for the slab pages when CONFIG_DEBUG_VM is enabled. As neither lru, nor kmem, nor slab page should have anything left in there by the time the page is freed, what we care about is whether the value of page->memcg_data is 0. So just directly access page->memcg_data here. Signed-off-by: Muchun Song Acked-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/page_alloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f10966e3b4a5..e5454b85a106 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1124,7 +1124,7 @@ static inline bool page_expected_state(struct page *page, if (unlikely((unsigned long)page->mapping | page_ref_count(page) | #ifdef CONFIG_MEMCG - (unsigned long)page_memcg(page) | + page->memcg_data | #endif (page->flags & check_flags))) return false; @@ -1149,7 +1149,7 @@ static const char *page_bad_reason(struct page *page, unsigned long flags) bad_reason = "PAGE_FLAGS_CHECK_AT_FREE flag(s) set"; } #ifdef CONFIG_MEMCG - if (unlikely(page_memcg(page))) + if (unlikely(page->memcg_data)) bad_reason = "page still charged to cgroup"; #endif return bad_reason; From patchwork Thu Mar 18 11:06:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12147919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1B1FC433E0 for ; Thu, 18 Mar 2021 11:08:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 26A9D6186A for ; Thu, 18 Mar 2021 11:08:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 26A9D6186A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B84B36B0073; Thu, 18 Mar 2021 07:08:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B5BEA6B0074; Thu, 18 Mar 2021 07:08:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A246F6B0075; Thu, 18 Mar 2021 07:08:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 886AD6B0073 for ; Thu, 18 Mar 2021 07:08:41 -0400 (EDT) Received: from smtpin37.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 4B9418249980 for ; Thu, 18 Mar 2021 11:08:41 +0000 (UTC) X-FDA: 77932722042.37.FA31CB5 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf24.hostedemail.com (Postfix) with ESMTP id D4D70A0009E0 for ; Thu, 18 Mar 2021 11:08:40 +0000 (UTC) Received: by mail-pl1-f181.google.com with SMTP id v8so1121032plz.10 for ; Thu, 18 Mar 2021 04:08:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=15LgLWnrDF9xrY/f9Qycq0PzYRn8kF+pr7olZciihDc=; b=KtCvKs+mdXNV7hiXR9LnggjmgOU3pD83nEi3aMALjUmJrnwc0VCJWtA33DzRM3ejR+ 8YCeNCjveM54mbqMhLrw7JP3SDAA/+rS/tHgOg0YaCpi3cyMXag0Dd61VsNKgMB2tvID 2b3pQVxuK/+w61Y4cHv1gZ2IyWU6+YiEJEGoFb3ThIPpM7fhElDTPdmC8c0tPj13uqN7 Cnc09uKri9MNkeloxXsBQiYNrzdIxFI+iQgNuZlIcQ/cYabVryhTdoTfUO6LWLt+9hxe LqHeBrO8JdxSUeNpZP/k85gDtzMZNOpwKW35Zg+IlzLlJg2zS4J2SFgP68SJqs4Lp1+l lemA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=15LgLWnrDF9xrY/f9Qycq0PzYRn8kF+pr7olZciihDc=; b=pQF9Ihx4bg2VpQizeup9WolnavCkU4nvoqw0QD1Ge3G3W6bT/NRC0diaRdAGN2KRno gP6Lz8q0w+OZm283ubeGxc/r8wA1p2EJ1Dhj4mMPaOqtJp63QP68tGHAkNE9ipnV7WWs go1crhlAK4tuq19c1Ni5FO71828bVlK76HRGnaVt+31s/EVTLRQjgFkfPiD4tqghtQr3 73vrNdZdApIsd+s0VxP2dEVh87nD4P1/a8HMTr2x1l7HATbH8yg4+AekCbm1rgsTb0/4 4gzBGrQmNKEj8Mx2LIQb5BOUmHEtwVgDcgUA/+yvVrCyfZqvhNyuNRBpfj/H+wVtt7Ai Ad6Q== X-Gm-Message-State: AOAM532NN2Fs5fVK/vEjmV+YinMk7ah1kvcvUHIweBnbsmV/WQAjLW9u xZd5H+4bcvxBGcNGXq6oo7WVPA== X-Google-Smtp-Source: ABdhPJwNlbzYvF2snPZSAkGAO2vl/mGld5g0amM9i3l7jk7w+k2UvRsVeWdXBYNJeKHXR8MS6D85HA== X-Received: by 2002:a17:90a:6e44:: with SMTP id s4mr3579552pjm.112.1616065720004; Thu, 18 Mar 2021 04:08:40 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.231]) by smtp.gmail.com with ESMTPSA id e21sm1779509pgv.74.2021.03.18.04.08.35 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 18 Mar 2021 04:08:39 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH v4 3/5] mm: memcontrol: change ug->dummy_page only if memcg changed Date: Thu, 18 Mar 2021 19:06:56 +0800 Message-Id: <20210318110658.60892-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210318110658.60892-1-songmuchun@bytedance.com> References: <20210318110658.60892-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: t81x5go5z4n5tuhppkqyz3wd87shdesq X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D4D70A0009E0 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf24; identity=mailfrom; envelope-from=""; helo=mail-pl1-f181.google.com; client-ip=209.85.214.181 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616065720-299741 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just like assignment to ug->memcg, we only need to update ug->dummy_page if memcg changed. So move it to there. This is a very small optimization. Signed-off-by: Muchun Song Acked-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/memcontrol.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index fc22da9805fb..104bddf21314 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6845,6 +6845,7 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) uncharge_gather_clear(ug); } ug->memcg = page_memcg(page); + ug->dummy_page = page; /* pairs with css_put in uncharge_batch */ css_get(&ug->memcg->css); @@ -6858,7 +6859,6 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) else ug->pgpgout++; - ug->dummy_page = page; page->memcg_data = 0; css_put(&ug->memcg->css); } From patchwork Thu Mar 18 11:06:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12147921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A038AC433DB for ; Thu, 18 Mar 2021 11:08:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CEEC864DD8 for ; Thu, 18 Mar 2021 11:08:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CEEC864DD8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5929C6B0074; Thu, 18 Mar 2021 07:08:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 569016B0075; Thu, 18 Mar 2021 07:08:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3BD066B0078; Thu, 18 Mar 2021 07:08:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0083.hostedemail.com [216.40.44.83]) by kanga.kvack.org (Postfix) with ESMTP id 1BD9C6B0074 for ; Thu, 18 Mar 2021 07:08:46 -0400 (EDT) Received: from smtpin32.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D655B8249980 for ; Thu, 18 Mar 2021 11:08:45 +0000 (UTC) X-FDA: 77932722210.32.04EEF36 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf22.hostedemail.com (Postfix) with ESMTP id 30305C0007C2 for ; Thu, 18 Mar 2021 11:08:45 +0000 (UTC) Received: by mail-pl1-f182.google.com with SMTP id k4so1122054plk.5 for ; Thu, 18 Mar 2021 04:08:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=08vyzPUPKvmLNfOadbgAGNQdLOXmbl0LbP58+XW4jVM=; b=lJOHSBb8un9AZY5DlX2KQY+f+BrK1KheoJjVbMIQI1taLuqBgAAC79D4bDNNekaNkK 5dOVtF4X/SgaPxsHRsKwNLC/F+0e6f75k1vwrNkP1aGpDmht/TN7JOIohR12MWxmSGOj QRCaI7Wa2X3D2QDtIVMvZkTXozhzTMRZe53uC24myEYqtEUVtiWACelEl3gdAudZNn3D g1Sd2TtPBtTamsEnjoCCHmCNA516e9wbQbcoIEbsGDxBhH2MbEvSYAeOAunWyjcOnos0 2WOtdXDde3C9EtwOwYlz00cNxxUf5JyOSCk9/hPmZjccOuM77a17Wyw4uL5nTVyi49z/ M1mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=08vyzPUPKvmLNfOadbgAGNQdLOXmbl0LbP58+XW4jVM=; b=RZzjpJ6pQ0SJimrxaZTE0HM+Sr6zKwKy2C8Oe2uktcjcDdr6cJcQtAQZ7aNXEr6dbU WUXJHn0VV2W9zEJGo48cGXv1rFygfWfCPv+QowtqYDt4k4roZpNga73DRRrK5haodfmZ lMt9gxKXwVMW8JAeQUhlNiHLtxjiC/lbMYKZww05Tgec2ihC4+dzTZnp6M5IwHzuRHtz nGws/L+yddVUz0q2A93ceCkyuPGQBSo6QlvTwFLGHkPBc34THKfVGUU0b1l5BmkM8m1V pOeG20D80sav1cPuST44r/MHVcgz/SRBNhbe/CSIhAwAdxG0tdvWX0iRz3/GEv3guzUJ nwag== X-Gm-Message-State: AOAM533HX4BnMldc5IYxczWRYdodbIUD1U3TAImGWo4RWb7EmjILG16A 7Mt9a9pEpe1CJcUnY9tyZytwSg== X-Google-Smtp-Source: ABdhPJz66Q481dL8WfSeW2L0N6YQirVOEzoQXHkK0DamES8JlQLHZuohYoFAPOn8UCn26YHqpE897g== X-Received: by 2002:a17:902:ea0e:b029:e4:81d4:ddae with SMTP id s14-20020a170902ea0eb02900e481d4ddaemr9248123plg.12.1616065724143; Thu, 18 Mar 2021 04:08:44 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.231]) by smtp.gmail.com with ESMTPSA id e21sm1779509pgv.74.2021.03.18.04.08.40 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 18 Mar 2021 04:08:43 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH v4 4/5] mm: memcontrol: use obj_cgroup APIs to charge kmem pages Date: Thu, 18 Mar 2021 19:06:57 +0800 Message-Id: <20210318110658.60892-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210318110658.60892-1-songmuchun@bytedance.com> References: <20210318110658.60892-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 30305C0007C2 X-Stat-Signature: tcctgxjrb9ynnrqmik57uoqmsx8kjkn8 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf22; identity=mailfrom; envelope-from=""; helo=mail-pl1-f182.google.com; client-ip=209.85.214.182 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616065725-275149 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since Roman series "The new cgroup slab memory controller" applied. All slab objects are charged via the new APIs of obj_cgroup. The new APIs introduce a struct obj_cgroup to charge slab objects. It prevents long-living objects from pinning the original memory cgroup in the memory. But there are still some corner objects (e.g. allocations larger than order-1 page on SLUB) which are not charged via the new APIs. Those objects (include the pages which are allocated from buddy allocator directly) are charged as kmem pages which still hold a reference to the memory cgroup. We want to reuse the obj_cgroup APIs to charge the kmem pages. If we do that, we should store an object cgroup pointer to page->memcg_data for the kmem pages. Finally, page->memcg_data will have 3 different meanings. 1) For the slab pages, page->memcg_data points to an object cgroups vector. 2) For the kmem pages (exclude the slab pages), page->memcg_data points to an object cgroup. 3) For the user pages (e.g. the LRU pages), page->memcg_data points to a memory cgroup. We do not change the behavior of page_memcg() and page_memcg_rcu(). They are also suitable for LRU pages and kmem pages. Why? Because memory allocations pinning memcgs for a long time - it exists at a larger scale and is causing recurring problems in the real world: page cache doesn't get reclaimed for a long time, or is used by the second, third, fourth, ... instance of the same job that was restarted into a new cgroup every time. Unreclaimable dying cgroups pile up, waste memory, and make page reclaim very inefficient. We can convert LRU pages and most other raw memcg pins to the objcg direction to fix this problem, and then the page->memcg will always point to an object cgroup pointer. At that time, LRU pages and kmem pages will be treated the same. The implementation of page_memcg() will remove the kmem page check. This patch aims to charge the kmem pages by using the new APIs of obj_cgroup. Finally, the page->memcg_data of the kmem page points to an object cgroup. We can use the __page_objcg() to get the object cgroup associated with a kmem page. Or we can use page_memcg() to get the memory cgroup associated with a kmem page, but caller must ensure that the returned memcg won't be released (e.g. acquire the rcu_read_lock or css_set_lock). Signed-off-by: Muchun Song Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 116 +++++++++++++++++++++++++++++++++++---------- mm/memcontrol.c | 101 ++++++++++++++++++++++++--------------- 2 files changed, 156 insertions(+), 61 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e6dc793d587d..395a113e4a3b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -358,6 +358,62 @@ enum page_memcg_data_flags { #define MEMCG_DATA_FLAGS_MASK (__NR_MEMCG_DATA_FLAGS - 1) +static inline bool PageMemcgKmem(struct page *page); + +/* + * After the initialization objcg->memcg is always pointing at + * a valid memcg, but can be atomically swapped to the parent memcg. + * + * The caller must ensure that the returned memcg won't be released: + * e.g. acquire the rcu_read_lock or css_set_lock. + */ +static inline struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *objcg) +{ + return READ_ONCE(objcg->memcg); +} + +/* + * __page_memcg - get the memory cgroup associated with a non-kmem page + * @page: a pointer to the page struct + * + * Returns a pointer to the memory cgroup associated with the page, + * or NULL. This function assumes that the page is known to have a + * proper memory cgroup pointer. It's not safe to call this function + * against some type of pages, e.g. slab pages or ex-slab pages or + * kmem pages. + */ +static inline struct mem_cgroup *__page_memcg(struct page *page) +{ + unsigned long memcg_data = page->memcg_data; + + VM_BUG_ON_PAGE(PageSlab(page), page); + VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_OBJCGS, page); + VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_KMEM, page); + + return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); +} + +/* + * __page_objcg - get the object cgroup associated with a kmem page + * @page: a pointer to the page struct + * + * Returns a pointer to the object cgroup associated with the page, + * or NULL. This function assumes that the page is known to have a + * proper object cgroup pointer. It's not safe to call this function + * against some type of pages, e.g. slab pages or ex-slab pages or + * LRU pages. + */ +static inline struct obj_cgroup *__page_objcg(struct page *page) +{ + unsigned long memcg_data = page->memcg_data; + + VM_BUG_ON_PAGE(PageSlab(page), page); + VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_OBJCGS, page); + VM_BUG_ON_PAGE(!(memcg_data & MEMCG_DATA_KMEM), page); + + return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); +} + /* * page_memcg - get the memory cgroup associated with a page * @page: a pointer to the page struct @@ -367,20 +423,23 @@ enum page_memcg_data_flags { * proper memory cgroup pointer. It's not safe to call this function * against some type of pages, e.g. slab pages or ex-slab pages. * - * Any of the following ensures page and memcg binding stability: + * For a non-kmem page any of the following ensures page and memcg binding + * stability: + * * - the page lock * - LRU isolation * - lock_page_memcg() * - exclusive reference + * + * For a kmem page a caller should hold an rcu read lock to protect memcg + * associated with a kmem page from being released. */ static inline struct mem_cgroup *page_memcg(struct page *page) { - unsigned long memcg_data = page->memcg_data; - - VM_BUG_ON_PAGE(PageSlab(page), page); - VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_OBJCGS, page); - - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + if (PageMemcgKmem(page)) + return obj_cgroup_memcg(__page_objcg(page)); + else + return __page_memcg(page); } /* @@ -394,11 +453,19 @@ static inline struct mem_cgroup *page_memcg(struct page *page) */ static inline struct mem_cgroup *page_memcg_rcu(struct page *page) { + unsigned long memcg_data = READ_ONCE(page->memcg_data); + VM_BUG_ON_PAGE(PageSlab(page), page); WARN_ON_ONCE(!rcu_read_lock_held()); - return (struct mem_cgroup *)(READ_ONCE(page->memcg_data) & - ~MEMCG_DATA_FLAGS_MASK); + if (memcg_data & MEMCG_DATA_KMEM) { + struct obj_cgroup *objcg; + + objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return obj_cgroup_memcg(objcg); + } + + return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } /* @@ -406,15 +473,21 @@ static inline struct mem_cgroup *page_memcg_rcu(struct page *page) * @page: a pointer to the page struct * * Returns a pointer to the memory cgroup associated with the page, - * or NULL. This function unlike page_memcg() can take any page + * or NULL. This function unlike page_memcg() can take any page * as an argument. It has to be used in cases when it's not known if a page - * has an associated memory cgroup pointer or an object cgroups vector. + * has an associated memory cgroup pointer or an object cgroups vector or + * an object cgroup. + * + * For a non-kmem page any of the following ensures page and memcg binding + * stability: * - * Any of the following ensures page and memcg binding stability: * - the page lock * - LRU isolation * - lock_page_memcg() * - exclusive reference + * + * For a kmem page a caller should hold an rcu read lock to protect memcg + * associated with a kmem page from being released. */ static inline struct mem_cgroup *page_memcg_check(struct page *page) { @@ -427,6 +500,13 @@ static inline struct mem_cgroup *page_memcg_check(struct page *page) if (memcg_data & MEMCG_DATA_OBJCGS) return NULL; + if (memcg_data & MEMCG_DATA_KMEM) { + struct obj_cgroup *objcg; + + objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return obj_cgroup_memcg(objcg); + } + return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } @@ -713,18 +793,6 @@ static inline void obj_cgroup_put(struct obj_cgroup *objcg) percpu_ref_put(&objcg->refcnt); } -/* - * After the initialization objcg->memcg is always pointing at - * a valid memcg, but can be atomically swapped to the parent memcg. - * - * The caller must ensure that the returned memcg won't be released: - * e.g. acquire the rcu_read_lock or css_set_lock. - */ -static inline struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *objcg) -{ - return READ_ONCE(objcg->memcg); -} - static inline void mem_cgroup_put(struct mem_cgroup *memcg) { if (memcg) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 104bddf21314..1cef20a2f116 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -855,18 +855,22 @@ void __mod_lruvec_page_state(struct page *page, enum node_stat_item idx, int val) { struct page *head = compound_head(page); /* rmap on tail pages */ - struct mem_cgroup *memcg = page_memcg(head); + struct mem_cgroup *memcg; pg_data_t *pgdat = page_pgdat(page); struct lruvec *lruvec; + rcu_read_lock(); + memcg = page_memcg(head); /* Untracked pages have no memcg, no lruvec. Update only the node */ if (!memcg) { + rcu_read_unlock(); __mod_node_page_state(pgdat, idx, val); return; } lruvec = mem_cgroup_lruvec(memcg, pgdat); __mod_lruvec_state(lruvec, idx, val); + rcu_read_unlock(); } EXPORT_SYMBOL(__mod_lruvec_page_state); @@ -2905,6 +2909,20 @@ static void commit_charge(struct page *page, struct mem_cgroup *memcg) page->memcg_data = (unsigned long)memcg; } +static inline struct mem_cgroup *get_obj_cgroup_memcg(struct obj_cgroup *objcg) +{ + struct mem_cgroup *memcg; + + rcu_read_lock(); +retry: + memcg = obj_cgroup_memcg(objcg); + if (unlikely(!css_tryget(&memcg->css))) + goto retry; + rcu_read_unlock(); + + return memcg; +} + #ifdef CONFIG_MEMCG_KMEM int memcg_alloc_page_obj_cgroups(struct page *page, struct kmem_cache *s, gfp_t gfp, bool new_page) @@ -3070,15 +3088,8 @@ static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, struct mem_cgroup *memcg; int ret; - rcu_read_lock(); -retry: - memcg = obj_cgroup_memcg(objcg); - if (unlikely(!css_tryget(&memcg->css))) - goto retry; - rcu_read_unlock(); - + memcg = get_obj_cgroup_memcg(objcg); ret = __memcg_kmem_charge(memcg, gfp, nr_pages); - css_put(&memcg->css); return ret; @@ -3143,18 +3154,18 @@ static void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_page */ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) { - struct mem_cgroup *memcg; + struct obj_cgroup *objcg; int ret = 0; - memcg = get_mem_cgroup_from_current(); - if (memcg && !mem_cgroup_is_root(memcg)) { - ret = __memcg_kmem_charge(memcg, gfp, 1 << order); + objcg = get_obj_cgroup_from_current(); + if (objcg) { + ret = obj_cgroup_charge_pages(objcg, gfp, 1 << order); if (!ret) { - page->memcg_data = (unsigned long)memcg | + page->memcg_data = (unsigned long)objcg | MEMCG_DATA_KMEM; return 0; } - css_put(&memcg->css); + obj_cgroup_put(objcg); } return ret; } @@ -3166,16 +3177,16 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) */ void __memcg_kmem_uncharge_page(struct page *page, int order) { - struct mem_cgroup *memcg = page_memcg(page); + struct obj_cgroup *objcg; unsigned int nr_pages = 1 << order; - if (!memcg) + if (!PageMemcgKmem(page)) return; - VM_BUG_ON_PAGE(mem_cgroup_is_root(memcg), page); - __memcg_kmem_uncharge(memcg, nr_pages); + objcg = __page_objcg(page); + obj_cgroup_uncharge_pages(objcg, nr_pages); page->memcg_data = 0; - css_put(&memcg->css); + obj_cgroup_put(objcg); } static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) @@ -6790,7 +6801,7 @@ int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask) struct uncharge_gather { struct mem_cgroup *memcg; - unsigned long nr_pages; + unsigned long nr_memory; unsigned long pgpgout; unsigned long nr_kmem; struct page *dummy_page; @@ -6805,10 +6816,10 @@ static void uncharge_batch(const struct uncharge_gather *ug) { unsigned long flags; - if (!mem_cgroup_is_root(ug->memcg)) { - page_counter_uncharge(&ug->memcg->memory, ug->nr_pages); + if (ug->nr_memory) { + page_counter_uncharge(&ug->memcg->memory, ug->nr_memory); if (do_memsw_account()) - page_counter_uncharge(&ug->memcg->memsw, ug->nr_pages); + page_counter_uncharge(&ug->memcg->memsw, ug->nr_memory); if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && ug->nr_kmem) page_counter_uncharge(&ug->memcg->kmem, ug->nr_kmem); memcg_oom_recover(ug->memcg); @@ -6816,7 +6827,7 @@ static void uncharge_batch(const struct uncharge_gather *ug) local_irq_save(flags); __count_memcg_events(ug->memcg, PGPGOUT, ug->pgpgout); - __this_cpu_add(ug->memcg->vmstats_percpu->nr_page_events, ug->nr_pages); + __this_cpu_add(ug->memcg->vmstats_percpu->nr_page_events, ug->nr_memory); memcg_check_events(ug->memcg, ug->dummy_page); local_irq_restore(flags); @@ -6827,40 +6838,56 @@ static void uncharge_batch(const struct uncharge_gather *ug) static void uncharge_page(struct page *page, struct uncharge_gather *ug) { unsigned long nr_pages; + struct mem_cgroup *memcg; + struct obj_cgroup *objcg; VM_BUG_ON_PAGE(PageLRU(page), page); - if (!page_memcg(page)) - return; - /* * Nobody should be changing or seriously looking at - * page_memcg(page) at this point, we have fully + * page memcg or objcg at this point, we have fully * exclusive access to the page. */ + if (PageMemcgKmem(page)) { + objcg = __page_objcg(page); + memcg = get_obj_cgroup_memcg(objcg); + } else { + memcg = __page_memcg(page); + } + + if (!memcg) + return; - if (ug->memcg != page_memcg(page)) { + if (ug->memcg != memcg) { if (ug->memcg) { uncharge_batch(ug); uncharge_gather_clear(ug); } - ug->memcg = page_memcg(page); + ug->memcg = memcg; ug->dummy_page = page; /* pairs with css_put in uncharge_batch */ - css_get(&ug->memcg->css); + css_get(&memcg->css); } nr_pages = compound_nr(page); - ug->nr_pages += nr_pages; - if (PageMemcgKmem(page)) + if (PageMemcgKmem(page)) { + ug->nr_memory += nr_pages; ug->nr_kmem += nr_pages; - else + + page->memcg_data = 0; + obj_cgroup_put(objcg); + } else { + /* LRU pages aren't accounted at the root level */ + if (!mem_cgroup_is_root(memcg)) + ug->nr_memory += nr_pages; ug->pgpgout++; - page->memcg_data = 0; - css_put(&ug->memcg->css); + page->memcg_data = 0; + } + + css_put(&memcg->css); } /** From patchwork Thu Mar 18 11:06:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12147923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1302FC433E0 for ; Thu, 18 Mar 2021 11:08:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A33E7601FD for ; Thu, 18 Mar 2021 11:08:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A33E7601FD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 320B36B0075; Thu, 18 Mar 2021 07:08:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F85A6B0078; Thu, 18 Mar 2021 07:08:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1BFE26B007B; Thu, 18 Mar 2021 07:08:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0142.hostedemail.com [216.40.44.142]) by kanga.kvack.org (Postfix) with ESMTP id BE9F96B0075 for ; Thu, 18 Mar 2021 07:08:49 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 8A42A83FB for ; Thu, 18 Mar 2021 11:08:49 +0000 (UTC) X-FDA: 77932722378.27.753B29D Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf19.hostedemail.com (Postfix) with ESMTP id 2BE2490009D6 for ; Thu, 18 Mar 2021 11:08:49 +0000 (UTC) Received: by mail-pf1-f181.google.com with SMTP id y5so3240314pfn.1 for ; Thu, 18 Mar 2021 04:08:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2a1Q2yqghil2SSuBrZCKn/WKXtEY0qZpSKwUUGtGYmY=; b=hX6/GKnUT+Ilw/SDCCSBWdFzs4WAT6mGp/EfDQfq8LGKDxZ+SKKA7NxoIHTfJ+a/nX VhS0xLPj5LjDtRdfrNYbeEO+LRjq69d8n50zryDPlHp2Ojva4nqea0rDXjk+UlaCJIab WyDJBosKmTyQw3uRCCHZjq25G8//ycGePxYRHo+ydN8I/qVMx1Vnts6J6F8Bu11BtY/9 IUflt7mGSnn+997hIM8VMWgmrhrO8wt94uFlH2jpVIuJR/H3UXMpvdxP0y8dpio8WM1z GL0ejD58nG7HecS6pj5zUo1KZKmgu/RjQ91OkKYJ7yeAjwBLYThZp+GYxSrcnRPjUVem jetg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2a1Q2yqghil2SSuBrZCKn/WKXtEY0qZpSKwUUGtGYmY=; b=h9S+6PUB1GUqgGZm/Hi777Zn/i5qVtjLoQ3Te6UHqn21Bwqdu93um6Tk5FNzkDTvHQ oN+uMADQub8ubfbG/eKiKzo+iePF+g66+nlZUQnCNAeOKywaJmYcfukwBhkkvybr79mD swCFzvyyIfMI4/JaEf9CULxWB1MdVLTTdcMAxwqmN+F9xse4pK9ASS/sDG0gwE1s6cih neCN2rtqbKP//RDd0Oq6BrAWXrFLmLAmXImxSbk9cn3fyyA1lRS2+lCfLqtrReMpRthf tGMf8s0ZTkqX8fJFn0AmwlwZWrOKcGsfH3zWwK5dJepyhpNoc8XEQymX7xrnSt53q5Xs B1sQ== X-Gm-Message-State: AOAM530yo7g1UmOXbfexHOcQIpA+jtwMfz7WnuTFChe2oBNOwlzMxFKw qRgYYBC7/TM5sfl00Eggm9W5sg== X-Google-Smtp-Source: ABdhPJxFBl9WuSrFTiRWvdlXHku6toNPWED+IUaMV77zPDYMhGWwbqGQh69e6odC2uF5ITmlXNk3BQ== X-Received: by 2002:aa7:8e51:0:b029:1ed:2928:18ff with SMTP id d17-20020aa78e510000b02901ed292818ffmr3514424pfr.76.1616065728142; Thu, 18 Mar 2021 04:08:48 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.231]) by smtp.gmail.com with ESMTPSA id e21sm1779509pgv.74.2021.03.18.04.08.44 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 18 Mar 2021 04:08:47 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH v4 5/5] mm: memcontrol: move PageMemcgKmem to the scope of CONFIG_MEMCG_KMEM Date: Thu, 18 Mar 2021 19:06:58 +0800 Message-Id: <20210318110658.60892-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210318110658.60892-1-songmuchun@bytedance.com> References: <20210318110658.60892-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: zag9hfwjciacawhmqhzoaphgzai4zc8u X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 2BE2490009D6 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf19; identity=mailfrom; envelope-from=""; helo=mail-pf1-f181.google.com; client-ip=209.85.210.181 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616065729-552528 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page only can be marked as kmem when CONFIG_MEMCG_KMEM is enabled. So move PageMemcgKmem() to the scope of the CONFIG_MEMCG_KMEM. As a bonus, on !CONFIG_MEMCG_KMEM build some code can be compiled out. Signed-off-by: Muchun Song Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 395a113e4a3b..7fdc92e1983e 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -510,6 +510,7 @@ static inline struct mem_cgroup *page_memcg_check(struct page *page) return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } +#ifdef CONFIG_MEMCG_KMEM /* * PageMemcgKmem - check if the page has MemcgKmem flag set * @page: a pointer to the page struct @@ -524,7 +525,6 @@ static inline bool PageMemcgKmem(struct page *page) return page->memcg_data & MEMCG_DATA_KMEM; } -#ifdef CONFIG_MEMCG_KMEM /* * page_objcgs - get the object cgroups vector associated with a page * @page: a pointer to the page struct @@ -566,6 +566,11 @@ static inline struct obj_cgroup **page_objcgs_check(struct page *page) } #else +static inline bool PageMemcgKmem(struct page *page) +{ + return false; +} + static inline struct obj_cgroup **page_objcgs(struct page *page) { return NULL;