From patchwork Fri Mar 19 16:38:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12151403 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD044C433C1 for ; Fri, 19 Mar 2021 16:38:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4F61361934 for ; Fri, 19 Mar 2021 16:38:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4F61361934 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D0CAB8D000E; Fri, 19 Mar 2021 12:38:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CBD2A6B008A; Fri, 19 Mar 2021 12:38:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5D048D000E; Fri, 19 Mar 2021 12:38:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0167.hostedemail.com [216.40.44.167]) by kanga.kvack.org (Postfix) with ESMTP id 997366B007B for ; Fri, 19 Mar 2021 12:38:49 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4B9E31801D01C for ; Fri, 19 Mar 2021 16:38:49 +0000 (UTC) X-FDA: 77937182778.15.CCB40C2 Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by imf23.hostedemail.com (Postfix) with ESMTP id AB5BBA0009CA for ; Fri, 19 Mar 2021 16:38:43 +0000 (UTC) Received: by mail-pg1-f175.google.com with SMTP id r17so4054715pgi.0 for ; Fri, 19 Mar 2021 09:38:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=q1x9F0RxytCzXg/daJ/mfbteZTYPjROB2MKRFPS4wa0=; b=DEMZOb2ZxxtpsbPJtNeCUdXVflxX0xnzNmd0okHPwRSj8fPXt7AUSn/411t7ZnSc1V mE6TsuZKMqrrwsil/GbrCJeIPHzCu+25dTYIxKzm72VWt6NO59WeZZ/QsJXqR4CRuZqT GBQHQIn2nlJmGOJnLFb4R9rm8pEvGE2Fj31DotWldMM1g9KRLtxBLhv0rtLzgXoYMTad ACquv1g/mbCZGDrX0KrYzoofSH5h9SeC1S2Qdo5hDoHUV/ZeC5WIruC2QrNrjBoJTyO8 vZ/tDIROyr5ka6LxC/5tIDXSLbKKMHZ8l6DyHC79+09L6/f/bFKZHMYFCJqtclTkXpT6 T12g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=q1x9F0RxytCzXg/daJ/mfbteZTYPjROB2MKRFPS4wa0=; b=bLPRCHdXN30cG9lmcIXWSjoAMdL64/+os79POPiH5brAHRIfBULz9hyDANmTJVDG8J WZrpxWXrBBhYUMwtw4MPzUcgHAUXi5mEhkW4hQyDrONQLNuY7OLu2bYlntwRaVko15Pt DdvRrw1sUJed4KNIAGxayft8wAQNRpp6gAwnhvt3x2Vq2A4O4pUfzz0Whzx/ols6jSiD LML+krecigljMeL411mw9RkhTTcyM83x7F/5wzqu74Yfu9Hy5P9MQ5D2qRXC7rivJJlR uG0tO+z9m1t4dy0m+Zv/zs5//YCizI8Ryv/nXUHeE3WqXq/IFaDToeEIyZ9sjhsIgNd5 7E0Q== X-Gm-Message-State: AOAM532MX1YFilmRIbOFOSQ3wDIo3RzZ+tDWFEAEsqlLmvS0ZVfHu6Qb Pnv0WDvnkstEB3D2Xossiem8Kg== X-Google-Smtp-Source: ABdhPJzv6KoFJsnEXmVe1wLnx5vlmo8q+uLsN4x2hPYnirz8Y5YOPeFFmr8wTjnjzbdpJXF6MUvWAQ== X-Received: by 2002:a63:fb18:: with SMTP id o24mr11855982pgh.55.1616171922411; Fri, 19 Mar 2021 09:38:42 -0700 (PDT) Received: from localhost.bytedance.net ([139.177.225.231]) by smtp.gmail.com with ESMTPSA id z25sm5860239pfn.37.2021.03.19.09.38.38 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Mar 2021 09:38:42 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH v5 1/7] mm: memcontrol: slab: fix obtain a reference to a freeing memcg Date: Sat, 20 Mar 2021 00:38:14 +0800 Message-Id: <20210319163821.20704-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210319163821.20704-1-songmuchun@bytedance.com> References: <20210319163821.20704-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: d3oegy9wo3jsuaahmex6haazpcruox5i X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: AB5BBA0009CA Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf23; identity=mailfrom; envelope-from=""; helo=mail-pg1-f175.google.com; client-ip=209.85.215.175 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616171923-424152 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The rcu_read_lock/unlock only can guarantee that the memcg will not be freed, but it cannot guarantee the success of css_get (which is in the refill_stock when cached memcg changed) to memcg. rcu_read_lock() memcg = obj_cgroup_memcg(old) __memcg_kmem_uncharge(memcg) refill_stock(memcg) if (stock->cached != memcg) // css_get can change the ref counter from 0 back to 1. css_get(&memcg->css) rcu_read_unlock() This fix is very like the commit: eefbfa7fd678 ("mm: memcg/slab: fix use after free in obj_cgroup_charge") Fix this by holding a reference to the memcg which is passed to the __memcg_kmem_uncharge() before calling __memcg_kmem_uncharge(). Fixes: 3de7d4f25a74 ("mm: memcg/slab: optimize objcg stock draining") Signed-off-by: Muchun Song Reviewed-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Roman Gushchin --- mm/memcontrol.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 845eec01ef9d..2cda76ff0629 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3181,9 +3181,17 @@ static void drain_obj_stock(struct memcg_stock_pcp *stock) unsigned int nr_bytes = stock->nr_bytes & (PAGE_SIZE - 1); if (nr_pages) { + struct mem_cgroup *memcg; + rcu_read_lock(); - __memcg_kmem_uncharge(obj_cgroup_memcg(old), nr_pages); +retry: + memcg = obj_cgroup_memcg(old); + if (unlikely(!css_tryget(&memcg->css))) + goto retry; rcu_read_unlock(); + + __memcg_kmem_uncharge(memcg, nr_pages); + css_put(&memcg->css); } /* From patchwork Fri Mar 19 16:38:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12151407 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 312ADC433C1 for ; Fri, 19 Mar 2021 16:39:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 97D5761934 for ; Fri, 19 Mar 2021 16:38:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 97D5761934 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0D77A8D0011; Fri, 19 Mar 2021 12:38:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0AF686B008C; Fri, 19 Mar 2021 12:38:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF51A8D0011; Fri, 19 Mar 2021 12:38:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0222.hostedemail.com [216.40.44.222]) by kanga.kvack.org (Postfix) with ESMTP id C19F06B008A for ; Fri, 19 Mar 2021 12:38:58 -0400 (EDT) Received: from smtpin37.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 8A02E180189E2 for ; Fri, 19 Mar 2021 16:38:58 +0000 (UTC) X-FDA: 77937183156.37.932D104 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf19.hostedemail.com (Postfix) with ESMTP id 501679001151 for ; Fri, 19 Mar 2021 16:38:47 +0000 (UTC) Received: by mail-pl1-f172.google.com with SMTP id g1so3213424plg.7 for ; Fri, 19 Mar 2021 09:38:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tK2Kyn55J1ACRGVq+2xCPNndPw6tNbpgN30+dQ6cs1M=; b=KlDrOsu3HXjqYEWfzW96OEELxMjlB+S8Aw6bTHI/LlQiy+ywS0J0DShcNA93lHkM20 OEDgjQOwtPsJk8R+HGZ7/R5Z4Qaa5HW6Ut14J7eJydjrNJsvIBRzHHOxwWcAe3dqzTh8 cNhr3GysdJxDaMDmOO1krGskM3Azh4LgDVPkBOehY+AILcU0bRItR3ZUqpYK0nbPg7qs deAZyDUWNtlbSriVmMeFcqsBGT/Ku27az6BYsrWqaJbxzZAYWeKz4s6pjgk9S30+jCY+ w/g8nP1rCZoTV6mkO6zVygUphY7HbszeLo8MZ0L6M0qi88MJU8PBPGF2AQVVWTnrvZ9/ zyKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tK2Kyn55J1ACRGVq+2xCPNndPw6tNbpgN30+dQ6cs1M=; b=sNXGrwCWjUkFNVog77KcRCLPMPk6k2XHPEiVjTagVCaAdy7aAYMFhdBrI6HrSJZFNY 3N/FNtX98hnlg0GZPZd8w/9caJjp1ZH9L6lo2fRngwdBswxk9HdjsmmMT63NPCvyPlmP f6EDDdThWwOFoSf6ivdFbqTYFloyuK/hYLFzAJTq/LkGmlx0anpg01eREejlvoVxsH+e b+PWGTWy3QSH6GvJgZ/mQpiT7eY+u7ocsvsBVWRkaOGRJ+Axe8eWkOsnDIxmUStKE/hn nrJDF29vsgYIyPmwhNEaU0YXgcJ26r4Xd6WxVBaTkDH3+fuBwjngcu0HvmV3XBfcb6Un 4i+w== X-Gm-Message-State: AOAM532VWUJsNWF146otjKixlyEw0L4W1NjDOicwhYdIwKtvdbxj95oe BdIP/cFS2j8kwrvHr+vn+PjvEA== X-Google-Smtp-Source: ABdhPJyO9aDWXR5ZTvV+MBsj7/ZU9SJfeDhvcDMisicQWhb4extbGfdF6avKGjschmd5J381U6rN2w== X-Received: by 2002:a17:90b:284:: with SMTP id az4mr10684970pjb.12.1616171926457; Fri, 19 Mar 2021 09:38:46 -0700 (PDT) Received: from localhost.bytedance.net ([139.177.225.231]) by smtp.gmail.com with ESMTPSA id z25sm5860239pfn.37.2021.03.19.09.38.42 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Mar 2021 09:38:46 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH v5 2/7] mm: memcontrol: introduce obj_cgroup_{un}charge_pages Date: Sat, 20 Mar 2021 00:38:15 +0800 Message-Id: <20210319163821.20704-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210319163821.20704-1-songmuchun@bytedance.com> References: <20210319163821.20704-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 501679001151 X-Stat-Signature: cnnujc4pjzbsh95zrqt76tss77eep46o Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf19; identity=mailfrom; envelope-from=""; helo=mail-pl1-f172.google.com; client-ip=209.85.214.172 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616171927-270385 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We know that the unit of slab object charging is bytes, the unit of kmem page charging is PAGE_SIZE. If we want to reuse obj_cgroup APIs to charge the kmem pages, we should pass PAGE_SIZE (as third parameter) to obj_cgroup_charge(). Because the size is already PAGE_SIZE, we can skip touch the objcg stock. And obj_cgroup_{un}charge_pages() are introduced to charge in units of page level. In the later patch, we also can reuse those two helpers to charge or uncharge a number of kernel pages to a object cgroup. This is just a code movement without any functional changes. Signed-off-by: Muchun Song Acked-by: Roman Gushchin Acked-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/memcontrol.c | 63 ++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 40 insertions(+), 23 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2cda76ff0629..9489c7fc1e04 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2905,6 +2905,20 @@ static void commit_charge(struct page *page, struct mem_cgroup *memcg) page->memcg_data = (unsigned long)memcg; } +static struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *objcg) +{ + struct mem_cgroup *memcg; + + rcu_read_lock(); +retry: + memcg = obj_cgroup_memcg(objcg); + if (unlikely(!css_tryget(&memcg->css))) + goto retry; + rcu_read_unlock(); + + return memcg; +} + #ifdef CONFIG_MEMCG_KMEM int memcg_alloc_page_obj_cgroups(struct page *page, struct kmem_cache *s, gfp_t gfp, bool new_page) @@ -3056,6 +3070,29 @@ static void memcg_free_cache_id(int id) ida_simple_remove(&memcg_cache_ida, id); } +static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, + unsigned int nr_pages) +{ + struct mem_cgroup *memcg; + + memcg = get_mem_cgroup_from_objcg(objcg); + __memcg_kmem_uncharge(memcg, nr_pages); + css_put(&memcg->css); +} + +static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, + unsigned int nr_pages) +{ + struct mem_cgroup *memcg; + int ret; + + memcg = get_mem_cgroup_from_objcg(objcg); + ret = __memcg_kmem_charge(memcg, gfp, nr_pages); + css_put(&memcg->css); + + return ret; +} + /** * __memcg_kmem_charge: charge a number of kernel pages to a memcg * @memcg: memory cgroup to charge @@ -3180,19 +3217,8 @@ static void drain_obj_stock(struct memcg_stock_pcp *stock) unsigned int nr_pages = stock->nr_bytes >> PAGE_SHIFT; unsigned int nr_bytes = stock->nr_bytes & (PAGE_SIZE - 1); - if (nr_pages) { - struct mem_cgroup *memcg; - - rcu_read_lock(); -retry: - memcg = obj_cgroup_memcg(old); - if (unlikely(!css_tryget(&memcg->css))) - goto retry; - rcu_read_unlock(); - - __memcg_kmem_uncharge(memcg, nr_pages); - css_put(&memcg->css); - } + if (nr_pages) + obj_cgroup_uncharge_pages(old, nr_pages); /* * The leftover is flushed to the centralized per-memcg value. @@ -3250,7 +3276,6 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size) { - struct mem_cgroup *memcg; unsigned int nr_pages, nr_bytes; int ret; @@ -3267,24 +3292,16 @@ int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size) * refill_obj_stock(), called from this function or * independently later. */ - rcu_read_lock(); -retry: - memcg = obj_cgroup_memcg(objcg); - if (unlikely(!css_tryget(&memcg->css))) - goto retry; - rcu_read_unlock(); - nr_pages = size >> PAGE_SHIFT; nr_bytes = size & (PAGE_SIZE - 1); if (nr_bytes) nr_pages += 1; - ret = __memcg_kmem_charge(memcg, gfp, nr_pages); + ret = obj_cgroup_charge_pages(objcg, gfp, nr_pages); if (!ret && nr_bytes) refill_obj_stock(objcg, PAGE_SIZE - nr_bytes); - css_put(&memcg->css); return ret; } From patchwork Fri Mar 19 16:38:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12151413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A3C4C433E0 for ; Fri, 19 Mar 2021 16:39:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B623161936 for ; Fri, 19 Mar 2021 16:39:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B623161936 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 592A08D0014; Fri, 19 Mar 2021 12:39:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 542226B008C; Fri, 19 Mar 2021 12:39:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3BC968D0014; Fri, 19 Mar 2021 12:39:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0059.hostedemail.com [216.40.44.59]) by kanga.kvack.org (Postfix) with ESMTP id 244A36B008A for ; Fri, 19 Mar 2021 12:39:15 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E25F6A768 for ; Fri, 19 Mar 2021 16:39:14 +0000 (UTC) X-FDA: 77937183828.03.047F302 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf11.hostedemail.com (Postfix) with ESMTP id DA16A2000390 for ; Fri, 19 Mar 2021 16:39:06 +0000 (UTC) Received: by mail-pg1-f177.google.com with SMTP id m3so4043633pga.1 for ; Fri, 19 Mar 2021 09:39:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EXGZ6vDG2a1eZeTzRSZB0jHuhMCuUpF3ItkaVaacSL8=; b=p7Y0nmmY5gp4ASuWh5hZmoWDSRrfcJZeWv6p/KSmtuKv7nZ21+azoq3I/s4Ij5Xx14 IRkgxHEf7kksT7v1/fZ8TEC4Nmx+tP6gmhhGb9lcv4Kxr/o/e8DsYhpLt5CMGMJterX9 ROFVTMT1IZtAkzrpW9saiD/E3ANHjzYINBid2I2x2URuIXZHvhJfOxCSqqMruh0COux8 Uy8ejr7QAHfIMoGRsGpumyeHP4ILai4ijNONHUwxwcZJsGEW/BLf0qIKe9NQ4u2B1Z+b s0HJ67rK77lXXUM/HwlX9pYTjWMvcCc2CguiDQ0Wi5IRwwu7d6h64ZQYzeYVxNPNC6BP oT0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EXGZ6vDG2a1eZeTzRSZB0jHuhMCuUpF3ItkaVaacSL8=; b=eNi0ov8HK1UY3s14q922XbnIs4KjBjplaDDInsXSZwRtgki8SdHQ9GMWl55bDcRaGP W29xorvrPYQ8kAwkLDCVWYqjN31acxy+SiGYsH2ChxuSTP2KzrO1PD4Cv1pc4j9LEMot XHZZvbmxH8bxnA0yI4y5YY/pVLtTGP6igYXlHLPYbahPsklVUrAVbVC7e9p2mcHkCJx0 jizboxQ1mGgqJN3MtWOMNF1AY3UPGqQuc93HKRSB32nlWS4SM7XJtmNa0PMTgVlVyqfB zzMv0YMX926gsIICmUM63UcszZcaBeXJTBZ4TJR1bZ05pTpbfqAyO+EJW3CiziQ6yfK7 tqfQ== X-Gm-Message-State: AOAM5334kz3fN7Xr9BIoh22/oCy9LgHc1IpGFt469B7umtCIc0FpfMTU UJbynhJc6fAxN8npMuYzrLEQsmApEK8n6Qlx X-Google-Smtp-Source: ABdhPJyhh2n5LgD32ubXUuyzP9bBzi7XNR7Nflj4NX51VQzHo76y7TCe9A8AahIe1urtB+xZFvKoxg== X-Received: by 2002:a63:4421:: with SMTP id r33mr12025068pga.247.1616171930332; Fri, 19 Mar 2021 09:38:50 -0700 (PDT) Received: from localhost.bytedance.net ([139.177.225.231]) by smtp.gmail.com with ESMTPSA id z25sm5860239pfn.37.2021.03.19.09.38.46 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Mar 2021 09:38:50 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH v5 3/7] mm: memcontrol: directly access page->memcg_data in mm/page_alloc.c Date: Sat, 20 Mar 2021 00:38:16 +0800 Message-Id: <20210319163821.20704-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210319163821.20704-1-songmuchun@bytedance.com> References: <20210319163821.20704-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: DA16A2000390 X-Stat-Signature: 979j9n79bajiiy7w9zr9yk59cqmymtgt Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf11; identity=mailfrom; envelope-from=""; helo=mail-pg1-f177.google.com; client-ip=209.85.215.177 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616171946-296295 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page_memcg() is not suitable for use by page_expected_state() and page_bad_reason(). Because it can BUG_ON() for the slab pages when CONFIG_DEBUG_VM is enabled. As neither lru, nor kmem, nor slab page should have anything left in there by the time the page is freed, what we care about is whether the value of page->memcg_data is 0. So just directly access page->memcg_data here. Signed-off-by: Muchun Song Acked-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/page_alloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f10966e3b4a5..e5454b85a106 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1124,7 +1124,7 @@ static inline bool page_expected_state(struct page *page, if (unlikely((unsigned long)page->mapping | page_ref_count(page) | #ifdef CONFIG_MEMCG - (unsigned long)page_memcg(page) | + page->memcg_data | #endif (page->flags & check_flags))) return false; @@ -1149,7 +1149,7 @@ static const char *page_bad_reason(struct page *page, unsigned long flags) bad_reason = "PAGE_FLAGS_CHECK_AT_FREE flag(s) set"; } #ifdef CONFIG_MEMCG - if (unlikely(page_memcg(page))) + if (unlikely(page->memcg_data)) bad_reason = "page still charged to cgroup"; #endif return bad_reason; From patchwork Fri Mar 19 16:38:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12151409 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3080C433C1 for ; Fri, 19 Mar 2021 16:39:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7DA6161934 for ; Fri, 19 Mar 2021 16:39:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7DA6161934 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1E9B38D0012; Fri, 19 Mar 2021 12:39:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C0546B008C; Fri, 19 Mar 2021 12:39:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 060DE8D0012; Fri, 19 Mar 2021 12:39:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0056.hostedemail.com [216.40.44.56]) by kanga.kvack.org (Postfix) with ESMTP id DD46F6B008A for ; Fri, 19 Mar 2021 12:39:09 -0400 (EDT) Received: from smtpin37.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 984613657 for ; Fri, 19 Mar 2021 16:39:09 +0000 (UTC) X-FDA: 77937183618.37.6A10DEA Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf13.hostedemail.com (Postfix) with ESMTP id 801D3E0003AD for ; Fri, 19 Mar 2021 16:39:00 +0000 (UTC) Received: by mail-pl1-f172.google.com with SMTP id k4so3214770plk.5 for ; Fri, 19 Mar 2021 09:38:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=j+pjxQiaXnPHaW0/aM9Ma4XbfsZ3VGpPRSn22gO/IJk=; b=hndwoIvTDp18phnZoOCAMilxvlnI5R4bA0e6iNMwiZ2EStHkNhhanShOAxQm/paJOL vrd258tkru/GarijoiTCdxlme4iCHlzw78VJjk6jR6gS8nQFoqbrbHcAY16uad1yuM4M Z7Cynrc7DTbGPAJ93R+Ee1nqRC89+tCS9In9PXNSX2XxHoD8v4I2VxZcrdxGwb/VLK6V e0OqJ087jjnsHeyWmUdjgEnRK+OAv+O7nGiG0gsoDo/HmZuzINONi8sptadFQnhycaeI huBd+7zO5WRDxnOHI+cNbxkvIv7NZSCUwhVxSqXR9nEH7q73J37fkqSpW2Ty+vFuYWUu 9WIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=j+pjxQiaXnPHaW0/aM9Ma4XbfsZ3VGpPRSn22gO/IJk=; b=EbOlXNJi1+yBqffNsaN5ZEiFrGeoI5jyiCD2vVrfVASZjJXaIxFJ5TiC4xBowsPeiY U0tgdRoj7fm4R5OoHqiNgJwt11Z2gOP6tFbeGS6yYbUw9+xak8Bt9vhqsNpS54OTsvbx NYuhBgWgu1pDWVkp7QE/h/ABQxBZJIebARerg7KutS7S+x/D/rcqekaE9IQCkDsR8WbU KvHbZaybDcqzKNz7Y1pcd9TCeY1uTkyI5fKaqjo58fiqloXTXYKbcLisJdTQ5b98fElL JquScCHNzqNYlkfaRZmIGNvz0R31p3GksBaBho+GULZsbGtiCZpNFMTTmgF5nxgE4/JI pYOQ== X-Gm-Message-State: AOAM531YObyl6ZpdzvFvK/R1nIG0HK8LNTSNRNs70tqpmBIX0SZpTfhY ZZuLCz+G1xqLc/BK1jfR1W2GOg== X-Google-Smtp-Source: ABdhPJxixIQ1BHBaesp4TbWicJ28kV8dAw86Zj6b62FphQwkOH+vVTVeZmrTYVNMmK7JAPjxFw1sNg== X-Received: by 2002:a17:902:c48c:b029:e4:c093:593a with SMTP id n12-20020a170902c48cb02900e4c093593amr14951679plx.1.1616171934202; Fri, 19 Mar 2021 09:38:54 -0700 (PDT) Received: from localhost.bytedance.net ([139.177.225.231]) by smtp.gmail.com with ESMTPSA id z25sm5860239pfn.37.2021.03.19.09.38.50 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Mar 2021 09:38:53 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH v5 4/7] mm: memcontrol: change ug->dummy_page only if memcg changed Date: Sat, 20 Mar 2021 00:38:17 +0800 Message-Id: <20210319163821.20704-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210319163821.20704-1-songmuchun@bytedance.com> References: <20210319163821.20704-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 801D3E0003AD X-Stat-Signature: eu6sx9ow6y1xfcmiekjdbbfzkkx3jfap Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf13; identity=mailfrom; envelope-from=""; helo=mail-pl1-f172.google.com; client-ip=209.85.214.172 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616171940-172407 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just like assignment to ug->memcg, we only need to update ug->dummy_page if memcg changed. So move it to there. This is a very small optimization. Signed-off-by: Muchun Song Acked-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/memcontrol.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9489c7fc1e04..8d28a5a2ee58 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6854,6 +6854,7 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) uncharge_gather_clear(ug); } ug->memcg = page_memcg(page); + ug->dummy_page = page; /* pairs with css_put in uncharge_batch */ css_get(&ug->memcg->css); @@ -6867,7 +6868,6 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) else ug->pgpgout++; - ug->dummy_page = page; page->memcg_data = 0; css_put(&ug->memcg->css); } From patchwork Fri Mar 19 16:38:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12151415 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDB53C433DB for ; Fri, 19 Mar 2021 16:39:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 70E246197D for ; Fri, 19 Mar 2021 16:39:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 70E246197D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 14C6E6B008A; Fri, 19 Mar 2021 12:39:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 124128D0016; Fri, 19 Mar 2021 12:39:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB9878D0015; Fri, 19 Mar 2021 12:39:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0162.hostedemail.com [216.40.44.162]) by kanga.kvack.org (Postfix) with ESMTP id CC6836B008A for ; Fri, 19 Mar 2021 12:39:22 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 85E3B8249980 for ; Fri, 19 Mar 2021 16:39:22 +0000 (UTC) X-FDA: 77937184164.10.CA76F37 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by imf22.hostedemail.com (Postfix) with ESMTP id 850A0C0007C1 for ; Fri, 19 Mar 2021 16:39:00 +0000 (UTC) Received: by mail-pg1-f170.google.com with SMTP id e33so4049828pgm.13 for ; Fri, 19 Mar 2021 09:39:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EmBuA/r+QurHtqpfxW42oGsEG8Apvr3KHhKly1xy83o=; b=aZQqcz7JVko6UPPEEBN3nrKpbT45dGZnBrUpOBP61X37n1d9BOG6o0ll5LNlhioyvT El2kFYrz2BIEghgNPJgTH3L0KEourUdLmxs9KX9yMCV5PWo5ymul2d3otYD+frUa/g+w 3zD3e8I9/Vgfvv9uabk4QIfh6trmKEL75AbACojXKKh3ICWUcNoRqOANEAx5x9F4FL9g cdrd0eBw+J8+Gn/4vP7D8Lq+fkJU8aDyN9q3lJP5GCITPrADy53rGAeCr63SsUbj7aht yuR1F1HZryIBHH2ChCqrweDuSJNOgg7CIdxfoa0ezuwyPO3B/Hlz8GdNKA6zYRNZRA85 f97w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EmBuA/r+QurHtqpfxW42oGsEG8Apvr3KHhKly1xy83o=; b=m0sOR7PrdR+nRUPWOwwlz0NF5MSIDelJ9BjvC/yYOyGJMJoGEko4D5tj6uv0En340v YFXD66QqoB483S8tuM/679XYkCB6O224Er76HHnuh5QJxE5IQSJ0XHiGA4HYycCYgQSy Vp1FVcC6GP8UPmrLue+2nSsGrRNX5KGttj44RuFdhQh2v1rne6UYma8+KAg68K7mTMii ih5EslMIMOC13Hem+15E+6CS03YnnicyRH6i+GNwyTVvMt4grtYUi274CjAohTCSIa3B gzxFYG9lGMRDuR5TGvIzlhWVSDwY9xhzi3fSaiZZVGx1tHmw+H+gMvQc+WOgSU/1pEUr dNyA== X-Gm-Message-State: AOAM533Q1k7/cyCT/j5vc3kTwvUCAyBMu2abK9459W4uOcFL7jowVOTm 5p4egU5E3yzmz6ojakypVCMiCQ== X-Google-Smtp-Source: ABdhPJwiLpxAfQjAZYRWBHTDmUFzm2ukdCITd6xH+YrO7TdxTEhp2chmlPPff0OLWeJn32rNjKW8Xw== X-Received: by 2002:a62:5e05:0:b029:20b:241e:4e18 with SMTP id s5-20020a625e050000b029020b241e4e18mr10016154pfb.1.1616171938301; Fri, 19 Mar 2021 09:38:58 -0700 (PDT) Received: from localhost.bytedance.net ([139.177.225.231]) by smtp.gmail.com with ESMTPSA id z25sm5860239pfn.37.2021.03.19.09.38.54 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Mar 2021 09:38:58 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH v5 5/7] mm: memcontrol: use obj_cgroup APIs to charge kmem pages Date: Sat, 20 Mar 2021 00:38:18 +0800 Message-Id: <20210319163821.20704-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210319163821.20704-1-songmuchun@bytedance.com> References: <20210319163821.20704-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 850A0C0007C1 X-Stat-Signature: 175euy7eyts3yb5147rmgcjj81inmp15 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf22; identity=mailfrom; envelope-from=""; helo=mail-pg1-f170.google.com; client-ip=209.85.215.170 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616171940-667440 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since Roman series "The new cgroup slab memory controller" applied. All slab objects are charged via the new APIs of obj_cgroup. The new APIs introduce a struct obj_cgroup to charge slab objects. It prevents long-living objects from pinning the original memory cgroup in the memory. But there are still some corner objects (e.g. allocations larger than order-1 page on SLUB) which are not charged via the new APIs. Those objects (include the pages which are allocated from buddy allocator directly) are charged as kmem pages which still hold a reference to the memory cgroup. We want to reuse the obj_cgroup APIs to charge the kmem pages. If we do that, we should store an object cgroup pointer to page->memcg_data for the kmem pages. Finally, page->memcg_data will have 3 different meanings. 1) For the slab pages, page->memcg_data points to an object cgroups vector. 2) For the kmem pages (exclude the slab pages), page->memcg_data points to an object cgroup. 3) For the user pages (e.g. the LRU pages), page->memcg_data points to a memory cgroup. We do not change the behavior of page_memcg() and page_memcg_rcu(). They are also suitable for LRU pages and kmem pages. Why? Because memory allocations pinning memcgs for a long time - it exists at a larger scale and is causing recurring problems in the real world: page cache doesn't get reclaimed for a long time, or is used by the second, third, fourth, ... instance of the same job that was restarted into a new cgroup every time. Unreclaimable dying cgroups pile up, waste memory, and make page reclaim very inefficient. We can convert LRU pages and most other raw memcg pins to the objcg direction to fix this problem, and then the page->memcg will always point to an object cgroup pointer. At that time, LRU pages and kmem pages will be treated the same. The implementation of page_memcg() will remove the kmem page check. This patch aims to charge the kmem pages by using the new APIs of obj_cgroup. Finally, the page->memcg_data of the kmem page points to an object cgroup. We can use the __page_objcg() to get the object cgroup associated with a kmem page. Or we can use page_memcg() to get the memory cgroup associated with a kmem page, but caller must ensure that the returned memcg won't be released (e.g. acquire the rcu_read_lock or css_set_lock). Signed-off-by: Muchun Song Acked-by: Johannes Weiner Reviewed-by: Shakeel Butt Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 116 +++++++++++++++++++++++++++++++++++---------- mm/memcontrol.c | 110 +++++++++++++++++++++--------------------- 2 files changed, 145 insertions(+), 81 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e6dc793d587d..395a113e4a3b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -358,6 +358,62 @@ enum page_memcg_data_flags { #define MEMCG_DATA_FLAGS_MASK (__NR_MEMCG_DATA_FLAGS - 1) +static inline bool PageMemcgKmem(struct page *page); + +/* + * After the initialization objcg->memcg is always pointing at + * a valid memcg, but can be atomically swapped to the parent memcg. + * + * The caller must ensure that the returned memcg won't be released: + * e.g. acquire the rcu_read_lock or css_set_lock. + */ +static inline struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *objcg) +{ + return READ_ONCE(objcg->memcg); +} + +/* + * __page_memcg - get the memory cgroup associated with a non-kmem page + * @page: a pointer to the page struct + * + * Returns a pointer to the memory cgroup associated with the page, + * or NULL. This function assumes that the page is known to have a + * proper memory cgroup pointer. It's not safe to call this function + * against some type of pages, e.g. slab pages or ex-slab pages or + * kmem pages. + */ +static inline struct mem_cgroup *__page_memcg(struct page *page) +{ + unsigned long memcg_data = page->memcg_data; + + VM_BUG_ON_PAGE(PageSlab(page), page); + VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_OBJCGS, page); + VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_KMEM, page); + + return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); +} + +/* + * __page_objcg - get the object cgroup associated with a kmem page + * @page: a pointer to the page struct + * + * Returns a pointer to the object cgroup associated with the page, + * or NULL. This function assumes that the page is known to have a + * proper object cgroup pointer. It's not safe to call this function + * against some type of pages, e.g. slab pages or ex-slab pages or + * LRU pages. + */ +static inline struct obj_cgroup *__page_objcg(struct page *page) +{ + unsigned long memcg_data = page->memcg_data; + + VM_BUG_ON_PAGE(PageSlab(page), page); + VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_OBJCGS, page); + VM_BUG_ON_PAGE(!(memcg_data & MEMCG_DATA_KMEM), page); + + return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); +} + /* * page_memcg - get the memory cgroup associated with a page * @page: a pointer to the page struct @@ -367,20 +423,23 @@ enum page_memcg_data_flags { * proper memory cgroup pointer. It's not safe to call this function * against some type of pages, e.g. slab pages or ex-slab pages. * - * Any of the following ensures page and memcg binding stability: + * For a non-kmem page any of the following ensures page and memcg binding + * stability: + * * - the page lock * - LRU isolation * - lock_page_memcg() * - exclusive reference + * + * For a kmem page a caller should hold an rcu read lock to protect memcg + * associated with a kmem page from being released. */ static inline struct mem_cgroup *page_memcg(struct page *page) { - unsigned long memcg_data = page->memcg_data; - - VM_BUG_ON_PAGE(PageSlab(page), page); - VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_OBJCGS, page); - - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + if (PageMemcgKmem(page)) + return obj_cgroup_memcg(__page_objcg(page)); + else + return __page_memcg(page); } /* @@ -394,11 +453,19 @@ static inline struct mem_cgroup *page_memcg(struct page *page) */ static inline struct mem_cgroup *page_memcg_rcu(struct page *page) { + unsigned long memcg_data = READ_ONCE(page->memcg_data); + VM_BUG_ON_PAGE(PageSlab(page), page); WARN_ON_ONCE(!rcu_read_lock_held()); - return (struct mem_cgroup *)(READ_ONCE(page->memcg_data) & - ~MEMCG_DATA_FLAGS_MASK); + if (memcg_data & MEMCG_DATA_KMEM) { + struct obj_cgroup *objcg; + + objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return obj_cgroup_memcg(objcg); + } + + return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } /* @@ -406,15 +473,21 @@ static inline struct mem_cgroup *page_memcg_rcu(struct page *page) * @page: a pointer to the page struct * * Returns a pointer to the memory cgroup associated with the page, - * or NULL. This function unlike page_memcg() can take any page + * or NULL. This function unlike page_memcg() can take any page * as an argument. It has to be used in cases when it's not known if a page - * has an associated memory cgroup pointer or an object cgroups vector. + * has an associated memory cgroup pointer or an object cgroups vector or + * an object cgroup. + * + * For a non-kmem page any of the following ensures page and memcg binding + * stability: * - * Any of the following ensures page and memcg binding stability: * - the page lock * - LRU isolation * - lock_page_memcg() * - exclusive reference + * + * For a kmem page a caller should hold an rcu read lock to protect memcg + * associated with a kmem page from being released. */ static inline struct mem_cgroup *page_memcg_check(struct page *page) { @@ -427,6 +500,13 @@ static inline struct mem_cgroup *page_memcg_check(struct page *page) if (memcg_data & MEMCG_DATA_OBJCGS) return NULL; + if (memcg_data & MEMCG_DATA_KMEM) { + struct obj_cgroup *objcg; + + objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return obj_cgroup_memcg(objcg); + } + return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } @@ -713,18 +793,6 @@ static inline void obj_cgroup_put(struct obj_cgroup *objcg) percpu_ref_put(&objcg->refcnt); } -/* - * After the initialization objcg->memcg is always pointing at - * a valid memcg, but can be atomically swapped to the parent memcg. - * - * The caller must ensure that the returned memcg won't be released: - * e.g. acquire the rcu_read_lock or css_set_lock. - */ -static inline struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *objcg) -{ - return READ_ONCE(objcg->memcg); -} - static inline void mem_cgroup_put(struct mem_cgroup *memcg) { if (memcg) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 8d28a5a2ee58..962499542531 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -855,18 +855,22 @@ void __mod_lruvec_page_state(struct page *page, enum node_stat_item idx, int val) { struct page *head = compound_head(page); /* rmap on tail pages */ - struct mem_cgroup *memcg = page_memcg(head); + struct mem_cgroup *memcg; pg_data_t *pgdat = page_pgdat(page); struct lruvec *lruvec; + rcu_read_lock(); + memcg = page_memcg(head); /* Untracked pages have no memcg, no lruvec. Update only the node */ if (!memcg) { + rcu_read_unlock(); __mod_node_page_state(pgdat, idx, val); return; } lruvec = mem_cgroup_lruvec(memcg, pgdat); __mod_lruvec_state(lruvec, idx, val); + rcu_read_unlock(); } EXPORT_SYMBOL(__mod_lruvec_page_state); @@ -1055,20 +1059,6 @@ static __always_inline struct mem_cgroup *active_memcg(void) return current->active_memcg; } -static __always_inline struct mem_cgroup *get_active_memcg(void) -{ - struct mem_cgroup *memcg; - - rcu_read_lock(); - memcg = active_memcg(); - /* remote memcg must hold a ref. */ - if (memcg && WARN_ON_ONCE(!css_tryget(&memcg->css))) - memcg = root_mem_cgroup; - rcu_read_unlock(); - - return memcg; -} - static __always_inline bool memcg_kmem_bypass(void) { /* Allow remote memcg charging from any context. */ @@ -1083,20 +1073,6 @@ static __always_inline bool memcg_kmem_bypass(void) } /** - * If active memcg is set, do not fallback to current->mm->memcg. - */ -static __always_inline struct mem_cgroup *get_mem_cgroup_from_current(void) -{ - if (memcg_kmem_bypass()) - return NULL; - - if (unlikely(active_memcg())) - return get_active_memcg(); - - return get_mem_cgroup_from_mm(current->mm); -} - -/** * mem_cgroup_iter - iterate over memory cgroup hierarchy * @root: hierarchy root * @prev: previously returned memcg, NULL on first invocation @@ -3152,18 +3128,18 @@ static void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_page */ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) { - struct mem_cgroup *memcg; + struct obj_cgroup *objcg; int ret = 0; - memcg = get_mem_cgroup_from_current(); - if (memcg && !mem_cgroup_is_root(memcg)) { - ret = __memcg_kmem_charge(memcg, gfp, 1 << order); + objcg = get_obj_cgroup_from_current(); + if (objcg) { + ret = obj_cgroup_charge_pages(objcg, gfp, 1 << order); if (!ret) { - page->memcg_data = (unsigned long)memcg | + page->memcg_data = (unsigned long)objcg | MEMCG_DATA_KMEM; return 0; } - css_put(&memcg->css); + obj_cgroup_put(objcg); } return ret; } @@ -3175,16 +3151,16 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) */ void __memcg_kmem_uncharge_page(struct page *page, int order) { - struct mem_cgroup *memcg = page_memcg(page); + struct obj_cgroup *objcg; unsigned int nr_pages = 1 << order; - if (!memcg) + if (!PageMemcgKmem(page)) return; - VM_BUG_ON_PAGE(mem_cgroup_is_root(memcg), page); - __memcg_kmem_uncharge(memcg, nr_pages); + objcg = __page_objcg(page); + obj_cgroup_uncharge_pages(objcg, nr_pages); page->memcg_data = 0; - css_put(&memcg->css); + obj_cgroup_put(objcg); } static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) @@ -6799,7 +6775,7 @@ int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask) struct uncharge_gather { struct mem_cgroup *memcg; - unsigned long nr_pages; + unsigned long nr_memory; unsigned long pgpgout; unsigned long nr_kmem; struct page *dummy_page; @@ -6814,10 +6790,10 @@ static void uncharge_batch(const struct uncharge_gather *ug) { unsigned long flags; - if (!mem_cgroup_is_root(ug->memcg)) { - page_counter_uncharge(&ug->memcg->memory, ug->nr_pages); + if (ug->nr_memory) { + page_counter_uncharge(&ug->memcg->memory, ug->nr_memory); if (do_memsw_account()) - page_counter_uncharge(&ug->memcg->memsw, ug->nr_pages); + page_counter_uncharge(&ug->memcg->memsw, ug->nr_memory); if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && ug->nr_kmem) page_counter_uncharge(&ug->memcg->kmem, ug->nr_kmem); memcg_oom_recover(ug->memcg); @@ -6825,7 +6801,7 @@ static void uncharge_batch(const struct uncharge_gather *ug) local_irq_save(flags); __count_memcg_events(ug->memcg, PGPGOUT, ug->pgpgout); - __this_cpu_add(ug->memcg->vmstats_percpu->nr_page_events, ug->nr_pages); + __this_cpu_add(ug->memcg->vmstats_percpu->nr_page_events, ug->nr_memory); memcg_check_events(ug->memcg, ug->dummy_page); local_irq_restore(flags); @@ -6836,40 +6812,60 @@ static void uncharge_batch(const struct uncharge_gather *ug) static void uncharge_page(struct page *page, struct uncharge_gather *ug) { unsigned long nr_pages; + struct mem_cgroup *memcg; + struct obj_cgroup *objcg; VM_BUG_ON_PAGE(PageLRU(page), page); - if (!page_memcg(page)) - return; - /* * Nobody should be changing or seriously looking at - * page_memcg(page) at this point, we have fully + * page memcg or objcg at this point, we have fully * exclusive access to the page. */ + if (PageMemcgKmem(page)) { + objcg = __page_objcg(page); + /* + * This get matches the put at the end of the function and + * kmem pages do not hold memcg references anymore. + */ + memcg = get_mem_cgroup_from_objcg(objcg); + } else { + memcg = __page_memcg(page); + } - if (ug->memcg != page_memcg(page)) { + if (!memcg) + return; + + if (ug->memcg != memcg) { if (ug->memcg) { uncharge_batch(ug); uncharge_gather_clear(ug); } - ug->memcg = page_memcg(page); + ug->memcg = memcg; ug->dummy_page = page; /* pairs with css_put in uncharge_batch */ - css_get(&ug->memcg->css); + css_get(&memcg->css); } nr_pages = compound_nr(page); - ug->nr_pages += nr_pages; - if (PageMemcgKmem(page)) + if (PageMemcgKmem(page)) { + ug->nr_memory += nr_pages; ug->nr_kmem += nr_pages; - else + + page->memcg_data = 0; + obj_cgroup_put(objcg); + } else { + /* LRU pages aren't accounted at the root level */ + if (!mem_cgroup_is_root(memcg)) + ug->nr_memory += nr_pages; ug->pgpgout++; - page->memcg_data = 0; - css_put(&ug->memcg->css); + page->memcg_data = 0; + } + + css_put(&memcg->css); } /** From patchwork Fri Mar 19 16:38:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12151411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 162B8C433C1 for ; Fri, 19 Mar 2021 16:39:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B030B61934 for ; Fri, 19 Mar 2021 16:39:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B030B61934 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 51DC68D0013; Fri, 19 Mar 2021 12:39:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F4A46B008C; Fri, 19 Mar 2021 12:39:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 399138D0013; Fri, 19 Mar 2021 12:39:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0179.hostedemail.com [216.40.44.179]) by kanga.kvack.org (Postfix) with ESMTP id 1C66F6B008A for ; Fri, 19 Mar 2021 12:39:13 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id CD822988D for ; Fri, 19 Mar 2021 16:39:12 +0000 (UTC) X-FDA: 77937183744.25.5215F6E Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf27.hostedemail.com (Postfix) with ESMTP id A94938019AF6 for ; Fri, 19 Mar 2021 16:39:03 +0000 (UTC) Received: by mail-pj1-f43.google.com with SMTP id s21so4952302pjq.1 for ; Fri, 19 Mar 2021 09:39:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lvoDDerEyyb4UPjtZfh5TD8N4MiB2jCVDZj0evdETsk=; b=fQAZGlvsU3OmzhfCCu/HWUmLUqVlSmLNPymtXoqiH7pS5/I0Kze/+9HQ83upaFWyGx 27vF1mZmZ0JHLI8Z8wq6ozzzU8AqboHGFbaj6Czj3czD/Ua7ReM1HfpWi/ebB0rkQZJL QN3YN+xGwz+97sGV5s/Q+Wa1T2XoZP1VGv00rN5XpMUWNMkACte4/mM3cAEhUqPG9A+g AhZHHDV7js9Fx2Y1DGTvB4kSZVDdPlKTrI3wfg/n9WForsmh7tqgzJ4t0qL8Awh/oHQE cbES+FQudokbFgW0+B2pkr3ARIYPrCMFjr16euT252iBOmH1BRmJPtoRYrCBMkRwwsFF v+8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lvoDDerEyyb4UPjtZfh5TD8N4MiB2jCVDZj0evdETsk=; b=oCxYmGvHpfZi91mdfKyjfEIadNQmGgUu30i5cdBBRGIFwkDjQ1yPk8STCqy73sIfjk kvwF2jgorCN26oB6I6HhvV4ovf6N4ZJW8BGVfrkghkqgtkyllrX0iNaNyT0HMCVYT8KS 9soVXb4v8xk3PeGp+2qTDC0sYGsBRmpKKxhsg6h6e0J73id749OJCW6l+9Dhvw0Z6cPL n4IKNy0N/vlYPZ7c4zEnoXA5vByU1/h2R0GujnSe/5ACdSr8UWg0KGIQcq/xIeohb9nT 9aBV5cSDP5fdgHKf/ocMlps74oq/+v/GkeuU3ttPWPQCWxNJ0dXy7qf/jOausP6l8/jV EgPg== X-Gm-Message-State: AOAM532mK6clOrqoQX8qEmO2SfUdiOrSq83Z96r5n31i7dxQQswibfoW /losvQbN2XblibGcgeC3gXkFsQ== X-Google-Smtp-Source: ABdhPJzTX5iyroetnM6aj9hw9+8hw65hqaf8cBYjqAVBt6aCABn+31HsHiv/OzYOJZnwh6rZk4GGaw== X-Received: by 2002:a17:90b:ece:: with SMTP id gz14mr10719879pjb.192.1616171942380; Fri, 19 Mar 2021 09:39:02 -0700 (PDT) Received: from localhost.bytedance.net ([139.177.225.231]) by smtp.gmail.com with ESMTPSA id z25sm5860239pfn.37.2021.03.19.09.38.58 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Mar 2021 09:39:02 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH v5 6/7] mm: memcontrol: inline __memcg_kmem_{un}charge() into obj_cgroup_{un}charge_pages() Date: Sat, 20 Mar 2021 00:38:19 +0800 Message-Id: <20210319163821.20704-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210319163821.20704-1-songmuchun@bytedance.com> References: <20210319163821.20704-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A94938019AF6 X-Stat-Signature: mscqkrzrpxuhpj51k3ep4o7jy9xzczjg Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf27; identity=mailfrom; envelope-from=""; helo=mail-pj1-f43.google.com; client-ip=209.85.216.43 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616171943-219620 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is only one user of __memcg_kmem_charge(), so manually inline __memcg_kmem_charge() to obj_cgroup_charge_pages(). Similarly manually inline __memcg_kmem_uncharge() into obj_cgroup_uncharge_pages() and call obj_cgroup_uncharge_pages() in obj_cgroup_release(). This is just code cleanup without any functionality changes. Signed-off-by: Muchun Song Reviewed-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Roman Gushchin --- mm/memcontrol.c | 63 +++++++++++++++++++++++---------------------------------- 1 file changed, 25 insertions(+), 38 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 962499542531..249bf6b4d94c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -255,10 +255,8 @@ struct cgroup_subsys_state *vmpressure_to_css(struct vmpressure *vmpr) #ifdef CONFIG_MEMCG_KMEM extern spinlock_t css_set_lock; -static int __memcg_kmem_charge(struct mem_cgroup *memcg, gfp_t gfp, - unsigned int nr_pages); -static void __memcg_kmem_uncharge(struct mem_cgroup *memcg, - unsigned int nr_pages); +static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, + unsigned int nr_pages); static void obj_cgroup_release(struct percpu_ref *ref) { @@ -295,7 +293,7 @@ static void obj_cgroup_release(struct percpu_ref *ref) spin_lock_irqsave(&css_set_lock, flags); memcg = obj_cgroup_memcg(objcg); if (nr_pages) - __memcg_kmem_uncharge(memcg, nr_pages); + obj_cgroup_uncharge_pages(objcg, nr_pages); list_del(&objcg->list); mem_cgroup_put(memcg); spin_unlock_irqrestore(&css_set_lock, flags); @@ -3046,46 +3044,45 @@ static void memcg_free_cache_id(int id) ida_simple_remove(&memcg_cache_ida, id); } +/* + * obj_cgroup_uncharge_pages: uncharge a number of kernel pages from a objcg + * @objcg: object cgroup to uncharge + * @nr_pages: number of pages to uncharge + */ static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, unsigned int nr_pages) { struct mem_cgroup *memcg; memcg = get_mem_cgroup_from_objcg(objcg); - __memcg_kmem_uncharge(memcg, nr_pages); - css_put(&memcg->css); -} -static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, - unsigned int nr_pages) -{ - struct mem_cgroup *memcg; - int ret; + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + page_counter_uncharge(&memcg->kmem, nr_pages); + refill_stock(memcg, nr_pages); - memcg = get_mem_cgroup_from_objcg(objcg); - ret = __memcg_kmem_charge(memcg, gfp, nr_pages); css_put(&memcg->css); - - return ret; } -/** - * __memcg_kmem_charge: charge a number of kernel pages to a memcg - * @memcg: memory cgroup to charge +/* + * obj_cgroup_charge_pages: charge a number of kernel pages to a objcg + * @objcg: object cgroup to charge * @gfp: reclaim mode * @nr_pages: number of pages to charge * * Returns 0 on success, an error code on failure. */ -static int __memcg_kmem_charge(struct mem_cgroup *memcg, gfp_t gfp, - unsigned int nr_pages) +static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, + unsigned int nr_pages) { struct page_counter *counter; + struct mem_cgroup *memcg; int ret; + memcg = get_mem_cgroup_from_objcg(objcg); + ret = try_charge(memcg, gfp, nr_pages); if (ret) - return ret; + goto out; if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && !page_counter_try_charge(&memcg->kmem, nr_pages, &counter)) { @@ -3097,25 +3094,15 @@ static int __memcg_kmem_charge(struct mem_cgroup *memcg, gfp_t gfp, */ if (gfp & __GFP_NOFAIL) { page_counter_charge(&memcg->kmem, nr_pages); - return 0; + goto out; } cancel_charge(memcg, nr_pages); - return -ENOMEM; + ret = -ENOMEM; } - return 0; -} - -/** - * __memcg_kmem_uncharge: uncharge a number of kernel pages from a memcg - * @memcg: memcg to uncharge - * @nr_pages: number of pages to uncharge - */ -static void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages) -{ - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) - page_counter_uncharge(&memcg->kmem, nr_pages); +out: + css_put(&memcg->css); - refill_stock(memcg, nr_pages); + return ret; } /** From patchwork Fri Mar 19 16:38:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12151417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3606C433E0 for ; Fri, 19 Mar 2021 16:39:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4B61F61934 for ; Fri, 19 Mar 2021 16:39:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4B61F61934 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E44C48D0015; Fri, 19 Mar 2021 12:39:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E1C5E6B0092; Fri, 19 Mar 2021 12:39:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE30D8D0015; Fri, 19 Mar 2021 12:39:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0180.hostedemail.com [216.40.44.180]) by kanga.kvack.org (Postfix) with ESMTP id B4D0C6B008C for ; Fri, 19 Mar 2021 12:39:26 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 67257180284FA for ; Fri, 19 Mar 2021 16:39:26 +0000 (UTC) X-FDA: 77937184332.24.C558344 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf26.hostedemail.com (Postfix) with ESMTP id D22E64000F50 for ; Fri, 19 Mar 2021 16:39:17 +0000 (UTC) Received: by mail-pl1-f170.google.com with SMTP id g1so3213875plg.7 for ; Fri, 19 Mar 2021 09:39:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2a1Q2yqghil2SSuBrZCKn/WKXtEY0qZpSKwUUGtGYmY=; b=ts76DRlSUdCqKznHIFlxmkaJlHr7i6lQtNzoO/RMbIkr50NiOPe8TQWRN2ixR3V33V MSo2854BMApkQb62U5JW9wkPsfPYtTGC2w0eNR2VJcDOkmI0OCxK2tskqXLpMsQo5k6D I+9MF+0de3Cbo3B/emGkSQTZ9b8kytzbLLh/j7hBjH3RSTOab9/HOsEY5k0TVE2bTxrT HDUz3y6thtT6ym4M39vfRuGNeMWpbqVXC+l9xBR5ds5TgAFyMxiHBqq/RqrWxutVnF6b 1EMyHafpSTsTHBvOaC6bx0muSqlYD9s4rvl0E+dFySVF/yoosEDrBi5U7q6ke41gt9lS 09FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2a1Q2yqghil2SSuBrZCKn/WKXtEY0qZpSKwUUGtGYmY=; b=eIB/00ckOC8YlidhBpXupu/SqmpCtcwkI2NhepLJtiHQa10UCarHTp+6Lhnrvgy/7u vgVTCaYjm+blzd1qohXoNwCKp30ywuconAXdZJvUS0lkUafRPSDWMDe3E9EbAkIw2JQ3 BCKT/8MZjHhXmpIeoGi3C/UqumlxsuQuR+lXN0MdHdDZu+QRYNzrVFKuu8QS953Uon+q WTs7TreSkMmkv0LHEVxTVkdVpwogZ7baXVkeEpk3Tj2QLj6TKozCyV1cAFhBf5YP1N5F XtDNN8QxMkcyOjd7P5lviSDLZD1eMHQMr515NKnKkIt7QgacP7gUNLrEa2KZBbzJh/QB GpUg== X-Gm-Message-State: AOAM530UKjYOHUziU9OYw2mK58tVKLzgGbo32AItl6U5mdfaNSjXPXwp 1kFglkcPeSgutSiH52EO2iU+gA== X-Google-Smtp-Source: ABdhPJxLMtY1/YANoUnuBedOZOnF4FLQqQRkWkg4E+mDOLjpNusqnH6PFcwrpTUh62KkTCQDt9cozA== X-Received: by 2002:a17:902:7682:b029:e6:2bc5:f005 with SMTP id m2-20020a1709027682b02900e62bc5f005mr14814810pll.32.1616171946396; Fri, 19 Mar 2021 09:39:06 -0700 (PDT) Received: from localhost.bytedance.net ([139.177.225.231]) by smtp.gmail.com with ESMTPSA id z25sm5860239pfn.37.2021.03.19.09.39.02 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Mar 2021 09:39:06 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH v5 7/7] mm: memcontrol: move PageMemcgKmem to the scope of CONFIG_MEMCG_KMEM Date: Sat, 20 Mar 2021 00:38:20 +0800 Message-Id: <20210319163821.20704-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210319163821.20704-1-songmuchun@bytedance.com> References: <20210319163821.20704-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: qrorkx37e6zqij4m7uur8ru1okqrb1ge X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: D22E64000F50 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=mail-pl1-f170.google.com; client-ip=209.85.214.170 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616171957-485663 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page only can be marked as kmem when CONFIG_MEMCG_KMEM is enabled. So move PageMemcgKmem() to the scope of the CONFIG_MEMCG_KMEM. As a bonus, on !CONFIG_MEMCG_KMEM build some code can be compiled out. Signed-off-by: Muchun Song Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 395a113e4a3b..7fdc92e1983e 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -510,6 +510,7 @@ static inline struct mem_cgroup *page_memcg_check(struct page *page) return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } +#ifdef CONFIG_MEMCG_KMEM /* * PageMemcgKmem - check if the page has MemcgKmem flag set * @page: a pointer to the page struct @@ -524,7 +525,6 @@ static inline bool PageMemcgKmem(struct page *page) return page->memcg_data & MEMCG_DATA_KMEM; } -#ifdef CONFIG_MEMCG_KMEM /* * page_objcgs - get the object cgroups vector associated with a page * @page: a pointer to the page struct @@ -566,6 +566,11 @@ static inline struct obj_cgroup **page_objcgs_check(struct page *page) } #else +static inline bool PageMemcgKmem(struct page *page) +{ + return false; +} + static inline struct obj_cgroup **page_objcgs(struct page *page) { return NULL;