From patchwork Wed Apr 28 09:49:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12228417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDC3BC433B4 for ; Wed, 28 Apr 2021 09:54:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A900661422 for ; Wed, 28 Apr 2021 09:54:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238453AbhD1Jyu (ORCPT ); Wed, 28 Apr 2021 05:54:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238070AbhD1Jyt (ORCPT ); Wed, 28 Apr 2021 05:54:49 -0400 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C778C061574 for ; Wed, 28 Apr 2021 02:54:05 -0700 (PDT) Received: by mail-pl1-x632.google.com with SMTP id h7so3951843plt.1 for ; Wed, 28 Apr 2021 02:54:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9oUrOqpAChDXaXDoRREdcI5L9MHP6oIlBRQpnJzUeII=; b=KjmVrYt4/mY4WAr2kGNuBttDXe558+QJuBQ0FarvEbVpYaJKm1byHI0esEdym/Sbms ro0NbPCCKfv5HOUetWMK0UzUV/wBvWj1pdq6vDvbMZJSDoWUsAEaM5qnn0Zn94mGyu76 CIdM0kJHKdvY2YznYTJNRCQjsDJ89lLa/6V9en+m4TrMBSXszIAsfpheCjK7mXscPLii f4WMz1stInaW9MZwHqi5sRS8/FdZ3f75T59LuLxAzP9za4ip+85ii2835JA9zLFnKkwg /7LBEqHUDEYbsKjDOXcN00GLzP6ZQcqmZwgmfvE5wno6F5pVKaC2ocCOaWtiQmLDvMik WWRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9oUrOqpAChDXaXDoRREdcI5L9MHP6oIlBRQpnJzUeII=; b=Gv7zf70dg1rMoxcTyzvSd4xbUSpUvmCOpGVuWz1sTYeupyfG9J1n7T6cKoLQnJSDEX tj714McJmsItMuUD74FDGktOz5DHBMnaAFUiBkwVvn+T87mGSAlXxJqSuhVhpAxxt+PV 4AbWxuq8jkkRoylHGoDHmy3cOCR6j5DxGYLHrx6/vB9ZQn93BUHjhmmrtDQXmLFpz09J 7Ub/Fu/9KWvBydwOpIpuhvjjwsH67oZANbp/ZEAv9MkMV8mEnMbw6DUvT30/MwcFTtR0 riVZEwF79//L35rmjsDkKgxEIsU7Vv4K6Az5jMuGeN5son6bIzHekMd4ZTokBwdskSQG i81Q== X-Gm-Message-State: AOAM532RSuUM1t2jQBfn7QTc7KR1y+ACd1k/8VlNgB9fm6IuuR3pMsSx yoGsNluhQhR+wwGv6sDk7Xs27w== X-Google-Smtp-Source: ABdhPJxBl/GM8wieywRB0WXvNHzOvK1MwlC3y0XOZRlQNnVIeOPVORj003QMYJqVZivfXIWTUHMcqg== X-Received: by 2002:a17:902:7d89:b029:ec:c084:d4bc with SMTP id a9-20020a1709027d89b02900ecc084d4bcmr29571045plm.18.1619603645050; Wed, 28 Apr 2021 02:54:05 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.233]) by smtp.gmail.com with ESMTPSA id x77sm4902365pfc.19.2021.04.28.02.54.00 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Apr 2021 02:54:04 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, alexander.h.duyck@linux.intel.com, richard.weiyang@gmail.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH 1/9] mm: list_lru: fix list_lru_count_one() return value Date: Wed, 28 Apr 2021 17:49:41 +0800 Message-Id: <20210428094949.43579-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210428094949.43579-1-songmuchun@bytedance.com> References: <20210428094949.43579-1-songmuchun@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Since commit 2788cf0c401c ("memcg: reparent list_lrus and free kmemcg_id on css offline"), the ->nr_items can be negative during memory cgroup reparenting. In this case, list_lru_count_one() can returns an unusually large value. In order to not surprise the user. So return zero when ->nr_items is negative. Signed-off-by: Muchun Song --- mm/list_lru.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index cd58790d0fb3..4962d48d4410 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -176,13 +176,16 @@ unsigned long list_lru_count_one(struct list_lru *lru, { struct list_lru_node *nlru = &lru->node[nid]; struct list_lru_one *l; - unsigned long count; + long count; rcu_read_lock(); l = list_lru_from_memcg_idx(nlru, memcg_cache_id(memcg)); count = READ_ONCE(l->nr_items); rcu_read_unlock(); + if (unlikely(count < 0)) + count = 0; + return count; } EXPORT_SYMBOL_GPL(list_lru_count_one); From patchwork Wed Apr 28 09:49:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12228419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77D50C433ED for ; Wed, 28 Apr 2021 09:54:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4102261422 for ; Wed, 28 Apr 2021 09:54:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238468AbhD1Jy5 (ORCPT ); Wed, 28 Apr 2021 05:54:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238456AbhD1Jyz (ORCPT ); Wed, 28 Apr 2021 05:54:55 -0400 Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B7530C06138B for ; Wed, 28 Apr 2021 02:54:10 -0700 (PDT) Received: by mail-pf1-x432.google.com with SMTP id a12so1056924pfc.7 for ; Wed, 28 Apr 2021 02:54:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=m1CFbU+mCktpHQ1dKqvusnGfjIT06imnu2MiKnHY6nk=; b=v4VjCZlNqoTtno1oSp5L516Q2Oq7puICK/nvOGS2G21b+Zf8dDbqpbPxbXBFaxZyPX +9XX3GF59psC7dYErKQ1Wd1WNsQ+vI7dYeuNft1Ozh+O1wGMo22QQASc86eHBg+p/taX 0nt80fb8lIzpAY70hZqGDF1imMiyDa8M/NsPH3xct7MNOLYD0cR3yy9FOLpKSE51hOFh hoyOIPv/848DsYPW8PmuRPui51Ovt41Nq0xw0czJphN9QGOR5l8mxqA0xVizRBGv80Dm V7n+ezDBiebnuPwdwgVwZO118XmqHJVIwU+D7QP07FQozcA5XNWoYM4DmScmZFDqI3Fp cdGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=m1CFbU+mCktpHQ1dKqvusnGfjIT06imnu2MiKnHY6nk=; b=dVvo2TVZUmGbL3OIJJRnyfiH5PYvdHd/eRlM+FbrAlI1DoYY2d8qCz07vsA7sqLhja pB/PZc+xrB3dJpEtYjnfgmk8STf00yusKKy22MRoclom/1zpJqPMMwuvWcd696sKGVH1 75vXNOWqzjDcJLws0sA8nKvOOuU7uejLYcGwRda0vZFj9bdOI0Sw6DuAsemjlNmVYEdi ZbkJgEBhhk26JyAu3yokSnwJrsPMYNwaYivzqCvo77YqmOJdMJAkA3TI2jq7rbffrRGV VhRcx4lrSaSoYgEUDidVhMMS4v2wAvPpPWh3Tr4LvzBDQ3Kfxbqv/QymmrDx2ALopeTg dGFA== X-Gm-Message-State: AOAM532e5pPgA+H39eX0pQa2D9IHBlCcMMQwAhorWfbyYTaBbzgtbyjW DybyFrYudF6fIYeW6iA5x63bLQ== X-Google-Smtp-Source: ABdhPJw6KaQEv1gmE2Ru2d5aoT7DK6/dT4q9Q9aoINopiR0PMuD4/fmPAfJYaLngGk2J5Pm10PwxBA== X-Received: by 2002:a62:7646:0:b029:27a:29b6:e10b with SMTP id r67-20020a6276460000b029027a29b6e10bmr8361616pfc.14.1619603650322; Wed, 28 Apr 2021 02:54:10 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.233]) by smtp.gmail.com with ESMTPSA id x77sm4902365pfc.19.2021.04.28.02.54.05 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Apr 2021 02:54:10 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, alexander.h.duyck@linux.intel.com, richard.weiyang@gmail.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH 2/9] mm: memcontrol: remove kmemcg_id reparenting Date: Wed, 28 Apr 2021 17:49:42 +0800 Message-Id: <20210428094949.43579-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210428094949.43579-1-songmuchun@bytedance.com> References: <20210428094949.43579-1-songmuchun@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Since slab objects and kmem pages are charged to object cgroup instead of memory cgroup, memcg_reparent_objcgs() will reparent this cgroup and all its descendants to the parent cgroup. It means that the new parent memory cgroup can be returned by mem_cgroup_from_obj() which is called from list_lru_from_kmem(). This can make further list_lru_add()'s add elements to the parent's list. So we do not need to change kmemcg_id of an offline cgroup to its parent's id. This is just waste CPU cycles. Just remove those redundant code. Signed-off-by: Muchun Song --- mm/memcontrol.c | 20 ++------------------ 1 file changed, 2 insertions(+), 18 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 64ada9e650a5..21e12312509c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3485,8 +3485,7 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) static void memcg_offline_kmem(struct mem_cgroup *memcg) { - struct cgroup_subsys_state *css; - struct mem_cgroup *parent, *child; + struct mem_cgroup *parent; int kmemcg_id; if (memcg->kmem_state != KMEM_ONLINE) @@ -3503,22 +3502,7 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) kmemcg_id = memcg->kmemcg_id; BUG_ON(kmemcg_id < 0); - /* - * Change kmemcg_id of this cgroup and all its descendants to the - * parent's id, and then move all entries from this cgroup's list_lrus - * to ones of the parent. After we have finished, all list_lrus - * corresponding to this cgroup are guaranteed to remain empty. The - * ordering is imposed by list_lru_node->lock taken by - * memcg_drain_all_list_lrus(). - */ - rcu_read_lock(); /* can be called from css_free w/o cgroup_mutex */ - css_for_each_descendant_pre(css, &memcg->css) { - child = mem_cgroup_from_css(css); - BUG_ON(child->kmemcg_id != kmemcg_id); - child->kmemcg_id = parent->kmemcg_id; - } - rcu_read_unlock(); - + /* memcg_reparent_objcgs() must be called before this. */ memcg_drain_all_list_lrus(kmemcg_id, parent); memcg_free_cache_id(kmemcg_id); From patchwork Wed Apr 28 09:49:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12228421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 659AEC433B4 for ; Wed, 28 Apr 2021 09:54:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 28C5B613E7 for ; Wed, 28 Apr 2021 09:54:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238500AbhD1JzC (ORCPT ); Wed, 28 Apr 2021 05:55:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238462AbhD1JzB (ORCPT ); Wed, 28 Apr 2021 05:55:01 -0400 Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 202EFC061574 for ; Wed, 28 Apr 2021 02:54:16 -0700 (PDT) Received: by mail-pf1-x432.google.com with SMTP id m11so197022pfc.11 for ; Wed, 28 Apr 2021 02:54:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DbfZmKXxQSOn7wHkCejhev2qWZcfPUS5M9gRiT8vHvs=; b=BqOFNyjA/Es8BddpzZ+9QxaIwEeZvSIuQO419b/fBSreqOqcUtciS6XnVplZDrZlG/ Imy6gNobcHN9VQvDMV/23CyMkdByJAqNWYXOjnXrqLQnlBCL3Q5MGNlt5tUXlbGe2fsh EzWGLAY4HuEZi8OeUdvhfwXr+AJyKi8UUff6lbxmCS5S1oHkh6xPZtngzOHzBOlXvwa7 mXj6jgUf+OY2OCpdvZPqdFr/f0e9CYhODs0mfdwqr9OPaDCGojvLQSHtoTaEfwj0nl7d rN6sbJqX27CKHTBysTUwLvfX/Ky7+llOZ3baxVl62+P2oov3Ev4KJJ6+DO2fgHGZetZS JHzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DbfZmKXxQSOn7wHkCejhev2qWZcfPUS5M9gRiT8vHvs=; b=LoFq4AFKRd5+72qHAEnuevaFhrF11CSHmaZyCS1fLzCSrhKf/Hijiou1PrcCfT/9YB fu1kZX4pQMilJKJAv4hyFk3z2pexcqse7EJOInJ7DeS+0cOeaOawGVYdULZMbueIPHMO MK6XjdeQv7mvGkPsCapJ64cpAS6t49IYfEMKcqTi2DVrBekJpGiwDmPCZU051QcQ9zLu qcPa1CvwnLxacmYbloTnRej2D8KcufqlpabHJPzRoA26GlfRQ0+6y4KHQlkikIMnpnOd r/efYXgtRjek4mvo5v/jMEZm0PGtKWwRSO2kW1SULIaWUzjxgWAYmqIbPcSySBM0fPkg 0oxA== X-Gm-Message-State: AOAM530jfSDMerzjTJk3OlQ/gjOQ6YZrIg0QCvapy08yE5tV2gXyOYVj btJsX/gyInAIoatoKjxJxz/ZKw== X-Google-Smtp-Source: ABdhPJw1XENVld3cLK+I71antXFfgkPHZ+eowp5+rG7qg4jm1nTaFVssJ6CW0IB96l43Y9shWLLlEQ== X-Received: by 2002:aa7:9806:0:b029:253:e613:4ada with SMTP id e6-20020aa798060000b0290253e6134adamr27192427pfl.65.1619603655719; Wed, 28 Apr 2021 02:54:15 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.233]) by smtp.gmail.com with ESMTPSA id x77sm4902365pfc.19.2021.04.28.02.54.10 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Apr 2021 02:54:15 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, alexander.h.duyck@linux.intel.com, richard.weiyang@gmail.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH 3/9] mm: list_lru: rename memcg_drain_all_list_lrus to memcg_reparent_list_lrus Date: Wed, 28 Apr 2021 17:49:43 +0800 Message-Id: <20210428094949.43579-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210428094949.43579-1-songmuchun@bytedance.com> References: <20210428094949.43579-1-songmuchun@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Since we do not change memcg->kmemcg_id before calling memcg_drain_all_list_lrus(), so we do not need to take kmemcg_id as parameter. The two parameters of memcg_drain_all_list_lrus() seems odd, one is kmemcg_id, another is memcg. Now we can change the kmemcg_id to the memcg. It is more consistent. Since the purpose of the memcg_drain_all_list_lrus() is list_lru reparenting. So also rename it to memcg_reparent_list_lrus(). The name is also consistent with memcg_reparent_objcgs(). Signed-off-by: Muchun Song --- include/linux/list_lru.h | 2 +- mm/list_lru.c | 23 ++++++++++++----------- mm/memcontrol.c | 2 +- 3 files changed, 14 insertions(+), 13 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 9dcaa3e582c9..e8a5e3a2c0dd 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -70,7 +70,7 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, __list_lru_init((lru), true, NULL, shrinker) int memcg_update_all_list_lrus(int num_memcgs); -void memcg_drain_all_list_lrus(int src_idx, struct mem_cgroup *dst_memcg); +void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent); /** * list_lru_add: add an element to the lru list's tail diff --git a/mm/list_lru.c b/mm/list_lru.c index 4962d48d4410..d78dba5a6dab 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -523,11 +523,11 @@ int memcg_update_all_list_lrus(int new_size) goto out; } -static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, - int src_idx, struct mem_cgroup *dst_memcg) +static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, + struct mem_cgroup *memcg, + struct mem_cgroup *parent) { struct list_lru_node *nlru = &lru->node[nid]; - int dst_idx = dst_memcg->kmemcg_id; struct list_lru_one *src, *dst; /* @@ -536,22 +536,23 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, */ spin_lock_irq(&nlru->lock); - src = list_lru_from_memcg_idx(nlru, src_idx); - dst = list_lru_from_memcg_idx(nlru, dst_idx); + src = list_lru_from_memcg_idx(nlru, memcg->kmemcg_id); + dst = list_lru_from_memcg_idx(nlru, parent->kmemcg_id); list_splice_init(&src->list, &dst->list); if (src->nr_items) { dst->nr_items += src->nr_items; - set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); + set_shrinker_bit(parent, nid, lru_shrinker_id(lru)); src->nr_items = 0; } spin_unlock_irq(&nlru->lock); } -static void memcg_drain_list_lru(struct list_lru *lru, - int src_idx, struct mem_cgroup *dst_memcg) +static void memcg_reparent_list_lru(struct list_lru *lru, + struct mem_cgroup *memcg, + struct mem_cgroup *parent) { int i; @@ -559,16 +560,16 @@ static void memcg_drain_list_lru(struct list_lru *lru, return; for_each_node(i) - memcg_drain_list_lru_node(lru, i, src_idx, dst_memcg); + memcg_reparent_list_lru_node(lru, i, memcg, parent); } -void memcg_drain_all_list_lrus(int src_idx, struct mem_cgroup *dst_memcg) +void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent) { struct list_lru *lru; mutex_lock(&list_lrus_mutex); list_for_each_entry(lru, &list_lrus, list) - memcg_drain_list_lru(lru, src_idx, dst_memcg); + memcg_reparent_list_lru(lru, memcg, parent); mutex_unlock(&list_lrus_mutex); } #else diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 21e12312509c..c1ce4fdba028 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3503,7 +3503,7 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) BUG_ON(kmemcg_id < 0); /* memcg_reparent_objcgs() must be called before this. */ - memcg_drain_all_list_lrus(kmemcg_id, parent); + memcg_reparent_list_lrus(memcg, parent); memcg_free_cache_id(kmemcg_id); } From patchwork Wed Apr 28 09:49:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12228423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64662C433ED for ; Wed, 28 Apr 2021 09:54:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2DBBD61419 for ; Wed, 28 Apr 2021 09:54:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238533AbhD1JzN (ORCPT ); Wed, 28 Apr 2021 05:55:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57142 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238481AbhD1JzJ (ORCPT ); Wed, 28 Apr 2021 05:55:09 -0400 Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 787AAC06138A for ; Wed, 28 Apr 2021 02:54:21 -0700 (PDT) Received: by mail-pj1-x1035.google.com with SMTP id z6-20020a17090a1706b0290155e8a752d8so328346pjd.4 for ; Wed, 28 Apr 2021 02:54:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NP/rxZesEiIIrfKiG+UYq1kkvOJzyQmEnIKOjy9l8kg=; b=Svyqhok6H6Syu6MAMe9UCLt3sv86mUklBNHn2V++/4jxLrT2f5AFZv9tKBSqsHVgiP ZoCf3jsDk5APQTo72uUAk2ZNYE4xWEseg/ya9aj/kyQzqUV0P6jWd7Mxu/vdl2t1ZEy5 Qo1yA2mysgdJliLeZxMv2OtZsL83h8tp0G5w837IQwOgUKEbPn/1f/zs2TfaS2/YR0s8 AKF0JFo43aaX3cyIP+5m3KmlrByf3LlDwqDNQM/sjmmhbHjgaVHa+UAb6KnrpRUuyc4X lGNSqcaj1YgmDEAaerlHT9cm+BQvw4HtgjshaqMQQV/vAZFH8Vz2aw87Eo8Gcx4MW+pC 0avg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NP/rxZesEiIIrfKiG+UYq1kkvOJzyQmEnIKOjy9l8kg=; b=WP/LLIhHp51gSYdxg287AR/Ai22MNjSE26ID8RK+XPG55mbUsntzRfJtSY9+xUCF/D M09z72YagyBgQThglF97ndv2kxa0kpkikWBLVs6I11aC1sFTjSWGTOmyPOxrAxDa0Vgq l9J2JErqOz0Yc3y3nNQ7Q+wzh9IJbuD/ETY9qsLC6yEQhuY2BS/RrjHXvV5rfCjX9ptw CLwV+goPVKBL+Ag+pO3unNBwwY6hWd7JipveV5RWqHJPr5IY8Y7Mjhr/YS9+la7YPoZX 0A1b3IzEGhWWPHnMabP5z3ciCTbaz9xorf52zgq3PcbJ4JE0zrwlt8gcrdSxnrvAeUhJ krOg== X-Gm-Message-State: AOAM530Y9cbgw7+7g+9iA5WMadpHH2j0w2TElm+UOPTDJeQ/UPWZnO1Z Ms/ev6VSp/LlREj5/0ZhJ0hDKg== X-Google-Smtp-Source: ABdhPJy2UAfP3e/NNwuaLzNawPPbdd+obpHPXRqeQPBax1hGR6109fKOvajSwFFeB/oc84VrtkReyg== X-Received: by 2002:a17:90a:c717:: with SMTP id o23mr25749439pjt.214.1619603661139; Wed, 28 Apr 2021 02:54:21 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.233]) by smtp.gmail.com with ESMTPSA id x77sm4902365pfc.19.2021.04.28.02.54.16 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Apr 2021 02:54:20 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, alexander.h.duyck@linux.intel.com, richard.weiyang@gmail.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH 4/9] mm: memcontrol: remove the kmem states Date: Wed, 28 Apr 2021 17:49:44 +0800 Message-Id: <20210428094949.43579-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210428094949.43579-1-songmuchun@bytedance.com> References: <20210428094949.43579-1-songmuchun@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Now the kmem states is only used to indicate whether the kmem is offlined. But we can use ->kmemcg_id to do the same things. So remove the kmem states to simplify the code. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 7 ------- mm/memcontrol.c | 10 ++-------- 2 files changed, 2 insertions(+), 15 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index c193be760709..6350c563c7b8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -182,12 +182,6 @@ struct mem_cgroup_thresholds { struct mem_cgroup_threshold_ary *spare; }; -enum memcg_kmem_state { - KMEM_NONE, - KMEM_ALLOCATED, - KMEM_ONLINE, -}; - #if defined(CONFIG_SMP) struct memcg_padding { char x[0]; @@ -320,7 +314,6 @@ struct mem_cgroup { #ifdef CONFIG_MEMCG_KMEM int kmemcg_id; - enum memcg_kmem_state kmem_state; struct obj_cgroup __rcu *objcg; struct list_head objcg_list; /* list of inherited objcgs */ #endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c1ce4fdba028..9b9a5368a3e9 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3461,7 +3461,6 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) return 0; BUG_ON(memcg->kmemcg_id >= 0); - BUG_ON(memcg->kmem_state); memcg_id = memcg_alloc_cache_id(); if (memcg_id < 0) @@ -3478,7 +3477,6 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) static_branch_enable(&memcg_kmem_enabled_key); memcg->kmemcg_id = memcg_id; - memcg->kmem_state = KMEM_ONLINE; return 0; } @@ -3488,11 +3486,6 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) struct mem_cgroup *parent; int kmemcg_id; - if (memcg->kmem_state != KMEM_ONLINE) - return; - - memcg->kmem_state = KMEM_ALLOCATED; - parent = parent_mem_cgroup(memcg); if (!parent) parent = root_mem_cgroup; @@ -3506,12 +3499,13 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) memcg_reparent_list_lrus(memcg, parent); memcg_free_cache_id(kmemcg_id); + memcg->kmemcg_id = -1; } static void memcg_free_kmem(struct mem_cgroup *memcg) { /* css_alloc() failed, offlining didn't happen */ - if (unlikely(memcg->kmem_state == KMEM_ONLINE)) + if (unlikely(memcg->kmemcg_id != -1)) memcg_offline_kmem(memcg); } #else From patchwork Wed Apr 28 09:49:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12228425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C0CEC43460 for ; Wed, 28 Apr 2021 09:54:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E3CA86142D for ; Wed, 28 Apr 2021 09:54:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238576AbhD1JzO (ORCPT ); Wed, 28 Apr 2021 05:55:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57170 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238512AbhD1JzM (ORCPT ); Wed, 28 Apr 2021 05:55:12 -0400 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5129DC06138C for ; Wed, 28 Apr 2021 02:54:27 -0700 (PDT) Received: by mail-pj1-x1036.google.com with SMTP id y22-20020a17090a8b16b0290150ae1a6d2bso8861744pjn.0 for ; Wed, 28 Apr 2021 02:54:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rR+OvhVrb+QAHQ+jiVrkYgB4Dzl1YoXeDesY75cVJEI=; b=QvZJqE3ta2E/c3ZVQ+FxrxKL1VJJR+NIP4aPVZ3vheUWyux1jDovf//k/iN0z68sQ3 ui2Zg4Owg7kHHJQnT+BuuFMMfwjIVtkab2zyIXz4WwGRigQA57PzG9HvF5d3kGm6kgPL ikcC3jjEBpbjZH8wnBtMWy158RctA+ANe6wb0kGi0s/bPMGhnu/yJ3nnbsP0Wzvcr3Ma yDMDnyyAPXKwlJJqCqCBanhk1okLxlS5l5OYz+QbIWLLn+h/6o+RIdFj9jQHbpxGZltj 8hXbvNQpnhO8PG8FRAUqnqC3y15yMwipWjvJYNQQygKtYiUsSS6InGx4uvkhdT/6CAus eyVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rR+OvhVrb+QAHQ+jiVrkYgB4Dzl1YoXeDesY75cVJEI=; b=QopTawAs5ndEq5ko2H5Kobq5E67xc8sr1ZgujiFr3OHT+uzGT0kjQziJO4vxrJ+CZc R57oOJFKzpaSkIGqF5cZTltFmy6vKs6tUUXlhuBntd72dJzaZgn6Se/uw8LmES17NAfs BrgaoZwCDZUlASCW4hfo4HuZbKwizVFu2UNRF2JnwLOKIxhoOxfyokPiS5ZGoRvNmOrW 7PEhZiRY6KdK1sS4cw2af7pCDc8Q09VQoMhsD8YRXqWSqaoJ0r2VX/4soEvAntOYJGJT j9W8wGzhp254FUt1cSNLTa5e0YtJ9DLcRn+BPZfmeiSjCaBS+JWoIqD4KpsQE+uCCOp0 J2cA== X-Gm-Message-State: AOAM532dgZWYS8IUW32QYyiFd8qRJHWrnmwLiZwE6Cb+m268Pz02xIbb u1ILApRZnxL1yWT2bnFS2rmkng== X-Google-Smtp-Source: ABdhPJx+BQwvCa+tWyaLztEOm3PIxKcql1j60oXvl+7x6kH9BoqBdjRVuQn6aVNZA3U5QJvl60B37w== X-Received: by 2002:a17:90a:6c23:: with SMTP id x32mr18667949pjj.62.1619603666977; Wed, 28 Apr 2021 02:54:26 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.233]) by smtp.gmail.com with ESMTPSA id x77sm4902365pfc.19.2021.04.28.02.54.21 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Apr 2021 02:54:26 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, alexander.h.duyck@linux.intel.com, richard.weiyang@gmail.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH 5/9] mm: memcontrol: move memcg_online_kmem() to mem_cgroup_css_online() Date: Wed, 28 Apr 2021 17:49:45 +0800 Message-Id: <20210428094949.43579-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210428094949.43579-1-songmuchun@bytedance.com> References: <20210428094949.43579-1-songmuchun@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Move memcg_online_kmem() to mem_cgroup_css_online() to simplify the code. In this case, we can remove memcg_free_kmem(). Signed-off-by: Muchun Song --- mm/memcontrol.c | 41 ++++++++++++++++------------------------- 1 file changed, 16 insertions(+), 25 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9b9a5368a3e9..1610d501e7b5 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3460,6 +3460,9 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) if (cgroup_memory_nokmem) return 0; + if (mem_cgroup_is_root(memcg)) + return 0; + BUG_ON(memcg->kmemcg_id >= 0); memcg_id = memcg_alloc_cache_id(); @@ -3486,6 +3489,9 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) struct mem_cgroup *parent; int kmemcg_id; + if (mem_cgroup_is_root(memcg)) + return; + parent = parent_mem_cgroup(memcg); if (!parent) parent = root_mem_cgroup; @@ -3499,14 +3505,6 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) memcg_reparent_list_lrus(memcg, parent); memcg_free_cache_id(kmemcg_id); - memcg->kmemcg_id = -1; -} - -static void memcg_free_kmem(struct mem_cgroup *memcg) -{ - /* css_alloc() failed, offlining didn't happen */ - if (unlikely(memcg->kmemcg_id != -1)) - memcg_offline_kmem(memcg); } #else static int memcg_online_kmem(struct mem_cgroup *memcg) @@ -3516,9 +3514,6 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) static void memcg_offline_kmem(struct mem_cgroup *memcg) { } -static void memcg_free_kmem(struct mem_cgroup *memcg) -{ -} #endif /* CONFIG_MEMCG_KMEM */ static int memcg_update_kmem_max(struct mem_cgroup *memcg, @@ -5047,7 +5042,6 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) { struct mem_cgroup *parent = mem_cgroup_from_css(parent_css); struct mem_cgroup *memcg, *old_memcg; - long error = -ENOMEM; old_memcg = set_active_memcg(parent); memcg = mem_cgroup_alloc(); @@ -5077,38 +5071,36 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) } /* The following stuff does not apply to the root */ - error = memcg_online_kmem(memcg); - if (error) - goto fail; - if (cgroup_subsys_on_dfl(memory_cgrp_subsys) && !cgroup_memory_nosocket) static_branch_inc(&memcg_sockets_enabled_key); return &memcg->css; -fail: - mem_cgroup_id_remove(memcg); - mem_cgroup_free(memcg); - return ERR_PTR(error); } static int mem_cgroup_css_online(struct cgroup_subsys_state *css) { struct mem_cgroup *memcg = mem_cgroup_from_css(css); + if (memcg_online_kmem(memcg)) + goto remove_id; + /* * A memcg must be visible for expand_shrinker_info() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (alloc_shrinker_info(memcg)) { - mem_cgroup_id_remove(memcg); - return -ENOMEM; - } + if (alloc_shrinker_info(memcg)) + goto offline_kmem; /* Online state pins memcg ID, memcg ID pins CSS */ refcount_set(&memcg->id.ref, 1); css_get(css); return 0; +offline_kmem: + memcg_offline_kmem(memcg); +remove_id: + mem_cgroup_id_remove(memcg); + return -ENOMEM; } static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) @@ -5166,7 +5158,6 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); free_shrinker_info(memcg); - memcg_free_kmem(memcg); mem_cgroup_free(memcg); } From patchwork Wed Apr 28 09:49:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12228427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E665C433ED for ; Wed, 28 Apr 2021 09:54:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 67A8C6141B for ; Wed, 28 Apr 2021 09:54:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238148AbhD1JzT (ORCPT ); Wed, 28 Apr 2021 05:55:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57194 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238608AbhD1JzS (ORCPT ); Wed, 28 Apr 2021 05:55:18 -0400 Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B836C06138B for ; Wed, 28 Apr 2021 02:54:32 -0700 (PDT) Received: by mail-pg1-x52a.google.com with SMTP id t22so9097987pgu.0 for ; Wed, 28 Apr 2021 02:54:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tsZpxnSCiHU3yOSRKuQ5Nn0zsBnruhO0+WHjObzR7x8=; b=CVN6j6GWu+63QHDHliyf2AZ1WZ20KwHrSvk5XqYiXCO7XP6C3pKHdUDBT4SxVeMhrP L7tMKIH47RxVvddI2lTDpeJ8zoSoyEYMuQ45LVFeb5DXE+Fh6X6ChHpirl3PFzwWAMLR Mq16ofgtQ7FlNKOSAT4pY+afeqGjvzzZCTuj0MZdV7TOq3YnjHw7tiV/vRs6qP2M2Xgm th2Wnk7lULXJi9N8kST37AJ0xLNcyrFfwLaX9ZH7VPC/rOXuYy0wadwRq00TeBP8sxq3 UWfh5WsU99QG0r6ObL1e/F7tLvVLs7RbR4LYSVxhVou+9gePD8fjRXBe3VlMtaRlw2Xe LMlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tsZpxnSCiHU3yOSRKuQ5Nn0zsBnruhO0+WHjObzR7x8=; b=cw8p5oTAUy/1JGueteBedGgtnZLqDnMUL209s9Y1CfPHaruKZtGpfEKtnA/ZwE4h2n O8lWRA/fvU6dQIDxjd/0Wa8Ida7VpCbdiDH5WyAyqpfJBeAGpBv69RpY+u0lovj4KC79 ofmgY90kEm2XJ+HIdbWuLRlZOPzN3wGFN5vrNfvqm+oR0IyyEeFQEycyCSraLYIRM20b SgtjE60/yFZb/Swvs5K9B1H9zA9j3rJDLv9UDBLm9aR05kxNV5lZHg5fbsMfVC6AJkUY 2s3kogudi86o5KctEeQ8aWLInN3jxf05Zai+QpTpzzJn2GEshk/sR1CIBEpOAg0tKzGT wdhA== X-Gm-Message-State: AOAM530DWrjjAL0atKdFbvsPdpYEjzNmwzv9hv0QtVzo+zsUHylDuYa1 USXw/5F7HPz4FB29aLW+AivImQ== X-Google-Smtp-Source: ABdhPJxeJwwMM87CerPEYoMJbuNWzwWZUAqJ1LCPLGY2k3lesIokTfkNVrUPpW+Jem/IzuHxzll8pA== X-Received: by 2002:a63:570b:: with SMTP id l11mr26184726pgb.193.1619603672214; Wed, 28 Apr 2021 02:54:32 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.233]) by smtp.gmail.com with ESMTPSA id x77sm4902365pfc.19.2021.04.28.02.54.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Apr 2021 02:54:31 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, alexander.h.duyck@linux.intel.com, richard.weiyang@gmail.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH 6/9] mm: list_lru: support for shrinking list lru Date: Wed, 28 Apr 2021 17:49:46 +0800 Message-Id: <20210428094949.43579-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210428094949.43579-1-songmuchun@bytedance.com> References: <20210428094949.43579-1-songmuchun@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Now memcg_update_all_list_lrus() only can increase the size of all the list lrus. This patch adds an ability to make it can shrink the size of all the list lrus. This can help us save memory when the user want to shrink the size. Signed-off-by: Muchun Song --- mm/list_lru.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 49 insertions(+), 4 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index d78dba5a6dab..3ee5239922c9 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -383,13 +383,11 @@ static void memcg_destroy_list_lru_node(struct list_lru_node *nlru) kvfree(memcg_lrus); } -static int memcg_update_list_lru_node(struct list_lru_node *nlru, - int old_size, int new_size) +static int memcg_list_lru_node_inc(struct list_lru_node *nlru, + int old_size, int new_size) { struct list_lru_memcg *old, *new; - BUG_ON(old_size > new_size); - old = rcu_dereference_protected(nlru->memcg_lrus, lockdep_is_held(&list_lrus_mutex)); new = kvmalloc(sizeof(*new) + new_size * sizeof(void *), GFP_KERNEL); @@ -418,11 +416,58 @@ static int memcg_update_list_lru_node(struct list_lru_node *nlru, return 0; } +/* This function always returns 0. */ +static int memcg_list_lru_node_dec(struct list_lru_node *nlru, + int old_size, int new_size) +{ + struct list_lru_memcg *old, *new; + + old = rcu_dereference_protected(nlru->memcg_lrus, + lockdep_is_held(&list_lrus_mutex)); + __memcg_destroy_list_lru_node(old, new_size, old_size); + + /* Reuse the old array if the allocation failures here. */ + new = kvmalloc(sizeof(*new) + new_size * sizeof(void *), GFP_KERNEL); + if (!new) + return 0; + + memcpy(&new->lru, &old->lru, new_size * sizeof(void *)); + + /* + * The locking below allows readers that hold nlru->lock avoid taking + * rcu_read_lock (see list_lru_from_memcg_idx). + * + * Since list_lru_{add,del} may be called under an IRQ-safe lock, + * we have to use IRQ-safe primitives here to avoid deadlock. + */ + spin_lock_irq(&nlru->lock); + rcu_assign_pointer(nlru->memcg_lrus, new); + spin_unlock_irq(&nlru->lock); + + kvfree_rcu(old, rcu); + return 0; +} + +static int memcg_update_list_lru_node(struct list_lru_node *nlru, + int old_size, int new_size) +{ + if (new_size > old_size) + return memcg_list_lru_node_inc(nlru, old_size, new_size); + else if (new_size < old_size) + return memcg_list_lru_node_dec(nlru, old_size, new_size); + + return 0; +} + static void memcg_cancel_update_list_lru_node(struct list_lru_node *nlru, int old_size, int new_size) { struct list_lru_memcg *memcg_lrus; + /* Nothing to do for the shrinking case. */ + if (old_size >= new_size) + return; + memcg_lrus = rcu_dereference_protected(nlru->memcg_lrus, lockdep_is_held(&list_lrus_mutex)); /* do not bother shrinking the array back to the old size, because we From patchwork Wed Apr 28 09:49:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12228429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A15F0C433B4 for ; Wed, 28 Apr 2021 09:54:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7860F6142C for ; Wed, 28 Apr 2021 09:54:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238660AbhD1JzX (ORCPT ); Wed, 28 Apr 2021 05:55:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57214 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238608AbhD1JzX (ORCPT ); Wed, 28 Apr 2021 05:55:23 -0400 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4DECC06138A for ; Wed, 28 Apr 2021 02:54:37 -0700 (PDT) Received: by mail-pj1-x1029.google.com with SMTP id t13so6440427pji.4 for ; Wed, 28 Apr 2021 02:54:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tOlUteNHINhRpZtpztBeELGSpydzeET01dFGAMvcB9w=; b=LEEZH5u64AUh+ynJYV212VHcTTq3jo7SOak1Z80frSXjEr35j/gMZeBApZgIac5EtK oyQLEU220WgDEPzQQt937Zemxc7veTiPvTy2Rem1vxKR5EMp3d0T0RkE6upT1Nx13pd3 ct69tBbCAl5PkeCcTgc9IYO1AqQ4C5KPm82xwGXsp+4PGuex1H2KlazxCnxlGyVZwc5j Fs3/VxauZ9qh06JmV+1Po+DoXkgn8GDqHvtx/snioyeBACqoFzuMhU4SBVX7lh2/yQ8s hU0c7U9jmWxlmiUBhSuIaccnhBlEYqFEurNWm6cRlhEbmb8D8OXWCCXdL2sH6LGIxV++ yNOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tOlUteNHINhRpZtpztBeELGSpydzeET01dFGAMvcB9w=; b=hEnDMayWGj6/B0xMP2YG5U3SAwplI1AB/SY3fNDfynJSUD1h5cUSWRO4scnowKcZLk Y0WptSm+2iNm+cevR0Uzt97a1Y9nppnladYMgYKifWhoe1qjKNEVywkNQ5fBz2HqiLTj Vau9ICZyf2eRrgsseKY5HY1wlOcXwR3kEpwdcQupmcSaherJnVslXIoTzGmqCql7J5P0 Fu830Fn9++cTHwRFkkq9p5jFZuhf8ga71o2T4nByUNgOiQ8oPVNtC6Wwds+0ktdif/Ki i2+1qnshQ3qJvptZRi0rlOite1xyF7YcTFKxRDB7adX0c7PRPjfGflPtoaTYew102agC ENFQ== X-Gm-Message-State: AOAM532mO0/YlKCKVUm5I0LBXQznLh/Ktk4+bQ20BuBKR7scnid/ndUH B3w4s2Hft20X7BuJf8nI+Thf9FZs2IBIMQ== X-Google-Smtp-Source: ABdhPJyrBDmZk/mG3JBVhVPM7uIF8K2OuTdn8hm9YJpozguzJZ16PR0+D5tUGPQdGb8zskw4RSE7hg== X-Received: by 2002:a17:90b:1bc1:: with SMTP id oa1mr30557077pjb.46.1619603677248; Wed, 28 Apr 2021 02:54:37 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.233]) by smtp.gmail.com with ESMTPSA id x77sm4902365pfc.19.2021.04.28.02.54.32 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Apr 2021 02:54:36 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, alexander.h.duyck@linux.intel.com, richard.weiyang@gmail.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH 7/9] ida: introduce ida_max() to return the maximum allocated ID Date: Wed, 28 Apr 2021 17:49:47 +0800 Message-Id: <20210428094949.43579-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210428094949.43579-1-songmuchun@bytedance.com> References: <20210428094949.43579-1-songmuchun@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Introduce ida_max() to return the maximum allocated ID. This will be used by memory cgroup in the later patch. Signed-off-by: Muchun Song --- include/linux/idr.h | 1 + lib/idr.c | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+) diff --git a/include/linux/idr.h b/include/linux/idr.h index a0dce14090a9..c3968a6348d1 100644 --- a/include/linux/idr.h +++ b/include/linux/idr.h @@ -255,6 +255,7 @@ struct ida { int ida_alloc_range(struct ida *, unsigned int min, unsigned int max, gfp_t); void ida_free(struct ida *, unsigned int id); void ida_destroy(struct ida *ida); +int ida_max(struct ida *ida); /** * ida_alloc() - Allocate an unused ID. diff --git a/lib/idr.c b/lib/idr.c index f4ab4f4aa3c7..bcfcaae89aa7 100644 --- a/lib/idr.c +++ b/lib/idr.c @@ -553,6 +553,46 @@ void ida_destroy(struct ida *ida) } EXPORT_SYMBOL(ida_destroy); +/** + * ida_max() - Return the maximum allocated ID. + * @ida: IDA handle. + * + * Context: Any context. It is safe to call this function without + * locking in your code. + * + * Return: The maximum allocated ID, or %-ENOSPC if the @ida is empty + */ +int ida_max(struct ida *ida) +{ + XA_STATE(xas, &ida->xa, 0); + struct ida_bitmap *curr, *prev; + unsigned long flags; + unsigned int bit, index; + + xas_lock_irqsave(&xas, flags); + if (ida_is_empty(ida)) { + xas_unlock_irqrestore(&xas, flags); + return -ENOSPC; + } + + xas_for_each(&xas, curr, ULONG_MAX) { + prev = curr; + index = xas.xa_index; + } + + if (xa_is_value(prev)) { + unsigned long val = xa_to_value(prev); + + bit = find_last_bit(&val, BITS_PER_XA_VALUE); + } else { + bit = find_last_bit(prev->bitmap, IDA_BITMAP_BITS); + } + + xas_unlock_irqrestore(&xas, flags); + + return index * IDA_BITMAP_BITS + bit; +} + #ifndef __KERNEL__ extern void xa_dump_index(unsigned long index, unsigned int shift); #define IDA_CHUNK_SHIFT ilog2(IDA_BITMAP_BITS) From patchwork Wed Apr 28 09:49:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12228431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C2C2C433B4 for ; Wed, 28 Apr 2021 09:54:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 620626113D for ; Wed, 28 Apr 2021 09:54:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238684AbhD1Jz3 (ORCPT ); Wed, 28 Apr 2021 05:55:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238675AbhD1Jz2 (ORCPT ); Wed, 28 Apr 2021 05:55:28 -0400 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 69F15C06138C for ; Wed, 28 Apr 2021 02:54:43 -0700 (PDT) Received: by mail-pl1-x62f.google.com with SMTP id v20so5240515plo.10 for ; Wed, 28 Apr 2021 02:54:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eVgbS1r6y4dIIx1w8PNRWtWwFVatPvGWznCWZtpnt6I=; b=Ye3xQ26M7eyxBvF1bS1McN7GEP62giojy9IF5P3ezIkyXipoDT7N/fkT6NCMcTVycO 7wweELpBf6jyeLY2t40FbNsG2yEJmQOMZqknkludEfimlm98g0muJrDzSN4TMKeBKWVo IdsaNDiqZVjUGqKQHFJidvKzW0JWGw9SfYEEniTlb/nFTAaMbDOFCxSxHQc8PU/nI1FF AW+vhhOSTAEq1pU1U8J7yA88C+NgjHgUhKShdfD5DRi825nWvy2Ev0gS3E+s8hHn4cbX xyhvAS+Ou5nHUyqvsyRpg4ure61j3lSlohGXXPmXLO3WATbp3rAQdQhi+JCJ7XIbBsO6 nSxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eVgbS1r6y4dIIx1w8PNRWtWwFVatPvGWznCWZtpnt6I=; b=pOWXX7sbCruMhdfNyUJ1qCjrrDkfpRO2WZhDCErkSI8z+4NWkGjBIzaa20RazQyXIC MW+wMG64MpfSeSkpF9HN+eTeKkfrcVHzGyD454N6OV9bz7MpWCVVTTfZgVXWclKRVT4D Na5aCaMd6/rG0JM7axy9ygbEL5pfJYP8BlzHprKNpRwbseqkfX5/BHq50pcHyEdO57jD IWiWZ/bvUnXUN/WdJ/4PK7zhIcyzYjjx1oRd0eJgYqSxnbQRR5lyH/d+wVOmQCvJ4xKE odzxzH4hezQyE8doZv4Y3aKC2743zUUeh9+7xuia1qJsZ5czBaMNOXYGZkDSQjMzroO+ 43kw== X-Gm-Message-State: AOAM530jMyfY/+R4cgA6vLOgyK6C7yPfC3nl43yEKbr+2bq6gm/a+egH ic03u2ypGu5pPGx2Ec6wKukb5g== X-Google-Smtp-Source: ABdhPJykGeKEYrTpape4UuVoYtuRgn6sXbKt1+jlCioXuVws9hGzmd0k8K7tcpxp1Y4c4yq3YegKxA== X-Received: by 2002:a17:90a:6687:: with SMTP id m7mr384642pjj.75.1619603682998; Wed, 28 Apr 2021 02:54:42 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.233]) by smtp.gmail.com with ESMTPSA id x77sm4902365pfc.19.2021.04.28.02.54.37 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Apr 2021 02:54:42 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, alexander.h.duyck@linux.intel.com, richard.weiyang@gmail.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH 8/9] mm: memcontrol: shrink the list lru size Date: Wed, 28 Apr 2021 17:49:48 +0800 Message-Id: <20210428094949.43579-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210428094949.43579-1-songmuchun@bytedance.com> References: <20210428094949.43579-1-songmuchun@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org In our server, we found a suspected memory leak problem. The kmalloc-32 consumes more than 6GB of memory. Other kmem_caches consume less than 2GB memory. After our in-depth analysis, the memory consumption of kmalloc-32 slab cache is the cause of list_lru_one allocation. crash> p memcg_nr_cache_ids memcg_nr_cache_ids = $2 = 24574 memcg_nr_cache_ids is very large and memory consumption of each list_lru can be calculated with the following formula. num_numa_node * memcg_nr_cache_ids * 32 (kmalloc-32) There are 4 numa nodes in our system, so each list_lru consumes ~3MB. crash> list super_blocks | wc -l 952 Every mount will register 2 list lrus, one is for inode, another is for dentry. There are 952 super_blocks. So the total memory is 952 * 2 * 3 MB (~5.6GB). But the number of memory cgroup is less than 500. So I guess more than 12286 containers have been deployed on this machine (I do not know why there are so many containers, it may be a user's bug or the user really want to do that). But now there are less than 500 containers in the system. And memcg_nr_cache_ids has not been reduced to a suitable value. This can waste a lot of memory. If we want to reduce memcg_nr_cache_ids, we have to reboot the server. This is not what we want. So this patch will dynamically adjust the value of memcg_nr_cache_ids to keep healthy memory consumption. In this case, we may be able to restore a healthy environment even if the users have created tens of thousands of memory cgroups. In this patch, I adjusted the calculation formula of memcg_nr_cache_ids from "size = 2 * (id + 1)" to "size = 2 * id" in memcg_alloc_cache_id(). Because this can make things more simple when shrink the list lru size. Signed-off-by: Muchun Song --- mm/memcontrol.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 49 insertions(+), 3 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1610d501e7b5..f8cdd87cf693 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -362,6 +362,8 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg, static DEFINE_IDA(memcg_cache_ida); int memcg_nr_cache_ids; +static int kmemcg_max_id; + /* Protects memcg_nr_cache_ids */ static DECLARE_RWSEM(memcg_cache_ids_sem); @@ -2856,8 +2858,11 @@ static int memcg_alloc_cache_id(void) if (id < 0) return id; - if (id < memcg_nr_cache_ids) + if (id < memcg_nr_cache_ids) { + if (id > kmemcg_max_id) + kmemcg_max_id = id; return id; + } /* * There's no space for the new id in memcg_caches arrays, @@ -2865,15 +2870,17 @@ static int memcg_alloc_cache_id(void) */ down_write(&memcg_cache_ids_sem); - size = 2 * (id + 1); + size = 2 * id; if (size < MEMCG_CACHES_MIN_SIZE) size = MEMCG_CACHES_MIN_SIZE; else if (size > MEMCG_CACHES_MAX_SIZE) size = MEMCG_CACHES_MAX_SIZE; err = memcg_update_all_list_lrus(size); - if (!err) + if (!err) { memcg_nr_cache_ids = size; + kmemcg_max_id = id; + } up_write(&memcg_cache_ids_sem); @@ -2884,9 +2891,48 @@ static int memcg_alloc_cache_id(void) return id; } +static inline int nearest_fit_id(int id) +{ + if (unlikely(id < MEMCG_CACHES_MIN_SIZE)) + return MEMCG_CACHES_MIN_SIZE; + + return 1 << (__fls(id) + 1); +} + +/* + * memcg_alloc_cache_id() and memcg_free_cache_id() are serialized by + * cgroup_mutex. So there is no race on kmemcg_max_id. + */ static void memcg_free_cache_id(int id) { ida_simple_remove(&memcg_cache_ida, id); + + if (kmemcg_max_id == id) { + /* + * In order to avoid @memcg_nr_cache_ids bouncing between + * @memcg_nr_cache_ids / 2 and @memcg_nr_cache_ids. We only + * shrink the list lru size when @kmemcg_max_id is smaller + * than @memcg_nr_cache_ids / 3. + */ + int size = memcg_nr_cache_ids / 3; + + kmemcg_max_id = ida_max(&memcg_cache_ida); + if (kmemcg_max_id < size) { + /* + * Find the first value greater than @kmemcg_max_id + * which can fit our need. And shrink the list lru + * to this size. + */ + size = nearest_fit_id(kmemcg_max_id); + + down_write(&memcg_cache_ids_sem); + if (size != memcg_nr_cache_ids) { + memcg_update_all_list_lrus(size); + memcg_nr_cache_ids = size; + } + up_write(&memcg_cache_ids_sem); + } + } } /* From patchwork Wed Apr 28 09:49:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12228433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80835C433B4 for ; Wed, 28 Apr 2021 09:54:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5806E61419 for ; Wed, 28 Apr 2021 09:54:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238675AbhD1Jzg (ORCPT ); Wed, 28 Apr 2021 05:55:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238683AbhD1Jzf (ORCPT ); Wed, 28 Apr 2021 05:55:35 -0400 Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D510BC06138F for ; Wed, 28 Apr 2021 02:54:48 -0700 (PDT) Received: by mail-pf1-x435.google.com with SMTP id h11so3732888pfn.0 for ; Wed, 28 Apr 2021 02:54:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hY8MMvczpGg13vjFG0pVN31ztr+LQshtCwbuU4Brm+8=; b=OVbU3euh7GNOL90p5IiOxYxd4FcV1VfX0Y9vH/DEq9PQ+YiGaCtAbF9kfcosIOpdNt +/+lxJEW370B9C2wMjLbygjflseJ/Bjqq3IiZmMZrz+aDvE97hM9gBX/TnJna/XR4twH ghtJswmWGKVLMbNUuVlgg+yv1Ui/e8jn7fEiLnj0my55Xpciv3n8/sSpBrnNSV17kx6g 6WqjfZh1HFaIBmkPWTn8wwv6NFUYza8HXULyd3PUKJpw32v5videUL6humajUOB0SVuH W95VSs9KRquN+EZQVRhc35Ml9pUhxcqB966YC+EuQPDvO8LdqrpLONVSsK59dHAyLB4I NXAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hY8MMvczpGg13vjFG0pVN31ztr+LQshtCwbuU4Brm+8=; b=FnriUK+kJYmahB6X7EEhaMyaAFgT/ltcdE+QjyefJucBeKYFlaD/+TO/uqYNngpQzz +QRXh0UezvHXdc8XE/hAn1J6W3RiAhrZ/C7pOagaquPLZaHbK/Mux1CIS5An5CUVpB0u y6z6LWrE5KDUwr7VQVWhiXJAz3vX9XyYxIO4mDRbysT88NV6yQvgA5hiZavECuJEygPl eSmHNxbe19VuP5IpLnE7s4qxxljP4KVhihJxLkIuotYQNZtWiYH/SXBUiG6AtVUMJ30c 1xFNrMCbJqAy0KiU0YDWdgfI9+z+VoxfeY6biqXUygXqahR7gsGyviqQKrpvSlRd11Zd EHpw== X-Gm-Message-State: AOAM530kzELVCwmRjKUwdm6OoV+K7kWrH3gRrMCmoSoq5BFhQwctbtDN oUPLfx0ZG5MnM/dqAfNbNC8wxQ== X-Google-Smtp-Source: ABdhPJyzJQIe5mcRTW0GlTyxzjQmNhmM1GNGv6hhvKBTTupwNZQlaXeqkzuhovHXs5jNeEzNcBllrw== X-Received: by 2002:a63:b52:: with SMTP id a18mr26475766pgl.276.1619603688418; Wed, 28 Apr 2021 02:54:48 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.233]) by smtp.gmail.com with ESMTPSA id x77sm4902365pfc.19.2021.04.28.02.54.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Apr 2021 02:54:48 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, alexander.h.duyck@linux.intel.com, richard.weiyang@gmail.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH 9/9] mm: memcontrol: rename memcg_{get,put}_cache_ids to memcg_list_lru_resize_{lock,unlock} Date: Wed, 28 Apr 2021 17:49:49 +0800 Message-Id: <20210428094949.43579-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210428094949.43579-1-songmuchun@bytedance.com> References: <20210428094949.43579-1-songmuchun@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The rwsem is held for writing during list lru arrays relocation and memcg_nr_cache_ids updates. Therefore memcg_get_cache_ids implies memcg_nr_cache_ids cannot be updated. It acts as a lock primitive. So rename it to a more suitable name. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 8 ++++---- mm/list_lru.c | 8 ++++---- mm/memcontrol.c | 4 ++-- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 6350c563c7b8..e8ba6ee1b369 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1635,8 +1635,8 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size); extern struct static_key_false memcg_kmem_enabled_key; extern int memcg_nr_cache_ids; -void memcg_get_cache_ids(void); -void memcg_put_cache_ids(void); +void memcg_list_lru_resize_lock(void); +void memcg_list_lru_resize_unlock(void); /* * Helper macro to loop through all memcg-specific caches. Callers must still @@ -1711,11 +1711,11 @@ static inline int memcg_cache_id(struct mem_cgroup *memcg) return -1; } -static inline void memcg_get_cache_ids(void) +static inline void memcg_list_lru_resize_lock(void) { } -static inline void memcg_put_cache_ids(void) +static inline void memcg_list_lru_resize_unlock(void) { } diff --git a/mm/list_lru.c b/mm/list_lru.c index 3ee5239922c9..e0ba0641b4e1 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -640,7 +640,7 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, else lru->shrinker_id = -1; #endif - memcg_get_cache_ids(); + memcg_list_lru_resize_lock(); lru->node = kcalloc(nr_node_ids, sizeof(*lru->node), GFP_KERNEL); if (!lru->node) @@ -663,7 +663,7 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, list_lru_register(lru); out: - memcg_put_cache_ids(); + memcg_list_lru_resize_unlock(); return err; } EXPORT_SYMBOL_GPL(__list_lru_init); @@ -674,7 +674,7 @@ void list_lru_destroy(struct list_lru *lru) if (!lru->node) return; - memcg_get_cache_ids(); + memcg_list_lru_resize_lock(); list_lru_unregister(lru); @@ -685,6 +685,6 @@ void list_lru_destroy(struct list_lru *lru) #ifdef CONFIG_MEMCG_KMEM lru->shrinker_id = -1; #endif - memcg_put_cache_ids(); + memcg_list_lru_resize_unlock(); } EXPORT_SYMBOL_GPL(list_lru_destroy); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f8cdd87cf693..437465611845 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -367,12 +367,12 @@ static int kmemcg_max_id; /* Protects memcg_nr_cache_ids */ static DECLARE_RWSEM(memcg_cache_ids_sem); -void memcg_get_cache_ids(void) +void memcg_list_lru_resize_lock(void) { down_read(&memcg_cache_ids_sem); } -void memcg_put_cache_ids(void) +void memcg_list_lru_resize_unlock(void) { up_read(&memcg_cache_ids_sem); }