From patchwork Wed Apr 28 09:49:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12228435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DAB3C433ED for ; Wed, 28 Apr 2021 09:54:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ABED0613E8 for ; Wed, 28 Apr 2021 09:54:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ABED0613E8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F28186B0036; Wed, 28 Apr 2021 05:54:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB0A76B006E; Wed, 28 Apr 2021 05:54:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D29F56B0070; Wed, 28 Apr 2021 05:54:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0009.hostedemail.com [216.40.44.9]) by kanga.kvack.org (Postfix) with ESMTP id B48B76B0036 for ; Wed, 28 Apr 2021 05:54:03 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 680513ABC for ; Wed, 28 Apr 2021 09:54:03 +0000 (UTC) X-FDA: 78081314766.05.D912085 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf30.hostedemail.com (Postfix) with ESMTP id 332F2E00011D for ; Wed, 28 Apr 2021 09:53:41 +0000 (UTC) Received: by mail-pf1-f174.google.com with SMTP id q2so11420828pfk.9 for ; Wed, 28 Apr 2021 02:54:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=0XVjixsLTLJGSFLESUPSKkiw4I4YXcT+TC9FDzs+gCA=; b=blgKr4FouT87ImEPqykKfSgJEIDTXvFoVNhAWBB5zIo8uOISObsgHvReBYKdv5GCtv rib0aeZCZbvNKbiQx9ldgpPQ5QerZp199KxpeuB5dFpzGi1r4Z6lLwDppS0Fx0qPnvTS DIRgGVBL1EOlxF73vE45CEmcYRrmBYnu/XBwkKMC5nDm4XRfwrgUz3yMfx11RbaYNq5s BkJhr+hZd9jZhn4bqig1cd9lgRo3AqlH27H1UlTHdF6upDQZnMkfTyVG1SHoalkzHFno vE3U7HIQ0R67RpYBZScYZCAd87Au3FEmQxeJkCPwiw4ZWzAUuGM8BucDwrraxHqoDIzI o8qA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=0XVjixsLTLJGSFLESUPSKkiw4I4YXcT+TC9FDzs+gCA=; b=cjtkZkVLhv/F5KyIqDB8XbAO7cK28bddhD8PVhrKAgwJnNb3h1oNqp1I9dYOR1Zqvw q/y7hZ61581ykYcnwZTf3inbaT/DDPPVJDcVlJ6eUiSodUXqSZB2N38Eq0vAyP2aZ8wD z9rLe2m7wp22nAS8AtxPcCJxr5xyCvTA7WaRJ3ZhgPSaV9T5nLBPIYROGirT3ir2eZ7x RnLhW5Jfom8BCbry+q9EEg48H2LzvcGT9rPtHuaA8anQ6jS1XS7jyFpwqFIl5COa1k1/ p30noVxZK9S46slscF9bF9QEvh6QDPYmgQt2FQr064oS28KW99sMs7wumNx1HtxMC2S4 aEfQ== X-Gm-Message-State: AOAM532B8g8lQCIXwLG6EysQ4uTw4MQn6PkGVqZ6DuJryMfewUnP5ji6 UIhzIgZRLmkNENTVyo2zTVm9Nw== X-Google-Smtp-Source: ABdhPJy7eMSsqpg9mqBYwGXKqweBIOvNTR20P0fg58PUahgtl2RvSFLbuwlpkiW9nkThOcGpuD+u/w== X-Received: by 2002:a63:6387:: with SMTP id x129mr2578387pgb.58.1619603639829; Wed, 28 Apr 2021 02:53:59 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.233]) by smtp.gmail.com with ESMTPSA id x77sm4902365pfc.19.2021.04.28.02.53.54 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Apr 2021 02:53:59 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, alexander.h.duyck@linux.intel.com, richard.weiyang@gmail.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH 0/9] Shrink the list lru size on memory cgroup removal Date: Wed, 28 Apr 2021 17:49:40 +0800 Message-Id: <20210428094949.43579-1-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) MIME-Version: 1.0 X-Rspamd-Queue-Id: 332F2E00011D X-Stat-Signature: hx5nuafnq9sq347wodadjhjqzyh6fgkh X-Rspamd-Server: rspam02 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf30; identity=mailfrom; envelope-from=""; helo=mail-pf1-f174.google.com; client-ip=209.85.210.174 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619603621-94322 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In our server, we found a suspected memory leak problem. The kmalloc-32 consumes more than 6GB of memory. Other kmem_caches consume less than 2GB memory. After our in-depth analysis, the memory consumption of kmalloc-32 slab cache is the cause of list_lru_one allocation. crash> p memcg_nr_cache_ids memcg_nr_cache_ids = $2 = 24574 memcg_nr_cache_ids is very large and memory consumption of each list_lru can be calculated with the following formula. num_numa_node * memcg_nr_cache_ids * 32 (kmalloc-32) There are 4 numa nodes in our system, so each list_lru consumes ~3MB. crash> list super_blocks | wc -l 952 Every mount will register 2 list lrus, one is for inode, another is for dentry. There are 952 super_blocks. So the total memory is 952 * 2 * 3 MB (~5.6GB). But the number of memory cgroup is less than 500. So I guess more than 12286 containers have been deployed on this machine (I do not know why there are so many containers, it may be a user's bug or the user really want to do that). But now there are less than 500 containers in the system. And memcg_nr_cache_ids has not been reduced to a suitable value. This can waste a lot of memory. If we want to reduce memcg_nr_cache_ids, we have to reboot the server. This is not what we want. So this patchset will dynamically adjust the value of memcg_nr_cache_ids to keep healthy memory consumption. In this case, we may be able to restore a healthy environment even if the users have created tens of thousands of memory cgroups and then destroyed those memory cgroups. This patchset also contains some code simplification. Muchun Song (9): mm: list_lru: fix list_lru_count_one() return value mm: memcontrol: remove kmemcg_id reparenting mm: list_lru: rename memcg_drain_all_list_lrus to memcg_reparent_list_lrus mm: memcontrol: remove the kmem states mm: memcontrol: move memcg_online_kmem() to mem_cgroup_css_online() mm: list_lru: support for shrinking list lru ida: introduce ida_max() to return the maximum allocated ID mm: memcontrol: shrink the list lru size mm: memcontrol: rename memcg_{get,put}_cache_ids to memcg_list_lru_resize_{lock,unlock} include/linux/idr.h | 1 + include/linux/list_lru.h | 2 +- include/linux/memcontrol.h | 15 ++---- lib/idr.c | 40 +++++++++++++++ mm/list_lru.c | 89 +++++++++++++++++++++++++-------- mm/memcontrol.c | 121 +++++++++++++++++++++++++-------------------- 6 files changed, 183 insertions(+), 85 deletions(-)