From patchwork Tue May 11 10:46:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53566C433B4 for ; Tue, 11 May 2021 10:51:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BA04361932 for ; Tue, 11 May 2021 10:51:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BA04361932 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4336E6B0071; Tue, 11 May 2021 06:51:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 40A276B0072; Tue, 11 May 2021 06:51:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 283676B0073; Tue, 11 May 2021 06:51:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0250.hostedemail.com [216.40.44.250]) by kanga.kvack.org (Postfix) with ESMTP id 0B78E6B0071 for ; Tue, 11 May 2021 06:51:24 -0400 (EDT) Received: from smtpin35.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C3BBC8249980 for ; Tue, 11 May 2021 10:51:23 +0000 (UTC) X-FDA: 78128633646.35.9C06AC8 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf20.hostedemail.com (Postfix) with ESMTP id 42534138 for ; Tue, 11 May 2021 10:51:23 +0000 (UTC) Received: by mail-pl1-f175.google.com with SMTP id b3so10619604plg.11 for ; Tue, 11 May 2021 03:51:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Fkv5wb+1+DGkoP3yZjv9oMKkFDk0tSiczNIe18RktTE=; b=qqcBn0jkVqV+KyeSBol7e4l5R1MpV3SwrDDEATmlvOxSVFKRCPBuK9jGUX9biY+jfO Z1oyy4lDOgQ+rT9wRD7JjUydOvramniAP+pJ0+/K8u/WtP1IEeJCisO0GbI8iaiCkVaE YJtymGVfa43s3ZgRj+N0RMBviHNacOv9TltFKxilWHLa899GoJ7PNjCXRfHUx8NLf5Z5 JwoPOsOLMd6ErK09p3/ireKgcFFaqtPqJPK2B/rPiJ9vY7hnLhMK9QHyZgTeWlR4EjHa 3qxntUlsNp0YDL6BGl2QE27Wl+oK4HQBuuXV12VJSJu8Q0rMM1I6o6vXbL2Oi1Cv3H0N ghUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Fkv5wb+1+DGkoP3yZjv9oMKkFDk0tSiczNIe18RktTE=; b=feSrfuvkzHgpcgohEqxhsO2u4nwNtEPkjxo/XDAEP04Sz/UOvK8ddWKdegT3kEu1UH 4DUIIcSo36mm9Q48lJ6TH5es0/XiiszYzt0EtPCW9QOWYM2WB2QVeZc6AgFVKHUFfcz8 IlTlc/TflQEfDtTRKSlhIiUO3HdBNXxQPnMxNaP1s6ACzBM01xVtLgz3t+Brcq0vC6Uz Nj6wFbVfUFq8mw+lL4iWvoXzW/6sVbA/cw7xhVgBduSXGOY+X4x5VHZPHw+ueYngnOoL nxYdOcKaiQkxbwvrJvJ+jqSCq3qxmvlUZO2E+r5VMdLEXIfrbmV00u1emKPpXUejz9Hr 24xg== X-Gm-Message-State: AOAM530yPu8gH5NjhI3WFsvQUXnckuotj85mtcM4j81T9jBwqwVLFW6+ hu4ZWDZ7LvN+Aye6aDe5JfTTag== X-Google-Smtp-Source: ABdhPJxC3I9XtMaDwx6Dx0AYAHNd26ktO1L37i+djgUgk6WSl8Trt5/KWMOtA3o9dfGojcV7D/p7bw== X-Received: by 2002:a17:90a:8902:: with SMTP id u2mr33066729pjn.143.1620730282400; Tue, 11 May 2021 03:51:22 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.51.15 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:51:22 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 01/17] mm: list_lru: fix list_lru_count_one() return value Date: Tue, 11 May 2021 18:46:31 +0800 Message-Id: <20210511104647.604-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=qqcBn0jk; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: 8s6e7tw3qjp3rfnqf9ux8xrj8itjdfmd X-Rspamd-Queue-Id: 42534138 X-Rspamd-Server: rspam02 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf20; identity=mailfrom; envelope-from=""; helo=mail-pl1-f175.google.com; client-ip=209.85.214.175 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730283-501243 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since commit 2788cf0c401c ("memcg: reparent list_lrus and free kmemcg_id on css offline"), the ->nr_items can be negative during memory cgroup reparenting. In this case, list_lru_count_one() returns an unusual and huge value. Ii can surprise users. So let it return zero when ->nr_items is negative. Signed-off-by: Muchun Song --- mm/list_lru.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index cd58790d0fb3..4962d48d4410 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -176,13 +176,16 @@ unsigned long list_lru_count_one(struct list_lru *lru, { struct list_lru_node *nlru = &lru->node[nid]; struct list_lru_one *l; - unsigned long count; + long count; rcu_read_lock(); l = list_lru_from_memcg_idx(nlru, memcg_cache_id(memcg)); count = READ_ONCE(l->nr_items); rcu_read_unlock(); + if (unlikely(count < 0)) + count = 0; + return count; } EXPORT_SYMBOL_GPL(list_lru_count_one); From patchwork Tue May 11 10:46:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34FB3C433ED for ; Tue, 11 May 2021 10:51:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B25F561927 for ; Tue, 11 May 2021 10:51:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B25F561927 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 348988D0002; Tue, 11 May 2021 06:51:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 31F576B0073; Tue, 11 May 2021 06:51:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C0C68D0002; Tue, 11 May 2021 06:51:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0080.hostedemail.com [216.40.44.80]) by kanga.kvack.org (Postfix) with ESMTP id 0000B6B0072 for ; Tue, 11 May 2021 06:51:30 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id AD1C8A774 for ; Tue, 11 May 2021 10:51:30 +0000 (UTC) X-FDA: 78128633940.13.E24C2E4 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf10.hostedemail.com (Postfix) with ESMTP id A48F040002E2 for ; Tue, 11 May 2021 10:51:14 +0000 (UTC) Received: by mail-pf1-f181.google.com with SMTP id 10so15921168pfl.1 for ; Tue, 11 May 2021 03:51:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lGjb9XHs9HEa3yuCIWtdN6OV6aUJlazGyEylQig/2qw=; b=1ZlSDkHHElJFkZnuHviAqONLGrsrF74n1N3r4603t2OcT6B8Zghjytatk5NUDWAMrc CWYBWR0DHY/FIT0z1qEOKkn3++hfRLeyeSa1exRX29IrunIVc2fcFbuABH9vof0rUXHF /D0FIOJckKHayu9vDAtiALygRnnhdSdfAQzTMup39o1wwGXCYbLgZkaitBjY1w7Uu2A3 Iwa73mJ/dlsPt/kwINQkxilRF3wFpWP+ZtWK1/44M7xGUcE9f9IuZbNNkoAJ0sZEnS+C wEfyogG2RhX8rGy/uRAUaZV5pvFsMsHZmUiC44od1xt/PD4GPgUcvi14//naVQmHZgD7 YrWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lGjb9XHs9HEa3yuCIWtdN6OV6aUJlazGyEylQig/2qw=; b=audNStVeZvolPoCCr00xCoBR9Ax8duA51NT4cCvmAFMrynlyICkknLLGc0fvTgzZsR UpgcIHJ9PP+SSPVFfp9R5oGkaS1GQTnx1ztJnHhI6QnsxU86pf5tlFCmJxc340UA4ok0 JBMBz6woZZ8XQnMyvDO59qc+nq9Fn4UlRv9p8/P1923Yc52Zn6YFoETeQCcVH0Eq6HCt xf119iQ+Oc0+a8ZJSNKuEf9/6fiJm47i66OyGLqCL6AgPrbNJHMZMC/hSPHZycZRgrCI iHTNm9RvUzFocZMyRrS8JKjXP2TExB8vhf8RkfX02NRJ4Wasy23N8zqjKx5oHTcWepdt ncow== X-Gm-Message-State: AOAM533OWzahyMYEQYzzsZPQNEVLBn5T/kYl6neVv/t1BcNQ7dxyDZZP zXINaRN3cPQOSaAr5TOGJkg7rw== X-Google-Smtp-Source: ABdhPJwrs5wlHj9JR4nN7o9mNIe/lOSuXtHDeccVzlqFnpwhk46tI6wfm+22UPitRWq14ew6ukG+hg== X-Received: by 2002:a65:5088:: with SMTP id r8mr29061106pgp.12.1620730289556; Tue, 11 May 2021 03:51:29 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.51.22 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:51:29 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 02/17] mm: memcontrol: remove kmemcg_id reparenting Date: Tue, 11 May 2021 18:46:32 +0800 Message-Id: <20210511104647.604-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=1ZlSDkHH; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf10.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: en5tt1ga8d3jjha734g7ep9f785neh11 X-Rspamd-Queue-Id: A48F040002E2 X-Rspamd-Server: rspam02 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf10; identity=mailfrom; envelope-from=""; helo=mail-pf1-f181.google.com; client-ip=209.85.210.181 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730274-587811 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since slab objects and kmem pages are charged to object cgroup instead of memory cgroup, memcg_reparent_objcgs() will reparent this cgroup and all its descendants to the parent cgroup. This already makes further list_lru_add()'s add elements to the parent's list. So we do not need to change kmemcg_id of an offline cgroup to its parent's id. It is just waste CPU cycles. Just remove those redundant code. Signed-off-by: Muchun Song --- mm/memcontrol.c | 20 ++------------------ 1 file changed, 2 insertions(+), 18 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 64ada9e650a5..21e12312509c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3485,8 +3485,7 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) static void memcg_offline_kmem(struct mem_cgroup *memcg) { - struct cgroup_subsys_state *css; - struct mem_cgroup *parent, *child; + struct mem_cgroup *parent; int kmemcg_id; if (memcg->kmem_state != KMEM_ONLINE) @@ -3503,22 +3502,7 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) kmemcg_id = memcg->kmemcg_id; BUG_ON(kmemcg_id < 0); - /* - * Change kmemcg_id of this cgroup and all its descendants to the - * parent's id, and then move all entries from this cgroup's list_lrus - * to ones of the parent. After we have finished, all list_lrus - * corresponding to this cgroup are guaranteed to remain empty. The - * ordering is imposed by list_lru_node->lock taken by - * memcg_drain_all_list_lrus(). - */ - rcu_read_lock(); /* can be called from css_free w/o cgroup_mutex */ - css_for_each_descendant_pre(css, &memcg->css) { - child = mem_cgroup_from_css(css); - BUG_ON(child->kmemcg_id != kmemcg_id); - child->kmemcg_id = parent->kmemcg_id; - } - rcu_read_unlock(); - + /* memcg_reparent_objcgs() must be called before this. */ memcg_drain_all_list_lrus(kmemcg_id, parent); memcg_free_cache_id(kmemcg_id); From patchwork Tue May 11 10:46:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A94F2C43462 for ; Tue, 11 May 2021 10:51:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3595B61132 for ; Tue, 11 May 2021 10:51:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3595B61132 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B5ED88E0001; Tue, 11 May 2021 06:51:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B35EA6B0073; Tue, 11 May 2021 06:51:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9AF768E0001; Tue, 11 May 2021 06:51:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0139.hostedemail.com [216.40.44.139]) by kanga.kvack.org (Postfix) with ESMTP id 8184D6B0072 for ; Tue, 11 May 2021 06:51:38 -0400 (EDT) Received: from smtpin34.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4AD2F180ACF9A for ; Tue, 11 May 2021 10:51:38 +0000 (UTC) X-FDA: 78128634276.34.B1BF8E2 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf16.hostedemail.com (Postfix) with ESMTP id 44F6580192E1 for ; Tue, 11 May 2021 10:51:26 +0000 (UTC) Received: by mail-pj1-f53.google.com with SMTP id t8-20020a17090abc48b029015ccb7f47daso91646pjv.0 for ; Tue, 11 May 2021 03:51:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=x4jDxCt8n77REyDHgYZFiAH9QsICwT/CfRTyFliAVkY=; b=06zHA1gUi1VqCeLYroahSAhcX58diigPj7gDH0kP9dNSgiSEAsd+8hjKAzQO//KZEB 50DRgwgI3IiAnTdaaw7RhOawjEaFaTST5LC9244nJUIqTKJg1nsw6h2f/zBKPFaEagVY 2Vw9KfZzxnd7N5d134QldOvuGpsFAs4FEaTTW693QPW/tizv+01rT786DHV91kz9pgPE Dhbh5eFkF75RBi6OiGIonOEHInDCkbXiyZy2c4mR8lfQc4R6R1jGSTbnv7vQ5z7Plarq BxWufxvjbl9CYmSBI5ct6TEk7D28bj49BaRszr3VF5gVU00IFSAXWh3+gDY+0omlR7UK 8mxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x4jDxCt8n77REyDHgYZFiAH9QsICwT/CfRTyFliAVkY=; b=cb4w7jb/3yCkiQuUSHiv7aoQgrzi87ujO1Y9OXPWvyLMfX4nI7Lc64yJXEhvpX1ZCm hgNjKh+OLCbppCXfRpcwXVrgE4tGKOdSvk2LdRhUPdc7qecAsg5h0CVNY7ALIkfyMnvR 8GeI2v6QwyVSoi6zrj4XfDxFGeOuPAOvGxJ49b9qb32WwEa1TX7jcQtNaHm7A8pjHW1x clQvCuD1JmwKxkeUUiRO3f4tckBpP41PJJq6QRnsYF27bh+OH/KZ3cm1z/1HVXcEMVQ9 hLdW0I6EAskcVxwWWVzPdULTvjxId60LwQ2fH1LJSEXrA0lrPiU9zmYvacOEnpfKwbhg C5pw== X-Gm-Message-State: AOAM532WaCojoviyg9+KLFyaRePySoZ29dLBwHnZEQI1zV0ds80L857Y 7USXdhkYPr/pLVq8ps+YQz8cbw== X-Google-Smtp-Source: ABdhPJz2S/+FDBdfIoJEY/9ohxgiE1dl2S6saMFngV85buTbmhClfdJI96FIYTAx6lhNks4IMYV45Q== X-Received: by 2002:a17:902:9685:b029:ef:70fd:a5a2 with SMTP id n5-20020a1709029685b02900ef70fda5a2mr180626plp.20.1620730297052; Tue, 11 May 2021 03:51:37 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.51.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:51:36 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 03/17] mm: memcontrol: remove the kmem states Date: Tue, 11 May 2021 18:46:33 +0800 Message-Id: <20210511104647.604-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 44F6580192E1 Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=06zHA1gU; spf=pass (imf16.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam04 X-Stat-Signature: w13p9n9p9knwhcbawohtdpej4bbdt8ty Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=mail-pj1-f53.google.com; client-ip=209.85.216.53 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730286-934916 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now the kmem states is only used to indicate whether the kmem is offlined. But we can use ->kmemcg_id to do the same thing. So remove the kmem states to simplify the code. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 7 ------- mm/memcontrol.c | 10 ++-------- 2 files changed, 2 insertions(+), 15 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index c193be760709..6350c563c7b8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -182,12 +182,6 @@ struct mem_cgroup_thresholds { struct mem_cgroup_threshold_ary *spare; }; -enum memcg_kmem_state { - KMEM_NONE, - KMEM_ALLOCATED, - KMEM_ONLINE, -}; - #if defined(CONFIG_SMP) struct memcg_padding { char x[0]; @@ -320,7 +314,6 @@ struct mem_cgroup { #ifdef CONFIG_MEMCG_KMEM int kmemcg_id; - enum memcg_kmem_state kmem_state; struct obj_cgroup __rcu *objcg; struct list_head objcg_list; /* list of inherited objcgs */ #endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 21e12312509c..e161a319982a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3461,7 +3461,6 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) return 0; BUG_ON(memcg->kmemcg_id >= 0); - BUG_ON(memcg->kmem_state); memcg_id = memcg_alloc_cache_id(); if (memcg_id < 0) @@ -3478,7 +3477,6 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) static_branch_enable(&memcg_kmem_enabled_key); memcg->kmemcg_id = memcg_id; - memcg->kmem_state = KMEM_ONLINE; return 0; } @@ -3488,11 +3486,6 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) struct mem_cgroup *parent; int kmemcg_id; - if (memcg->kmem_state != KMEM_ONLINE) - return; - - memcg->kmem_state = KMEM_ALLOCATED; - parent = parent_mem_cgroup(memcg); if (!parent) parent = root_mem_cgroup; @@ -3506,12 +3499,13 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) memcg_drain_all_list_lrus(kmemcg_id, parent); memcg_free_cache_id(kmemcg_id); + memcg->kmemcg_id = -1; } static void memcg_free_kmem(struct mem_cgroup *memcg) { /* css_alloc() failed, offlining didn't happen */ - if (unlikely(memcg->kmem_state == KMEM_ONLINE)) + if (unlikely(memcg->kmemcg_id != -1)) memcg_offline_kmem(memcg); } #else From patchwork Tue May 11 10:46:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E78AC43460 for ; Tue, 11 May 2021 10:51:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B23C561132 for ; Tue, 11 May 2021 10:51:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B23C561132 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 40D066B0072; Tue, 11 May 2021 06:51:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E39B940008; Tue, 11 May 2021 06:51:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 23A70940007; Tue, 11 May 2021 06:51:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0022.hostedemail.com [216.40.44.22]) by kanga.kvack.org (Postfix) with ESMTP id 04C526B0072 for ; Tue, 11 May 2021 06:51:45 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B5A57181AF5CC for ; Tue, 11 May 2021 10:51:45 +0000 (UTC) X-FDA: 78128634570.12.AF6327C Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by imf30.hostedemail.com (Postfix) with ESMTP id 7A036E00201C for ; Tue, 11 May 2021 10:51:16 +0000 (UTC) Received: by mail-pf1-f175.google.com with SMTP id c17so15793469pfn.6 for ; Tue, 11 May 2021 03:51:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=570qZ6BW+Bw5Veu1Tn32KJSDsrgLb0WocPAS7l9UKjo=; b=cXkuqPg26sg7zCH8Zlm/L88hhl+2M29S3NHF0UZeW5xgiIgj31m3I8ZyPpcszdrz7c 15RY5Bsx8gNF/FQ4bejRpw932IgFMX1JZRbJ5gE7Y8QdG+g18lfChibOAzcm65JIaIrw 86LR4GShsjV0SzJt73Sbhs2DwfFfqzUOJufF5icNTSDzL+3bKpmAUn0IrL+P0ZmGNZyR 6vGPdtR0+qN8zCBTgpJCOEdgIxd8ZYNFQ/wlCXv14MwQj3gTzslwBhxFrUi6Vuqx6mr7 8nE9r+O6raKN2Gprocb5v5qTnkk9+rRfWZKcBZB4W6l7Eae5w5CMcEdB8xg+9+J7GhM6 yYXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=570qZ6BW+Bw5Veu1Tn32KJSDsrgLb0WocPAS7l9UKjo=; b=iAHXJr06ZbDYagxrclf9B15tXgv+fq6WheWIE4ufC6MP7YGzrJAiqdlKLz12fkYUkq 7IrF+4XCByc1QRzjQRbKxXBGdMHorK5HyqIfIl9/G3yKK4Fg1f44kpx9fnZP/H6LHGq3 3TIFgnSmV60aYdUrw7PmZRDU+JyVHxnWih+IXoNc05FOg6ci4E+47vNMNJxNQPOJwVtm 07S+3vMEqdZHugHx0Lm55nPSjYP9P5OiYZRMMWto3F4pWNR8lAEKpkVmKIzQh6Ponwv8 osHoEc7E8Hm3M8jzERjj1Po5SKwl4BzNTvlgShBa+HJ4b0+Ss3vlRd4kjCjpPxa8Q4tQ NFZg== X-Gm-Message-State: AOAM533nnrQ/O6N2RwCt2DXIOHki1AVvJjruIvvSU/msJC0HxLw7xr9+ GK3GEECqG/8v0xQ44eej7rkyug== X-Google-Smtp-Source: ABdhPJx7H8aic1vSgbaFmAPVPSrhFZQeZuy+H/+JFDYMlJNa7Yea+LAnclzj4B/AdQ6QtWZrhZyQZw== X-Received: by 2002:aa7:8a87:0:b029:27d:a1e:bc71 with SMTP id a7-20020aa78a870000b029027d0a1ebc71mr12280361pfc.14.1620730304649; Tue, 11 May 2021 03:51:44 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.51.37 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:51:44 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 04/17] mm: memcontrol: move memcg_online_kmem() to mem_cgroup_css_online() Date: Tue, 11 May 2021 18:46:34 +0800 Message-Id: <20210511104647.604-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=cXkuqPg2; spf=pass (imf30.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 7A036E00201C X-Stat-Signature: eg7uuaun9sutbkrrjy5m4n5jx9gqgf9n Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf30; identity=mailfrom; envelope-from=""; helo=mail-pf1-f175.google.com; client-ip=209.85.210.175 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730276-774676 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move memcg_online_kmem() to mem_cgroup_css_online() to simplify the code. In this case, we can remove memcg_free_kmem(). Signed-off-by: Muchun Song --- mm/memcontrol.c | 41 ++++++++++++++++------------------------- 1 file changed, 16 insertions(+), 25 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e161a319982a..0d1c09873bad 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3460,6 +3460,9 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) if (cgroup_memory_nokmem) return 0; + if (mem_cgroup_is_root(memcg)) + return 0; + BUG_ON(memcg->kmemcg_id >= 0); memcg_id = memcg_alloc_cache_id(); @@ -3486,6 +3489,9 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) struct mem_cgroup *parent; int kmemcg_id; + if (mem_cgroup_is_root(memcg)) + return; + parent = parent_mem_cgroup(memcg); if (!parent) parent = root_mem_cgroup; @@ -3499,14 +3505,6 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) memcg_drain_all_list_lrus(kmemcg_id, parent); memcg_free_cache_id(kmemcg_id); - memcg->kmemcg_id = -1; -} - -static void memcg_free_kmem(struct mem_cgroup *memcg) -{ - /* css_alloc() failed, offlining didn't happen */ - if (unlikely(memcg->kmemcg_id != -1)) - memcg_offline_kmem(memcg); } #else static int memcg_online_kmem(struct mem_cgroup *memcg) @@ -3516,9 +3514,6 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) static void memcg_offline_kmem(struct mem_cgroup *memcg) { } -static void memcg_free_kmem(struct mem_cgroup *memcg) -{ -} #endif /* CONFIG_MEMCG_KMEM */ static int memcg_update_kmem_max(struct mem_cgroup *memcg, @@ -5047,7 +5042,6 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) { struct mem_cgroup *parent = mem_cgroup_from_css(parent_css); struct mem_cgroup *memcg, *old_memcg; - long error = -ENOMEM; old_memcg = set_active_memcg(parent); memcg = mem_cgroup_alloc(); @@ -5077,38 +5071,36 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) } /* The following stuff does not apply to the root */ - error = memcg_online_kmem(memcg); - if (error) - goto fail; - if (cgroup_subsys_on_dfl(memory_cgrp_subsys) && !cgroup_memory_nosocket) static_branch_inc(&memcg_sockets_enabled_key); return &memcg->css; -fail: - mem_cgroup_id_remove(memcg); - mem_cgroup_free(memcg); - return ERR_PTR(error); } static int mem_cgroup_css_online(struct cgroup_subsys_state *css) { struct mem_cgroup *memcg = mem_cgroup_from_css(css); + if (memcg_online_kmem(memcg)) + goto remove_id; + /* * A memcg must be visible for expand_shrinker_info() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (alloc_shrinker_info(memcg)) { - mem_cgroup_id_remove(memcg); - return -ENOMEM; - } + if (alloc_shrinker_info(memcg)) + goto offline_kmem; /* Online state pins memcg ID, memcg ID pins CSS */ refcount_set(&memcg->id.ref, 1); css_get(css); return 0; +offline_kmem: + memcg_offline_kmem(memcg); +remove_id: + mem_cgroup_id_remove(memcg); + return -ENOMEM; } static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) @@ -5166,7 +5158,6 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); free_shrinker_info(memcg); - memcg_free_kmem(memcg); mem_cgroup_free(memcg); } From patchwork Tue May 11 10:46:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4B71C43461 for ; Tue, 11 May 2021 10:51:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 516B2613CD for ; Tue, 11 May 2021 10:51:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 516B2613CD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DA9B86B0073; Tue, 11 May 2021 06:51:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D3225940008; Tue, 11 May 2021 06:51:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAB6D940007; Tue, 11 May 2021 06:51:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0224.hostedemail.com [216.40.44.224]) by kanga.kvack.org (Postfix) with ESMTP id 9E3926B0073 for ; Tue, 11 May 2021 06:51:53 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6644499B3 for ; Tue, 11 May 2021 10:51:53 +0000 (UTC) X-FDA: 78128634906.28.65C3ECA Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf24.hostedemail.com (Postfix) with ESMTP id E4AE9A0009F4 for ; Tue, 11 May 2021 10:51:38 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id z18so7039856plg.8 for ; Tue, 11 May 2021 03:51:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XKsKB4GYgRd6f8D/mWUIDZKZ0++0gc+VH9lM9TpP6RU=; b=VVp+A4ZecBFGFSRbIs1i2QqI6JPeI905+XDW/+4NUVYBpfJKdo2mbnRAZqBaYDT4fj 4z4sAgYuQMO2Gzd88guaqjzr7QsSzWk0M6u3aSFhwuIAnhT5BWQL/0cgKQPTOI5FsaUz /gzItuJCZ+WqT7RPthjOskKDSNBtyhJ8fagVIok9ix6HJrCNF0IioVjIH4hIUBZQtzQP +jY9uc0H9L0xxB0RWb1XZatZrdQmzbCtqEVPUBVor82Q9kSw1DprU++yxJg4Ht9CyskU T1sCsNzpcnN61qhYTe6qncS6dSI8vpgLFlRmEJ5qpeRyJ1pQjYp/3sFEKP5R1+o1JB57 mjxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XKsKB4GYgRd6f8D/mWUIDZKZ0++0gc+VH9lM9TpP6RU=; b=IQERA70zxmRvRRwECBeFe47wjAy9GGupThR9Iw7+r3HKYRefipHzFCnFwVknd6hlyj kOKIUo5Ky/MgJE9yvfnQ+Fb+mSNKBhr2gQRseqGtk5HPi/hjdg/vMYYC7NgLwEgTndJY vBkgOitqQ/ds80en1weXDAHUargGmYx1PoaX80nTesDhNUq+KHRLMgWvcqsmhr2CzO0M TzNl2tOyu+DEAUXX8agzQU4FrvUMa/yj/+7sps6oE+0i7ccAYz1huoP/41WnjJWMSgxn 8EFLV5JFq3oy90hg4Uc4637rjg/6Or2gQIbI/lOJ0lGJRY+fbbz/4zrOjqrknR9D055g Vo4g== X-Gm-Message-State: AOAM531OMK00ZsWhpckhjwxNjpVVjbON5q8VkplH6Dt1iF8dA1brxvo0 kUOH/WRwaI4Ro5bRSDCsrr3CNg== X-Google-Smtp-Source: ABdhPJydRosyrrbHMxbhHLsJ0w0mphTejG9nrk4d+o8QXBnZoaP/o2HTRqwJkBfCPH2wNNO8V3cQCg== X-Received: by 2002:a17:902:e8cb:b029:ee:f963:4fd8 with SMTP id v11-20020a170902e8cbb02900eef9634fd8mr28864251plg.40.1620730312287; Tue, 11 May 2021 03:51:52 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.51.45 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:51:51 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 05/17] mm: list_lru: remove holding lru node lock Date: Tue, 11 May 2021 18:46:35 +0800 Message-Id: <20210511104647.604-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E4AE9A0009F4 Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=VVp+A4Ze; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf24.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam03 X-Stat-Signature: hp7cducnnpypqi5dxbmokf8gpxaatrc3 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf24; identity=mailfrom; envelope-from=""; helo=mail-pl1-f171.google.com; client-ip=209.85.214.171 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730298-595992 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since commit e5bc3af7734f ("rcu: Consolidate PREEMPT and !PREEMPT synchronize_rcu()"), the critical section of spinlock can serve as RCU read-side critical section. So we can remove redundant locking from memcg_update_list_lru_node(). Signed-off-by: Muchun Song --- mm/list_lru.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index 4962d48d4410..e86d4d055d3c 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -403,18 +403,9 @@ static int memcg_update_list_lru_node(struct list_lru_node *nlru, memcpy(&new->lru, &old->lru, old_size * sizeof(void *)); - /* - * The locking below allows readers that hold nlru->lock avoid taking - * rcu_read_lock (see list_lru_from_memcg_idx). - * - * Since list_lru_{add,del} may be called under an IRQ-safe lock, - * we have to use IRQ-safe primitives here to avoid deadlock. - */ - spin_lock_irq(&nlru->lock); rcu_assign_pointer(nlru->memcg_lrus, new); - spin_unlock_irq(&nlru->lock); - kvfree_rcu(old, rcu); + return 0; } From patchwork Tue May 11 10:46:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF730C433B4 for ; Tue, 11 May 2021 10:52:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3C3C461132 for ; Tue, 11 May 2021 10:52:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3C3C461132 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BD3E66B0074; Tue, 11 May 2021 06:52:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BAACE940008; Tue, 11 May 2021 06:52:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A23DB940007; Tue, 11 May 2021 06:52:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0023.hostedemail.com [216.40.44.23]) by kanga.kvack.org (Postfix) with ESMTP id 88F2F6B0074 for ; Tue, 11 May 2021 06:52:00 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 4ACFB99B3 for ; Tue, 11 May 2021 10:52:00 +0000 (UTC) X-FDA: 78128635200.25.E733F28 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf29.hostedemail.com (Postfix) with ESMTP id 70B8E3CA for ; Tue, 11 May 2021 10:51:53 +0000 (UTC) Received: by mail-pl1-f172.google.com with SMTP id z18so7039966plg.8 for ; Tue, 11 May 2021 03:51:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=11A9hpa5Km9RVMW4601u7SbycLIFDxGvAvOSrlim2Xc=; b=yvPuzFw8N21facJrGJT4nrD88AXfMowTGYWSysddZzFx/clZ7i+LX0/i80M1CQJC7s hDg1wMBVRyNqoW+qSMU0KfF4kqCu3tdVj3ab7VQwAZIc0K+VgO6dmODMUDM60L51DXi+ 9YcldnlL3DtH7o5LHcBhHQcmGVMH6UvDbERiGnRtomQvVaFJEVqN/hZP/teDZ3ziC5WH WAFw9a80wFGLpiIUUXa8sX4iu5e0M4qf0FZDf85MS2U3v9D1Xd8y9wXYyClWDPBjcthL y5BwP2U8SnzmIrEeiyadG1itAlx9lLrZzT2WpzdEmAdRGrBOKlk+nPAMLvmoMlTGds3k WN8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=11A9hpa5Km9RVMW4601u7SbycLIFDxGvAvOSrlim2Xc=; b=GlMx2QZANhmWvUwe1pzwQbI9IYD0XZmYA4IACGyF0lmXLGbltbeUkE+g6wM/9Y2pEy 8vV70P6cZC1B9QhCfvCaPBcrn4ZfKm9iHs+jc/BOy6Iyn3zb1QL/TOdq+lBFeyVXI2Kj mwAhbSvhAuvD47h9L2oOxgvg1vI/fSYrFFhSXpEvmJu65GeObjo9SfOwoTEvwfd4+wXo dtawAYhlYYUgAFRuW524tatdI9OQ6A8Cq/iPFrj7wT/wO32crDibDwGEzUI3reBG/u5g 8qoLt6LLCSwgCekb/cnWlCNGl2EW/ufMI+oLDRwilMmjeIVVoetoUs3YSmvAwylVCYqI oEjg== X-Gm-Message-State: AOAM533S9NjQduMKrt3tZdnTDPmnkb8tRs0PjQYY+we0q9qhFRmjoOtq G9GD6VzYdyEXLZq4f+fx1lM0Eg== X-Google-Smtp-Source: ABdhPJxl/y4TP7fRrZPnVbX3UHYYrhhayxnsGFGvdtwDYk3OoIIY/8ZJmdVs7gcENlBfgEL4O2dyJg== X-Received: by 2002:a17:90a:2a84:: with SMTP id j4mr32249851pjd.42.1620730319173; Tue, 11 May 2021 03:51:59 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.51.52 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:51:58 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 06/17] mm: list_lru: only add the memcg aware lrus to the list Date: Tue, 11 May 2021 18:46:36 +0800 Message-Id: <20210511104647.604-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=yvPuzFw8; spf=pass (imf29.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 70B8E3CA X-Stat-Signature: bnyg5wh531wusdcq4dq8shbc4xcktij9 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=mail-pl1-f172.google.com; client-ip=209.85.214.172 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730313-877592 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We need to traverse every lru in the list_lrus in some routines, but skip the non memcg aware lru. Actually, we can only add the memcg aware lrus to the list. This can be efficient. Signed-off-by: Muchun Song --- mm/list_lru.c | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index e86d4d055d3c..bed699edabe5 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -18,8 +18,16 @@ static LIST_HEAD(list_lrus); static DEFINE_MUTEX(list_lrus_mutex); +static inline bool list_lru_memcg_aware(struct list_lru *lru) +{ + return lru->memcg_aware; +} + static void list_lru_register(struct list_lru *lru) { + if (!list_lru_memcg_aware(lru)) + return; + mutex_lock(&list_lrus_mutex); list_add(&lru->list, &list_lrus); mutex_unlock(&list_lrus_mutex); @@ -27,6 +35,9 @@ static void list_lru_register(struct list_lru *lru) static void list_lru_unregister(struct list_lru *lru) { + if (!list_lru_memcg_aware(lru)) + return; + mutex_lock(&list_lrus_mutex); list_del(&lru->list); mutex_unlock(&list_lrus_mutex); @@ -37,11 +48,6 @@ static int lru_shrinker_id(struct list_lru *lru) return lru->shrinker_id; } -static inline bool list_lru_memcg_aware(struct list_lru *lru) -{ - return lru->memcg_aware; -} - static inline struct list_lru_one * list_lru_from_memcg_idx(struct list_lru_node *nlru, int idx) { @@ -460,9 +466,6 @@ static int memcg_update_list_lru(struct list_lru *lru, { int i; - if (!list_lru_memcg_aware(lru)) - return 0; - for_each_node(i) { if (memcg_update_list_lru_node(&lru->node[i], old_size, new_size)) @@ -485,9 +488,6 @@ static void memcg_cancel_update_list_lru(struct list_lru *lru, { int i; - if (!list_lru_memcg_aware(lru)) - return; - for_each_node(i) memcg_cancel_update_list_lru_node(&lru->node[i], old_size, new_size); @@ -546,9 +546,6 @@ static void memcg_drain_list_lru(struct list_lru *lru, { int i; - if (!list_lru_memcg_aware(lru)) - return; - for_each_node(i) memcg_drain_list_lru_node(lru, i, src_idx, dst_memcg); } From patchwork Tue May 11 10:46:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C687C433B4 for ; Tue, 11 May 2021 10:52:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0743761370 for ; Tue, 11 May 2021 10:52:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0743761370 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 89F53940007; Tue, 11 May 2021 06:52:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 875A86B0078; Tue, 11 May 2021 06:52:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C8FB940007; Tue, 11 May 2021 06:52:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0037.hostedemail.com [216.40.44.37]) by kanga.kvack.org (Postfix) with ESMTP id 510BF6B0075 for ; Tue, 11 May 2021 06:52:08 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id F2B01180ACF9A for ; Tue, 11 May 2021 10:52:07 +0000 (UTC) X-FDA: 78128635494.28.80DA08D Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf08.hostedemail.com (Postfix) with ESMTP id 9AA6680192F3 for ; Tue, 11 May 2021 10:51:40 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id t4so10634997plc.6 for ; Tue, 11 May 2021 03:52:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jeTdKVRo5QuCEVZvKj9gqqAVw4NEW3vKLIlTKdR8pEg=; b=WOPHoX6BIlEp8VCErN6A7MSF0B/or02EmfQESVTCay6cbvtkIwijQDJkB5QcnN8yaG pknIj8oCwjzhmrUjKxfue2hrDZM/dwXZO7XNwqGLyEstGsyT8pEiabjUg5vcuo4Yr1rE UUiKYUR2S9AeIAqGsg8fcxjOHtwtgH0lAN2MAUXXngRNP2NAHafIGo3tG7RPSeAI/Ihr V/t0txXkFOvLBanEtbQIijkBBQ+4ECWiKOvds4GrV0QtWdiyYxi+d1FjOEM5sJkPy9Xe BHQ8DBvNd+MbpDCI7My51VARp9Oup4dbaNVX9bGjRueqSESvyyh2wlc293POFgkgeGMp BXeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jeTdKVRo5QuCEVZvKj9gqqAVw4NEW3vKLIlTKdR8pEg=; b=VVUCyoppNpOmeJdAVghpRNkkMABaKKZ3k3mKGaaIdxNHmiHR82boZA7oKewSS6Z3wC CT+pj+zL+mq+y7EbibAZodNZAG7T+cTddaV1y2aQbfX+/LoA5rtpQq7ZcKMLh7bGUjys ayF4LFddpIHbJLKdLbjm3VR/6uvJW28Vs8MkLHuwzS8SQdRZoLUaWPRIRTZR4QBcGFFd UnZYM7Ekrw4yr0vpUKF/uDZSWvd1h8li4wcY+vy1HJe8kZ6ltlrndD1Kq89EcM271aYl lXZ5MrTkkj19N55vHqAHQ5vwMhAxeBQCUZoiFDqRwQhoHUpn42mLUA2XQEekrblGUdu0 mPyg== X-Gm-Message-State: AOAM531ToLNBhBNPSb1S5x1SoErlzq4fjZbiuk6TfuKgRaehoTRbX4va QvSOb6pJ3G1P984fmOmZYRyK4Q== X-Google-Smtp-Source: ABdhPJxot+KaqQBQ93sZ5shlFuZodNdl7RCg71lYf+C98aDm4ASSHdeHf7imeXBDKqNZSd+EyI7mwg== X-Received: by 2002:a17:90a:1d41:: with SMTP id u1mr19116963pju.20.1620730326736; Tue, 11 May 2021 03:52:06 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.51.59 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:52:06 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 07/17] mm: list_lru: optimize the array of per memcg lists Date: Tue, 11 May 2021 18:46:37 +0800 Message-Id: <20210511104647.604-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9AA6680192F3 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=WOPHoX6B; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf08.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam03 X-Stat-Signature: kktr9h8i9s418bybuaoxg6cyk58cr83j Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf08; identity=mailfrom; envelope-from=""; helo=mail-pl1-f171.google.com; client-ip=209.85.214.171 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730300-626351 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The list_lru use an array to store the list_lru_one pointers, which is per memcg per node. What if we run 10k+ containers in the system? The size of the arrays for every list lru can be 10k * number_of_node * sizeof(void *). We can convert the array to per memcg instead of per memcg per node. It can help us save memory especially when there are many nodes in the system. And also simplify the code. Signed-off-by: Muchun Song --- include/linux/list_lru.h | 17 +++-- mm/list_lru.c | 192 +++++++++++++++++------------------------------ 2 files changed, 80 insertions(+), 129 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 9dcaa3e582c9..228bef9fdd0b 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -31,10 +31,15 @@ struct list_lru_one { long nr_items; }; +struct list_lru_per_memcg { + /* array of per cgroup per node lists, indexed by node id */ + struct list_lru_one nodes[0]; +}; + struct list_lru_memcg { - struct rcu_head rcu; + struct rcu_head rcu; /* array of per cgroup lists, indexed by memcg_cache_id */ - struct list_lru_one *lru[]; + struct list_lru_per_memcg *lrus[0]; }; struct list_lru_node { @@ -42,11 +47,7 @@ struct list_lru_node { spinlock_t lock; /* global list, used for the root cgroup in cgroup aware lrus */ struct list_lru_one lru; -#ifdef CONFIG_MEMCG_KMEM - /* for cgroup aware lrus points to per cgroup lists, otherwise NULL */ - struct list_lru_memcg __rcu *memcg_lrus; -#endif - long nr_items; + long nr_items; } ____cacheline_aligned_in_smp; struct list_lru { @@ -55,6 +56,8 @@ struct list_lru { struct list_head list; int shrinker_id; bool memcg_aware; + /* for cgroup aware lrus points to per cgroup lists, otherwise NULL */ + struct list_lru_memcg __rcu *memcg_lrus; #endif }; diff --git a/mm/list_lru.c b/mm/list_lru.c index bed699edabe5..57c55916cc1c 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -49,35 +49,38 @@ static int lru_shrinker_id(struct list_lru *lru) } static inline struct list_lru_one * -list_lru_from_memcg_idx(struct list_lru_node *nlru, int idx) +list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) { struct list_lru_memcg *memcg_lrus; + struct list_lru_node *nlru = &lru->node[nid]; + /* * Either lock or RCU protects the array of per cgroup lists - * from relocation (see memcg_update_list_lru_node). + * from relocation (see memcg_update_list_lru). */ - memcg_lrus = rcu_dereference_check(nlru->memcg_lrus, + memcg_lrus = rcu_dereference_check(lru->memcg_lrus, lockdep_is_held(&nlru->lock)); if (memcg_lrus && idx >= 0) - return memcg_lrus->lru[idx]; + return &memcg_lrus->lrus[idx]->nodes[nid]; return &nlru->lru; } static inline struct list_lru_one * -list_lru_from_kmem(struct list_lru_node *nlru, void *ptr, +list_lru_from_kmem(struct list_lru *lru, int nid, void *ptr, struct mem_cgroup **memcg_ptr) { + struct list_lru_node *nlru = &lru->node[nid]; struct list_lru_one *l = &nlru->lru; struct mem_cgroup *memcg = NULL; - if (!nlru->memcg_lrus) + if (!lru->memcg_lrus) goto out; memcg = mem_cgroup_from_obj(ptr); if (!memcg) goto out; - l = list_lru_from_memcg_idx(nlru, memcg_cache_id(memcg)); + l = list_lru_from_memcg_idx(lru, nid, memcg_cache_id(memcg)); out: if (memcg_ptr) *memcg_ptr = memcg; @@ -103,18 +106,18 @@ static inline bool list_lru_memcg_aware(struct list_lru *lru) } static inline struct list_lru_one * -list_lru_from_memcg_idx(struct list_lru_node *nlru, int idx) +list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) { - return &nlru->lru; + return &lru->node[nid].lru; } static inline struct list_lru_one * -list_lru_from_kmem(struct list_lru_node *nlru, void *ptr, +list_lru_from_kmem(struct list_lru *lru, int nid, void *ptr, struct mem_cgroup **memcg_ptr) { if (memcg_ptr) *memcg_ptr = NULL; - return &nlru->lru; + return &lru->node[nid].lru; } #endif /* CONFIG_MEMCG_KMEM */ @@ -127,7 +130,7 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item) spin_lock(&nlru->lock); if (list_empty(item)) { - l = list_lru_from_kmem(nlru, item, &memcg); + l = list_lru_from_kmem(lru, nid, item, &memcg); list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) @@ -150,7 +153,7 @@ bool list_lru_del(struct list_lru *lru, struct list_head *item) spin_lock(&nlru->lock); if (!list_empty(item)) { - l = list_lru_from_kmem(nlru, item, NULL); + l = list_lru_from_kmem(lru, nid, item, NULL); list_del_init(item); l->nr_items--; nlru->nr_items--; @@ -180,12 +183,11 @@ EXPORT_SYMBOL_GPL(list_lru_isolate_move); unsigned long list_lru_count_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg) { - struct list_lru_node *nlru = &lru->node[nid]; struct list_lru_one *l; long count; rcu_read_lock(); - l = list_lru_from_memcg_idx(nlru, memcg_cache_id(memcg)); + l = list_lru_from_memcg_idx(lru, nid, memcg_cache_id(memcg)); count = READ_ONCE(l->nr_items); rcu_read_unlock(); @@ -206,16 +208,16 @@ unsigned long list_lru_count_node(struct list_lru *lru, int nid) EXPORT_SYMBOL_GPL(list_lru_count_node); static unsigned long -__list_lru_walk_one(struct list_lru_node *nlru, int memcg_idx, +__list_lru_walk_one(struct list_lru *lru, int nid, int memcg_idx, list_lru_walk_cb isolate, void *cb_arg, unsigned long *nr_to_walk) { - + struct list_lru_node *nlru = &lru->node[nid]; struct list_lru_one *l; struct list_head *item, *n; unsigned long isolated = 0; - l = list_lru_from_memcg_idx(nlru, memcg_idx); + l = list_lru_from_memcg_idx(lru, nid, memcg_idx); restart: list_for_each_safe(item, n, &l->list) { enum lru_status ret; @@ -272,8 +274,8 @@ list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg, unsigned long ret; spin_lock(&nlru->lock); - ret = __list_lru_walk_one(nlru, memcg_cache_id(memcg), isolate, cb_arg, - nr_to_walk); + ret = __list_lru_walk_one(lru, nid, memcg_cache_id(memcg), isolate, + cb_arg, nr_to_walk); spin_unlock(&nlru->lock); return ret; } @@ -288,8 +290,8 @@ list_lru_walk_one_irq(struct list_lru *lru, int nid, struct mem_cgroup *memcg, unsigned long ret; spin_lock_irq(&nlru->lock); - ret = __list_lru_walk_one(nlru, memcg_cache_id(memcg), isolate, cb_arg, - nr_to_walk); + ret = __list_lru_walk_one(lru, nid, memcg_cache_id(memcg), isolate, + cb_arg, nr_to_walk); spin_unlock_irq(&nlru->lock); return ret; } @@ -308,7 +310,7 @@ unsigned long list_lru_walk_node(struct list_lru *lru, int nid, struct list_lru_node *nlru = &lru->node[nid]; spin_lock(&nlru->lock); - isolated += __list_lru_walk_one(nlru, memcg_idx, + isolated += __list_lru_walk_one(lru, nid, memcg_idx, isolate, cb_arg, nr_to_walk); spin_unlock(&nlru->lock); @@ -328,169 +330,115 @@ static void init_one_lru(struct list_lru_one *l) } #ifdef CONFIG_MEMCG_KMEM -static void __memcg_destroy_list_lru_node(struct list_lru_memcg *memcg_lrus, - int begin, int end) +static void memcg_destroy_list_lru_range(struct list_lru_memcg *memcg_lrus, + int begin, int end) { int i; for (i = begin; i < end; i++) - kfree(memcg_lrus->lru[i]); + kfree(memcg_lrus->lrus[i]); } -static int __memcg_init_list_lru_node(struct list_lru_memcg *memcg_lrus, - int begin, int end) +static int memcg_init_list_lru_range(struct list_lru_memcg *memcg_lrus, + int begin, int end) { int i; for (i = begin; i < end; i++) { - struct list_lru_one *l; + int nid; + struct list_lru_per_memcg *lru; - l = kmalloc(sizeof(struct list_lru_one), GFP_KERNEL); - if (!l) + lru = kmalloc(sizeof(*lru) + nr_node_ids * sizeof(lru->nodes[0]), + GFP_KERNEL); + if (!lru) goto fail; - init_one_lru(l); - memcg_lrus->lru[i] = l; + for_each_node(nid) + init_one_lru(&lru->nodes[nid]); + memcg_lrus->lrus[i] = lru; } return 0; fail: - __memcg_destroy_list_lru_node(memcg_lrus, begin, i); + memcg_destroy_list_lru_range(memcg_lrus, begin, i); return -ENOMEM; } -static int memcg_init_list_lru_node(struct list_lru_node *nlru) +static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) { struct list_lru_memcg *memcg_lrus; int size = memcg_nr_cache_ids; + lru->memcg_aware = memcg_aware; + if (!memcg_aware) + return 0; + memcg_lrus = kvmalloc(sizeof(*memcg_lrus) + - size * sizeof(void *), GFP_KERNEL); + size * sizeof(memcg_lrus->lrus[0]), GFP_KERNEL); if (!memcg_lrus) return -ENOMEM; - if (__memcg_init_list_lru_node(memcg_lrus, 0, size)) { + if (memcg_init_list_lru_range(memcg_lrus, 0, size)) { kvfree(memcg_lrus); return -ENOMEM; } - RCU_INIT_POINTER(nlru->memcg_lrus, memcg_lrus); + RCU_INIT_POINTER(lru->memcg_lrus, memcg_lrus); return 0; } -static void memcg_destroy_list_lru_node(struct list_lru_node *nlru) +static void memcg_destroy_list_lru(struct list_lru *lru) { struct list_lru_memcg *memcg_lrus; + + if (!list_lru_memcg_aware(lru)) + return; + /* * This is called when shrinker has already been unregistered, * and nobody can use it. So, there is no need to use kvfree_rcu(). */ - memcg_lrus = rcu_dereference_protected(nlru->memcg_lrus, true); - __memcg_destroy_list_lru_node(memcg_lrus, 0, memcg_nr_cache_ids); + memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, true); + memcg_destroy_list_lru_range(memcg_lrus, 0, memcg_nr_cache_ids); kvfree(memcg_lrus); } -static int memcg_update_list_lru_node(struct list_lru_node *nlru, - int old_size, int new_size) +static int memcg_update_list_lru(struct list_lru *lru, int old_size, int new_size) { struct list_lru_memcg *old, *new; BUG_ON(old_size > new_size); - old = rcu_dereference_protected(nlru->memcg_lrus, + old = rcu_dereference_protected(lru->memcg_lrus, lockdep_is_held(&list_lrus_mutex)); - new = kvmalloc(sizeof(*new) + new_size * sizeof(void *), GFP_KERNEL); + new = kvmalloc(sizeof(*new) + new_size * sizeof(new->lrus[0]), GFP_KERNEL); if (!new) return -ENOMEM; - if (__memcg_init_list_lru_node(new, old_size, new_size)) { + if (memcg_init_list_lru_range(new, old_size, new_size)) { kvfree(new); return -ENOMEM; } - memcpy(&new->lru, &old->lru, old_size * sizeof(void *)); + memcpy(&new->lrus, &old->lrus, old_size * sizeof(new->lrus[0])); - rcu_assign_pointer(nlru->memcg_lrus, new); + rcu_assign_pointer(lru->memcg_lrus, new); kvfree_rcu(old, rcu); return 0; } -static void memcg_cancel_update_list_lru_node(struct list_lru_node *nlru, - int old_size, int new_size) -{ - struct list_lru_memcg *memcg_lrus; - - memcg_lrus = rcu_dereference_protected(nlru->memcg_lrus, - lockdep_is_held(&list_lrus_mutex)); - /* do not bother shrinking the array back to the old size, because we - * cannot handle allocation failures here */ - __memcg_destroy_list_lru_node(memcg_lrus, old_size, new_size); -} - -static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) -{ - int i; - - lru->memcg_aware = memcg_aware; - - if (!memcg_aware) - return 0; - - for_each_node(i) { - if (memcg_init_list_lru_node(&lru->node[i])) - goto fail; - } - return 0; -fail: - for (i = i - 1; i >= 0; i--) { - if (!lru->node[i].memcg_lrus) - continue; - memcg_destroy_list_lru_node(&lru->node[i]); - } - return -ENOMEM; -} - -static void memcg_destroy_list_lru(struct list_lru *lru) -{ - int i; - - if (!list_lru_memcg_aware(lru)) - return; - - for_each_node(i) - memcg_destroy_list_lru_node(&lru->node[i]); -} - -static int memcg_update_list_lru(struct list_lru *lru, - int old_size, int new_size) -{ - int i; - - for_each_node(i) { - if (memcg_update_list_lru_node(&lru->node[i], - old_size, new_size)) - goto fail; - } - return 0; -fail: - for (i = i - 1; i >= 0; i--) { - if (!lru->node[i].memcg_lrus) - continue; - - memcg_cancel_update_list_lru_node(&lru->node[i], - old_size, new_size); - } - return -ENOMEM; -} - static void memcg_cancel_update_list_lru(struct list_lru *lru, int old_size, int new_size) { - int i; + struct list_lru_memcg *memcg_lrus; - for_each_node(i) - memcg_cancel_update_list_lru_node(&lru->node[i], - old_size, new_size); + memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, + lockdep_is_held(&list_lrus_mutex)); + /* + * Do not bother shrinking the array back to the old size, because we + * cannot handle allocation failures here. + */ + memcg_destroy_list_lru_range(memcg_lrus, old_size, new_size); } int memcg_update_all_list_lrus(int new_size) @@ -527,8 +475,8 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, */ spin_lock_irq(&nlru->lock); - src = list_lru_from_memcg_idx(nlru, src_idx); - dst = list_lru_from_memcg_idx(nlru, dst_idx); + src = list_lru_from_memcg_idx(lru, nid, src_idx); + dst = list_lru_from_memcg_idx(lru, nid, dst_idx); list_splice_init(&src->list, &dst->list); From patchwork Tue May 11 10:46:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250611 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F6BAC43461 for ; Tue, 11 May 2021 10:52:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2F7D5611ED for ; Tue, 11 May 2021 10:52:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2F7D5611ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B8487940008; Tue, 11 May 2021 06:52:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B34866B0078; Tue, 11 May 2021 06:52:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D5AB940008; Tue, 11 May 2021 06:52:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0171.hostedemail.com [216.40.44.171]) by kanga.kvack.org (Postfix) with ESMTP id 7E1476B0075 for ; Tue, 11 May 2021 06:52:15 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 4356A8249980 for ; Tue, 11 May 2021 10:52:15 +0000 (UTC) X-FDA: 78128635830.19.35859EE Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf12.hostedemail.com (Postfix) with ESMTP id 1A6473DC for ; Tue, 11 May 2021 10:51:58 +0000 (UTC) Received: by mail-pj1-f49.google.com with SMTP id t2-20020a17090ae502b029015b0fbfbc50so909450pjy.3 for ; Tue, 11 May 2021 03:52:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tthoopsr5G2zL3tdafUCx6W9EvXjjGg7T7Xsd3uJmoE=; b=DgYc8YN57QkMjkN4APmwzgLRQgmYPr6XAFw2JfSibXgtfC++yeyD2TcEF54woH1iN2 n3zN6l/Hg1pIpYz72F+QESwZX61Xfx7RRwihrrrf+301eP5i0+TuQIlLLZCYHcPYMKEi sqROIWtzd+oljqufTEt62BJ2ZGFMQCcdQ/7NBm24/ZFNp18uXHsVWlyZJtrrcrGtBLe7 w+2xnmvaEXXaoTitxY1U0UcOOAmXKlF+xzrr/Fw3xbHDy/W+es+jMUhSgsoTuThZu+1x cPBKwUaBjcUcHmj8lVtNXisgc7gTBD+sc/LxGtS+WWGhyPrPvtMCPoENtEBVhLHHQ//a 5evg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tthoopsr5G2zL3tdafUCx6W9EvXjjGg7T7Xsd3uJmoE=; b=QHG/OZ5iEh51W+wGGc5cjFSUle3JOCb/6UIVL0DwuYU8D3lzVsf3pQDjJIdMnQ/OCw NtNEfetjf1aa1lsYkTaM58CFQpZ2jKdU6t/CwvL36Xwd9At70hRWXcH2m5/9LVtnObhl 0CmdkwI6WNrk2ZrDp7MGD9yw1YH9RjcLtU9nEoypcIOwUKWa0Xddre1mIVSLlUpmPZXb US6Azox2RE4xEND17b34iESBWPOwW+SL+lFHynP7RDQbffNOe/ziVWT8tjUKrFCnpNh4 msFK7MbNnalBiMttHaJmjf9y0xtoO0FEo+xjK13MN6QKR0F/hmEXkoDI7trY4vuiuq0X 2xLQ== X-Gm-Message-State: AOAM533uTDSm7T46sHMsbmMyprfROKdxDEogtkE7MBAvrjFK1h7mGSBz ICRCbBDHkG6Q2KamDY+lg3/sAQ== X-Google-Smtp-Source: ABdhPJywOW5mRiqRiX3AMrSpDX1DjzEOYzvjOm7zqQXU9JNa6vfWHW/5engdadWoRg8kIR60CTh+Ng== X-Received: by 2002:a17:90a:8902:: with SMTP id u2mr33070016pjn.143.1620730334181; Tue, 11 May 2021 03:52:14 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.52.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:52:13 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 08/17] mm: list_lru: remove memcg_aware from struct list_lru Date: Tue, 11 May 2021 18:46:38 +0800 Message-Id: <20210511104647.604-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1A6473DC Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=DgYc8YN5; spf=pass (imf12.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam04 X-Stat-Signature: 9mq7ajzrtdg4nhdam3muwf7a9korjsam Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf12; identity=mailfrom; envelope-from=""; helo=mail-pj1-f49.google.com; client-ip=209.85.216.49 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730318-284859 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We can use ->memcg_lrus to indicate if the lru is memcg aware. So ->memcg_aware is redundant and remove it. Signed-off-by: Muchun Song --- include/linux/list_lru.h | 1 - mm/list_lru.c | 9 +++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 228bef9fdd0b..00a13ad6128a 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -55,7 +55,6 @@ struct list_lru { #ifdef CONFIG_MEMCG_KMEM struct list_head list; int shrinker_id; - bool memcg_aware; /* for cgroup aware lrus points to per cgroup lists, otherwise NULL */ struct list_lru_memcg __rcu *memcg_lrus; #endif diff --git a/mm/list_lru.c b/mm/list_lru.c index 57c55916cc1c..cf17ff5a5940 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -20,7 +20,7 @@ static DEFINE_MUTEX(list_lrus_mutex); static inline bool list_lru_memcg_aware(struct list_lru *lru) { - return lru->memcg_aware; + return !!lru->memcg_lrus; } static void list_lru_register(struct list_lru *lru) @@ -73,7 +73,7 @@ list_lru_from_kmem(struct list_lru *lru, int nid, void *ptr, struct list_lru_one *l = &nlru->lru; struct mem_cgroup *memcg = NULL; - if (!lru->memcg_lrus) + if (!list_lru_memcg_aware(lru)) goto out; memcg = mem_cgroup_from_obj(ptr); @@ -368,9 +368,10 @@ static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) struct list_lru_memcg *memcg_lrus; int size = memcg_nr_cache_ids; - lru->memcg_aware = memcg_aware; - if (!memcg_aware) + if (!memcg_aware) { + lru->memcg_lrus = NULL; return 0; + } memcg_lrus = kvmalloc(sizeof(*memcg_lrus) + size * sizeof(memcg_lrus->lrus[0]), GFP_KERNEL); From patchwork Tue May 11 10:46:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8153BC433B4 for ; Tue, 11 May 2021 10:52:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DEA5561132 for ; Tue, 11 May 2021 10:52:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DEA5561132 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 718D06B0036; Tue, 11 May 2021 06:52:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6EFB86B0075; Tue, 11 May 2021 06:52:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 541DD6B0078; Tue, 11 May 2021 06:52:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0192.hostedemail.com [216.40.44.192]) by kanga.kvack.org (Postfix) with ESMTP id 324416B0036 for ; Tue, 11 May 2021 06:52:24 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id E5F3A180ACF9A for ; Tue, 11 May 2021 10:52:23 +0000 (UTC) X-FDA: 78128636166.12.19830E9 Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by imf11.hostedemail.com (Postfix) with ESMTP id 77C66200027C for ; Tue, 11 May 2021 10:52:03 +0000 (UTC) Received: by mail-pj1-f46.google.com with SMTP id v11-20020a17090a6b0bb029015cba7c6bdeso876498pjj.0 for ; Tue, 11 May 2021 03:52:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=P2RS8dO5UeAjVo7MTGjLxWlO3kHvdlom3A5w3Rrk59A=; b=cpuih/0EtSTX8oRzjZYkFzBFmTcUUacux67JTkxporxAHvbQjpxaYD66lV0Ft3ZRJs rQM19A9YgAPm2BhkAL2o5SQboVjyQX6tVywqZBV3YLOYvYd7iFfWW23926Z1XE9j8Tn7 TgXKyQoNOviUuEQYcHZLw56yUMMdF3S9P1+reUmmMTYuIfEdIDvdoKi9sPXHKAHg56pf xXx/c0G1O3b6suFZXHRClDJTB8YV9aEuP3weF+Cl+vpzqwA+TfOiEi/mE7WSpOdV5dMC b8EiUq7A0Uo0a+H+jsy488J8gHdYIh1r7EFejhYNntcq1LXG1b+OlzXCnZaTOohjWIdr 66UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=P2RS8dO5UeAjVo7MTGjLxWlO3kHvdlom3A5w3Rrk59A=; b=gszLCQePRka3Ay47kPUAhT2nLjleIZc7KjWSQisRtcZdFxVf41mZD14XXLGfqcnt9G RPPl1bKmteJm1+Tgm4EDnF0UNlPb/ndSw7rGY7XxGEZ0iWrtv0P089/SXWA87QO41y2K wmvjj9oEjAZzmKuzTR2WPi+btPBv7UGv7dLMksyv6KoPSY4QVmrOgeoR73yS0v0eG64Z 38XmOE+QCyovJIJkcB8xaNYQL9HG5zAxYmR3X5dNWCL8YSsWxGGd704V6dvBsofNvvC3 dFia6hYqFjyfd3SbnGXxuDfJJqojvYU9QQXDc/BUHAwhpTNPTiC4hTDqAB93Ena0VzsH PC1g== X-Gm-Message-State: AOAM533lLauQ9iPK72CpaBh6IqA9N+xJCnTb5HYTjhmqOiR9TH9tsRwS hAB/zQzkOtL7UDQ3SoqN4cyezA== X-Google-Smtp-Source: ABdhPJyDYPd0wZ5zYLIjWIFYoyh6d1gsXjfbZ0G/Sgne26mpdqYEpoOH6sigLlXgFT/bd438RC0enw== X-Received: by 2002:a17:90a:5e4a:: with SMTP id u10mr32812100pji.202.1620730341296; Tue, 11 May 2021 03:52:21 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.52.14 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:52:20 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 09/17] mm: introduce kmem_cache_alloc_lru Date: Tue, 11 May 2021 18:46:39 +0800 Message-Id: <20210511104647.604-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b="cpuih/0E"; spf=pass (imf11.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 77C66200027C X-Stat-Signature: ks3p16ufm7mz5f985n3qxraki6so3jbt Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf11; identity=mailfrom; envelope-from=""; helo=mail-pj1-f46.google.com; client-ip=209.85.216.46 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730323-798840 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We currently allocate scope for every memcg to be able to tracked on every not on every superblock instantiated in the system, regardless of whether that superblock is even accessible to that memcg. These huge memcg counts come from container hosts where memcgs are confined to just a small subset of the total number of superblocks that instantiated at any given point in time. For these systems with huge container counts, list_lru does not need the capability of tracking every memcg on every superblock. What it comes down to is that adding the memcg to the list_lru at the first insert. So introduce kmem_cache_alloc_lru to allocate objects and its list_lru. In the later patch, we will convert all inode and dentry allocation from kmem_cache_alloc to kmem_cache_alloc_lru. Signed-off-by: Muchun Song --- include/linux/list_lru.h | 3 ++ include/linux/memcontrol.h | 14 ++++++ include/linux/slab.h | 4 ++ mm/list_lru.c | 116 ++++++++++++++++++++++++++++++++++++++++----- mm/memcontrol.c | 14 ------ mm/slab.c | 39 ++++++++++----- mm/slab.h | 17 ++++++- mm/slub.c | 42 ++++++++++------ 8 files changed, 197 insertions(+), 52 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 00a13ad6128a..2b2fdb4de96a 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -55,11 +55,14 @@ struct list_lru { #ifdef CONFIG_MEMCG_KMEM struct list_head list; int shrinker_id; + /* protects ->memcg_lrus->lrus[i] */ + spinlock_t lock; /* for cgroup aware lrus points to per cgroup lists, otherwise NULL */ struct list_lru_memcg __rcu *memcg_lrus; #endif }; +int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t gfp); void list_lru_destroy(struct list_lru *lru); int __list_lru_init(struct list_lru *lru, bool memcg_aware, struct lock_class_key *key, struct shrinker *shrinker); diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 6350c563c7b8..96a85cff248a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -517,6 +517,20 @@ static inline struct mem_cgroup *page_memcg_check(struct page *page) return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } +static inline struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *objcg) +{ + struct mem_cgroup *memcg; + + rcu_read_lock(); +retry: + memcg = obj_cgroup_memcg(objcg); + if (unlikely(!css_tryget(&memcg->css))) + goto retry; + rcu_read_unlock(); + + return memcg; +} + #ifdef CONFIG_MEMCG_KMEM /* * PageMemcgKmem - check if the page has MemcgKmem flag set diff --git a/include/linux/slab.h b/include/linux/slab.h index 2e09a0ac5d60..88aac607e968 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -392,8 +392,12 @@ static __always_inline unsigned int kmalloc_index(size_t size) } #endif /* !CONFIG_SLOB */ +struct list_lru; + void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __malloc; void *kmem_cache_alloc(struct kmem_cache *, gfp_t flags) __assume_slab_alignment __malloc; +void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, + gfp_t gfpflags) __assume_slab_alignment __malloc; void kmem_cache_free(struct kmem_cache *, void *); /* diff --git a/mm/list_lru.c b/mm/list_lru.c index cf17ff5a5940..182cb2a2e64f 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -339,23 +339,30 @@ static void memcg_destroy_list_lru_range(struct list_lru_memcg *memcg_lrus, kfree(memcg_lrus->lrus[i]); } +static struct list_lru_per_memcg *list_lru_per_memcg_alloc(gfp_t gfp) +{ + int nid; + struct list_lru_per_memcg *lru; + + lru = kmalloc(sizeof(*lru) + nr_node_ids * sizeof(lru->nodes[0]), gfp); + if (!lru) + return NULL; + + for_each_node(nid) + init_one_lru(&lru->nodes[nid]); + + return lru; +} + static int memcg_init_list_lru_range(struct list_lru_memcg *memcg_lrus, int begin, int end) { int i; for (i = begin; i < end; i++) { - int nid; - struct list_lru_per_memcg *lru; - - lru = kmalloc(sizeof(*lru) + nr_node_ids * sizeof(lru->nodes[0]), - GFP_KERNEL); - if (!lru) + memcg_lrus->lrus[i] = list_lru_per_memcg_alloc(GFP_KERNEL); + if (!memcg_lrus->lrus[i]) goto fail; - - for_each_node(nid) - init_one_lru(&lru->nodes[nid]); - memcg_lrus->lrus[i] = lru; } return 0; fail: @@ -373,6 +380,8 @@ static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) return 0; } + spin_lock_init(&lru->lock); + memcg_lrus = kvmalloc(sizeof(*memcg_lrus) + size * sizeof(memcg_lrus->lrus[0]), GFP_KERNEL); if (!memcg_lrus) @@ -420,9 +429,11 @@ static int memcg_update_list_lru(struct list_lru *lru, int old_size, int new_siz return -ENOMEM; } + spin_lock_irq(&lru->lock); memcpy(&new->lrus, &old->lrus, old_size * sizeof(new->lrus[0])); - rcu_assign_pointer(lru->memcg_lrus, new); + spin_unlock_irq(&lru->lock); + kvfree_rcu(old, rcu); return 0; @@ -508,6 +519,89 @@ void memcg_drain_all_list_lrus(int src_idx, struct mem_cgroup *dst_memcg) memcg_drain_list_lru(lru, src_idx, dst_memcg); mutex_unlock(&list_lrus_mutex); } + +static bool list_lru_per_memcg_allocated(struct list_lru *lru, + struct mem_cgroup *memcg) +{ + struct list_lru_memcg *memcg_lrus; + int idx = memcg_cache_id(memcg); + + if (unlikely(idx < 0)) + return true; + + rcu_read_lock(); + memcg_lrus = rcu_dereference(lru->memcg_lrus); + if (memcg_lrus->lrus[idx]) { + rcu_read_unlock(); + return true; + } + rcu_read_unlock(); + + return false; +} + +int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t gfp) +{ + unsigned long flags; + struct list_lru_memcg *memcg_lrus; + int i; + + struct list_lru_memcg { + struct list_lru_per_memcg *mlru; + struct mem_cgroup *memcg; + } *table; + + if (!list_lru_memcg_aware(lru)) + return 0; + + if (list_lru_per_memcg_allocated(lru, memcg)) + return 0; + + /* + * The allocated list_lru_per_memcg array is not accounted directly. + * Moreover, it should not come from DMA buffer and is not readily + * reclaimable. So those GFP bits should be masked off. + */ + gfp &= ~(__GFP_DMA | __GFP_RECLAIMABLE | __GFP_ACCOUNT | __GFP_ZERO); + table = kmalloc_array(memcg->css.cgroup->level, sizeof(*table), gfp); + if (!table) + return -ENOMEM; + + /* + * Because the list_lru can be reparented to the parent cgroup's + * list_lru, we should make sure that this cgroup and all its + * ancestors have allocated list_lru_per_memcg. + */ + for (i = 0; memcg; memcg = parent_mem_cgroup(memcg), i++) { + if (list_lru_per_memcg_allocated(lru, memcg)) + break; + + table[i].memcg = memcg; + table[i].mlru = list_lru_per_memcg_alloc(gfp); + if (!table[i].mlru) { + while (i--) + kfree(table[i].mlru); + kfree(table); + return -ENOMEM; + } + } + + spin_lock_irqsave(&lru->lock, flags); + memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, true); + while (i--) { + int idx = memcg_cache_id(table[i].memcg); + + if (memcg_lrus->lrus[idx]) + kfree(table[i].mlru); + else + memcg_lrus->lrus[idx] = table[i].mlru; + } + spin_unlock_irqrestore(&lru->lock, flags); + + kfree(table); + + return 0; +} #else static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0d1c09873bad..6aace39fd1a4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2724,20 +2724,6 @@ static void commit_charge(struct page *page, struct mem_cgroup *memcg) page->memcg_data = (unsigned long)memcg; } -static struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *objcg) -{ - struct mem_cgroup *memcg; - - rcu_read_lock(); -retry: - memcg = obj_cgroup_memcg(objcg); - if (unlikely(!css_tryget(&memcg->css))) - goto retry; - rcu_read_unlock(); - - return memcg; -} - #ifdef CONFIG_MEMCG_KMEM int memcg_alloc_page_obj_cgroups(struct page *page, struct kmem_cache *s, gfp_t gfp, bool new_page) diff --git a/mm/slab.c b/mm/slab.c index d0f725637663..9a001aabc77b 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3219,7 +3219,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_ bool init = false; flags &= gfp_allowed_mask; - cachep = slab_pre_alloc_hook(cachep, &objcg, 1, flags); + cachep = slab_pre_alloc_hook(cachep, NULL, &objcg, 1, flags); if (unlikely(!cachep)) return NULL; @@ -3295,7 +3295,8 @@ __do_cache_alloc(struct kmem_cache *cachep, gfp_t flags) #endif /* CONFIG_NUMA */ static __always_inline void * -slab_alloc(struct kmem_cache *cachep, gfp_t flags, size_t orig_size, unsigned long caller) +slab_alloc(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags, + size_t orig_size, unsigned long caller) { unsigned long save_flags; void *objp; @@ -3303,7 +3304,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, size_t orig_size, unsigned lo bool init = false; flags &= gfp_allowed_mask; - cachep = slab_pre_alloc_hook(cachep, &objcg, 1, flags); + cachep = slab_pre_alloc_hook(cachep, lru, &objcg, 1, flags); if (unlikely(!cachep)) return NULL; @@ -3492,6 +3493,18 @@ void ___cache_free(struct kmem_cache *cachep, void *objp, __free_one(ac, objp); } +static __always_inline +void *__kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, + gfp_t flags) +{ + void *ret = slab_alloc(cachep, lru, flags, cachep->object_size, _RET_IP_); + + trace_kmem_cache_alloc(_RET_IP_, ret, + cachep->object_size, cachep->size, flags); + + return ret; +} + /** * kmem_cache_alloc - Allocate an object * @cachep: The cache to allocate from. @@ -3504,15 +3517,17 @@ void ___cache_free(struct kmem_cache *cachep, void *objp, */ void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) { - void *ret = slab_alloc(cachep, flags, cachep->object_size, _RET_IP_); - - trace_kmem_cache_alloc(_RET_IP_, ret, - cachep->object_size, cachep->size, flags); - - return ret; + return __kmem_cache_alloc_lru(cachep, NULL, flags); } EXPORT_SYMBOL(kmem_cache_alloc); +void *kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, + gfp_t flags) +{ + return __kmem_cache_alloc_lru(cachep, lru, flags); +} +EXPORT_SYMBOL(kmem_cache_alloc_lru); + static __always_inline void cache_alloc_debugcheck_after_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p, unsigned long caller) @@ -3529,7 +3544,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, size_t i; struct obj_cgroup *objcg = NULL; - s = slab_pre_alloc_hook(s, &objcg, size, flags); + s = slab_pre_alloc_hook(s, NULL, &objcg, size, flags); if (!s) return 0; @@ -3570,7 +3585,7 @@ kmem_cache_alloc_trace(struct kmem_cache *cachep, gfp_t flags, size_t size) { void *ret; - ret = slab_alloc(cachep, flags, size, _RET_IP_); + ret = slab_alloc(cachep, NULL, flags, size, _RET_IP_); ret = kasan_kmalloc(cachep, ret, size, flags); trace_kmalloc(_RET_IP_, ret, @@ -3697,7 +3712,7 @@ static __always_inline void *__do_kmalloc(size_t size, gfp_t flags, cachep = kmalloc_slab(size, flags); if (unlikely(ZERO_OR_NULL_PTR(cachep))) return cachep; - ret = slab_alloc(cachep, flags, size, caller); + ret = slab_alloc(cachep, NULL, flags, size, caller); ret = kasan_kmalloc(cachep, ret, size, flags); trace_kmalloc(caller, ret, diff --git a/mm/slab.h b/mm/slab.h index a7268072f017..55e4dea357c3 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -46,6 +46,7 @@ struct kmem_cache { #include #include #include +#include /* * State of the slab allocator. @@ -262,6 +263,7 @@ static inline size_t obj_full_size(struct kmem_cache *s) * Returns false if the allocation should fail. */ static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, + struct list_lru *lru, struct obj_cgroup **objcgp, size_t objects, gfp_t flags) { @@ -277,6 +279,17 @@ static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, if (!objcg) return true; + if (lru) { + struct mem_cgroup *memcg = get_mem_cgroup_from_objcg(objcg); + + if (list_lru_memcg_alloc(lru, memcg, flags)) { + css_put(&memcg->css); + obj_cgroup_put(objcg); + return false; + } + css_put(&memcg->css); + } + if (obj_cgroup_charge(objcg, flags, objects * obj_full_size(s))) { obj_cgroup_put(objcg); return false; @@ -394,6 +407,7 @@ static inline void memcg_free_page_obj_cgroups(struct page *page) } static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, + struct list_lru *lru, struct obj_cgroup **objcgp, size_t objects, gfp_t flags) { @@ -492,6 +506,7 @@ static inline size_t slab_ksize(const struct kmem_cache *s) } static inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, + struct list_lru *lru, struct obj_cgroup **objcgp, size_t size, gfp_t flags) { @@ -502,7 +517,7 @@ static inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, if (should_failslab(s, flags)) return NULL; - if (!memcg_slab_pre_alloc_hook(s, objcgp, size, flags)) + if (!memcg_slab_pre_alloc_hook(s, lru, objcgp, size, flags)) return NULL; return s; diff --git a/mm/slub.c b/mm/slub.c index cb62496d7122..6566b0b2b062 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2832,7 +2832,7 @@ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s, * * Otherwise we can simply pick the next object from the lockless free list. */ -static __always_inline void *slab_alloc_node(struct kmem_cache *s, +static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags, int node, unsigned long addr, size_t orig_size) { void *object; @@ -2842,7 +2842,7 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct obj_cgroup *objcg = NULL; bool init = false; - s = slab_pre_alloc_hook(s, &objcg, 1, gfpflags); + s = slab_pre_alloc_hook(s, lru, &objcg, 1, gfpflags); if (!s) return NULL; @@ -2926,27 +2926,41 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, return object; } -static __always_inline void *slab_alloc(struct kmem_cache *s, +static __always_inline void *slab_alloc(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags, unsigned long addr, size_t orig_size) { - return slab_alloc_node(s, gfpflags, NUMA_NO_NODE, addr, orig_size); + return slab_alloc_node(s, lru, gfpflags, NUMA_NO_NODE, addr, orig_size); } -void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) +static __always_inline +void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, + gfp_t gfpflags) { - void *ret = slab_alloc(s, gfpflags, _RET_IP_, s->object_size); + void *ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size); trace_kmem_cache_alloc(_RET_IP_, ret, s->object_size, s->size, gfpflags); return ret; } + +void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) +{ + return __kmem_cache_alloc_lru(s, NULL, gfpflags); +} EXPORT_SYMBOL(kmem_cache_alloc); +void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, + gfp_t gfpflags) +{ + return __kmem_cache_alloc_lru(s, lru, gfpflags); +} +EXPORT_SYMBOL(kmem_cache_alloc_lru); + #ifdef CONFIG_TRACING void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) { - void *ret = slab_alloc(s, gfpflags, _RET_IP_, size); + void *ret = slab_alloc(s, NULL, gfpflags, _RET_IP_, size); trace_kmalloc(_RET_IP_, ret, size, s->size, gfpflags); ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; @@ -2957,7 +2971,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_trace); #ifdef CONFIG_NUMA void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) { - void *ret = slab_alloc_node(s, gfpflags, node, _RET_IP_, s->object_size); + void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size); trace_kmem_cache_alloc_node(_RET_IP_, ret, s->object_size, s->size, gfpflags, node); @@ -2971,7 +2985,7 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, int node, size_t size) { - void *ret = slab_alloc_node(s, gfpflags, node, _RET_IP_, size); + void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, size); trace_kmalloc_node(_RET_IP_, ret, size, s->size, gfpflags, node); @@ -3322,7 +3336,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, struct obj_cgroup *objcg = NULL; /* memcg and kmem_cache debug support */ - s = slab_pre_alloc_hook(s, &objcg, size, flags); + s = slab_pre_alloc_hook(s, NULL, &objcg, size, flags); if (unlikely(!s)) return false; /* @@ -4071,7 +4085,7 @@ void *__kmalloc(size_t size, gfp_t flags) if (unlikely(ZERO_OR_NULL_PTR(s))) return s; - ret = slab_alloc(s, flags, _RET_IP_, size); + ret = slab_alloc(s, NULL, flags, _RET_IP_, size); trace_kmalloc(_RET_IP_, ret, size, s->size, flags); @@ -4119,7 +4133,7 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node) if (unlikely(ZERO_OR_NULL_PTR(s))) return s; - ret = slab_alloc_node(s, flags, node, _RET_IP_, size); + ret = slab_alloc_node(s, NULL, flags, node, _RET_IP_, size); trace_kmalloc_node(_RET_IP_, ret, size, s->size, flags, node); @@ -4581,7 +4595,7 @@ void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller) if (unlikely(ZERO_OR_NULL_PTR(s))) return s; - ret = slab_alloc(s, gfpflags, caller, size); + ret = slab_alloc(s, NULL, gfpflags, caller, size); /* Honor the call site pointer we received. */ trace_kmalloc(caller, ret, size, s->size, gfpflags); @@ -4612,7 +4626,7 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags, if (unlikely(ZERO_OR_NULL_PTR(s))) return s; - ret = slab_alloc_node(s, gfpflags, node, caller, size); + ret = slab_alloc_node(s, NULL, gfpflags, node, caller, size); /* Honor the call site pointer we received. */ trace_kmalloc_node(caller, ret, size, s->size, gfpflags, node); From patchwork Tue May 11 10:46:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27571C43462 for ; Tue, 11 May 2021 10:52:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 67C8D61132 for ; Tue, 11 May 2021 10:52:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 67C8D61132 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 57B8F940009; Tue, 11 May 2021 06:52:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DB306B0075; Tue, 11 May 2021 06:52:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D4199940009; Tue, 11 May 2021 06:52:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0040.hostedemail.com [216.40.44.40]) by kanga.kvack.org (Postfix) with ESMTP id 984D06B0071 for ; Tue, 11 May 2021 06:52:30 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 524208249980 for ; Tue, 11 May 2021 10:52:30 +0000 (UTC) X-FDA: 78128636460.23.BDF4B54 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf16.hostedemail.com (Postfix) with ESMTP id 467928019382 for ; Tue, 11 May 2021 10:52:18 +0000 (UTC) Received: by mail-pj1-f43.google.com with SMTP id md17so11451500pjb.0 for ; Tue, 11 May 2021 03:52:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fDVkMdVlv8miwozjHsh7GB2JUg1dHMhy3ePeK1EeLfQ=; b=PrUKKX+IbTYrG6FrLDBTBqaAODbo06EoJIkRuGXBPXkcfqRWCToKK2zDIY+X7Pcdk9 H1h5TdRNjFBwsyOMi2GpxLJMh3EadWSLtG3SxjMrnWdI79OdezkDzfnG+G5AfvbfevN9 gcYew4igaIg0fPMjZ2csLdrDLGbhRN2bn7tCZuiNXHAPcnyM+wFPb9leFWl1wu4ceR92 GScV63NF8LpI0vQKgTE7Qr73LrxXdgLpsw/WiPuXqSQ/TOSjnHMqoR3hY5mdsiAEpiC5 DDJlmFuvQ77kEQpH83fxjKrdZ8iuDKkMn/GcHgtvFFhmhHNDIzPiVv/jmxx3igcTAyxO lIxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fDVkMdVlv8miwozjHsh7GB2JUg1dHMhy3ePeK1EeLfQ=; b=XO6oEO7Z+UpqJ1tPe8Pz/AscYkQEJ/Orj47funwAStUguPRtRY5lN996elcZyVldT8 XRHo9UIN3SFrZeixLnaqzFqRPRL3h7Zk0d1/L+E0u/liZgvG6b5FW8V33ucw5LC/RxbY quFCXp+0OCJbGjEw/PkrcNAgYtL5+NG/8UmO5xuLr9+4BkrQxA5NoCa/y8VqOA5fiXyU LxZZRWN/U2i5EoPtXlWTJGNI8K0uyWdZ0GcG9pzc4MKZ0ad2JZVG/ejGkdma2fWkVBNi emfWRwVa0zp7iH+eVJuTqKKHZJDqZsjZ4Eenq2G0lGBW2I5xWv66TOtE6op62ejne5Rt Mr2w== X-Gm-Message-State: AOAM531bukbwSNERdEoCv2lJwNXO0HsVl7pr1NBp67YOG46deWVHEmqa ZSkgCevBFtunlscUu3zW5mLWqg== X-Google-Smtp-Source: ABdhPJzUQgX4EQDAVnICVpPGPzOizsrTZ/06l+VKMq2ca5pTUrG1ja598LWNzyIuEJT9fqzlK2kiDg== X-Received: by 2002:a17:902:f203:b029:ee:af8f:8e4b with SMTP id m3-20020a170902f203b02900eeaf8f8e4bmr28587257plc.23.1620730348945; Tue, 11 May 2021 03:52:28 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.52.21 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:52:28 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 10/17] fs: introduce alloc_inode_sb() to allocate filesystems specific inode Date: Tue, 11 May 2021 18:46:40 +0800 Message-Id: <20210511104647.604-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=PrUKKX+I; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 467928019382 X-Stat-Signature: c4ug6mp31qir7bz6d41rxj9fqqrbgprt Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=mail-pj1-f43.google.com; client-ip=209.85.216.43 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730338-692372 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The allocated inode cache will be added into its memcg lru list later, but we do not allocate list_lru in the later patch. So the caller should call kmem_cache_alloc_lru() to allocate inode and related list_lru. Introduce alloc_inode_sb() to do that and convert all inodes allocation to it. Signed-off-by: Muchun Song --- drivers/dax/super.c | 2 +- fs/9p/vfs_inode.c | 2 +- fs/adfs/super.c | 2 +- fs/affs/super.c | 2 +- fs/afs/super.c | 2 +- fs/befs/linuxvfs.c | 2 +- fs/bfs/inode.c | 2 +- fs/block_dev.c | 2 +- fs/btrfs/inode.c | 2 +- fs/ceph/inode.c | 2 +- fs/cifs/cifsfs.c | 2 +- fs/coda/inode.c | 2 +- fs/ecryptfs/super.c | 2 +- fs/efs/super.c | 2 +- fs/erofs/super.c | 2 +- fs/exfat/super.c | 2 +- fs/ext2/super.c | 2 +- fs/ext4/super.c | 2 +- fs/f2fs/super.c | 2 +- fs/fat/inode.c | 2 +- fs/freevxfs/vxfs_super.c | 2 +- fs/fuse/inode.c | 2 +- fs/gfs2/super.c | 2 +- fs/hfs/super.c | 2 +- fs/hfsplus/super.c | 2 +- fs/hostfs/hostfs_kern.c | 2 +- fs/hpfs/super.c | 2 +- fs/hugetlbfs/inode.c | 2 +- fs/inode.c | 2 +- fs/isofs/inode.c | 2 +- fs/jffs2/super.c | 2 +- fs/jfs/super.c | 2 +- fs/minix/inode.c | 2 +- fs/nfs/inode.c | 2 +- fs/nilfs2/super.c | 2 +- fs/ntfs/inode.c | 2 +- fs/ocfs2/dlmfs/dlmfs.c | 2 +- fs/ocfs2/super.c | 2 +- fs/openpromfs/inode.c | 2 +- fs/orangefs/super.c | 2 +- fs/overlayfs/super.c | 2 +- fs/proc/inode.c | 2 +- fs/qnx4/inode.c | 2 +- fs/qnx6/inode.c | 2 +- fs/reiserfs/super.c | 2 +- fs/romfs/super.c | 2 +- fs/squashfs/super.c | 2 +- fs/sysv/inode.c | 2 +- fs/ubifs/super.c | 2 +- fs/udf/super.c | 2 +- fs/ufs/super.c | 2 +- fs/vboxsf/super.c | 2 +- fs/xfs/xfs_icache.c | 3 ++- fs/zonefs/super.c | 2 +- include/linux/fs.h | 7 +++++++ ipc/mqueue.c | 2 +- mm/shmem.c | 2 +- net/socket.c | 2 +- net/sunrpc/rpc_pipe.c | 2 +- 59 files changed, 66 insertions(+), 58 deletions(-) diff --git a/drivers/dax/super.c b/drivers/dax/super.c index 5fa6ae9dbc8b..30474ba337c8 100644 --- a/drivers/dax/super.c +++ b/drivers/dax/super.c @@ -460,7 +460,7 @@ static struct inode *dax_alloc_inode(struct super_block *sb) struct dax_device *dax_dev; struct inode *inode; - dax_dev = kmem_cache_alloc(dax_cache, GFP_KERNEL); + dax_dev = alloc_inode_sb(sb, dax_cache, GFP_KERNEL); if (!dax_dev) return NULL; diff --git a/fs/9p/vfs_inode.c b/fs/9p/vfs_inode.c index 795706520b5e..5311f35accf5 100644 --- a/fs/9p/vfs_inode.c +++ b/fs/9p/vfs_inode.c @@ -223,7 +223,7 @@ v9fs_blank_wstat(struct p9_wstat *wstat) struct inode *v9fs_alloc_inode(struct super_block *sb) { struct v9fs_inode *v9inode; - v9inode = kmem_cache_alloc(v9fs_inode_cache, GFP_KERNEL); + v9inode = alloc_inode_sb(sb, v9fs_inode_cache, GFP_KERNEL); if (!v9inode) return NULL; #ifdef CONFIG_9P_FSCACHE diff --git a/fs/adfs/super.c b/fs/adfs/super.c index bdbd26e571ed..e8bfc38239cd 100644 --- a/fs/adfs/super.c +++ b/fs/adfs/super.c @@ -220,7 +220,7 @@ static struct kmem_cache *adfs_inode_cachep; static struct inode *adfs_alloc_inode(struct super_block *sb) { struct adfs_inode_info *ei; - ei = kmem_cache_alloc(adfs_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, adfs_inode_cachep, GFP_KERNEL); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/fs/affs/super.c b/fs/affs/super.c index c6c2a513ec92..67458f63a1ab 100644 --- a/fs/affs/super.c +++ b/fs/affs/super.c @@ -100,7 +100,7 @@ static struct inode *affs_alloc_inode(struct super_block *sb) { struct affs_inode_info *i; - i = kmem_cache_alloc(affs_inode_cachep, GFP_KERNEL); + i = alloc_inode_sb(sb, affs_inode_cachep, GFP_KERNEL); if (!i) return NULL; diff --git a/fs/afs/super.c b/fs/afs/super.c index e38bb1e7a4d2..0ecea5a94af9 100644 --- a/fs/afs/super.c +++ b/fs/afs/super.c @@ -677,7 +677,7 @@ static struct inode *afs_alloc_inode(struct super_block *sb) { struct afs_vnode *vnode; - vnode = kmem_cache_alloc(afs_inode_cachep, GFP_KERNEL); + vnode = alloc_inode_sb(sb, afs_inode_cachep, GFP_KERNEL); if (!vnode) return NULL; diff --git a/fs/befs/linuxvfs.c b/fs/befs/linuxvfs.c index c1ba13d19024..b4b3567ac655 100644 --- a/fs/befs/linuxvfs.c +++ b/fs/befs/linuxvfs.c @@ -277,7 +277,7 @@ befs_alloc_inode(struct super_block *sb) { struct befs_inode_info *bi; - bi = kmem_cache_alloc(befs_inode_cachep, GFP_KERNEL); + bi = alloc_inode_sb(sb, befs_inode_cachep, GFP_KERNEL); if (!bi) return NULL; return &bi->vfs_inode; diff --git a/fs/bfs/inode.c b/fs/bfs/inode.c index fd691e4815c5..1926bec2c850 100644 --- a/fs/bfs/inode.c +++ b/fs/bfs/inode.c @@ -239,7 +239,7 @@ static struct kmem_cache *bfs_inode_cachep; static struct inode *bfs_alloc_inode(struct super_block *sb) { struct bfs_inode_info *bi; - bi = kmem_cache_alloc(bfs_inode_cachep, GFP_KERNEL); + bi = alloc_inode_sb(sb, bfs_inode_cachep, GFP_KERNEL); if (!bi) return NULL; return &bi->vfs_inode; diff --git a/fs/block_dev.c b/fs/block_dev.c index 9114e0a0e7b4..4fc2ddb9e5ca 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -796,7 +796,7 @@ static struct kmem_cache * bdev_cachep __read_mostly; static struct inode *bdev_alloc_inode(struct super_block *sb) { - struct bdev_inode *ei = kmem_cache_alloc(bdev_cachep, GFP_KERNEL); + struct bdev_inode *ei = alloc_inode_sb(sb, bdev_cachep, GFP_KERNEL); if (!ei) return NULL; diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 4af336008b12..2262f58c0179 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -8830,7 +8830,7 @@ struct inode *btrfs_alloc_inode(struct super_block *sb) struct btrfs_inode *ei; struct inode *inode; - ei = kmem_cache_alloc(btrfs_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, btrfs_inode_cachep, GFP_KERNEL); if (!ei) return NULL; diff --git a/fs/ceph/inode.c b/fs/ceph/inode.c index 689e3ffd29d7..d5e95b524293 100644 --- a/fs/ceph/inode.c +++ b/fs/ceph/inode.c @@ -443,7 +443,7 @@ struct inode *ceph_alloc_inode(struct super_block *sb) struct ceph_inode_info *ci; int i; - ci = kmem_cache_alloc(ceph_inode_cachep, GFP_NOFS); + ci = alloc_inode_sb(sb, ceph_inode_cachep, GFP_NOFS); if (!ci) return NULL; diff --git a/fs/cifs/cifsfs.c b/fs/cifs/cifsfs.c index e0c5e860a0ee..7cd5c389fbf9 100644 --- a/fs/cifs/cifsfs.c +++ b/fs/cifs/cifsfs.c @@ -360,7 +360,7 @@ static struct inode * cifs_alloc_inode(struct super_block *sb) { struct cifsInodeInfo *cifs_inode; - cifs_inode = kmem_cache_alloc(cifs_inode_cachep, GFP_KERNEL); + cifs_inode = alloc_inode_sb(sb, cifs_inode_cachep, GFP_KERNEL); if (!cifs_inode) return NULL; cifs_inode->cifsAttrs = 0x20; /* default */ diff --git a/fs/coda/inode.c b/fs/coda/inode.c index d9f1bd7153df..2185328b65c7 100644 --- a/fs/coda/inode.c +++ b/fs/coda/inode.c @@ -43,7 +43,7 @@ static struct kmem_cache * coda_inode_cachep; static struct inode *coda_alloc_inode(struct super_block *sb) { struct coda_inode_info *ei; - ei = kmem_cache_alloc(coda_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, coda_inode_cachep, GFP_KERNEL); if (!ei) return NULL; memset(&ei->c_fid, 0, sizeof(struct CodaFid)); diff --git a/fs/ecryptfs/super.c b/fs/ecryptfs/super.c index 39116af0390f..0b1c878317ab 100644 --- a/fs/ecryptfs/super.c +++ b/fs/ecryptfs/super.c @@ -38,7 +38,7 @@ static struct inode *ecryptfs_alloc_inode(struct super_block *sb) struct ecryptfs_inode_info *inode_info; struct inode *inode = NULL; - inode_info = kmem_cache_alloc(ecryptfs_inode_info_cache, GFP_KERNEL); + inode_info = alloc_inode_sb(sb, ecryptfs_inode_info_cache, GFP_KERNEL); if (unlikely(!inode_info)) goto out; if (ecryptfs_init_crypt_stat(&inode_info->crypt_stat)) { diff --git a/fs/efs/super.c b/fs/efs/super.c index 62b155b9366b..b287f47c165b 100644 --- a/fs/efs/super.c +++ b/fs/efs/super.c @@ -69,7 +69,7 @@ static struct kmem_cache * efs_inode_cachep; static struct inode *efs_alloc_inode(struct super_block *sb) { struct efs_inode_info *ei; - ei = kmem_cache_alloc(efs_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, efs_inode_cachep, GFP_KERNEL); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/fs/erofs/super.c b/fs/erofs/super.c index bbf3bbd908e0..03f2dd684f8d 100644 --- a/fs/erofs/super.c +++ b/fs/erofs/super.c @@ -83,7 +83,7 @@ static void erofs_inode_init_once(void *ptr) static struct inode *erofs_alloc_inode(struct super_block *sb) { struct erofs_inode *vi = - kmem_cache_alloc(erofs_inode_cachep, GFP_KERNEL); + alloc_inode_sb(sb, erofs_inode_cachep, GFP_KERNEL); if (!vi) return NULL; diff --git a/fs/exfat/super.c b/fs/exfat/super.c index d38d17a77e76..c88e0f218df0 100644 --- a/fs/exfat/super.c +++ b/fs/exfat/super.c @@ -182,7 +182,7 @@ static struct inode *exfat_alloc_inode(struct super_block *sb) { struct exfat_inode_info *ei; - ei = kmem_cache_alloc(exfat_inode_cachep, GFP_NOFS); + ei = alloc_inode_sb(sb, exfat_inode_cachep, GFP_NOFS); if (!ei) return NULL; diff --git a/fs/ext2/super.c b/fs/ext2/super.c index 21e09fbaa46f..7eba9d6fbdc6 100644 --- a/fs/ext2/super.c +++ b/fs/ext2/super.c @@ -180,7 +180,7 @@ static struct kmem_cache * ext2_inode_cachep; static struct inode *ext2_alloc_inode(struct super_block *sb) { struct ext2_inode_info *ei; - ei = kmem_cache_alloc(ext2_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, ext2_inode_cachep, GFP_KERNEL); if (!ei) return NULL; ei->i_block_alloc_info = NULL; diff --git a/fs/ext4/super.c b/fs/ext4/super.c index 7dc94f3e18e6..b2c486d224b4 100644 --- a/fs/ext4/super.c +++ b/fs/ext4/super.c @@ -1275,7 +1275,7 @@ static struct inode *ext4_alloc_inode(struct super_block *sb) { struct ext4_inode_info *ei; - ei = kmem_cache_alloc(ext4_inode_cachep, GFP_NOFS); + ei = alloc_inode_sb(sb, ext4_inode_cachep, GFP_NOFS); if (!ei) return NULL; diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index 7d325bfaf65a..444134950ae4 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -1169,7 +1169,7 @@ static struct inode *f2fs_alloc_inode(struct super_block *sb) { struct f2fs_inode_info *fi; - fi = kmem_cache_alloc(f2fs_inode_cachep, GFP_F2FS_ZERO); + fi = alloc_inode_sb(sb, f2fs_inode_cachep, GFP_F2FS_ZERO); if (!fi) return NULL; diff --git a/fs/fat/inode.c b/fs/fat/inode.c index bab9b202b496..adc8a525553a 100644 --- a/fs/fat/inode.c +++ b/fs/fat/inode.c @@ -744,7 +744,7 @@ static struct kmem_cache *fat_inode_cachep; static struct inode *fat_alloc_inode(struct super_block *sb) { struct msdos_inode_info *ei; - ei = kmem_cache_alloc(fat_inode_cachep, GFP_NOFS); + ei = alloc_inode_sb(sb, fat_inode_cachep, GFP_NOFS); if (!ei) return NULL; diff --git a/fs/freevxfs/vxfs_super.c b/fs/freevxfs/vxfs_super.c index 578a5062706e..22eed5a73ac2 100644 --- a/fs/freevxfs/vxfs_super.c +++ b/fs/freevxfs/vxfs_super.c @@ -124,7 +124,7 @@ static struct inode *vxfs_alloc_inode(struct super_block *sb) { struct vxfs_inode_info *vi; - vi = kmem_cache_alloc(vxfs_inode_cachep, GFP_KERNEL); + vi = alloc_inode_sb(sb, vxfs_inode_cachep, GFP_KERNEL); if (!vi) return NULL; inode_init_once(&vi->vfs_inode); diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 393e36b74dc4..3fe7d1484ae6 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -73,7 +73,7 @@ static struct inode *fuse_alloc_inode(struct super_block *sb) { struct fuse_inode *fi; - fi = kmem_cache_alloc(fuse_inode_cachep, GFP_KERNEL); + fi = alloc_inode_sb(sb, fuse_inode_cachep, GFP_KERNEL); if (!fi) return NULL; diff --git a/fs/gfs2/super.c b/fs/gfs2/super.c index 4d4ceb0b6903..e9e8ad6f0c66 100644 --- a/fs/gfs2/super.c +++ b/fs/gfs2/super.c @@ -1496,7 +1496,7 @@ static struct inode *gfs2_alloc_inode(struct super_block *sb) { struct gfs2_inode *ip; - ip = kmem_cache_alloc(gfs2_inode_cachep, GFP_KERNEL); + ip = alloc_inode_sb(sb, gfs2_inode_cachep, GFP_KERNEL); if (!ip) return NULL; ip->i_flags = 0; diff --git a/fs/hfs/super.c b/fs/hfs/super.c index 44d07c9e3a7f..e40334c93183 100644 --- a/fs/hfs/super.c +++ b/fs/hfs/super.c @@ -162,7 +162,7 @@ static struct inode *hfs_alloc_inode(struct super_block *sb) { struct hfs_inode_info *i; - i = kmem_cache_alloc(hfs_inode_cachep, GFP_KERNEL); + i = alloc_inode_sb(sb, hfs_inode_cachep, GFP_KERNEL); return i ? &i->vfs_inode : NULL; } diff --git a/fs/hfsplus/super.c b/fs/hfsplus/super.c index b9e3db3f855f..8479add998b5 100644 --- a/fs/hfsplus/super.c +++ b/fs/hfsplus/super.c @@ -624,7 +624,7 @@ static struct inode *hfsplus_alloc_inode(struct super_block *sb) { struct hfsplus_inode_info *i; - i = kmem_cache_alloc(hfsplus_inode_cachep, GFP_KERNEL); + i = alloc_inode_sb(sb, hfsplus_inode_cachep, GFP_KERNEL); return i ? &i->vfs_inode : NULL; } diff --git a/fs/hostfs/hostfs_kern.c b/fs/hostfs/hostfs_kern.c index 7d0c3dbb2898..96dc97ae48e4 100644 --- a/fs/hostfs/hostfs_kern.c +++ b/fs/hostfs/hostfs_kern.c @@ -222,7 +222,7 @@ static struct inode *hostfs_alloc_inode(struct super_block *sb) { struct hostfs_inode_info *hi; - hi = kmem_cache_alloc(hostfs_inode_cache, GFP_KERNEL_ACCOUNT); + hi = alloc_inode_sb(sb, hostfs_inode_cache, GFP_KERNEL_ACCOUNT); if (hi == NULL) return NULL; hi->fd = -1; diff --git a/fs/hpfs/super.c b/fs/hpfs/super.c index a7dbfc892022..1cb89595b875 100644 --- a/fs/hpfs/super.c +++ b/fs/hpfs/super.c @@ -232,7 +232,7 @@ static struct kmem_cache * hpfs_inode_cachep; static struct inode *hpfs_alloc_inode(struct super_block *sb) { struct hpfs_inode_info *ei; - ei = kmem_cache_alloc(hpfs_inode_cachep, GFP_NOFS); + ei = alloc_inode_sb(sb, hpfs_inode_cachep, GFP_NOFS); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index a2a42335e8fd..1403b4bd6514 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -1103,7 +1103,7 @@ static struct inode *hugetlbfs_alloc_inode(struct super_block *sb) if (unlikely(!hugetlbfs_dec_free_inodes(sbinfo))) return NULL; - p = kmem_cache_alloc(hugetlbfs_inode_cachep, GFP_KERNEL); + p = alloc_inode_sb(sb, hugetlbfs_inode_cachep, GFP_KERNEL); if (unlikely(!p)) { hugetlbfs_inc_free_inodes(sbinfo); return NULL; diff --git a/fs/inode.c b/fs/inode.c index af48d1b722f0..68b9167ad18b 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -232,7 +232,7 @@ static struct inode *alloc_inode(struct super_block *sb) if (ops->alloc_inode) inode = ops->alloc_inode(sb); else - inode = kmem_cache_alloc(inode_cachep, GFP_KERNEL); + inode = alloc_inode_sb(sb, inode_cachep, GFP_KERNEL); if (!inode) return NULL; diff --git a/fs/isofs/inode.c b/fs/isofs/inode.c index 21edc423b79f..ef6c29501702 100644 --- a/fs/isofs/inode.c +++ b/fs/isofs/inode.c @@ -70,7 +70,7 @@ static struct kmem_cache *isofs_inode_cachep; static struct inode *isofs_alloc_inode(struct super_block *sb) { struct iso_inode_info *ei; - ei = kmem_cache_alloc(isofs_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, isofs_inode_cachep, GFP_KERNEL); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/fs/jffs2/super.c b/fs/jffs2/super.c index 81ca58c10b72..7ea37f49f1e1 100644 --- a/fs/jffs2/super.c +++ b/fs/jffs2/super.c @@ -39,7 +39,7 @@ static struct inode *jffs2_alloc_inode(struct super_block *sb) { struct jffs2_inode_info *f; - f = kmem_cache_alloc(jffs2_inode_cachep, GFP_KERNEL); + f = alloc_inode_sb(sb, jffs2_inode_cachep, GFP_KERNEL); if (!f) return NULL; return &f->vfs_inode; diff --git a/fs/jfs/super.c b/fs/jfs/super.c index 1f0ffabbde56..65255541d44e 100644 --- a/fs/jfs/super.c +++ b/fs/jfs/super.c @@ -102,7 +102,7 @@ static struct inode *jfs_alloc_inode(struct super_block *sb) { struct jfs_inode_info *jfs_inode; - jfs_inode = kmem_cache_alloc(jfs_inode_cachep, GFP_NOFS); + jfs_inode = alloc_inode_sb(sb, jfs_inode_cachep, GFP_NOFS); if (!jfs_inode) return NULL; #ifdef CONFIG_QUOTA diff --git a/fs/minix/inode.c b/fs/minix/inode.c index a532a99bbe81..f30b1daac386 100644 --- a/fs/minix/inode.c +++ b/fs/minix/inode.c @@ -63,7 +63,7 @@ static struct kmem_cache * minix_inode_cachep; static struct inode *minix_alloc_inode(struct super_block *sb) { struct minix_inode_info *ei; - ei = kmem_cache_alloc(minix_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, minix_inode_cachep, GFP_KERNEL); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c index 529c4099f482..fa3d8e8b695f 100644 --- a/fs/nfs/inode.c +++ b/fs/nfs/inode.c @@ -2240,7 +2240,7 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr) struct inode *nfs_alloc_inode(struct super_block *sb) { struct nfs_inode *nfsi; - nfsi = kmem_cache_alloc(nfs_inode_cachep, GFP_KERNEL); + nfsi = alloc_inode_sb(sb, nfs_inode_cachep, GFP_KERNEL); if (!nfsi) return NULL; nfsi->flags = 0UL; diff --git a/fs/nilfs2/super.c b/fs/nilfs2/super.c index 4abd928b0bc8..e7d7ad7a1e50 100644 --- a/fs/nilfs2/super.c +++ b/fs/nilfs2/super.c @@ -151,7 +151,7 @@ struct inode *nilfs_alloc_inode(struct super_block *sb) { struct nilfs_inode_info *ii; - ii = kmem_cache_alloc(nilfs_inode_cachep, GFP_NOFS); + ii = alloc_inode_sb(sb, nilfs_inode_cachep, GFP_NOFS); if (!ii) return NULL; ii->i_bh = NULL; diff --git a/fs/ntfs/inode.c b/fs/ntfs/inode.c index f5c058b3192c..1e053bb58ead 100644 --- a/fs/ntfs/inode.c +++ b/fs/ntfs/inode.c @@ -310,7 +310,7 @@ struct inode *ntfs_alloc_big_inode(struct super_block *sb) ntfs_inode *ni; ntfs_debug("Entering."); - ni = kmem_cache_alloc(ntfs_big_inode_cache, GFP_NOFS); + ni = alloc_inode_sb(sb, ntfs_big_inode_cache, GFP_NOFS); if (likely(ni != NULL)) { ni->state = 0; return VFS_I(ni); diff --git a/fs/ocfs2/dlmfs/dlmfs.c b/fs/ocfs2/dlmfs/dlmfs.c index fa0a14f199eb..e360543ad7e7 100644 --- a/fs/ocfs2/dlmfs/dlmfs.c +++ b/fs/ocfs2/dlmfs/dlmfs.c @@ -280,7 +280,7 @@ static struct inode *dlmfs_alloc_inode(struct super_block *sb) { struct dlmfs_inode_private *ip; - ip = kmem_cache_alloc(dlmfs_inode_cache, GFP_NOFS); + ip = alloc_inode_sb(sb, dlmfs_inode_cache, GFP_NOFS); if (!ip) return NULL; diff --git a/fs/ocfs2/super.c b/fs/ocfs2/super.c index c86bd4e60e20..cf044448130f 100644 --- a/fs/ocfs2/super.c +++ b/fs/ocfs2/super.c @@ -549,7 +549,7 @@ static struct inode *ocfs2_alloc_inode(struct super_block *sb) { struct ocfs2_inode_info *oi; - oi = kmem_cache_alloc(ocfs2_inode_cachep, GFP_NOFS); + oi = alloc_inode_sb(sb, ocfs2_inode_cachep, GFP_NOFS); if (!oi) return NULL; diff --git a/fs/openpromfs/inode.c b/fs/openpromfs/inode.c index f825176ff4ed..f0b7f4d51a17 100644 --- a/fs/openpromfs/inode.c +++ b/fs/openpromfs/inode.c @@ -335,7 +335,7 @@ static struct inode *openprom_alloc_inode(struct super_block *sb) { struct op_inode_info *oi; - oi = kmem_cache_alloc(op_inode_cachep, GFP_KERNEL); + oi = alloc_inode_sb(sb, op_inode_cachep, GFP_KERNEL); if (!oi) return NULL; diff --git a/fs/orangefs/super.c b/fs/orangefs/super.c index ee5efdc35cc1..5731ce285c98 100644 --- a/fs/orangefs/super.c +++ b/fs/orangefs/super.c @@ -106,7 +106,7 @@ static struct inode *orangefs_alloc_inode(struct super_block *sb) { struct orangefs_inode_s *orangefs_inode; - orangefs_inode = kmem_cache_alloc(orangefs_inode_cache, GFP_KERNEL); + orangefs_inode = alloc_inode_sb(sb, orangefs_inode_cache, GFP_KERNEL); if (!orangefs_inode) return NULL; diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c index b01d4147520d..6d8d16bf4b57 100644 --- a/fs/overlayfs/super.c +++ b/fs/overlayfs/super.c @@ -174,7 +174,7 @@ static struct kmem_cache *ovl_inode_cachep; static struct inode *ovl_alloc_inode(struct super_block *sb) { - struct ovl_inode *oi = kmem_cache_alloc(ovl_inode_cachep, GFP_KERNEL); + struct ovl_inode *oi = alloc_inode_sb(sb, ovl_inode_cachep, GFP_KERNEL); if (!oi) return NULL; diff --git a/fs/proc/inode.c b/fs/proc/inode.c index 599eb724ff2d..cc0a406d3a19 100644 --- a/fs/proc/inode.c +++ b/fs/proc/inode.c @@ -66,7 +66,7 @@ static struct inode *proc_alloc_inode(struct super_block *sb) { struct proc_inode *ei; - ei = kmem_cache_alloc(proc_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, proc_inode_cachep, GFP_KERNEL); if (!ei) return NULL; ei->pid = NULL; diff --git a/fs/qnx4/inode.c b/fs/qnx4/inode.c index 3fb7fc819b4f..a635bb6615e9 100644 --- a/fs/qnx4/inode.c +++ b/fs/qnx4/inode.c @@ -338,7 +338,7 @@ static struct kmem_cache *qnx4_inode_cachep; static struct inode *qnx4_alloc_inode(struct super_block *sb) { struct qnx4_inode_info *ei; - ei = kmem_cache_alloc(qnx4_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, qnx4_inode_cachep, GFP_KERNEL); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/fs/qnx6/inode.c b/fs/qnx6/inode.c index 61191f7bdf62..9d8e7e9788a1 100644 --- a/fs/qnx6/inode.c +++ b/fs/qnx6/inode.c @@ -597,7 +597,7 @@ static struct kmem_cache *qnx6_inode_cachep; static struct inode *qnx6_alloc_inode(struct super_block *sb) { struct qnx6_inode_info *ei; - ei = kmem_cache_alloc(qnx6_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, qnx6_inode_cachep, GFP_KERNEL); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/fs/reiserfs/super.c b/fs/reiserfs/super.c index 3ffafc73acf0..d9e8b0368ce9 100644 --- a/fs/reiserfs/super.c +++ b/fs/reiserfs/super.c @@ -639,7 +639,7 @@ static struct kmem_cache *reiserfs_inode_cachep; static struct inode *reiserfs_alloc_inode(struct super_block *sb) { struct reiserfs_inode_info *ei; - ei = kmem_cache_alloc(reiserfs_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, reiserfs_inode_cachep, GFP_KERNEL); if (!ei) return NULL; atomic_set(&ei->openers, 0); diff --git a/fs/romfs/super.c b/fs/romfs/super.c index 259f684d9236..9e6bbb4219de 100644 --- a/fs/romfs/super.c +++ b/fs/romfs/super.c @@ -375,7 +375,7 @@ static struct inode *romfs_alloc_inode(struct super_block *sb) { struct romfs_inode_info *inode; - inode = kmem_cache_alloc(romfs_inode_cachep, GFP_KERNEL); + inode = alloc_inode_sb(sb, romfs_inode_cachep, GFP_KERNEL); return inode ? &inode->vfs_inode : NULL; } diff --git a/fs/squashfs/super.c b/fs/squashfs/super.c index 88cc94be1076..16b785cb78c4 100644 --- a/fs/squashfs/super.c +++ b/fs/squashfs/super.c @@ -466,7 +466,7 @@ static void __exit exit_squashfs_fs(void) static struct inode *squashfs_alloc_inode(struct super_block *sb) { struct squashfs_inode_info *ei = - kmem_cache_alloc(squashfs_inode_cachep, GFP_KERNEL); + alloc_inode_sb(sb, squashfs_inode_cachep, GFP_KERNEL); return ei ? &ei->vfs_inode : NULL; } diff --git a/fs/sysv/inode.c b/fs/sysv/inode.c index be47263b8605..9e8d4a6fb2f3 100644 --- a/fs/sysv/inode.c +++ b/fs/sysv/inode.c @@ -306,7 +306,7 @@ static struct inode *sysv_alloc_inode(struct super_block *sb) { struct sysv_inode_info *si; - si = kmem_cache_alloc(sysv_inode_cachep, GFP_KERNEL); + si = alloc_inode_sb(sb, sysv_inode_cachep, GFP_KERNEL); if (!si) return NULL; return &si->vfs_inode; diff --git a/fs/ubifs/super.c b/fs/ubifs/super.c index 7b572e1414ba..6851040c598e 100644 --- a/fs/ubifs/super.c +++ b/fs/ubifs/super.c @@ -268,7 +268,7 @@ static struct inode *ubifs_alloc_inode(struct super_block *sb) { struct ubifs_inode *ui; - ui = kmem_cache_alloc(ubifs_inode_slab, GFP_NOFS); + ui = alloc_inode_sb(sb, ubifs_inode_slab, GFP_NOFS); if (!ui) return NULL; diff --git a/fs/udf/super.c b/fs/udf/super.c index 2f83c1204e20..bdeaa5d88a9e 100644 --- a/fs/udf/super.c +++ b/fs/udf/super.c @@ -141,7 +141,7 @@ static struct kmem_cache *udf_inode_cachep; static struct inode *udf_alloc_inode(struct super_block *sb) { struct udf_inode_info *ei; - ei = kmem_cache_alloc(udf_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, udf_inode_cachep, GFP_KERNEL); if (!ei) return NULL; diff --git a/fs/ufs/super.c b/fs/ufs/super.c index 74028b5a7b0a..ba83ad6e9d39 100644 --- a/fs/ufs/super.c +++ b/fs/ufs/super.c @@ -1443,7 +1443,7 @@ static struct inode *ufs_alloc_inode(struct super_block *sb) { struct ufs_inode_info *ei; - ei = kmem_cache_alloc(ufs_inode_cachep, GFP_NOFS); + ei = alloc_inode_sb(sb, ufs_inode_cachep, GFP_NOFS); if (!ei) return NULL; diff --git a/fs/vboxsf/super.c b/fs/vboxsf/super.c index 4f5e59f06284..050ef855158b 100644 --- a/fs/vboxsf/super.c +++ b/fs/vboxsf/super.c @@ -244,7 +244,7 @@ static struct inode *vboxsf_alloc_inode(struct super_block *sb) { struct vboxsf_inode *sf_i; - sf_i = kmem_cache_alloc(vboxsf_inode_cachep, GFP_NOFS); + sf_i = alloc_inode_sb(sb, vboxsf_inode_cachep, GFP_NOFS); if (!sf_i) return NULL; diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c index 3c81daca0e9a..8c8b059e1e26 100644 --- a/fs/xfs/xfs_icache.c +++ b/fs/xfs/xfs_icache.c @@ -40,7 +40,8 @@ xfs_inode_alloc( * XXX: If this didn't occur in transactions, we could drop GFP_NOFAIL * and return NULL here on ENOMEM. */ - ip = kmem_cache_alloc(xfs_inode_zone, GFP_KERNEL | __GFP_NOFAIL); + ip = alloc_inode_sb(mp->m_super, xfs_inode_zone, + GFP_KERNEL | __GFP_NOFAIL); if (inode_init_always(mp->m_super, VFS_I(ip))) { kmem_cache_free(xfs_inode_zone, ip); diff --git a/fs/zonefs/super.c b/fs/zonefs/super.c index cd145d318b17..335b58251b52 100644 --- a/fs/zonefs/super.c +++ b/fs/zonefs/super.c @@ -1152,7 +1152,7 @@ static struct inode *zonefs_alloc_inode(struct super_block *sb) { struct zonefs_inode_info *zi; - zi = kmem_cache_alloc(zonefs_inode_cachep, GFP_KERNEL); + zi = alloc_inode_sb(sb, zonefs_inode_cachep, GFP_KERNEL); if (!zi) return NULL; diff --git a/include/linux/fs.h b/include/linux/fs.h index c3c88fdb9b2a..d8d5d4eb68d6 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -41,6 +41,7 @@ #include #include #include +#include #include #include @@ -3200,6 +3201,12 @@ extern void free_inode_nonrcu(struct inode *inode); extern int should_remove_suid(struct dentry *); extern int file_remove_privs(struct file *); +static inline void * +alloc_inode_sb(struct super_block *sb, struct kmem_cache *cache, gfp_t gfp) +{ + return kmem_cache_alloc_lru(cache, &sb->s_inode_lru, gfp); +} + extern void __insert_inode_hash(struct inode *, unsigned long hashval); static inline void insert_inode_hash(struct inode *inode) { diff --git a/ipc/mqueue.c b/ipc/mqueue.c index 8031464ed4ae..815d7289bf93 100644 --- a/ipc/mqueue.c +++ b/ipc/mqueue.c @@ -483,7 +483,7 @@ static struct inode *mqueue_alloc_inode(struct super_block *sb) { struct mqueue_inode_info *ei; - ei = kmem_cache_alloc(mqueue_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, mqueue_inode_cachep, GFP_KERNEL); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/mm/shmem.c b/mm/shmem.c index a08cedefbfaa..dad27ca340ef 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3772,7 +3772,7 @@ static struct kmem_cache *shmem_inode_cachep; static struct inode *shmem_alloc_inode(struct super_block *sb) { struct shmem_inode_info *info; - info = kmem_cache_alloc(shmem_inode_cachep, GFP_KERNEL); + info = alloc_inode_sb(sb, shmem_inode_cachep, GFP_KERNEL); if (!info) return NULL; return &info->vfs_inode; diff --git a/net/socket.c b/net/socket.c index 27e3e7d53f8e..d59844b02a2d 100644 --- a/net/socket.c +++ b/net/socket.c @@ -250,7 +250,7 @@ static struct inode *sock_alloc_inode(struct super_block *sb) { struct socket_alloc *ei; - ei = kmem_cache_alloc(sock_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, sock_inode_cachep, GFP_KERNEL); if (!ei) return NULL; init_waitqueue_head(&ei->socket.wq.wait); diff --git a/net/sunrpc/rpc_pipe.c b/net/sunrpc/rpc_pipe.c index 09c000d490a1..30707f39694e 100644 --- a/net/sunrpc/rpc_pipe.c +++ b/net/sunrpc/rpc_pipe.c @@ -197,7 +197,7 @@ static struct inode * rpc_alloc_inode(struct super_block *sb) { struct rpc_inode *rpci; - rpci = kmem_cache_alloc(rpc_inode_cachep, GFP_KERNEL); + rpci = alloc_inode_sb(sb, rpc_inode_cachep, GFP_KERNEL); if (!rpci) return NULL; return &rpci->vfs_inode; From patchwork Tue May 11 10:46:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2392BC433ED for ; Tue, 11 May 2021 10:52:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AF77D61132 for ; Tue, 11 May 2021 10:52:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AF77D61132 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 395366B0071; Tue, 11 May 2021 06:52:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 36C0A6B0075; Tue, 11 May 2021 06:52:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C0668D0002; Tue, 11 May 2021 06:52:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0116.hostedemail.com [216.40.44.116]) by kanga.kvack.org (Postfix) with ESMTP id 0171D6B0071 for ; Tue, 11 May 2021 06:52:37 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C03248249980 for ; Tue, 11 May 2021 10:52:37 +0000 (UTC) X-FDA: 78128636754.30.005DAC8 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by imf06.hostedemail.com (Postfix) with ESMTP id E05B6C000C79 for ; Tue, 11 May 2021 10:52:37 +0000 (UTC) Received: by mail-pg1-f170.google.com with SMTP id m190so15527276pga.2 for ; Tue, 11 May 2021 03:52:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NL60CwduwWyPJJzwINsmFcWyjAvC0A20QSD6WsxPODY=; b=OJw/5jAsOLZ3Uy7wKezj2tHz0wKrzoK0KxiKSuUqTZu3Th14oGo++0Q5g2ZwuuMBG0 9mOQ3JAJeR4l8W4IxHBSvbOnXz8s3fy3MCgjimjUH6i706PaiZfBVWn5DrY273AYfSAI pwc9Woa+GovmT2AIp1/iSMFw6JSKRd+g+itDf76pZBxBka0Vc67I2p7DOX2pov7MDM6W iVK82IKji3+exFHjd7BCDPtjLw8R2P/ZDKN/MeV/QQoNCUXu5qQI7496pBpcHt3UBiIL /uOjnJUlWUrmPrPKG6i9+ieV0orbFpBHt9BLmD/q6IUtHjVaDV/O1MoF2dATJSnsMt29 aXAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NL60CwduwWyPJJzwINsmFcWyjAvC0A20QSD6WsxPODY=; b=T+QSP8ZLsS/bk8sp49lcKpf8a3Q39/eJvEgi7ULzIv5GItHipViAdIrdIT718oBijN +XcundulPSQbZYPWXFNe8WUYXj5nHJemgGSGEtFwHAglyjRMrrEfISDZgMcr1BTOndUN xkNy3EZuRg+mHoQOapOKxWqboj3EZQB8YEQPMO3eXFgpZvLxa2DNr+Z+3ugWqmLLKE8E Lt+MT1lGOHPTvqsW4u+o0qN1LedARw3O3vevAg2JxxRJST56pMmBxX+4rllO7PKJyOAv RulZ91keThlBwCC+W5AQ3ovzqMcc2CqObCtzENulz6zcj0SxYPYV+vhnetYijOfq/GgL n7sA== X-Gm-Message-State: AOAM5329iawaxt3nO2py4PNct1Z2FRo0YjgX2wWVLAQ7OIOv13MEV36P JFQTreH6+oM0pbtJJQ+vR18BIw== X-Google-Smtp-Source: ABdhPJxgWSmEtEI8VOcDf8wYmVXuR0IRBvYI5h/0m5KiK4ZRrYceGvZXI1rutj4/RpYSUEb9ZamaGw== X-Received: by 2002:a63:e515:: with SMTP id r21mr29761202pgh.412.1620730356530; Tue, 11 May 2021 03:52:36 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.52.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:52:36 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 11/17] mm: dcache: use kmem_cache_alloc_lru() to allocate dentry Date: Tue, 11 May 2021 18:46:41 +0800 Message-Id: <20210511104647.604-12-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b="OJw/5jAs"; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf06.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: mjf77sm58pus45dr1e59489bbtciq3sg X-Rspamd-Queue-Id: E05B6C000C79 X-Rspamd-Server: rspam02 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=mail-pg1-f170.google.com; client-ip=209.85.215.170 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730357-191895 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The dentry will be added to its list_lru, so we should also allocate the lru list. Replace kmem_cache_alloc() with kmem_cache_alloc_lru(). Signed-off-by: Muchun Song --- fs/dcache.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/dcache.c b/fs/dcache.c index cf871a81f4fd..36d4806d7284 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -1741,7 +1741,8 @@ static struct dentry *__d_alloc(struct super_block *sb, const struct qstr *name) char *dname; int err; - dentry = kmem_cache_alloc(dentry_cache, GFP_KERNEL); + dentry = kmem_cache_alloc_lru(dentry_cache, &sb->s_dentry_lru, + GFP_KERNEL); if (!dentry) return NULL; From patchwork Tue May 11 10:46:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A531CC43603 for ; Tue, 11 May 2021 10:52:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4EFA5613C5 for ; Tue, 11 May 2021 10:52:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4EFA5613C5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CBDC08D0002; Tue, 11 May 2021 06:52:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C951C6B0078; Tue, 11 May 2021 06:52:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B0E098D0002; Tue, 11 May 2021 06:52:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0204.hostedemail.com [216.40.44.204]) by kanga.kvack.org (Postfix) with ESMTP id 95CD56B0075 for ; Tue, 11 May 2021 06:52:46 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 572D16C16 for ; Tue, 11 May 2021 10:52:46 +0000 (UTC) X-FDA: 78128637132.10.5113FD1 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf03.hostedemail.com (Postfix) with ESMTP id DEFC1C0007C8 for ; Tue, 11 May 2021 10:52:37 +0000 (UTC) Received: by mail-pf1-f179.google.com with SMTP id i13so15907543pfu.2 for ; Tue, 11 May 2021 03:52:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dR6RIg5hyVJsVLc68ckt9CCgyf302MrLbCg8peXJRjo=; b=VBuLofk9lyOSqLeeg7/jQTYCthw90DVKtc7rNVrh5jIKJ41W70kfvg9hW/j1w+K2VH VkQ+BTcahJEEPdpJJqN9GSIazNNgZe68QFx/Zo7UdfiHnUMMCJVCcIlvV1Wor4G8MXSk g08Sh6jlV/uvmsDv02zPD6mLYbH72bl3Cj7h/b9lghcfJMui+accLDzg2A8ERg8H+ew7 EX5J8LmOxSx0xWhhufvvNYx6l1G5vdb1QwBjJuDeNC9I4w6AP+OESOeqYOBgsr5sNS33 I67LpEGxIMcifEVH3yMgOPh/y9/Z6+rGe9FgFudQ99bSpKb66r1LPTXiCwRc3f3hn8Uq IUfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dR6RIg5hyVJsVLc68ckt9CCgyf302MrLbCg8peXJRjo=; b=g7BSJbDHN3PIxelf8kL+DcAaO/V9ZnE2glNK9JtpmyXF4bV6lZH6YhqhES4pIay9UX RjV0/HXvOkZMF87MP3PFXCpTgaPT/SXuz5iu5bIorf6QPPRrVMh/a/CkGtzVpT5CBlsY pqRFC6Bmm90MGNG4K9ltslV9oM9s+kMvnT/lBGD6/LqHlsAxwQ2vZ+OXZtNpzmILHyzQ LYuAuXMZ0+dzmayomjAD3Kela/obQlzxMSV31ud9n5fI9SppwRBciLb9htDJTz/iuqut Th70r8igP0KpmZX5gu4Rbp3rvc/1sPKd/3oEG6x2HkMBVoN7rN0cD6I86Mjk/TCd2JP8 M6Cw== X-Gm-Message-State: AOAM530AIkaJF6UL108qaNGuzCBsH+juGKUlbGp+YOKxohrd7150iyLr eJkVBweoAaZ5evSI8NrvHON3eQ== X-Google-Smtp-Source: ABdhPJzlX94Z8yeVkpxAS93di/OHph7eE4hW5452r85AqbzXMhnJLw8TMexcsIsRyI//CGbSIAWizA== X-Received: by 2002:aa7:9731:0:b029:28e:46dd:97c7 with SMTP id k17-20020aa797310000b029028e46dd97c7mr29476357pfg.27.1620730365136; Tue, 11 May 2021 03:52:45 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.52.36 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:52:44 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 12/17] xarray: replace kmem_cache_alloc with kmem_cache_alloc_lru Date: Tue, 11 May 2021 18:46:42 +0800 Message-Id: <20210511104647.604-13-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=VBuLofk9; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf03.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: a83ikf6uejj3tcxp8ubt6b7ix8ttau6k X-Rspamd-Queue-Id: DEFC1C0007C8 X-Rspamd-Server: rspam02 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf03; identity=mailfrom; envelope-from=""; helo=mail-pf1-f179.google.com; client-ip=209.85.210.179 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730357-381534 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The workingset will add the xa_node to the shadow_nodes list. So the allocation of xa_node should be done by kmem_cache_alloc_lru(). The user of the xarray can xas_set_lru() to pass the list_lru. Signed-off-by: Muchun Song --- include/linux/xarray.h | 9 ++++++++- lib/xarray.c | 10 +++++----- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index a91e3d90df8a..31f3e5ef3c7b 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -1317,6 +1317,7 @@ struct xa_state { struct xa_node *xa_node; struct xa_node *xa_alloc; xa_update_node_t xa_update; + struct list_lru *xa_lru; }; /* @@ -1336,7 +1337,8 @@ struct xa_state { .xa_pad = 0, \ .xa_node = XAS_RESTART, \ .xa_alloc = NULL, \ - .xa_update = NULL \ + .xa_update = NULL, \ + .xa_lru = NULL, \ } /** @@ -1613,6 +1615,11 @@ static inline void xas_set_update(struct xa_state *xas, xa_update_node_t update) xas->xa_update = update; } +static inline void xas_set_lru(struct xa_state *xas, struct list_lru *lru) +{ + xas->xa_lru = lru; +} + /** * xas_next_entry() - Advance iterator to next present entry. * @xas: XArray operation state. diff --git a/lib/xarray.c b/lib/xarray.c index f5d8f54907b4..e9b818abc823 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -302,7 +302,7 @@ bool xas_nomem(struct xa_state *xas, gfp_t gfp) } if (xas->xa->xa_flags & XA_FLAGS_ACCOUNT) gfp |= __GFP_ACCOUNT; - xas->xa_alloc = kmem_cache_alloc(radix_tree_node_cachep, gfp); + xas->xa_alloc = kmem_cache_alloc_lru(radix_tree_node_cachep, xas->xa_lru, gfp); if (!xas->xa_alloc) return false; xas->xa_alloc->parent = NULL; @@ -334,10 +334,10 @@ static bool __xas_nomem(struct xa_state *xas, gfp_t gfp) gfp |= __GFP_ACCOUNT; if (gfpflags_allow_blocking(gfp)) { xas_unlock_type(xas, lock_type); - xas->xa_alloc = kmem_cache_alloc(radix_tree_node_cachep, gfp); + xas->xa_alloc = kmem_cache_alloc_lru(radix_tree_node_cachep, xas->xa_lru, gfp); xas_lock_type(xas, lock_type); } else { - xas->xa_alloc = kmem_cache_alloc(radix_tree_node_cachep, gfp); + xas->xa_alloc = kmem_cache_alloc_lru(radix_tree_node_cachep, xas->xa_lru, gfp); } if (!xas->xa_alloc) return false; @@ -371,7 +371,7 @@ static void *xas_alloc(struct xa_state *xas, unsigned int shift) if (xas->xa->xa_flags & XA_FLAGS_ACCOUNT) gfp |= __GFP_ACCOUNT; - node = kmem_cache_alloc(radix_tree_node_cachep, gfp); + node = kmem_cache_alloc_lru(radix_tree_node_cachep, xas->xa_lru, gfp); if (!node) { xas_set_err(xas, -ENOMEM); return NULL; @@ -1014,7 +1014,7 @@ void xas_split_alloc(struct xa_state *xas, void *entry, unsigned int order, void *sibling = NULL; struct xa_node *node; - node = kmem_cache_alloc(radix_tree_node_cachep, gfp); + node = kmem_cache_alloc_lru(radix_tree_node_cachep, xas->xa_lru, gfp); if (!node) goto nomem; node->array = xas->xa; From patchwork Tue May 11 10:46:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5761C433ED for ; Tue, 11 May 2021 10:52:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5D2FA613C5 for ; Tue, 11 May 2021 10:52:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5D2FA613C5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DF64E8E0001; Tue, 11 May 2021 06:52:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DCC1C6B0075; Tue, 11 May 2021 06:52:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C473C8E0001; Tue, 11 May 2021 06:52:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0109.hostedemail.com [216.40.44.109]) by kanga.kvack.org (Postfix) with ESMTP id AAD336B0072 for ; Tue, 11 May 2021 06:52:53 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6861598BE for ; Tue, 11 May 2021 10:52:53 +0000 (UTC) X-FDA: 78128637426.15.10560E7 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by imf02.hostedemail.com (Postfix) with ESMTP id A6B1B40002C7 for ; Tue, 11 May 2021 10:52:19 +0000 (UTC) Received: by mail-pg1-f181.google.com with SMTP id k15so2710593pgb.10 for ; Tue, 11 May 2021 03:52:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UC6OZyaIhxumzS7XDWC0Pam7E2Uxa7uVbRoAHU6H0rc=; b=mvEtMpGAnQS8X8pRY7QVvYJqqFPUGuS6zE62rkByxR5oNroFVjCS/xKDGbYaocAf7z hH2gWvCBgKxEHy1XHZd1y2KTeOizDfHXJrv50DMYgnnhFlp1AEWzArmF/Ku8n2QKSQrw n0UxiRXkdGjhWXGofRNp7yl60YhBicMzxGfCQgcsFthk50VTyfFnVioBZCT6AnKa8Gel cUU1AQbGq6zAZMkEMMfKyksQht4k9fxrkdQdtKIQhRzpWK35znHAgqk3t5Bx1RDVKacM nKsk4K4cj9yoezeHv+dsCU7Is+7nMQ3Nk4ADCqY+FyBsQV/QqcmlmBAcD1ZLfyqdNc3V UYrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UC6OZyaIhxumzS7XDWC0Pam7E2Uxa7uVbRoAHU6H0rc=; b=FsQdE1KsUixdiUznku1LyVDFszAVf7pTglCXAy/NwHdoRJdL2l6kYLb0VPGgsvPiQ3 Ur1zFgQjvLN/TkZRn/19q5tEplfpfXINkktah5pNh9rGkxUp4iVXh7rT00HUWN4X3kZv AavdPErhTwyaOC1hH5wJAs/EQQA3ZdVVR4QWKhxycfa1pYA4ehukSzC1/0yHU51LKFHy aGb7RzGKozda3+4aixDQCLO9kNui5t1u3TTTvcmG2+L3asibe+548sYg3CIM0+ndCoP8 aZfr76xwJrE5kIOXIqU5aZ2BirMOj+Rt2dFWIzCSWO40BfgTUreColdaE9JksBB3+ZwI NJ5Q== X-Gm-Message-State: AOAM530uwW0UCO77WtrjXXKRPkY3V8QW1wRwWx/BKt33jyVIBvwrim4h PyCDvHKwK0XV3K+WG7j/IRB2jw== X-Google-Smtp-Source: ABdhPJxTL3K0RlFgs0YRY1Okx+0Y055XXZQ5WoXKr6gJFjTzXss/+4eK36NqLpJQOuB3HmVlCM+m1Q== X-Received: by 2002:a63:aa48:: with SMTP id x8mr6373173pgo.359.1620730372289; Tue, 11 May 2021 03:52:52 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.52.45 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:52:52 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 13/17] mm: workingset: allocate list_lru on xa_node allocation Date: Tue, 11 May 2021 18:46:43 +0800 Message-Id: <20210511104647.604-14-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=mvEtMpGA; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf02.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.181 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: 11kdwbd7em3k77sd6qff18bbrbw7jt4i X-Rspamd-Queue-Id: A6B1B40002C7 X-Rspamd-Server: rspam02 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf02; identity=mailfrom; envelope-from=""; helo=mail-pg1-f181.google.com; client-ip=209.85.215.181 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730339-147076 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The workingset will add the xa_node to its list_lru, we should use xas_set_lru() to pass which lru we want to insert. Signed-off-by: Muchun Song --- include/linux/swap.h | 5 ++++- mm/workingset.c | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 144727041e78..a2418ebad13d 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -322,9 +322,12 @@ void workingset_activation(struct page *page); /* Only track the nodes of mappings with shadow entries */ void workingset_update_node(struct xa_node *node); +extern struct list_lru shadow_nodes; #define mapping_set_update(xas, mapping) do { \ - if (!dax_mapping(mapping) && !shmem_mapping(mapping)) \ + if (!dax_mapping(mapping) && !shmem_mapping(mapping)) { \ xas_set_update(xas, workingset_update_node); \ + xas_set_lru(xas, &shadow_nodes); \ + } \ } while (0) /* linux/mm/page_alloc.c */ diff --git a/mm/workingset.c b/mm/workingset.c index b7cdeca5a76d..f6923865be47 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -426,7 +426,7 @@ void workingset_activation(struct page *page) * point where they would still be useful. */ -static struct list_lru shadow_nodes; +struct list_lru shadow_nodes; void workingset_update_node(struct xa_node *node) { From patchwork Tue May 11 10:46:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29FC9C433B4 for ; Tue, 11 May 2021 10:53:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B3A21611ED for ; Tue, 11 May 2021 10:53:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B3A21611ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3957D94000A; Tue, 11 May 2021 06:53:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 36CCF6B0075; Tue, 11 May 2021 06:53:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 25B3294000A; Tue, 11 May 2021 06:53:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0056.hostedemail.com [216.40.44.56]) by kanga.kvack.org (Postfix) with ESMTP id 098FF6B0072 for ; Tue, 11 May 2021 06:53:01 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id BCF3C181AF5CC for ; Tue, 11 May 2021 10:53:00 +0000 (UTC) X-FDA: 78128637720.31.B56D509 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by imf20.hostedemail.com (Postfix) with ESMTP id 348EBF3 for ; Tue, 11 May 2021 10:53:00 +0000 (UTC) Received: by mail-pj1-f48.google.com with SMTP id l10-20020a17090a850ab0290155b06f6267so1083138pjn.5 for ; Tue, 11 May 2021 03:53:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=W7N2ct9tvwpvEjrUOuEEAnTDqNZXNpAiEvvqkroYr+g=; b=mWGtONSkiGQTSidIKAiOi86zc4lRvtDLkmgp92k6lwi/xWGV6Z8f0RmEfk8KHxrylw FNPkkF7D8R4f7sQBN6KwptrKBO64QIShh6wCISgF9YAPhO1VikQ5gMpRU5yW8aukjolB A1px8LnZpEbJGzGpYNIdzONxb+IKCWIDi84djEbcyXDYA1TVWv2ZOWSRwRtFYBtjGf3y 7eE+HNX5kRHnpECIc7KcTnaNoFO4qNHgLIZefQfQXXAGKA5D9uhlBZjegq9AsdII25jI vlklpo3owHbENwSAAAdfKR+cYydbp5MD/DVz+oSKyzNJY2FcTlKYiKeTT7MZziTOrvzJ dMag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=W7N2ct9tvwpvEjrUOuEEAnTDqNZXNpAiEvvqkroYr+g=; b=L5VvUb3GWSNZGF3oBOizGAZHcdj1u/ulqbTu5mnqFbU2o/VQA0NIjvCXTMy7x2M5RB x1e9xtDoESgARO+JZaHjtURjX9W/i1mPDAJKXeBVO6RWREXIopm0tMgkpl9SHqawVNSA OKDj0hdyc6aaYYS+O9S2MUBz2DTRwuFHMpM0vFgf3EsbNz7gMgsskrGGFd45bY+dqLtz 8kUaXD+QFcjloolpfAYpXKVKsbAsRp/5o1HYr3+wNKp/9+xWPeeXYlBE5Ay1iznAKslk 5jblREKVkT+LDZFg3ZtB52a6DtTYg5vAHuPY842w3eURvKV2hzKIr330/vPLGcFvq7kH QM8A== X-Gm-Message-State: AOAM5314pBUv2hY/NoRMejnqEpiighid/x5l8f2IagxIKsG9wXTEekEN 3jlh67YAdYfdKbY4rrvJtBebVQ== X-Google-Smtp-Source: ABdhPJwOJA2BD0y65E44ULMn40Js/gJ8kg9txboXr5CMo2V3MkRk4Nj+rr9hq45AQ03KLQl8ug0oQA== X-Received: by 2002:a17:903:3106:b029:e9:15e8:250e with SMTP id w6-20020a1709033106b02900e915e8250emr29193253plc.33.1620730379548; Tue, 11 May 2021 03:52:59 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.52.52 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:52:59 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 14/17] nfs42: use a specific kmem_cache to allocate nfs4_xattr_entry Date: Tue, 11 May 2021 18:46:44 +0800 Message-Id: <20210511104647.604-15-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=mWGtONSk; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: ujortwnpq65yr5cw5qryeiexjnmm9gez X-Rspamd-Queue-Id: 348EBF3 X-Rspamd-Server: rspam02 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf20; identity=mailfrom; envelope-from=""; helo=mail-pj1-f48.google.com; client-ip=209.85.216.48 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730380-475987 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we want to add the allocated objects to its list_lru, we should use kmem_cache_alloc_lru() to allocate objects. So intruduce nfs4_xattr_entry_cachep which is used to allocate nfs4_xattr_entry. Signed-off-by: Muchun Song --- fs/nfs/nfs42xattr.c | 95 ++++++++++++++++++++++++++--------------------------- 1 file changed, 47 insertions(+), 48 deletions(-) diff --git a/fs/nfs/nfs42xattr.c b/fs/nfs/nfs42xattr.c index 1c4d2a05b401..5b7af9080db0 100644 --- a/fs/nfs/nfs42xattr.c +++ b/fs/nfs/nfs42xattr.c @@ -81,7 +81,7 @@ struct nfs4_xattr_entry { struct hlist_node hnode; struct list_head lru; struct list_head dispose; - char *xattr_name; + const char *xattr_name; void *xattr_value; size_t xattr_size; struct nfs4_xattr_bucket *bucket; @@ -98,6 +98,7 @@ static struct list_lru nfs4_xattr_entry_lru; static struct list_lru nfs4_xattr_large_entry_lru; static struct kmem_cache *nfs4_xattr_cache_cachep; +static struct kmem_cache *nfs4_xattr_entry_cachep; /* * Hashing helper functions. @@ -177,49 +178,27 @@ nfs4_xattr_alloc_entry(const char *name, const void *value, { struct nfs4_xattr_entry *entry; void *valp; - char *namep; - size_t alloclen, slen; - char *buf; - uint32_t flags; + const char *namep; + uint32_t flags = len > PAGE_SIZE ? NFS4_XATTR_ENTRY_EXTVAL : 0; + gfp_t gfp = GFP_KERNEL_ACCOUNT | GFP_NOFS; + struct list_lru *lru; BUILD_BUG_ON(sizeof(struct nfs4_xattr_entry) + XATTR_NAME_MAX + 1 > PAGE_SIZE); - alloclen = sizeof(struct nfs4_xattr_entry); - if (name != NULL) { - slen = strlen(name) + 1; - alloclen += slen; - } else - slen = 0; - - if (alloclen + len <= PAGE_SIZE) { - alloclen += len; - flags = 0; - } else { - flags = NFS4_XATTR_ENTRY_EXTVAL; - } - - buf = kmalloc(alloclen, GFP_KERNEL_ACCOUNT | GFP_NOFS); - if (buf == NULL) + lru = flags & NFS4_XATTR_ENTRY_EXTVAL ? &nfs4_xattr_large_entry_lru : + &nfs4_xattr_entry_lru; + entry = kmem_cache_alloc_lru(nfs4_xattr_entry_cachep, lru, gfp); + if (!entry) return NULL; - entry = (struct nfs4_xattr_entry *)buf; - - if (name != NULL) { - namep = buf + sizeof(struct nfs4_xattr_entry); - memcpy(namep, name, slen); - } else { - namep = NULL; - } - - - if (flags & NFS4_XATTR_ENTRY_EXTVAL) { - valp = kvmalloc(len, GFP_KERNEL_ACCOUNT | GFP_NOFS); - if (valp == NULL) { - kfree(buf); - return NULL; - } - } else if (len != 0) { - valp = buf + sizeof(struct nfs4_xattr_entry) + slen; + namep = kstrdup_const(name, gfp); + if (!namep && name) + goto free_buf; + + if (len != 0) { + valp = kvmalloc(len, gfp); + if (!valp) + goto free_name; } else valp = NULL; @@ -232,23 +211,23 @@ nfs4_xattr_alloc_entry(const char *name, const void *value, entry->flags = flags; entry->xattr_value = valp; - kref_init(&entry->ref); entry->xattr_name = namep; entry->xattr_size = len; - entry->bucket = NULL; - INIT_LIST_HEAD(&entry->lru); - INIT_LIST_HEAD(&entry->dispose); - INIT_HLIST_NODE(&entry->hnode); return entry; +free_name: + kfree_const(namep); +free_buf: + kmem_cache_free(nfs4_xattr_entry_cachep, entry); + return NULL; } static void nfs4_xattr_free_entry(struct nfs4_xattr_entry *entry) { - if (entry->flags & NFS4_XATTR_ENTRY_EXTVAL) - kvfree(entry->xattr_value); - kfree(entry); + kvfree(entry->xattr_value); + kfree_const(entry->xattr_name); + kmem_cache_free(nfs4_xattr_entry_cachep, entry); } static void @@ -289,7 +268,7 @@ nfs4_xattr_alloc_cache(void) { struct nfs4_xattr_cache *cache; - cache = kmem_cache_alloc(nfs4_xattr_cache_cachep, + cache = kmem_cache_alloc_lru(nfs4_xattr_cache_cachep, &nfs4_xattr_cache_lru, GFP_KERNEL_ACCOUNT | GFP_NOFS); if (cache == NULL) return NULL; @@ -992,6 +971,17 @@ static void nfs4_xattr_cache_init_once(void *p) INIT_LIST_HEAD(&cache->dispose); } +static void nfs4_xattr_entry_init_once(void *p) +{ + struct nfs4_xattr_entry *entry = p; + + kref_init(&entry->ref); + entry->bucket = NULL; + INIT_LIST_HEAD(&entry->lru); + INIT_LIST_HEAD(&entry->dispose); + INIT_HLIST_NODE(&entry->hnode); +} + int __init nfs4_xattr_cache_init(void) { int ret = 0; @@ -1003,6 +993,13 @@ int __init nfs4_xattr_cache_init(void) if (nfs4_xattr_cache_cachep == NULL) return -ENOMEM; + nfs4_xattr_entry_cachep = kmem_cache_create("nfs4_xattr_entry", + sizeof(struct nfs4_xattr_entry), 0, + (SLAB_RECLAIM_ACCOUNT | SLAB_MEM_SPREAD | SLAB_ACCOUNT), + nfs4_xattr_entry_init_once); + if (!nfs4_xattr_entry_cachep) + goto out5; + ret = list_lru_init_memcg(&nfs4_xattr_large_entry_lru, &nfs4_xattr_large_entry_shrinker); if (ret) @@ -1040,6 +1037,8 @@ int __init nfs4_xattr_cache_init(void) out3: list_lru_destroy(&nfs4_xattr_large_entry_lru); out4: + kmem_cache_destroy(nfs4_xattr_entry_cachep); +out5: kmem_cache_destroy(nfs4_xattr_cache_cachep); return ret; From patchwork Tue May 11 10:46:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2FD1C433ED for ; Tue, 11 May 2021 10:53:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5A5A7611ED for ; Tue, 11 May 2021 10:53:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5A5A7611ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DA7D094000B; Tue, 11 May 2021 06:53:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D7E216B0075; Tue, 11 May 2021 06:53:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF83C94000B; Tue, 11 May 2021 06:53:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0196.hostedemail.com [216.40.44.196]) by kanga.kvack.org (Postfix) with ESMTP id A2D086B0072 for ; Tue, 11 May 2021 06:53:08 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 603C2181AF5CC for ; Tue, 11 May 2021 10:53:08 +0000 (UTC) X-FDA: 78128638056.13.1123180 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf12.hostedemail.com (Postfix) with ESMTP id 0EB743CA for ; Tue, 11 May 2021 10:52:51 +0000 (UTC) Received: by mail-pl1-f169.google.com with SMTP id b21so10668921plz.0 for ; Tue, 11 May 2021 03:53:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AIJ6Bm+nra3MZOT6RE/mgqHKmSIemQHQMXkUmKw30r0=; b=pl7J7yLOZ4rOnofgbxw2wUdk44kF9RmMAtEsMMXd/H7V7yElZCN3YgC7zID7YH9w2k V1iDFYyYJxobgTVULxxayQo38dXv96Bj5YxCevPD62DVdOol0rN7G+Z4+P+IZDTU/eCT +Bvu2eJ4mHzI0mYjLcbqGrhq3sYoZ2QZqeo+xkKhAQqzePyYwsfv+J2eYM4qpNVxN0gC Sjz1+RnRurRW55DriTN/MjiPBM1LAWqk103Ikry3uY6de9gHLISjxV+Fjyo0G7cQKOp4 vQBlDMpr5vTnvbVpKjZE2Fn/PzHBTovB3mutLhmY1z7fGOjgRd62j+TrEte/kYcgmL14 JAzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AIJ6Bm+nra3MZOT6RE/mgqHKmSIemQHQMXkUmKw30r0=; b=kkGeDbKjjTeFMqI7gi0rsWzN2bpB1CGlHRcUl5x9jZFPlgqpt4fxBumy02UrbqxS1v DP17rO4hlnE1iAhBA8snMqj3MZ4Ty1gJ6rLBoAKX07k56VIrV1OVhUjlDkiqMlENckED RtHkAiXqmK2ZQoFq2zNpq0pzhq4NgX16NJWthEt0H64WmfMfv1hVEp2cNm1C4hs2Grf8 AA3M/c2mpKo9fIYQe7oSGXuSX3uYU+NcEFxxOI93ppbj0cKcTB07MjVqjMbByFMPtBGa DC5+8lFf45vzsnOQvXKMSQuIkeAKuGaXH1e2ii4ivvou1IZfiLd5vHJi3KYUPqL0HukC lW7w== X-Gm-Message-State: AOAM531wTJ5Ib5FqqZQ2tLj4arY0f4M3bmGz9oM8oUPipiVrs5mq6UO8 uVT/s6AbFO33XqihZhw9kqgGiA== X-Google-Smtp-Source: ABdhPJwS4BSVz9Do8hecfmV7Egr+Ohgvhurob05IY2qYSmo8rnJdjXQn+1WWEd7HbkBJW2ihWC+ioQ== X-Received: by 2002:a17:90a:8e82:: with SMTP id f2mr32078079pjo.45.1620730387003; Tue, 11 May 2021 03:53:07 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.52.59 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:53:06 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 15/17] mm: list_lru: allocate list_lru_one only when needed Date: Tue, 11 May 2021 18:46:45 +0800 Message-Id: <20210511104647.604-16-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=pl7J7yLO; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf12.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 0EB743CA X-Stat-Signature: pitsoxo1pqitzq9a6bt3gz57qcafdt9n Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf12; identity=mailfrom; envelope-from=""; helo=mail-pl1-f169.google.com; client-ip=209.85.214.169 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730371-930368 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In our server, we found a suspected memory leak problem. The kmalloc-32 consumes more than 6GB of memory. Other kmem_caches consume less than 2GB memory. After our in-depth analysis, the memory consumption of kmalloc-32 slab cache is the cause of list_lru_one allocation. crash> p memcg_nr_cache_ids memcg_nr_cache_ids = $2 = 24574 memcg_nr_cache_ids is very large and memory consumption of each list_lru can be calculated with the following formula. num_numa_node * memcg_nr_cache_ids * 32 (kmalloc-32) There are 4 numa nodes in our system, so each list_lru consumes ~3MB. crash> list super_blocks | wc -l 952 Every mount will register 2 list lrus, one is for inode, another is for dentry. There are 952 super_blocks. So the total memory is 952 * 2 * 3 MB (~5.6GB). But the number of memory cgroup is less than 500. So I guess more than 12286 containers have been deployed on this machine (I do not know why there are so many containers, it may be a user's bug or the user really want to do that). And memcg_nr_cache_ids has not been reduced to a suitable value. This can waste a lot of memory. Now the infrastructure for dynamic memory allocation is ready, so remove statically allocated memory code to save memory. Signed-off-by: Muchun Song --- include/linux/list_lru.h | 6 ++- mm/list_lru.c | 122 +++++++++++++++++++++++++---------------------- mm/memcontrol.c | 6 ++- 3 files changed, 74 insertions(+), 60 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 2b2fdb4de96a..7d4346b93b24 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -32,6 +32,7 @@ struct list_lru_one { }; struct list_lru_per_memcg { + struct rcu_head rcu; /* array of per cgroup per node lists, indexed by node id */ struct list_lru_one nodes[0]; }; @@ -39,7 +40,7 @@ struct list_lru_per_memcg { struct list_lru_memcg { struct rcu_head rcu; /* array of per cgroup lists, indexed by memcg_cache_id */ - struct list_lru_per_memcg *lrus[0]; + struct list_lru_per_memcg __rcu *lrus[0]; }; struct list_lru_node { @@ -75,7 +76,8 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, __list_lru_init((lru), true, NULL, shrinker) int memcg_update_all_list_lrus(int num_memcgs); -void memcg_drain_all_list_lrus(int src_idx, struct mem_cgroup *dst_memcg); +void memcg_drain_all_list_lrus(struct mem_cgroup *src_memcg, + struct mem_cgroup *dst_memcg); /** * list_lru_add: add an element to the lru list's tail diff --git a/mm/list_lru.c b/mm/list_lru.c index 182cb2a2e64f..b5ed6b797a48 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -60,8 +60,12 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) */ memcg_lrus = rcu_dereference_check(lru->memcg_lrus, lockdep_is_held(&nlru->lock)); - if (memcg_lrus && idx >= 0) - return &memcg_lrus->lrus[idx]->nodes[nid]; + if (memcg_lrus && idx >= 0) { + struct list_lru_per_memcg *mlru; + + mlru = rcu_dereference_check(memcg_lrus->lrus[idx], true); + return mlru ? &mlru->nodes[nid] : NULL; + } return &nlru->lru; } @@ -184,11 +188,12 @@ unsigned long list_lru_count_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg) { struct list_lru_one *l; - long count; + long count = 0; rcu_read_lock(); l = list_lru_from_memcg_idx(lru, nid, memcg_cache_id(memcg)); - count = READ_ONCE(l->nr_items); + if (l) + count = READ_ONCE(l->nr_items); rcu_read_unlock(); if (unlikely(count < 0)) @@ -217,8 +222,11 @@ __list_lru_walk_one(struct list_lru *lru, int nid, int memcg_idx, struct list_head *item, *n; unsigned long isolated = 0; - l = list_lru_from_memcg_idx(lru, nid, memcg_idx); restart: + l = list_lru_from_memcg_idx(lru, nid, memcg_idx); + if (!l) + goto out; + list_for_each_safe(item, n, &l->list) { enum lru_status ret; @@ -262,6 +270,7 @@ __list_lru_walk_one(struct list_lru *lru, int nid, int memcg_idx, BUG(); } } +out: return isolated; } @@ -354,22 +363,6 @@ static struct list_lru_per_memcg *list_lru_per_memcg_alloc(gfp_t gfp) return lru; } -static int memcg_init_list_lru_range(struct list_lru_memcg *memcg_lrus, - int begin, int end) -{ - int i; - - for (i = begin; i < end; i++) { - memcg_lrus->lrus[i] = list_lru_per_memcg_alloc(GFP_KERNEL); - if (!memcg_lrus->lrus[i]) - goto fail; - } - return 0; -fail: - memcg_destroy_list_lru_range(memcg_lrus, begin, i); - return -ENOMEM; -} - static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) { struct list_lru_memcg *memcg_lrus; @@ -382,15 +375,11 @@ static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) spin_lock_init(&lru->lock); - memcg_lrus = kvmalloc(sizeof(*memcg_lrus) + + memcg_lrus = kvzalloc(sizeof(*memcg_lrus) + size * sizeof(memcg_lrus->lrus[0]), GFP_KERNEL); if (!memcg_lrus) return -ENOMEM; - if (memcg_init_list_lru_range(memcg_lrus, 0, size)) { - kvfree(memcg_lrus); - return -ENOMEM; - } RCU_INIT_POINTER(lru->memcg_lrus, memcg_lrus); return 0; @@ -424,13 +413,9 @@ static int memcg_update_list_lru(struct list_lru *lru, int old_size, int new_siz if (!new) return -ENOMEM; - if (memcg_init_list_lru_range(new, old_size, new_size)) { - kvfree(new); - return -ENOMEM; - } - spin_lock_irq(&lru->lock); memcpy(&new->lrus, &old->lrus, old_size * sizeof(new->lrus[0])); + memset(&new->lrus[old_size], 0, (new_size - old_size) * sizeof(new->lrus[0])); rcu_assign_pointer(lru->memcg_lrus, new); spin_unlock_irq(&lru->lock); @@ -439,20 +424,6 @@ static int memcg_update_list_lru(struct list_lru *lru, int old_size, int new_siz return 0; } -static void memcg_cancel_update_list_lru(struct list_lru *lru, - int old_size, int new_size) -{ - struct list_lru_memcg *memcg_lrus; - - memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, - lockdep_is_held(&list_lrus_mutex)); - /* - * Do not bother shrinking the array back to the old size, because we - * cannot handle allocation failures here. - */ - memcg_destroy_list_lru_range(memcg_lrus, old_size, new_size); -} - int memcg_update_all_list_lrus(int new_size) { int ret = 0; @@ -463,15 +434,10 @@ int memcg_update_all_list_lrus(int new_size) list_for_each_entry(lru, &list_lrus, list) { ret = memcg_update_list_lru(lru, old_size, new_size); if (ret) - goto fail; + break; } -out: mutex_unlock(&list_lrus_mutex); return ret; -fail: - list_for_each_entry_continue_reverse(lru, &list_lrus, list) - memcg_cancel_update_list_lru(lru, old_size, new_size); - goto out; } static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, @@ -488,6 +454,8 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, spin_lock_irq(&nlru->lock); src = list_lru_from_memcg_idx(lru, nid, src_idx); + if (!src) + goto out; dst = list_lru_from_memcg_idx(lru, nid, dst_idx); list_splice_init(&src->list, &dst->list); @@ -497,10 +465,32 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0; } - +out: spin_unlock_irq(&nlru->lock); } +static void list_lru_per_memcg_free(struct list_lru *lru, int src_idx) +{ + struct list_lru_memcg *memcg_lrus; + struct list_lru_per_memcg *mlru; + + spin_lock_irq(&lru->lock); + memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, true); + mlru = rcu_dereference_protected(memcg_lrus->lrus[src_idx], true); + if (mlru) + rcu_assign_pointer(memcg_lrus->lrus[src_idx], NULL); + spin_unlock_irq(&lru->lock); + + /* + * The __list_lru_walk_one() can walk the list of this node. + * We need kvfree_rcu() here. And the walking of the list + * is under lru->node[nid]->lock, which can serve as a RCU + * read-side critical section. + */ + if (mlru) + kvfree_rcu(mlru, rcu); +} + static void memcg_drain_list_lru(struct list_lru *lru, int src_idx, struct mem_cgroup *dst_memcg) { @@ -508,11 +498,28 @@ static void memcg_drain_list_lru(struct list_lru *lru, for_each_node(i) memcg_drain_list_lru_node(lru, i, src_idx, dst_memcg); + + list_lru_per_memcg_free(lru, src_idx); } -void memcg_drain_all_list_lrus(int src_idx, struct mem_cgroup *dst_memcg) +void memcg_drain_all_list_lrus(struct mem_cgroup *src_memcg, + struct mem_cgroup *dst_memcg) { struct list_lru *lru; + int src_idx = src_memcg->kmemcg_id; + + /* + * Change kmemcg_id of this cgroup to the parent's id, and then move + * all entries from this cgroup's list_lrus to ones of the parent. + * + * After we have finished, all list_lrus corresponding to this cgroup + * are guaranteed to remain empty. So we can safely free this cgroup's + * list lrus which is implemented in list_lru_per_memcg_free(). + * Changing ->kmemcg_id to the parent can prevent list_lru_memcg_alloc() + * from allocating list lrus for this cgroup after calling + * list_lru_per_memcg_free(). + */ + src_memcg->kmemcg_id = dst_memcg->kmemcg_id; mutex_lock(&list_lrus_mutex); list_for_each_entry(lru, &list_lrus, list) @@ -531,7 +538,7 @@ static bool list_lru_per_memcg_allocated(struct list_lru *lru, rcu_read_lock(); memcg_lrus = rcu_dereference(lru->memcg_lrus); - if (memcg_lrus->lrus[idx]) { + if (rcu_access_pointer(memcg_lrus->lrus[idx])) { rcu_read_unlock(); return true; } @@ -590,11 +597,12 @@ int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t g memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, true); while (i--) { int idx = memcg_cache_id(table[i].memcg); + struct list_lru_per_memcg *mlru = table[i].mlru; - if (memcg_lrus->lrus[idx]) - kfree(table[i].mlru); + if (idx < 0 || rcu_dereference_protected(memcg_lrus->lrus[idx], true)) + kfree(mlru); else - memcg_lrus->lrus[idx] = table[i].mlru; + rcu_assign_pointer(memcg_lrus->lrus[idx], mlru); } spin_unlock_irqrestore(&lru->lock, flags); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6aace39fd1a4..879d2ff8d81f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3484,11 +3484,15 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) memcg_reparent_objcgs(memcg, parent); + /* + * memcg_drain_all_list_lrus() can change memcg->kmemcg_id. + * Cache it to @kmemcg_id. + */ kmemcg_id = memcg->kmemcg_id; BUG_ON(kmemcg_id < 0); /* memcg_reparent_objcgs() must be called before this. */ - memcg_drain_all_list_lrus(kmemcg_id, parent); + memcg_drain_all_list_lrus(memcg, parent); memcg_free_cache_id(kmemcg_id); } From patchwork Tue May 11 10:46:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250627 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DB28C433B4 for ; Tue, 11 May 2021 10:53:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9A8F7611ED for ; Tue, 11 May 2021 10:53:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9A8F7611ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 123ED94000C; Tue, 11 May 2021 06:53:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B9706B0075; Tue, 11 May 2021 06:53:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A7D2394000C; Tue, 11 May 2021 06:53:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0220.hostedemail.com [216.40.44.220]) by kanga.kvack.org (Postfix) with ESMTP id 80D6D6B0072 for ; Tue, 11 May 2021 06:53:15 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 3D000758E for ; Tue, 11 May 2021 10:53:15 +0000 (UTC) X-FDA: 78128638350.22.CA08BB2 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf16.hostedemail.com (Postfix) with ESMTP id 3DDE28019382 for ; Tue, 11 May 2021 10:53:03 +0000 (UTC) Received: by mail-pj1-f52.google.com with SMTP id cl24-20020a17090af698b0290157efd14899so1129324pjb.2 for ; Tue, 11 May 2021 03:53:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=l8N5ZUdYb8l/oRbGA6g7N4k0yzncAYFXCS7KEjhs0TU=; b=pE/EHpen5n4IYXpbRh8rAp8XuzT/8oWVSC4E6SwrUfStEJI8TITTEghKAeCrkLNFZI Q8U/OIBh/EJ3sZqvIvd5t2PPwjaw5PX1aWZcJlv0aAEG3q+uY3KUEf6xwBywKBp4OF9c vDT2wtI7BsdYMX6ltbRtlTC9hmlJ+l0O8yefFrC+65WYut3Dx+8hIHmi6kbs3CLdenAi jf89YMOOS1MDOlKuWkBLfqpSAjwKzqBuIZWoD/Lz8cXePLeDTZ1fkT9ELjjIMsv5UFOV b25XostjxyOgCTo4y5Mz/UwLe4pnqKJkCC++Gs65Y0p46H27uIpS3V9j29mk4eFuLgcV O3uA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l8N5ZUdYb8l/oRbGA6g7N4k0yzncAYFXCS7KEjhs0TU=; b=t5N171xkMplB0Qn1sMO9Wr/uC08aJrO839t7ieiUSAqfPi8SVyKim2N4dPeCir8CDT 7FQU9o5mi8BPvHIGGOkoaNL1DtXbtFaPHkQBu3aIqSe/ZooCb7ICUobMEQPcyuBT3+Mb nX1jTugO0o14NQjxhoF6nkXd53f2J3EVZbiQ+cqRhJa2QDY8h1X+iMVFYtWi/Fv1NtXb Uc4jlrVHsDXBOoRyedeQG3sYQbX4Mx3lf69esuZhHfgm5l7CXwfBTfAhC2wlUERHmA+8 qlzBdPUgwxUkudyFvkub0r8Yd1bPndp2AqAidbzwkM4inxGPs4hUnAvhUO3SLVcK83KM lQKg== X-Gm-Message-State: AOAM530m3zfJhTMZtGuWmNrpRDOPLJhD9EX9Pn64l+brGoXsSgv/VZ/s e/N9f1k9lHBvyODF8Y77c/Y6qA== X-Google-Smtp-Source: ABdhPJyh1wXx5mYfbkgRhiGJmuigd7/EclP3NGsv7ZdHfD+hX+newjmF7fotu22j9mscSwJYcjyXXQ== X-Received: by 2002:a17:902:e804:b029:ed:5748:9047 with SMTP id u4-20020a170902e804b02900ed57489047mr29202382plg.36.1620730394053; Tue, 11 May 2021 03:53:14 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.53.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:53:13 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 16/17] mm: list_lru: rename memcg_drain_all_list_lrus to memcg_reparent_list_lrus Date: Tue, 11 May 2021 18:46:46 +0800 Message-Id: <20210511104647.604-17-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3DDE28019382 Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b="pE/EHpen"; spf=pass (imf16.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam04 X-Stat-Signature: 1ngtzn4ux4i3qwx4b7zhh7jgct5i1e3f Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=mail-pj1-f52.google.com; client-ip=209.85.216.52 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730383-76859 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The purpose of the memcg_drain_all_list_lrus() is list_lrus reparenting. It is very similar to memcg_reparent_objcgs(). Rename it to memcg_reparent_list_lrus() so that the name can more consistent with memcg_reparent_objcgs(). Signed-off-by: Muchun Song --- include/linux/list_lru.h | 3 +-- mm/list_lru.c | 19 +++++++++---------- mm/memcontrol.c | 4 ++-- 3 files changed, 12 insertions(+), 14 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 7d4346b93b24..9222d0295d30 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -76,8 +76,7 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, __list_lru_init((lru), true, NULL, shrinker) int memcg_update_all_list_lrus(int num_memcgs); -void memcg_drain_all_list_lrus(struct mem_cgroup *src_memcg, - struct mem_cgroup *dst_memcg); +void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent); /** * list_lru_add: add an element to the lru list's tail diff --git a/mm/list_lru.c b/mm/list_lru.c index b5ed6b797a48..6d7ae24a4a70 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -440,8 +440,8 @@ int memcg_update_all_list_lrus(int new_size) return ret; } -static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, - int src_idx, struct mem_cgroup *dst_memcg) +static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, + int src_idx, struct mem_cgroup *dst_memcg) { struct list_lru_node *nlru = &lru->node[nid]; int dst_idx = dst_memcg->kmemcg_id; @@ -491,22 +491,21 @@ static void list_lru_per_memcg_free(struct list_lru *lru, int src_idx) kvfree_rcu(mlru, rcu); } -static void memcg_drain_list_lru(struct list_lru *lru, - int src_idx, struct mem_cgroup *dst_memcg) +static void memcg_reparent_list_lru(struct list_lru *lru, + int src_idx, struct mem_cgroup *dst_memcg) { int i; for_each_node(i) - memcg_drain_list_lru_node(lru, i, src_idx, dst_memcg); + memcg_reparent_list_lru_node(lru, i, src_idx, dst_memcg); list_lru_per_memcg_free(lru, src_idx); } -void memcg_drain_all_list_lrus(struct mem_cgroup *src_memcg, - struct mem_cgroup *dst_memcg) +void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent) { struct list_lru *lru; - int src_idx = src_memcg->kmemcg_id; + int src_idx = memcg->kmemcg_id; /* * Change kmemcg_id of this cgroup to the parent's id, and then move @@ -519,11 +518,11 @@ void memcg_drain_all_list_lrus(struct mem_cgroup *src_memcg, * from allocating list lrus for this cgroup after calling * list_lru_per_memcg_free(). */ - src_memcg->kmemcg_id = dst_memcg->kmemcg_id; + memcg->kmemcg_id = parent->kmemcg_id; mutex_lock(&list_lrus_mutex); list_for_each_entry(lru, &list_lrus, list) - memcg_drain_list_lru(lru, src_idx, dst_memcg); + memcg_reparent_list_lru(lru, src_idx, parent); mutex_unlock(&list_lrus_mutex); } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 879d2ff8d81f..02a65ff3b77a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3485,14 +3485,14 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) memcg_reparent_objcgs(memcg, parent); /* - * memcg_drain_all_list_lrus() can change memcg->kmemcg_id. + * memcg_reparent_list_lrus() can change memcg->kmemcg_id. * Cache it to @kmemcg_id. */ kmemcg_id = memcg->kmemcg_id; BUG_ON(kmemcg_id < 0); /* memcg_reparent_objcgs() must be called before this. */ - memcg_drain_all_list_lrus(memcg, parent); + memcg_reparent_list_lrus(memcg, parent); memcg_free_cache_id(kmemcg_id); } From patchwork Tue May 11 10:46:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12250629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32C7DC43460 for ; Tue, 11 May 2021 10:53:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9563E613CF for ; Tue, 11 May 2021 10:53:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9563E613CF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 235386B0072; Tue, 11 May 2021 06:53:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 20DA194000E; Tue, 11 May 2021 06:53:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05E4294000D; Tue, 11 May 2021 06:53:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0096.hostedemail.com [216.40.44.96]) by kanga.kvack.org (Postfix) with ESMTP id DEE956B0072 for ; Tue, 11 May 2021 06:53:22 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 9D55B8249980 for ; Tue, 11 May 2021 10:53:22 +0000 (UTC) X-FDA: 78128638644.24.89A07DF Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf07.hostedemail.com (Postfix) with ESMTP id 23399A000390 for ; Tue, 11 May 2021 10:53:19 +0000 (UTC) Received: by mail-pf1-f182.google.com with SMTP id i190so15731230pfc.12 for ; Tue, 11 May 2021 03:53:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BBiRN9Qb+L7tb15SHS6kXb3JqkS88GevZJZWamU/u9c=; b=RwHXRgnH+vesEcY0QYx5vjyGMyWGtRl7ssMu2L+o1OZbrXw9oH47daTHT86t1N0+cC CeFdYZ56YuEm/WOp3p3PiEJkokskY6QD5daN2uN+tuGmItQrX65bS9BwYyuUGk6xlLRr IU7TeL8xysn3lC2xJ4PkdHRviuX31HYttxULrTo0NXXroDeRuwuMc27bDjANGK0H14Xe F8sLncp3qx76vH9jEtzlfH97iARBgbMWajYsGfd4kNzm9+O4AMYYKGiZo8mZ44iNXt/l Ix2g8+NvBdo7Ir6EtBqSfwUjS382ygGoRLs1xCrpeC7n0zJXZDXZbgN2z+9uobH/ibLZ gpcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BBiRN9Qb+L7tb15SHS6kXb3JqkS88GevZJZWamU/u9c=; b=BTtshAe5MOosG1LSHhMt8D/PReeM5HSDjzLrNSdnM5KMRzdT2QyQIGkhWWCeZuI6eB /2nogcNqK5KerwrINfcUHo4l0SKqK75M+4azwnbH3m213n803c6BXake9KbGI8jQrKM5 8Z3QLYTqqtVP42G8a5AWtzHh4Xy6tljDE0g4awtTC7ZUOcwpvBPytipNM3EK77rSc6f4 r4qgGBKdUrS9fvMeUkpxmW57EY6z4QwHbSOAW4Il28WbMO+AFNPIBnFoNltM0ZoxvVcC nDlj1g1+LiL0U3Rnubi3/SWfeMZ2mcBuDBjV5HLdwhYdAes9/e6DxvWeTLEqbxIuyeLF jx8A== X-Gm-Message-State: AOAM530ZExGmpztr6OpqtZx2nxps8bybroKOE6cCRKQtUAuYQjlYldru viE3Lq98JUDc+owihO8bz5Rem8du2eWR0w== X-Google-Smtp-Source: ABdhPJxB+lrsf+yZbPyYfyf6tVDPr19ruZ2p/Y4axHmt6tZal6TNte7vwgxqvU4Wrb1jm76kqRMudg== X-Received: by 2002:a63:5a19:: with SMTP id o25mr20998682pgb.122.1620730401291; Tue, 11 May 2021 03:53:21 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id n18sm13501952pgj.71.2021.05.11.03.53.14 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 May 2021 03:53:21 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH 17/17] mm: list_lru: replace linear array with xarray Date: Tue, 11 May 2021 18:46:47 +0800 Message-Id: <20210511104647.604-18-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210511104647.604-1-songmuchun@bytedance.com> References: <20210511104647.604-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=RwHXRgnH; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf07.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: fkkmxmd5w6ufeb4douoedc647dnbz5jg X-Rspamd-Queue-Id: 23399A000390 X-Rspamd-Server: rspam02 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=mail-pf1-f182.google.com; client-ip=209.85.210.182 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620730399-938100 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we run 10k containers in the system, the size of the list_lru_memcg->lrus can be ~96KB per list_lru. When we decrease the number containers, the size of the array will not be shrinked. It is not scalable. The xarray is a good choice for this case. We can save a lot of memory when there are tens of thousands continers in the system. If we use xarray, we also can remove the logic code of resizing array, which can simplify the code. Signed-off-by: Muchun Song --- include/linux/list_lru.h | 6 +- include/linux/memcontrol.h | 10 --- mm/list_lru.c | 159 ++++++++++++--------------------------------- mm/memcontrol.c | 71 ++------------------ 4 files changed, 49 insertions(+), 197 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 9222d0295d30..b57556698de0 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -11,6 +11,7 @@ #include #include #include +#include struct mem_cgroup; @@ -56,10 +57,8 @@ struct list_lru { #ifdef CONFIG_MEMCG_KMEM struct list_head list; int shrinker_id; - /* protects ->memcg_lrus->lrus[i] */ - spinlock_t lock; /* for cgroup aware lrus points to per cgroup lists, otherwise NULL */ - struct list_lru_memcg __rcu *memcg_lrus; + struct xarray *xa; #endif }; @@ -75,7 +74,6 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, #define list_lru_init_memcg(lru, shrinker) \ __list_lru_init((lru), true, NULL, shrinker) -int memcg_update_all_list_lrus(int num_memcgs); void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent); /** diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 96a85cff248a..a0c77e8ec61a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1649,8 +1649,6 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size); extern struct static_key_false memcg_kmem_enabled_key; extern int memcg_nr_cache_ids; -void memcg_get_cache_ids(void); -void memcg_put_cache_ids(void); /* * Helper macro to loop through all memcg-specific caches. Callers must still @@ -1725,14 +1723,6 @@ static inline int memcg_cache_id(struct mem_cgroup *memcg) return -1; } -static inline void memcg_get_cache_ids(void) -{ -} - -static inline void memcg_put_cache_ids(void) -{ -} - static inline struct mem_cgroup *mem_cgroup_from_obj(void *p) { return NULL; diff --git a/mm/list_lru.c b/mm/list_lru.c index 6d7ae24a4a70..9878b08d44e4 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -20,7 +20,7 @@ static DEFINE_MUTEX(list_lrus_mutex); static inline bool list_lru_memcg_aware(struct list_lru *lru) { - return !!lru->memcg_lrus; + return !!lru->xa; } static void list_lru_register(struct list_lru *lru) @@ -51,22 +51,12 @@ static int lru_shrinker_id(struct list_lru *lru) static inline struct list_lru_one * list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) { - struct list_lru_memcg *memcg_lrus; - struct list_lru_node *nlru = &lru->node[nid]; + if (list_lru_memcg_aware(lru) && idx >= 0) { + struct list_lru_per_memcg *mlru = xa_load(lru->xa, idx); - /* - * Either lock or RCU protects the array of per cgroup lists - * from relocation (see memcg_update_list_lru). - */ - memcg_lrus = rcu_dereference_check(lru->memcg_lrus, - lockdep_is_held(&nlru->lock)); - if (memcg_lrus && idx >= 0) { - struct list_lru_per_memcg *mlru; - - mlru = rcu_dereference_check(memcg_lrus->lrus[idx], true); return mlru ? &mlru->nodes[nid] : NULL; } - return &nlru->lru; + return &lru->node[nid].lru; } static inline struct list_lru_one * @@ -310,16 +300,18 @@ unsigned long list_lru_walk_node(struct list_lru *lru, int nid, unsigned long *nr_to_walk) { long isolated = 0; - int memcg_idx; isolated += list_lru_walk_one(lru, nid, NULL, isolate, cb_arg, nr_to_walk); if (*nr_to_walk > 0 && list_lru_memcg_aware(lru)) { - for_each_memcg_cache_index(memcg_idx) { + struct list_lru_per_memcg *mlru; + unsigned long index; + + xa_for_each(lru->xa, index, mlru) { struct list_lru_node *nlru = &lru->node[nid]; spin_lock(&nlru->lock); - isolated += __list_lru_walk_one(lru, nid, memcg_idx, + isolated += __list_lru_walk_one(lru, nid, index, isolate, cb_arg, nr_to_walk); spin_unlock(&nlru->lock); @@ -339,15 +331,6 @@ static void init_one_lru(struct list_lru_one *l) } #ifdef CONFIG_MEMCG_KMEM -static void memcg_destroy_list_lru_range(struct list_lru_memcg *memcg_lrus, - int begin, int end) -{ - int i; - - for (i = begin; i < end; i++) - kfree(memcg_lrus->lrus[i]); -} - static struct list_lru_per_memcg *list_lru_per_memcg_alloc(gfp_t gfp) { int nid; @@ -365,79 +348,35 @@ static struct list_lru_per_memcg *list_lru_per_memcg_alloc(gfp_t gfp) static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) { - struct list_lru_memcg *memcg_lrus; - int size = memcg_nr_cache_ids; - if (!memcg_aware) { - lru->memcg_lrus = NULL; + lru->xa = NULL; return 0; } - spin_lock_init(&lru->lock); - - memcg_lrus = kvzalloc(sizeof(*memcg_lrus) + - size * sizeof(memcg_lrus->lrus[0]), GFP_KERNEL); - if (!memcg_lrus) + lru->xa = kmalloc(sizeof(*lru->xa), GFP_KERNEL); + if (!lru->xa) return -ENOMEM; - - RCU_INIT_POINTER(lru->memcg_lrus, memcg_lrus); + xa_init_flags(lru->xa, XA_FLAGS_LOCK_IRQ); return 0; } static void memcg_destroy_list_lru(struct list_lru *lru) { - struct list_lru_memcg *memcg_lrus; + XA_STATE(xas, lru->xa, 0); + struct list_lru_per_memcg *mlru; if (!list_lru_memcg_aware(lru)) return; - /* - * This is called when shrinker has already been unregistered, - * and nobody can use it. So, there is no need to use kvfree_rcu(). - */ - memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, true); - memcg_destroy_list_lru_range(memcg_lrus, 0, memcg_nr_cache_ids); - kvfree(memcg_lrus); -} - -static int memcg_update_list_lru(struct list_lru *lru, int old_size, int new_size) -{ - struct list_lru_memcg *old, *new; - - BUG_ON(old_size > new_size); - - old = rcu_dereference_protected(lru->memcg_lrus, - lockdep_is_held(&list_lrus_mutex)); - new = kvmalloc(sizeof(*new) + new_size * sizeof(new->lrus[0]), GFP_KERNEL); - if (!new) - return -ENOMEM; - - spin_lock_irq(&lru->lock); - memcpy(&new->lrus, &old->lrus, old_size * sizeof(new->lrus[0])); - memset(&new->lrus[old_size], 0, (new_size - old_size) * sizeof(new->lrus[0])); - rcu_assign_pointer(lru->memcg_lrus, new); - spin_unlock_irq(&lru->lock); - - kvfree_rcu(old, rcu); - - return 0; -} - -int memcg_update_all_list_lrus(int new_size) -{ - int ret = 0; - struct list_lru *lru; - int old_size = memcg_nr_cache_ids; - - mutex_lock(&list_lrus_mutex); - list_for_each_entry(lru, &list_lrus, list) { - ret = memcg_update_list_lru(lru, old_size, new_size); - if (ret) - break; + xas_lock_irq(&xas); + xas_for_each(&xas, mlru, ULONG_MAX) { + kfree(mlru); + xas_store(&xas, NULL); } - mutex_unlock(&list_lrus_mutex); - return ret; + xas_unlock_irq(&xas); + + kfree(lru->xa); } static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, @@ -471,15 +410,7 @@ static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, static void list_lru_per_memcg_free(struct list_lru *lru, int src_idx) { - struct list_lru_memcg *memcg_lrus; - struct list_lru_per_memcg *mlru; - - spin_lock_irq(&lru->lock); - memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, true); - mlru = rcu_dereference_protected(memcg_lrus->lrus[src_idx], true); - if (mlru) - rcu_assign_pointer(memcg_lrus->lrus[src_idx], NULL); - spin_unlock_irq(&lru->lock); + struct list_lru_per_memcg *mlru = xa_erase_irq(lru->xa, src_idx); /* * The __list_lru_walk_one() can walk the list of this node. @@ -529,28 +460,18 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren static bool list_lru_per_memcg_allocated(struct list_lru *lru, struct mem_cgroup *memcg) { - struct list_lru_memcg *memcg_lrus; int idx = memcg_cache_id(memcg); - if (unlikely(idx < 0)) + if (unlikely(idx < 0) || xa_load(lru->xa, idx)) return true; - - rcu_read_lock(); - memcg_lrus = rcu_dereference(lru->memcg_lrus); - if (rcu_access_pointer(memcg_lrus->lrus[idx])) { - rcu_read_unlock(); - return true; - } - rcu_read_unlock(); - return false; } int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t gfp) { + XA_STATE(xas, lru->xa, 0); unsigned long flags; - struct list_lru_memcg *memcg_lrus; - int i; + int i, ret = 0; struct list_lru_memcg { struct list_lru_per_memcg *mlru; @@ -592,22 +513,31 @@ int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t g } } - spin_lock_irqsave(&lru->lock, flags); - memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, true); + xas_lock_irqsave(&xas, flags); while (i--) { int idx = memcg_cache_id(table[i].memcg); struct list_lru_per_memcg *mlru = table[i].mlru; - if (idx < 0 || rcu_dereference_protected(memcg_lrus->lrus[idx], true)) + xas_set(&xas, idx); +retry: + if (unlikely(ret || idx < 0 || xas_load(&xas))) { kfree(mlru); - else - rcu_assign_pointer(memcg_lrus->lrus[idx], mlru); + } else { + ret = xa_err(xas_store(&xas, mlru)); + if (ret) { + xas_unlock_irqrestore(&xas, flags); + if (xas_nomem(&xas, gfp)) + ret = 0; + xas_lock_irqsave(&xas, flags); + goto retry; + } + } } - spin_unlock_irqrestore(&lru->lock, flags); + xas_unlock_irqrestore(&xas, flags); kfree(table); - return 0; + return ret; } #else static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) @@ -632,7 +562,6 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, else lru->shrinker_id = -1; #endif - memcg_get_cache_ids(); lru->node = kcalloc(nr_node_ids, sizeof(*lru->node), GFP_KERNEL); if (!lru->node) @@ -655,7 +584,6 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, list_lru_register(lru); out: - memcg_put_cache_ids(); return err; } EXPORT_SYMBOL_GPL(__list_lru_init); @@ -666,8 +594,6 @@ void list_lru_destroy(struct list_lru *lru) if (!lru->node) return; - memcg_get_cache_ids(); - list_lru_unregister(lru); memcg_destroy_list_lru(lru); @@ -677,6 +603,5 @@ void list_lru_destroy(struct list_lru *lru) #ifdef CONFIG_MEMCG_KMEM lru->shrinker_id = -1; #endif - memcg_put_cache_ids(); } EXPORT_SYMBOL_GPL(list_lru_destroy); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 02a65ff3b77a..b30832266bf7 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -352,28 +352,9 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg, * This will be used as a shrinker list's index. * The main reason for not using cgroup id for this: * this works better in sparse environments, where we have a lot of memcgs, - * but only a few kmem-limited. Or also, if we have, for instance, 200 - * memcgs, and none but the 200th is kmem-limited, we'd have to have a - * 200 entry array for that. - * - * The current size of the caches array is stored in memcg_nr_cache_ids. It - * will double each time we have to increase it. + * but only a few kmem-limited. */ static DEFINE_IDA(memcg_cache_ida); -int memcg_nr_cache_ids; - -/* Protects memcg_nr_cache_ids */ -static DECLARE_RWSEM(memcg_cache_ids_sem); - -void memcg_get_cache_ids(void) -{ - down_read(&memcg_cache_ids_sem); -} - -void memcg_put_cache_ids(void) -{ - up_read(&memcg_cache_ids_sem); -} /* * MIN_SIZE is different than 1, because we would like to avoid going through @@ -2832,49 +2813,6 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) return objcg; } -static int memcg_alloc_cache_id(void) -{ - int id, size; - int err; - - id = ida_simple_get(&memcg_cache_ida, - 0, MEMCG_CACHES_MAX_SIZE, GFP_KERNEL); - if (id < 0) - return id; - - if (id < memcg_nr_cache_ids) - return id; - - /* - * There's no space for the new id in memcg_caches arrays, - * so we have to grow them. - */ - down_write(&memcg_cache_ids_sem); - - size = 2 * (id + 1); - if (size < MEMCG_CACHES_MIN_SIZE) - size = MEMCG_CACHES_MIN_SIZE; - else if (size > MEMCG_CACHES_MAX_SIZE) - size = MEMCG_CACHES_MAX_SIZE; - - err = memcg_update_all_list_lrus(size); - if (!err) - memcg_nr_cache_ids = size; - - up_write(&memcg_cache_ids_sem); - - if (err) { - ida_simple_remove(&memcg_cache_ida, id); - return err; - } - return id; -} - -static void memcg_free_cache_id(int id) -{ - ida_simple_remove(&memcg_cache_ida, id); -} - /* * obj_cgroup_uncharge_pages: uncharge a number of kernel pages from a objcg * @objcg: object cgroup to uncharge @@ -3451,13 +3389,14 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) BUG_ON(memcg->kmemcg_id >= 0); - memcg_id = memcg_alloc_cache_id(); + memcg_id = ida_simple_get(&memcg_cache_ida, 0, MEMCG_CACHES_MAX_SIZE, + GFP_KERNEL); if (memcg_id < 0) return memcg_id; objcg = obj_cgroup_alloc(); if (!objcg) { - memcg_free_cache_id(memcg_id); + ida_simple_remove(&memcg_cache_ida, memcg_id); return -ENOMEM; } objcg->memcg = memcg; @@ -3494,7 +3433,7 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) /* memcg_reparent_objcgs() must be called before this. */ memcg_reparent_list_lrus(memcg, parent); - memcg_free_cache_id(kmemcg_id); + ida_simple_remove(&memcg_cache_ida, kmemcg_id); } #else static int memcg_online_kmem(struct mem_cgroup *memcg)