From patchwork Thu May 27 06:21:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68638C4707F for ; Thu, 27 May 2021 06:24:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BF71A610CC for ; Thu, 27 May 2021 06:24:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BF71A610CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5C5A86B006E; Thu, 27 May 2021 02:24:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 59BDA6B0070; Thu, 27 May 2021 02:24:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43D946B0071; Thu, 27 May 2021 02:24:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0097.hostedemail.com [216.40.44.97]) by kanga.kvack.org (Postfix) with ESMTP id 0EB766B006E for ; Thu, 27 May 2021 02:24:23 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 914CA180ACEEB for ; Thu, 27 May 2021 06:24:22 +0000 (UTC) X-FDA: 78186021564.03.1B860BF Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) by imf13.hostedemail.com (Postfix) with ESMTP id 99988E00080A for ; Thu, 27 May 2021 06:24:14 +0000 (UTC) Received: by mail-pg1-f182.google.com with SMTP id l70so2947147pga.1 for ; Wed, 26 May 2021 23:24:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7xcEWYskDBGu/bK3eKJT7YkQNy1YoQQPMH2C2+EMstk=; b=Q6/3D8pP+YPIY9c5gcugaTZWMDQovoBE/dn87+UA9E08IM6mZBijQrgLYo4IpMpwaM xdy8cDIDZRsXcysP+1i5noZsePA/87hpIsdudzFRoSVmyo6s86mwl+K+May6cNCegfkI Iy6GQGsUUytBgDnzvk2ByMnQBqgCshQLnunSjNfLLuZ7SJbQ8bSvTGmABFaPDY/N0l+j KxGtMcsR8YMEwc/ruSFE3PO+Jc0zJUDvz/FWuNZVCubkBKePUU4aNwu/7S/tQLDCiC5P u7OexXHYaa4olTIx7NkV5gtaPpWNlgCqKaTaK9gWuuWGhCsffUOgvT4jEgZxQ1DBUGzC cWTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7xcEWYskDBGu/bK3eKJT7YkQNy1YoQQPMH2C2+EMstk=; b=Of418zNM/rErVgHEnflG1OhkTRRSaC5vMYFLviUUw3O0CRCvV2+BOguJrTFIMDTptK wmFJg315tDosPoRs1kKPkRb/J4dwFJLJMFkIa7CE11zQZj480Z6DtrDZjvL6Ru7zNzJ7 253BCnxFYv1Bao9yh5y8ezTdxyunrTYwW995LiDaHN28R/WfWfsUPTJlpC6zX3kdPZl5 JpxT5kaBa8u6FmCzjpnrco4wZZM9k2CGF9Htinqz/0DKILutQ74/C1MO7GWiLaoE/2pH iU29Ypvr0NqH7wNKlbxWfZrYkrnOJVhyP5QCVKlXdjbsF104tt6wyfkNo2RB+Nt8mbDB G3Ng== X-Gm-Message-State: AOAM5310Y49mjP9An3MVQufXejoQJY6CQgaUc2dZuFve/9ZgNrFeq1Of E9+biyAnnf8Jgi4fQUfTgcs39g== X-Google-Smtp-Source: ABdhPJzrZBmPL3AYKSHKsLxo8L8abN9IDlFQW8JyPdoRceqGeMwyxDBF0enNTeguNaomtOhemc8nEw== X-Received: by 2002:a63:4d52:: with SMTP id n18mr2233081pgl.147.1622096661406; Wed, 26 May 2021 23:24:21 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.24.13 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:24:21 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 01/21] mm: list_lru: fix list_lru_count_one() return value Date: Thu, 27 May 2021 14:21:28 +0800 Message-Id: <20210527062148.9361-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b="Q6/3D8pP"; spf=pass (imf13.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.182 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 99988E00080A X-Stat-Signature: yz6uwe4q4s9ijxa7b5xqbxnyrpbzxegn X-HE-Tag: 1622096654-669763 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since commit 2788cf0c401c ("memcg: reparent list_lrus and free kmemcg_id on css offline"), the ->nr_items can be negative during memory cgroup reparenting. In this case, list_lru_count_one() returns an unusual and huge value. It can surprise users. So let it return zero when ->nr_items is negative. Signed-off-by: Muchun Song --- mm/list_lru.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index cd58790d0fb3..4962d48d4410 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -176,13 +176,16 @@ unsigned long list_lru_count_one(struct list_lru *lru, { struct list_lru_node *nlru = &lru->node[nid]; struct list_lru_one *l; - unsigned long count; + long count; rcu_read_lock(); l = list_lru_from_memcg_idx(nlru, memcg_cache_id(memcg)); count = READ_ONCE(l->nr_items); rcu_read_unlock(); + if (unlikely(count < 0)) + count = 0; + return count; } EXPORT_SYMBOL_GPL(list_lru_count_one); From patchwork Thu May 27 06:21:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AB62C4707F for ; Thu, 27 May 2021 06:24:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DEB90613C9 for ; Thu, 27 May 2021 06:24:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DEB90613C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E696B6B0070; Thu, 27 May 2021 02:24:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E10CF6B0071; Thu, 27 May 2021 02:24:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81A6E6B0072; Thu, 27 May 2021 02:24:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 23FB26B0070 for ; Thu, 27 May 2021 02:24:30 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id BEB3899B2 for ; Thu, 27 May 2021 06:24:29 +0000 (UTC) X-FDA: 78186021858.22.0A92D72 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by imf01.hostedemail.com (Postfix) with ESMTP id B7A8D500166D for ; Thu, 27 May 2021 06:24:21 +0000 (UTC) Received: by mail-pg1-f181.google.com with SMTP id j12so2918548pgh.7 for ; Wed, 26 May 2021 23:24:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Q7vJMAr6kcKAU4X210/nbOxFkuInBwsFSpIer8aaFUI=; b=MJSyMYT7SXXVY5RKfMgD9gPHEWPZfFgJ4hyxtY+uBDRdgQ5e7Oit1Zx3N5s4m0xdRY jCfKSSNkrcdj4bSRih64lYEPhpVfJ8HvTeZpPvnpy2xV0P99YtQdGo/K8pndCpAFNasx BNf2rQQzAIJ7upbjmJzl+74rmrW+Xc0nOiWprLlOVWCUBpLYVyR/sHGux9KprYmeuEZN 1EV6dB1bpOPrIWrVk+bseaN9DWHVOP4BrUdVSLwsaxxx4+tfRwrKQfnLKPEmAciAVwOa IBmHlybm2A/ZVmivR0R/FIl07T1NMscahzhc1EBy2pbgolJGn8qGnKVAd0epw0ZzjuEI zgPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Q7vJMAr6kcKAU4X210/nbOxFkuInBwsFSpIer8aaFUI=; b=T/Zb9nqKqN4rrcWR5Nmizo+7973UknGBouLR2PO3Hxke6VJ1WKHZftUF6kEaHQOpIc UHrbpCTtHCsWyDotPNX1o4H/p6Pg2XA+zgke3XqzcgpC9kpOMg4+NiMS5m4x/kYcaKAU pezLNZDFJsv2QX6sFu9jQ+cj/mZZfecNga40lWOPROelrl2rmDS8jWml970lwNGajS8B XJX8Xf1MP4LV3t2s/ityaWWVyNkuS+MacAmwZjML0WGBasb6U/Rt6+Q08lS6YolrxqUP aIaRfBHxxSWeGn8w1e/uk3sXOtILfDyfaqhsRI/oOf+Ds1yQrr9wenYiDpAjEV2TXkVi ojGg== X-Gm-Message-State: AOAM533osm1ucZ7EY07lCEFzsajAaEfbMCj7KoiMmG4+wNaFib+K4vuX H3n8h8oN7Xt5KLVtCD9KJbsGFA== X-Google-Smtp-Source: ABdhPJy8Bw2nz9uQDINiomY4QIvSyT7wuAqcIJ8o4jUTbLhRRcBzsCGLwm+NacVvN/K7nNkk9DkQ/Q== X-Received: by 2002:aa7:8491:0:b029:2dc:b1cc:5532 with SMTP id u17-20020aa784910000b02902dcb1cc5532mr1862562pfn.3.1622096668689; Wed, 26 May 2021 23:24:28 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.24.21 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:24:28 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 02/21] mm: memcontrol: remove kmemcg_id reparenting Date: Thu, 27 May 2021 14:21:29 +0800 Message-Id: <20210527062148.9361-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B7A8D500166D Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=MJSyMYT7; spf=pass (imf01.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.181 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam04 X-Stat-Signature: gntzqoae3tmw5f8kau4s1e6z69im5kgp X-HE-Tag: 1622096661-691676 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since slab objects and kmem pages are charged to object cgroup instead of memory cgroup, memcg_reparent_objcgs() will reparent this cgroup and all its descendants to the parent cgroup. This already makes further list_lru_add()'s add elements to the parent's list. So we do not need to change kmemcg_id of an offline cgroup to its parent's id. It is just waste CPU cycles. Just remove those redundant code. Signed-off-by: Muchun Song --- mm/memcontrol.c | 20 ++------------------ 1 file changed, 2 insertions(+), 18 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index db29b96f7311..9add859f69d7 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3654,8 +3654,7 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) static void memcg_offline_kmem(struct mem_cgroup *memcg) { - struct cgroup_subsys_state *css; - struct mem_cgroup *parent, *child; + struct mem_cgroup *parent; int kmemcg_id; if (memcg->kmem_state != KMEM_ONLINE) @@ -3672,22 +3671,7 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) kmemcg_id = memcg->kmemcg_id; BUG_ON(kmemcg_id < 0); - /* - * Change kmemcg_id of this cgroup and all its descendants to the - * parent's id, and then move all entries from this cgroup's list_lrus - * to ones of the parent. After we have finished, all list_lrus - * corresponding to this cgroup are guaranteed to remain empty. The - * ordering is imposed by list_lru_node->lock taken by - * memcg_drain_all_list_lrus(). - */ - rcu_read_lock(); /* can be called from css_free w/o cgroup_mutex */ - css_for_each_descendant_pre(css, &memcg->css) { - child = mem_cgroup_from_css(css); - BUG_ON(child->kmemcg_id != kmemcg_id); - child->kmemcg_id = parent->kmemcg_id; - } - rcu_read_unlock(); - + /* memcg_reparent_objcgs() must be called before this. */ memcg_drain_all_list_lrus(kmemcg_id, parent); memcg_free_cache_id(kmemcg_id); From patchwork Thu May 27 06:21:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93229C4708B for ; Thu, 27 May 2021 06:24:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 222286135F for ; Thu, 27 May 2021 06:24:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 222286135F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B1C2C6B0071; Thu, 27 May 2021 02:24:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ACCD66B0072; Thu, 27 May 2021 02:24:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 947746B0073; Thu, 27 May 2021 02:24:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0117.hostedemail.com [216.40.44.117]) by kanga.kvack.org (Postfix) with ESMTP id 63BFE6B0071 for ; Thu, 27 May 2021 02:24:37 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 007AC181AF5C6 for ; Thu, 27 May 2021 06:24:36 +0000 (UTC) X-FDA: 78186022194.25.164D4EA Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf25.hostedemail.com (Postfix) with ESMTP id B5D996000250 for ; Thu, 27 May 2021 06:24:28 +0000 (UTC) Received: by mail-pj1-f50.google.com with SMTP id v13-20020a17090abb8db029015f9f7d7290so4917542pjr.0 for ; Wed, 26 May 2021 23:24:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YJFialfFytJYqFwPaZ9fuqS4bMePk6YbpU38QlCLQ58=; b=UnH3qWYxoooroXQ84YWu+wKRLhX33XZ8fReFOkElDGPZjIkYvD7Jr2qj/bawaxtuVk 0wrpttY3vPBp708taxt6/qzJsuldK0fBy3bLgTCrPL3d0sC9numUAvWqXjuZXeELsvc/ LMxaiK1APmuWQ9LaWXQKeYFHr4fqK8x8yylDOlhBWr+nvuDOhjHjszRXm7qi22/ExL70 sxt45gqta9o5nuddGF1vkcXYREgjkAsocjKBfinC9o9HaNao8SOR+yFPVe7teNDGoP1E XyyvfKZ0MpR5fgkMVUA/dPdakTQo0UE9CyPKCmNj8sZ0YHL2McUxOmM5NWOicVtKPJqo CFTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YJFialfFytJYqFwPaZ9fuqS4bMePk6YbpU38QlCLQ58=; b=Vxf+2m9qMsg/vibGru3Ysv14GS0zMMjbLgopk0BBwnrfqBzaaSQQq+x+n7kNzC2ynp iIXhtHLqawN4He0TX/T4trphPq75SBUfGDcZIOQ7NaMp9K3qz85X+mYjALdDPKzvaeD6 XwiMaY062agOGypR9vD4suWa6fXRhm5VdptQo2FXvZb0jJfPVo1dtc5xWfGzgVVvbeGb k+UjqtJjQAY5ZhXnWB0qcalZuc0lxlWWvBcFmK8iIXMJArk7BAwvyzdReSbeB5lyXaX2 swcDHQFDemsCmt5Z676u1GQrxQ2cQ06xMbQ8k8dHXaPYGED8X1tgWjzVjoz267gA4MWI rW1g== X-Gm-Message-State: AOAM530FFPcMDIib6sQd8jNs7I9+wa/13eyfqHGAR1NphbG92fMzfdeS 6W0DHP1+3FWYGxSghDPKGQFXlg== X-Google-Smtp-Source: ABdhPJwVyZdBlb+jbI2xQkR+YujTB8jkU0xSIH8ff8LC+cL41/CANvCFkoeDu7g/QOWAnNTmHbCb1g== X-Received: by 2002:a17:90b:4ac5:: with SMTP id mh5mr2108699pjb.226.1622096675874; Wed, 26 May 2021 23:24:35 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.24.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:24:35 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 03/21] mm: memcontrol: remove the kmem states Date: Thu, 27 May 2021 14:21:30 +0800 Message-Id: <20210527062148.9361-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=UnH3qWYx; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf25.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.50 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: B5D996000250 X-Stat-Signature: 4w7c66tw6ymehpsopmuhtpzwmt7pmko8 X-HE-Tag: 1622096668-429872 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now the kmem states is only used to indicate whether the kmem is offline. Because css_alloc() could fail, then we didn't make the kmem offline. In this case, we need the kmem state to mark this so that memcg_free_kmem() can make the kmem offline. However, we can set ->kmemcg_id to -1 to indicate the kmem has been offline. So we can remove the kmem states to simplify the code. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 7 ------- mm/memcontrol.c | 9 +++------ 2 files changed, 3 insertions(+), 13 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 0ce97eff79e2..6d0638e13fc1 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -182,12 +182,6 @@ struct mem_cgroup_thresholds { struct mem_cgroup_threshold_ary *spare; }; -enum memcg_kmem_state { - KMEM_NONE, - KMEM_ALLOCATED, - KMEM_ONLINE, -}; - #if defined(CONFIG_SMP) struct memcg_padding { char x[0]; @@ -320,7 +314,6 @@ struct mem_cgroup { #ifdef CONFIG_MEMCG_KMEM int kmemcg_id; - enum memcg_kmem_state kmem_state; struct obj_cgroup __rcu *objcg; struct list_head objcg_list; /* list of inherited objcgs */ #endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9add859f69d7..23a9fc8dc143 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3630,7 +3630,6 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) return 0; BUG_ON(memcg->kmemcg_id >= 0); - BUG_ON(memcg->kmem_state); memcg_id = memcg_alloc_cache_id(); if (memcg_id < 0) @@ -3647,7 +3646,6 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) static_branch_enable(&memcg_kmem_enabled_key); memcg->kmemcg_id = memcg_id; - memcg->kmem_state = KMEM_ONLINE; return 0; } @@ -3657,11 +3655,9 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) struct mem_cgroup *parent; int kmemcg_id; - if (memcg->kmem_state != KMEM_ONLINE) + if (cgroup_memory_nokmem) return; - memcg->kmem_state = KMEM_ALLOCATED; - parent = parent_mem_cgroup(memcg); if (!parent) parent = root_mem_cgroup; @@ -3675,12 +3671,13 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) memcg_drain_all_list_lrus(kmemcg_id, parent); memcg_free_cache_id(kmemcg_id); + memcg->kmemcg_id = -1; } static void memcg_free_kmem(struct mem_cgroup *memcg) { /* css_alloc() failed, offlining didn't happen */ - if (unlikely(memcg->kmem_state == KMEM_ONLINE)) + if (unlikely(memcg->kmemcg_id != -1)) memcg_offline_kmem(memcg); } #else From patchwork Thu May 27 06:21:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D0A7C47089 for ; Thu, 27 May 2021 06:24:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A7FBD6135F for ; Thu, 27 May 2021 06:24:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A7FBD6135F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4DB966B0072; Thu, 27 May 2021 02:24:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4B3B96B0073; Thu, 27 May 2021 02:24:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 353298D0001; Thu, 27 May 2021 02:24:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0163.hostedemail.com [216.40.44.163]) by kanga.kvack.org (Postfix) with ESMTP id 003666B0072 for ; Thu, 27 May 2021 02:24:44 -0400 (EDT) Received: from smtpin33.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 9328F180ACEEB for ; Thu, 27 May 2021 06:24:44 +0000 (UTC) X-FDA: 78186022488.33.D3FDEFC Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by imf06.hostedemail.com (Postfix) with ESMTP id D8BB7C0007CB for ; Thu, 27 May 2021 06:24:36 +0000 (UTC) Received: by mail-pj1-f54.google.com with SMTP id g24so2145792pji.4 for ; Wed, 26 May 2021 23:24:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=69t5JvkMUpa4S3zHMzIPphToxILDMFyOxSV05k21NRo=; b=HKrLitcBnnklOOWAEzdjePdIbJuUgcw2SjWSInye4lqApH/rU0YcYtdFyq/JhyyFoG Su+eQeNmI0dL5rm6AjU4TNwh7Cg/2BG2fW/54JrJNPMdw0DSbUCF98FyR6DMchWC742O YEbNH8cBvRolyAvZ7fPJm3xEFzSKD6S7nZZ4chf7nnrx5f7Oj4yoKFFVxp2noXxKTiQr 04fVYRoe8f/H88BRNeu84IxYynvj3YflDbnQIR7hks2nJU1fXSZCf7AuwIkfagDooa/9 7JXKTpGfB0m4oKQFs/dL4v6rMwUMkavZlMi5+jXC+PVmOxDWraZ5CiqdbxoSm2TH5bgw YSyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=69t5JvkMUpa4S3zHMzIPphToxILDMFyOxSV05k21NRo=; b=Nm8263XyqO1fluiMiJVtvN5hy6T2gliOcgFn96AOXT0EV1A6fqHKycrsUUn7oURFd6 QKm/zrPR+HFa29AlzmeBJR7DW6md04iaZUUQTGWoUWsNg/P6Zwp7FujALMs+rF+iFZyl X+/oHTtBvNz/MpPN73JZPSnXxlykPuMNFrnhnF3Eqfoln7giqT6edkRVfzIPeHU5W7Qo kOqsu0OtS4qW+xA2rbsipOVHWtuqc50iiBz5aJAqMKtMR2PBBHfOwe2sYR1eVztfwUUr MXkrb+HLxJ9pUboxvuaYYtmu6qPtmLhyn71FtJlowv5v2Dtpxu5xaENEwymAMa3zWdDa hA8w== X-Gm-Message-State: AOAM532P9AU1i/2+doXjChb0ue1cU4Z5YYmoiEXWZjXKSMc1DrGc6B+9 ZR35TmfLqv/TYitctp047wt7sA== X-Google-Smtp-Source: ABdhPJxmvTKJ6I8q166ESGz1b/+UAvPTvctOrfMJF1LBcEehrkaOaWTcdudYGcw9acZPesyY84UDFw== X-Received: by 2002:a17:902:db0f:b029:f3:e5f4:87f1 with SMTP id m15-20020a170902db0fb02900f3e5f487f1mr1856086plx.26.1622096683363; Wed, 26 May 2021 23:24:43 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.24.36 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:24:43 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 04/21] mm: memcontrol: do it in mem_cgroup_css_online to make the kmem online Date: Thu, 27 May 2021 14:21:31 +0800 Message-Id: <20210527062148.9361-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D8BB7C0007CB Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=HKrLitcB; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf06.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam03 X-Stat-Signature: pwyix1wu7crrtqdcsg9ga71nb8xtc5cs X-HE-Tag: 1622096676-337432 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we do it in the mem_cgroup_css_online() to make the kmem online, we do not need to set ->kmemcg_id when the kmem is offline. And we also can remove memcg_free_kmem(). So just do that to simplify the code. Signed-off-by: Muchun Song --- mm/memcontrol.c | 42 +++++++++++++++--------------------------- 1 file changed, 15 insertions(+), 27 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 23a9fc8dc143..377ec9847179 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3629,7 +3629,8 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) if (cgroup_memory_nokmem) return 0; - BUG_ON(memcg->kmemcg_id >= 0); + if (unlikely(mem_cgroup_is_root(memcg))) + return 0; memcg_id = memcg_alloc_cache_id(); if (memcg_id < 0) @@ -3658,6 +3659,9 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) if (cgroup_memory_nokmem) return; + if (unlikely(mem_cgroup_is_root(memcg))) + return; + parent = parent_mem_cgroup(memcg); if (!parent) parent = root_mem_cgroup; @@ -3665,20 +3669,11 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) memcg_reparent_objcgs(memcg, parent); kmemcg_id = memcg->kmemcg_id; - BUG_ON(kmemcg_id < 0); /* memcg_reparent_objcgs() must be called before this. */ memcg_drain_all_list_lrus(kmemcg_id, parent); memcg_free_cache_id(kmemcg_id); - memcg->kmemcg_id = -1; -} - -static void memcg_free_kmem(struct mem_cgroup *memcg) -{ - /* css_alloc() failed, offlining didn't happen */ - if (unlikely(memcg->kmemcg_id != -1)) - memcg_offline_kmem(memcg); } #else static int memcg_online_kmem(struct mem_cgroup *memcg) @@ -3688,9 +3683,6 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) static void memcg_offline_kmem(struct mem_cgroup *memcg) { } -static void memcg_free_kmem(struct mem_cgroup *memcg) -{ -} #endif /* CONFIG_MEMCG_KMEM */ static int memcg_update_kmem_max(struct mem_cgroup *memcg, @@ -5219,7 +5211,6 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) { struct mem_cgroup *parent = mem_cgroup_from_css(parent_css); struct mem_cgroup *memcg, *old_memcg; - long error = -ENOMEM; old_memcg = set_active_memcg(parent); memcg = mem_cgroup_alloc(); @@ -5249,38 +5240,36 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) } /* The following stuff does not apply to the root */ - error = memcg_online_kmem(memcg); - if (error) - goto fail; - if (cgroup_subsys_on_dfl(memory_cgrp_subsys) && !cgroup_memory_nosocket) static_branch_inc(&memcg_sockets_enabled_key); return &memcg->css; -fail: - mem_cgroup_id_remove(memcg); - mem_cgroup_free(memcg); - return ERR_PTR(error); } static int mem_cgroup_css_online(struct cgroup_subsys_state *css) { struct mem_cgroup *memcg = mem_cgroup_from_css(css); + if (memcg_online_kmem(memcg)) + goto remove_id; + /* * A memcg must be visible for expand_shrinker_info() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (alloc_shrinker_info(memcg)) { - mem_cgroup_id_remove(memcg); - return -ENOMEM; - } + if (alloc_shrinker_info(memcg)) + goto offline_kmem; /* Online state pins memcg ID, memcg ID pins CSS */ refcount_set(&memcg->id.ref, 1); css_get(css); return 0; +offline_kmem: + memcg_offline_kmem(memcg); +remove_id: + mem_cgroup_id_remove(memcg); + return -ENOMEM; } static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) @@ -5338,7 +5327,6 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); free_shrinker_info(memcg); - memcg_free_kmem(memcg); mem_cgroup_free(memcg); } From patchwork Thu May 27 06:21:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 713FDC47089 for ; Thu, 27 May 2021 06:24:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1E356613D8 for ; Thu, 27 May 2021 06:24:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1E356613D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AF6356B0073; Thu, 27 May 2021 02:24:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ACB036B0074; Thu, 27 May 2021 02:24:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 920626B0075; Thu, 27 May 2021 02:24:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0138.hostedemail.com [216.40.44.138]) by kanga.kvack.org (Postfix) with ESMTP id 5EC316B0073 for ; Thu, 27 May 2021 02:24:54 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id F3ACDA2BE for ; Thu, 27 May 2021 06:24:53 +0000 (UTC) X-FDA: 78186022866.13.94BD70D Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf25.hostedemail.com (Postfix) with ESMTP id BB479600025E for ; Thu, 27 May 2021 06:24:45 +0000 (UTC) Received: by mail-pf1-f181.google.com with SMTP id c12so2861099pfl.3 for ; Wed, 26 May 2021 23:24:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4Skb3PkDwb/3WBbW5QytVqXcn8HE8kv5ZTlpyL+++g4=; b=0i7WfGFj/SwesZLMTohEbH+Evib1sQbS+c7e0TNLaQ9zFvhes7YR7vBubu1DEN7o65 U4kaKhsu0ps7yC0WmKlbpnH0Yp1uTp7xsDGZtgw9mvOCUbAxovpLxfdgb1Un1HEs1kD6 vPZkVvlpwqjulQqgd6Lx6rRF9WdZktw/surrmqDbPHFe0cmXZ97ypZ4oLDN8fzh2IxHc 48ovLGD59G7RK+b3/KI/CpYUUsU53ThIfbrik6hoygjAJR2Eib6bSSH51q+9wJb/G0ql 37NfnccShXUqf+ftAHTWbtj9kN/fR4pwwzGjRik/DodaikwAh5Ue66hT25q3+VkSQG3k 6asQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4Skb3PkDwb/3WBbW5QytVqXcn8HE8kv5ZTlpyL+++g4=; b=Tu5cWwYMLuCIJo1rrQjF8w3Yi3EKJzNQJegOY7+Uaov0RnELotndY7nGf4Kvt3aACx ZA99wgQ1XhDi6iRYE65hh1HDTbgHixPfjApmgfEeDypNrQxlgP6tbM7m8x131BFLte8c qIwkQQ9sc3gEefIn+nehQCyct1kCpoJP1g0PmpAOy0yMouncVBQPtZQuH+O6Z67qcJDm jlB1vx9Nme46KnDJTLafE+iy/zxk3sqzJ3VyC7ZoQqgqsGeqFXik9nc8cekA1eO3dvMA RVoLTFwDrWmj97FTb57kPl1H4iBi18kIROnhF/ppqlAAd/kGYhI1X/+m8I8Ug/oDZUem n2vg== X-Gm-Message-State: AOAM530mCbOHD05vqEi7/uH3lNeg0kBlSqMnd+zpOZx8h6grms+1SKzh r89xcKzny6wGx+GcvR4PrinmtQ== X-Google-Smtp-Source: ABdhPJw32otAEAGQ7wbA7Yy6KLbhe+rjlxJYa8gEkHEt45un6/ZtOcrGlPaiuyYswbLoNoB4qg5JRg== X-Received: by 2002:a63:b243:: with SMTP id t3mr2257632pgo.253.1622096692910; Wed, 26 May 2021 23:24:52 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.24.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:24:52 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 05/21] mm: list_lru: remove lru node locking from memcg_update_list_lru_node Date: Thu, 27 May 2021 14:21:32 +0800 Message-Id: <20210527062148.9361-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BB479600025E Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=0i7WfGFj; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf25.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam03 X-Stat-Signature: 6by8t85xsznzqr8ey5tze8z5bna3egj9 X-HE-Tag: 1622096685-623689 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since commit e5bc3af7734f ("rcu: Consolidate PREEMPT and !PREEMPT synchronize_rcu()"), the critical section of spinlock can serve as RCU read-side critical section. So we can remove the locking from memcg_update_list_lru_node(). Signed-off-by: Muchun Song --- mm/list_lru.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index 4962d48d4410..e86d4d055d3c 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -403,18 +403,9 @@ static int memcg_update_list_lru_node(struct list_lru_node *nlru, memcpy(&new->lru, &old->lru, old_size * sizeof(void *)); - /* - * The locking below allows readers that hold nlru->lock avoid taking - * rcu_read_lock (see list_lru_from_memcg_idx). - * - * Since list_lru_{add,del} may be called under an IRQ-safe lock, - * we have to use IRQ-safe primitives here to avoid deadlock. - */ - spin_lock_irq(&nlru->lock); rcu_assign_pointer(nlru->memcg_lrus, new); - spin_unlock_irq(&nlru->lock); - kvfree_rcu(old, rcu); + return 0; } From patchwork Thu May 27 06:21:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1CBEC4708A for ; Thu, 27 May 2021 06:25:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A7A2A613C9 for ; Thu, 27 May 2021 06:25:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A7A2A613C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E3DBB6B0074; Thu, 27 May 2021 02:25:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DE9736B0075; Thu, 27 May 2021 02:25:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B5EF6B0078; Thu, 27 May 2021 02:25:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0102.hostedemail.com [216.40.44.102]) by kanga.kvack.org (Postfix) with ESMTP id 47B2A6B0074 for ; Thu, 27 May 2021 02:25:01 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id CEA80824999B for ; Thu, 27 May 2021 06:25:00 +0000 (UTC) X-FDA: 78186023160.04.26D2859 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf28.hostedemail.com (Postfix) with ESMTP id 4307F2000981 for ; Thu, 27 May 2021 06:24:53 +0000 (UTC) Received: by mail-pf1-f171.google.com with SMTP id p39so2857707pfw.8 for ; Wed, 26 May 2021 23:25:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HEia8IE8kslbq8l2doyQoIh1s6NP1wi5+X3AcZEFjeo=; b=kvlV9ariGPa/KOzsdvMjL84J7KWbdBZljXXTqzBfy1gdD9aUiqOXYY7jJQcZ6EvphQ g5z1HggN5fovKKWheUdFdNevBNnCo1Y5FChVJM1/DuUAV36d/srrFP8AiZwJCy+M1ATd 3xhBPwrQGJbKj2Kl3PGvzcn+Dg2smkoyiPmIwtsfxEKF6SrN6b04pXobPQjFpXbR6spG lJc/mkpgOWaK5OEFoF0G+i+xICYWPwDNN/lYDf5Mj138CEBpr7jh8ghx1EP+yJo8N+z8 QxZdlGmO13yEIskhtzWSkcu7NE88svZEfcAegqOuCoCVD2QjYW395gNPHg9oHw5ejkrV /ysg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HEia8IE8kslbq8l2doyQoIh1s6NP1wi5+X3AcZEFjeo=; b=RGHBdhOhfHakFpDr7YJ8H82j4/fXCCYuRE6aCuAGcrW1maxZcnXf8XCxlzBTllO63A NufeGgmZ362YlHqQSieLdeyKkoZtrESwywlvroqvTDveZOu7k/wLwMq+JuCDLVcIOTyv xarxIWf8GZ9ODQ5GlAJE1lNEVqqfdgFaCIfdYcppQTLqpizLNoc6RQQEV3eB8cDOvzHC 1wvHaqJMPL1gnJcw+LxjVZe7Rg97NSR8RqdGsh4tUPJ8w7ItbDJn0sef8I/lDCb91ZNh mqDuxioAD0pflSZu6GBVBWYJtSBXNMBpyJKu1tB5ptZ1wLcBzt/ivs/yFmK6YvXTp6QC o4Xg== X-Gm-Message-State: AOAM530SjC9bmYyLtWAExPBX6qrq4nvwwLtIitDzyOSHIp8bqu0J53IE tHbJNXDJgbNoTLDBb1NSGN4ncw== X-Google-Smtp-Source: ABdhPJy8A0vvkxkOQSVawjHK6CAOJlYjigmh+AJP/eE52bt7Q22D37+MA/80ZY8HArWIAUCkU+9QRQ== X-Received: by 2002:a65:64cc:: with SMTP id t12mr2330420pgv.64.1622096699772; Wed, 26 May 2021 23:24:59 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.24.53 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:24:59 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 06/21] mm: list_lru: only add the memcg aware lrus to the list_lrus Date: Thu, 27 May 2021 14:21:33 +0800 Message-Id: <20210527062148.9361-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4307F2000981 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=kvlV9ari; spf=pass (imf28.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam04 X-Stat-Signature: taj1xkhapfqra13c1ftkt9mhfbs3mp3i X-HE-Tag: 1622096693-263160 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We need to traverse every lru in the list_lrus in some routines, but skip memcg non-aware lrus. However, we can only add the memcg aware lrus to the list_lrus. This can be efficient to traverse the list. Signed-off-by: Muchun Song --- mm/list_lru.c | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index e86d4d055d3c..bed699edabe5 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -18,8 +18,16 @@ static LIST_HEAD(list_lrus); static DEFINE_MUTEX(list_lrus_mutex); +static inline bool list_lru_memcg_aware(struct list_lru *lru) +{ + return lru->memcg_aware; +} + static void list_lru_register(struct list_lru *lru) { + if (!list_lru_memcg_aware(lru)) + return; + mutex_lock(&list_lrus_mutex); list_add(&lru->list, &list_lrus); mutex_unlock(&list_lrus_mutex); @@ -27,6 +35,9 @@ static void list_lru_register(struct list_lru *lru) static void list_lru_unregister(struct list_lru *lru) { + if (!list_lru_memcg_aware(lru)) + return; + mutex_lock(&list_lrus_mutex); list_del(&lru->list); mutex_unlock(&list_lrus_mutex); @@ -37,11 +48,6 @@ static int lru_shrinker_id(struct list_lru *lru) return lru->shrinker_id; } -static inline bool list_lru_memcg_aware(struct list_lru *lru) -{ - return lru->memcg_aware; -} - static inline struct list_lru_one * list_lru_from_memcg_idx(struct list_lru_node *nlru, int idx) { @@ -460,9 +466,6 @@ static int memcg_update_list_lru(struct list_lru *lru, { int i; - if (!list_lru_memcg_aware(lru)) - return 0; - for_each_node(i) { if (memcg_update_list_lru_node(&lru->node[i], old_size, new_size)) @@ -485,9 +488,6 @@ static void memcg_cancel_update_list_lru(struct list_lru *lru, { int i; - if (!list_lru_memcg_aware(lru)) - return; - for_each_node(i) memcg_cancel_update_list_lru_node(&lru->node[i], old_size, new_size); @@ -546,9 +546,6 @@ static void memcg_drain_list_lru(struct list_lru *lru, { int i; - if (!list_lru_memcg_aware(lru)) - return; - for_each_node(i) memcg_drain_list_lru_node(lru, i, src_idx, dst_memcg); } From patchwork Thu May 27 06:21:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FAA9C4707F for ; Thu, 27 May 2021 06:25:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9F04D613C9 for ; Thu, 27 May 2021 06:25:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9F04D613C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4184E6B0075; Thu, 27 May 2021 02:25:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EE006B0078; Thu, 27 May 2021 02:25:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26A7C6B007B; Thu, 27 May 2021 02:25:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0039.hostedemail.com [216.40.44.39]) by kanga.kvack.org (Postfix) with ESMTP id E85DB6B0075 for ; Thu, 27 May 2021 02:25:08 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 85120A2D9 for ; Thu, 27 May 2021 06:25:08 +0000 (UTC) X-FDA: 78186023496.30.41738C3 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by imf11.hostedemail.com (Postfix) with ESMTP id BE3172000991 for ; Thu, 27 May 2021 06:24:58 +0000 (UTC) Received: by mail-pg1-f181.google.com with SMTP id f22so2917168pgb.9 for ; Wed, 26 May 2021 23:25:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0eTzwbGvY3CeE+TpYrKkohuxs+OEth+NadypEgOkaJk=; b=BLRFzNEGNdR3foZPmFTep+tFknCyuLCTIY800eKx21ZgtrUSzYmcTuSLA97UO/af+u eMV43ouw74DPjAop/i9B0z8H3Hm7Uh3OUYAY1Ve83BJmZmpai95PzbDaUYX12HB9srWD GlvKtv/ECD2FF93sn3jumrTn7gptqmG0wFcC4NwA0dF5cZz2rnMMD9X1rsnn5lIQsEtz nN3WllBjsBBGKk5oN/x94pyrsdpYqRgpY5suCmQZqRTZQIKJEvEZupdvjYdx1akI8ULk BrnG3TEQAm63dTx/UL99DM0xR+7hzFutf9ViuSK/dA/PLJg9COf51B2hVpj9aP163el2 BYQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0eTzwbGvY3CeE+TpYrKkohuxs+OEth+NadypEgOkaJk=; b=jIL3SeN68G4pidicyvY9Kr4h20tgydGKAtjgJ4TmjJ3O6CJzYlOzi5xQutSd2aw3ni ir4V5F5gXPFqUpNy3xk73csAwWpFpdKnjVElOBFNEq2aGi6mIkniE78rfMipkJwHjcIj 5T7yb4b+sEIyeQzXmu7S4zukjS6mt8UhJyvH96qnU1NYZlOAumnB8RX5TDBjDle8gkMm AOnmmc+h8c+wMcfirybdVprlymNUm2jHWaKc6wruS32HpOnDOU7stgECCokbdYVNwbEe xjb2zmibfgfBbb+8AeAwDhmYf6Ge513YVn1bAowGBijl/6c/84Lf5BkDT6c1q5qYItVk DPSA== X-Gm-Message-State: AOAM532KdVle/wnS4THVY0Hpw0BMKXkWIInRoiV+yOq9t+4yx5coRPU8 RHtIQA1OUL/5uOEQR1fj3MwNvQ== X-Google-Smtp-Source: ABdhPJx69UQxplc9r/Oe9C7QDynhj361rLe45eMKQsH8ZPDIMauhXxR1Gke1KhDNIyf85fbct6qSgg== X-Received: by 2002:a63:224d:: with SMTP id t13mr2259684pgm.283.1622096707266; Wed, 26 May 2021 23:25:07 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.25.00 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:25:06 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 07/21] mm: list_lru: optimize the array of per memcg lists memory consumption Date: Thu, 27 May 2021 14:21:34 +0800 Message-Id: <20210527062148.9361-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=BLRFzNEG; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf11.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.181 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: BE3172000991 X-Stat-Signature: 5kqz1aszohsr66jntw5u8cep4kgafa8q X-HE-Tag: 1622096698-567961 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The list_lru use an array to store the list_lru_one pointers, which is per-memcg per-node. What if we run 10k containers in the system? The size of the array of every list_lru can be 10k * number_of_node * sizeof(void *). The array size becomes very big, the more numa node in the system, the more memory we consume. We can convert the array to per memcg instead of per-memcg per-node. It can save memory especially when there are many numa nodes in the system. And also simplify the code. Signed-off-by: Muchun Song --- include/linux/list_lru.h | 17 +++-- mm/list_lru.c | 191 +++++++++++++++++------------------------------ 2 files changed, 79 insertions(+), 129 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 9dcaa3e582c9..20a43904001d 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -31,10 +31,15 @@ struct list_lru_one { long nr_items; }; +struct list_lru_per_memcg { + /* array of per cgroup per node lists, indexed by node id */ + struct list_lru_one nodes[0]; +}; + struct list_lru_memcg { - struct rcu_head rcu; + struct rcu_head rcu; /* array of per cgroup lists, indexed by memcg_cache_id */ - struct list_lru_one *lru[]; + struct list_lru_per_memcg *lrus[]; }; struct list_lru_node { @@ -42,11 +47,7 @@ struct list_lru_node { spinlock_t lock; /* global list, used for the root cgroup in cgroup aware lrus */ struct list_lru_one lru; -#ifdef CONFIG_MEMCG_KMEM - /* for cgroup aware lrus points to per cgroup lists, otherwise NULL */ - struct list_lru_memcg __rcu *memcg_lrus; -#endif - long nr_items; + long nr_items; } ____cacheline_aligned_in_smp; struct list_lru { @@ -55,6 +56,8 @@ struct list_lru { struct list_head list; int shrinker_id; bool memcg_aware; + /* for cgroup aware lrus points to per cgroup lists, otherwise NULL */ + struct list_lru_memcg __rcu *memcg_lrus; #endif }; diff --git a/mm/list_lru.c b/mm/list_lru.c index bed699edabe5..e52f5a91fa0f 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -49,35 +49,38 @@ static int lru_shrinker_id(struct list_lru *lru) } static inline struct list_lru_one * -list_lru_from_memcg_idx(struct list_lru_node *nlru, int idx) +list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) { struct list_lru_memcg *memcg_lrus; + struct list_lru_node *nlru = &lru->node[nid]; + /* * Either lock or RCU protects the array of per cgroup lists - * from relocation (see memcg_update_list_lru_node). + * from relocation (see memcg_update_list_lru). */ - memcg_lrus = rcu_dereference_check(nlru->memcg_lrus, + memcg_lrus = rcu_dereference_check(lru->memcg_lrus, lockdep_is_held(&nlru->lock)); if (memcg_lrus && idx >= 0) - return memcg_lrus->lru[idx]; + return &memcg_lrus->lrus[idx]->nodes[nid]; return &nlru->lru; } static inline struct list_lru_one * -list_lru_from_kmem(struct list_lru_node *nlru, void *ptr, +list_lru_from_kmem(struct list_lru *lru, int nid, void *ptr, struct mem_cgroup **memcg_ptr) { + struct list_lru_node *nlru = &lru->node[nid]; struct list_lru_one *l = &nlru->lru; struct mem_cgroup *memcg = NULL; - if (!nlru->memcg_lrus) + if (!lru->memcg_lrus) goto out; memcg = mem_cgroup_from_obj(ptr); if (!memcg) goto out; - l = list_lru_from_memcg_idx(nlru, memcg_cache_id(memcg)); + l = list_lru_from_memcg_idx(lru, nid, memcg_cache_id(memcg)); out: if (memcg_ptr) *memcg_ptr = memcg; @@ -103,18 +106,18 @@ static inline bool list_lru_memcg_aware(struct list_lru *lru) } static inline struct list_lru_one * -list_lru_from_memcg_idx(struct list_lru_node *nlru, int idx) +list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) { - return &nlru->lru; + return &lru->node[nid].lru; } static inline struct list_lru_one * -list_lru_from_kmem(struct list_lru_node *nlru, void *ptr, +list_lru_from_kmem(struct list_lru *lru, int nid, void *ptr, struct mem_cgroup **memcg_ptr) { if (memcg_ptr) *memcg_ptr = NULL; - return &nlru->lru; + return &lru->node[nid].lru; } #endif /* CONFIG_MEMCG_KMEM */ @@ -127,7 +130,7 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item) spin_lock(&nlru->lock); if (list_empty(item)) { - l = list_lru_from_kmem(nlru, item, &memcg); + l = list_lru_from_kmem(lru, nid, item, &memcg); list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) @@ -150,7 +153,7 @@ bool list_lru_del(struct list_lru *lru, struct list_head *item) spin_lock(&nlru->lock); if (!list_empty(item)) { - l = list_lru_from_kmem(nlru, item, NULL); + l = list_lru_from_kmem(lru, nid, item, NULL); list_del_init(item); l->nr_items--; nlru->nr_items--; @@ -180,12 +183,11 @@ EXPORT_SYMBOL_GPL(list_lru_isolate_move); unsigned long list_lru_count_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg) { - struct list_lru_node *nlru = &lru->node[nid]; struct list_lru_one *l; long count; rcu_read_lock(); - l = list_lru_from_memcg_idx(nlru, memcg_cache_id(memcg)); + l = list_lru_from_memcg_idx(lru, nid, memcg_cache_id(memcg)); count = READ_ONCE(l->nr_items); rcu_read_unlock(); @@ -206,16 +208,16 @@ unsigned long list_lru_count_node(struct list_lru *lru, int nid) EXPORT_SYMBOL_GPL(list_lru_count_node); static unsigned long -__list_lru_walk_one(struct list_lru_node *nlru, int memcg_idx, +__list_lru_walk_one(struct list_lru *lru, int nid, int memcg_idx, list_lru_walk_cb isolate, void *cb_arg, unsigned long *nr_to_walk) { - + struct list_lru_node *nlru = &lru->node[nid]; struct list_lru_one *l; struct list_head *item, *n; unsigned long isolated = 0; - l = list_lru_from_memcg_idx(nlru, memcg_idx); + l = list_lru_from_memcg_idx(lru, nid, memcg_idx); restart: list_for_each_safe(item, n, &l->list) { enum lru_status ret; @@ -272,8 +274,8 @@ list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg, unsigned long ret; spin_lock(&nlru->lock); - ret = __list_lru_walk_one(nlru, memcg_cache_id(memcg), isolate, cb_arg, - nr_to_walk); + ret = __list_lru_walk_one(lru, nid, memcg_cache_id(memcg), isolate, + cb_arg, nr_to_walk); spin_unlock(&nlru->lock); return ret; } @@ -288,8 +290,8 @@ list_lru_walk_one_irq(struct list_lru *lru, int nid, struct mem_cgroup *memcg, unsigned long ret; spin_lock_irq(&nlru->lock); - ret = __list_lru_walk_one(nlru, memcg_cache_id(memcg), isolate, cb_arg, - nr_to_walk); + ret = __list_lru_walk_one(lru, nid, memcg_cache_id(memcg), isolate, + cb_arg, nr_to_walk); spin_unlock_irq(&nlru->lock); return ret; } @@ -308,7 +310,7 @@ unsigned long list_lru_walk_node(struct list_lru *lru, int nid, struct list_lru_node *nlru = &lru->node[nid]; spin_lock(&nlru->lock); - isolated += __list_lru_walk_one(nlru, memcg_idx, + isolated += __list_lru_walk_one(lru, nid, memcg_idx, isolate, cb_arg, nr_to_walk); spin_unlock(&nlru->lock); @@ -328,169 +330,114 @@ static void init_one_lru(struct list_lru_one *l) } #ifdef CONFIG_MEMCG_KMEM -static void __memcg_destroy_list_lru_node(struct list_lru_memcg *memcg_lrus, - int begin, int end) +static void memcg_destroy_list_lru_range(struct list_lru_memcg *memcg_lrus, + int begin, int end) { int i; for (i = begin; i < end; i++) - kfree(memcg_lrus->lru[i]); + kfree(memcg_lrus->lrus[i]); } -static int __memcg_init_list_lru_node(struct list_lru_memcg *memcg_lrus, - int begin, int end) +static int memcg_init_list_lru_range(struct list_lru_memcg *memcg_lrus, + int begin, int end) { int i; for (i = begin; i < end; i++) { - struct list_lru_one *l; + int nid; + struct list_lru_per_memcg *lru; - l = kmalloc(sizeof(struct list_lru_one), GFP_KERNEL); - if (!l) + lru = kmalloc(struct_size(lru, nodes, nr_node_ids), GFP_KERNEL); + if (!lru) goto fail; - init_one_lru(l); - memcg_lrus->lru[i] = l; + for_each_node(nid) + init_one_lru(&lru->nodes[nid]); + memcg_lrus->lrus[i] = lru; } return 0; fail: - __memcg_destroy_list_lru_node(memcg_lrus, begin, i); + memcg_destroy_list_lru_range(memcg_lrus, begin, i); return -ENOMEM; } -static int memcg_init_list_lru_node(struct list_lru_node *nlru) +static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) { struct list_lru_memcg *memcg_lrus; int size = memcg_nr_cache_ids; + lru->memcg_aware = memcg_aware; + if (!memcg_aware) + return 0; + memcg_lrus = kvmalloc(sizeof(*memcg_lrus) + - size * sizeof(void *), GFP_KERNEL); + size * sizeof(memcg_lrus->lrus[0]), GFP_KERNEL); if (!memcg_lrus) return -ENOMEM; - if (__memcg_init_list_lru_node(memcg_lrus, 0, size)) { + if (memcg_init_list_lru_range(memcg_lrus, 0, size)) { kvfree(memcg_lrus); return -ENOMEM; } - RCU_INIT_POINTER(nlru->memcg_lrus, memcg_lrus); + RCU_INIT_POINTER(lru->memcg_lrus, memcg_lrus); return 0; } -static void memcg_destroy_list_lru_node(struct list_lru_node *nlru) +static void memcg_destroy_list_lru(struct list_lru *lru) { struct list_lru_memcg *memcg_lrus; + + if (!list_lru_memcg_aware(lru)) + return; + /* * This is called when shrinker has already been unregistered, * and nobody can use it. So, there is no need to use kvfree_rcu(). */ - memcg_lrus = rcu_dereference_protected(nlru->memcg_lrus, true); - __memcg_destroy_list_lru_node(memcg_lrus, 0, memcg_nr_cache_ids); + memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, true); + memcg_destroy_list_lru_range(memcg_lrus, 0, memcg_nr_cache_ids); kvfree(memcg_lrus); } -static int memcg_update_list_lru_node(struct list_lru_node *nlru, - int old_size, int new_size) +static int memcg_update_list_lru(struct list_lru *lru, int old_size, int new_size) { struct list_lru_memcg *old, *new; BUG_ON(old_size > new_size); - old = rcu_dereference_protected(nlru->memcg_lrus, + old = rcu_dereference_protected(lru->memcg_lrus, lockdep_is_held(&list_lrus_mutex)); - new = kvmalloc(sizeof(*new) + new_size * sizeof(void *), GFP_KERNEL); + new = kvmalloc(sizeof(*new) + new_size * sizeof(new->lrus[0]), GFP_KERNEL); if (!new) return -ENOMEM; - if (__memcg_init_list_lru_node(new, old_size, new_size)) { + if (memcg_init_list_lru_range(new, old_size, new_size)) { kvfree(new); return -ENOMEM; } - memcpy(&new->lru, &old->lru, old_size * sizeof(void *)); + memcpy(&new->lrus, &old->lrus, old_size * sizeof(new->lrus[0])); - rcu_assign_pointer(nlru->memcg_lrus, new); + rcu_assign_pointer(lru->memcg_lrus, new); kvfree_rcu(old, rcu); return 0; } -static void memcg_cancel_update_list_lru_node(struct list_lru_node *nlru, - int old_size, int new_size) -{ - struct list_lru_memcg *memcg_lrus; - - memcg_lrus = rcu_dereference_protected(nlru->memcg_lrus, - lockdep_is_held(&list_lrus_mutex)); - /* do not bother shrinking the array back to the old size, because we - * cannot handle allocation failures here */ - __memcg_destroy_list_lru_node(memcg_lrus, old_size, new_size); -} - -static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) -{ - int i; - - lru->memcg_aware = memcg_aware; - - if (!memcg_aware) - return 0; - - for_each_node(i) { - if (memcg_init_list_lru_node(&lru->node[i])) - goto fail; - } - return 0; -fail: - for (i = i - 1; i >= 0; i--) { - if (!lru->node[i].memcg_lrus) - continue; - memcg_destroy_list_lru_node(&lru->node[i]); - } - return -ENOMEM; -} - -static void memcg_destroy_list_lru(struct list_lru *lru) -{ - int i; - - if (!list_lru_memcg_aware(lru)) - return; - - for_each_node(i) - memcg_destroy_list_lru_node(&lru->node[i]); -} - -static int memcg_update_list_lru(struct list_lru *lru, - int old_size, int new_size) -{ - int i; - - for_each_node(i) { - if (memcg_update_list_lru_node(&lru->node[i], - old_size, new_size)) - goto fail; - } - return 0; -fail: - for (i = i - 1; i >= 0; i--) { - if (!lru->node[i].memcg_lrus) - continue; - - memcg_cancel_update_list_lru_node(&lru->node[i], - old_size, new_size); - } - return -ENOMEM; -} - static void memcg_cancel_update_list_lru(struct list_lru *lru, int old_size, int new_size) { - int i; + struct list_lru_memcg *memcg_lrus; - for_each_node(i) - memcg_cancel_update_list_lru_node(&lru->node[i], - old_size, new_size); + memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, + lockdep_is_held(&list_lrus_mutex)); + /* + * Do not bother shrinking the array back to the old size, because we + * cannot handle allocation failures here. + */ + memcg_destroy_list_lru_range(memcg_lrus, old_size, new_size); } int memcg_update_all_list_lrus(int new_size) @@ -527,8 +474,8 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, */ spin_lock_irq(&nlru->lock); - src = list_lru_from_memcg_idx(nlru, src_idx); - dst = list_lru_from_memcg_idx(nlru, dst_idx); + src = list_lru_from_memcg_idx(lru, nid, src_idx); + dst = list_lru_from_memcg_idx(lru, nid, dst_idx); list_splice_init(&src->list, &dst->list); From patchwork Thu May 27 06:21:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0574DC4708A for ; Thu, 27 May 2021 06:25:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AA66A613C9 for ; Thu, 27 May 2021 06:25:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AA66A613C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 48CB06B0078; Thu, 27 May 2021 02:25:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 463D36B007B; Thu, 27 May 2021 02:25:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 32BF68D0001; Thu, 27 May 2021 02:25:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 013396B0078 for ; Thu, 27 May 2021 02:25:15 -0400 (EDT) Received: from smtpin40.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 8CD79181AF5C6 for ; Thu, 27 May 2021 06:25:15 +0000 (UTC) X-FDA: 78186023790.40.13E374C Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by imf28.hostedemail.com (Postfix) with ESMTP id CA73B20007E0 for ; Thu, 27 May 2021 06:25:07 +0000 (UTC) Received: by mail-pg1-f176.google.com with SMTP id m190so2937331pga.2 for ; Wed, 26 May 2021 23:25:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1X6OOf9lCjkP/bI+mce/Si1A6k74NmBkIKRRCYe+/To=; b=yUKu6uMU39gp5llazQNnfoBCQzbX3FSs734k0SCSX8eNSn4HYwd1czfoKj055JVrMK CnPyzs2aR5c5ooegedgR+MK1baVRLvuu4aaLxBxqw48Jbs0oyxGq42SqIG9r71sO/nPi 1RTAx3wwATjb3m8mzeajh12oPVduLm/csn2woaCGJfZUXaMfFHiM5ttScvImxUUroHX3 706O90OYql9Rkba+VivEBHx9xKY2ACadQYxTE5LmBt83UR4NVtDoreEy4bKyu9wlZb1I WTYyhuNCFAvi27uN8C6UFzeJ4zCF0A8iIk2VMFlWF+xjEJTVn449dE97X9MymaxPQWfm pn1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1X6OOf9lCjkP/bI+mce/Si1A6k74NmBkIKRRCYe+/To=; b=RraZpOitHBo5u6iDsusvuSW3l04JdWRvxNZgRxZd2Jh8w4wbbc5LOhtgqQUXbu6Ddy 6oNCx01O6TbBmvu9r9/Rq0skNHKKC/LhPvW0MFnuYC1mS83UGB18wUHJVN210db2l0Rw b5UYVPVyWt3pEGEjEgE1PU4Zqa4p0YYe6v1LSYzj2Q0SzB8cDMfhbQO43ahjUIKegVKz 3dE7hmBXF7macpYyLUAFEW7INZFQEpHsCngs36tjV2xQ28Wj2C3LcTNiq2aGlrwjcZEA Pi+FrGuV4k1fipEWvuEHYg5pIWMTohemnIGOe2krquw7IUU7fNxevFWeWxni0VPCK6UV 7s6w== X-Gm-Message-State: AOAM533ThRtMItqVCEZAkMpjn1KwfJRLLSJSBITVV8Cf+IpybriH2Yf6 9wa98XG3nqcru+vHgs/6OxxFUg== X-Google-Smtp-Source: ABdhPJw8+YzYI06AWPrDB7x80mQ4G/SrhgscISaIe7PFwscFc5ZGQOT6F8IPqbvcXg3UKbUcia50kg== X-Received: by 2002:a63:5322:: with SMTP id h34mr2320692pgb.182.1622096714294; Wed, 26 May 2021 23:25:14 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.25.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:25:14 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 08/21] mm: list_lru: remove memcg_aware field from struct list_lru Date: Thu, 27 May 2021 14:21:35 +0800 Message-Id: <20210527062148.9361-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=yUKu6uMU; spf=pass (imf28.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.176 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: CA73B20007E0 X-Stat-Signature: 4n7u93e8druzwnfrd5jnm3x1rksub5bc X-HE-Tag: 1622096707-385435 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We can use ->memcg_lrus to indicate if the list_lru is memcg aware. So ->memcg_aware is redundant and just remove it. Signed-off-by: Muchun Song --- include/linux/list_lru.h | 1 - mm/list_lru.c | 9 +++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 20a43904001d..4a9e4aaecc69 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -55,7 +55,6 @@ struct list_lru { #ifdef CONFIG_MEMCG_KMEM struct list_head list; int shrinker_id; - bool memcg_aware; /* for cgroup aware lrus points to per cgroup lists, otherwise NULL */ struct list_lru_memcg __rcu *memcg_lrus; #endif diff --git a/mm/list_lru.c b/mm/list_lru.c index e52f5a91fa0f..8006c0fcc506 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -20,7 +20,7 @@ static DEFINE_MUTEX(list_lrus_mutex); static inline bool list_lru_memcg_aware(struct list_lru *lru) { - return lru->memcg_aware; + return !!rcu_access_pointer(lru->memcg_lrus); } static void list_lru_register(struct list_lru *lru) @@ -73,7 +73,7 @@ list_lru_from_kmem(struct list_lru *lru, int nid, void *ptr, struct list_lru_one *l = &nlru->lru; struct mem_cgroup *memcg = NULL; - if (!lru->memcg_lrus) + if (!list_lru_memcg_aware(lru)) goto out; memcg = mem_cgroup_from_obj(ptr); @@ -367,9 +367,10 @@ static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) struct list_lru_memcg *memcg_lrus; int size = memcg_nr_cache_ids; - lru->memcg_aware = memcg_aware; - if (!memcg_aware) + if (!memcg_aware) { + lru->memcg_lrus = NULL; return 0; + } memcg_lrus = kvmalloc(sizeof(*memcg_lrus) + size * sizeof(memcg_lrus->lrus[0]), GFP_KERNEL); From patchwork Thu May 27 06:21:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283485 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47F5CC4707F for ; Thu, 27 May 2021 06:25:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CAAF4610CC for ; Thu, 27 May 2021 06:25:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CAAF4610CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6AFB26B007B; Thu, 27 May 2021 02:25:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 685608D0001; Thu, 27 May 2021 02:25:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4080D6B007E; Thu, 27 May 2021 02:25:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0007.hostedemail.com [216.40.44.7]) by kanga.kvack.org (Postfix) with ESMTP id DC7326B007B for ; Thu, 27 May 2021 02:25:22 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 852E6180AE816 for ; Thu, 27 May 2021 06:25:22 +0000 (UTC) X-FDA: 78186024084.07.622E5A7 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf06.hostedemail.com (Postfix) with ESMTP id C13E9C0007D8 for ; Thu, 27 May 2021 06:25:14 +0000 (UTC) Received: by mail-pl1-f169.google.com with SMTP id n8so1827168plf.7 for ; Wed, 26 May 2021 23:25:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=S96t8FV/KVNYLav3maTnbrPOMh4mzE7td/+61i8wGYU=; b=ZzjS56RR/gxspe+uPrYR7tN3OvcenOUl4lHdbQYCqpGeWtNBE+MN1sPQM4ddTrn4Ci 3Aw8aKmWyhCmaAhHS0LmcUssEmhpVaNeysyT9fzF2U+JziP5mlZTeJK8ZhoZa90jSdy+ AsmMCpO4aBjil+2oYTtjl5uEcuwT5V+So1dvuyWx75TGTCcx/DD8l7bHCiqgtISAxqRI e65lltGTG2iRXhufqP/pPCxITpv1goQSakxaslrT8UbcEboHiCnPc2RCB1NyzCM/gagF EdVKjMPh3hP87/1cOZEfNbDa7EbZ4wASdW70QlwkmkPHvHiof67zOYYgUw92+V7hpUW9 SwaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=S96t8FV/KVNYLav3maTnbrPOMh4mzE7td/+61i8wGYU=; b=XZXh/f4Cm2TJWcbYKHupXXxo9D/7CBWz8gZObnDdHWEI2DpoxZZrJ4j8xvYWwRVQme oXVFO+bNA31YjW4P8sqJ6DUzRcq7VRsiXoMWSvcdmptqwCWt2crPMe7Q+bQQ8DqCTdsw 345/gSYfn3sIaGr5AvzmE1ijxcvGB1KfEOCaToQ3TGZ+eB6DHim6wpR11NsMoMTfNdFg BrLFFp7J+ogRjNfOHHj43FrII/wpLIHCWaWfjV38VXnZj8DeNvapqFHNdLzmQaSXzul2 ODdRLkf/iciT/FQZmJigAj3zygdnQjtPLVXQB9TKE+KnBnKssBZK5o7fxhGnEsLYfhG1 vbSw== X-Gm-Message-State: AOAM532hjiMovnFxeza/7bsK1GPwn0YEYEJ0kohTSTDoI3YDaAHQCP9G RNTTOpbMaoi9WKi4V+IduQfKaT38AV5BpJJO X-Google-Smtp-Source: ABdhPJzjTQ6qtKCqzsaaiaYadkTszCwHc8sq/qdllmh3hpdht+UN80mUw23TYDzYYWnTiO4wAUd5jw== X-Received: by 2002:a17:90b:3709:: with SMTP id mg9mr7572081pjb.149.1622096721286; Wed, 26 May 2021 23:25:21 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.25.14 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:25:20 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 09/21] mm: introduce kmem_cache_alloc_lru Date: Thu, 27 May 2021 14:21:36 +0800 Message-Id: <20210527062148.9361-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=ZzjS56RR; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf06.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: 15a35iqsd16f9eo5hotcx9wuswbxupe1 X-Rspamd-Queue-Id: C13E9C0007D8 X-Rspamd-Server: rspam02 X-HE-Tag: 1622096714-602309 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We currently allocate scope for every memcg to be able to tracked on every superblock instantiated in the system, regardless of whether that superblock is even accessible to that memcg. These huge memcg counts come from container hosts where memcgs are confined to just a small subset of the total number of superblocks that instantiated at any given point in time. For these systems with huge container counts, list_lru does not need the capability of tracking every memcg on every superblock. What it comes down to is that adding the memcg to the list_lru at the first insert. So introduce kmem_cache_alloc_lru to allocate objects and its list_lru. In the later patch, we will convert all inode and dentry allocation from kmem_cache_alloc to kmem_cache_alloc_lru. Signed-off-by: Muchun Song --- include/linux/list_lru.h | 3 ++ include/linux/memcontrol.h | 14 ++++++ include/linux/slab.h | 4 ++ mm/list_lru.c | 115 +++++++++++++++++++++++++++++++++++++++++---- mm/memcontrol.c | 14 ------ mm/slab.c | 39 ++++++++++----- mm/slab.h | 17 ++++++- mm/slub.c | 42 +++++++++++------ 8 files changed, 197 insertions(+), 51 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 4a9e4aaecc69..2083f4f2701f 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -55,11 +55,14 @@ struct list_lru { #ifdef CONFIG_MEMCG_KMEM struct list_head list; int shrinker_id; + /* protects ->memcg_lrus->lrus[i] */ + spinlock_t lock; /* for cgroup aware lrus points to per cgroup lists, otherwise NULL */ struct list_lru_memcg __rcu *memcg_lrus; #endif }; +int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t gfp); void list_lru_destroy(struct list_lru *lru); int __list_lru_init(struct list_lru *lru, bool memcg_aware, struct lock_class_key *key, struct shrinker *shrinker); diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 6d0638e13fc1..84a70c219e85 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -517,6 +517,20 @@ static inline struct mem_cgroup *page_memcg_check(struct page *page) return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } +static inline struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *objcg) +{ + struct mem_cgroup *memcg; + + rcu_read_lock(); +retry: + memcg = obj_cgroup_memcg(objcg); + if (unlikely(!css_tryget(&memcg->css))) + goto retry; + rcu_read_unlock(); + + return memcg; +} + #ifdef CONFIG_MEMCG_KMEM /* * PageMemcgKmem - check if the page has MemcgKmem flag set diff --git a/include/linux/slab.h b/include/linux/slab.h index 9d316aac0aba..346698208258 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -424,8 +424,12 @@ static __always_inline unsigned int __kmalloc_index(size_t size, #define kmalloc_index(s) __kmalloc_index(s, true) #endif /* !CONFIG_SLOB */ +struct list_lru; + void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __malloc; void *kmem_cache_alloc(struct kmem_cache *, gfp_t flags) __assume_slab_alignment __malloc; +void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, + gfp_t gfpflags) __assume_slab_alignment __malloc; void kmem_cache_free(struct kmem_cache *, void *); /* diff --git a/mm/list_lru.c b/mm/list_lru.c index 8006c0fcc506..4ba1db6d4409 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -339,22 +339,30 @@ static void memcg_destroy_list_lru_range(struct list_lru_memcg *memcg_lrus, kfree(memcg_lrus->lrus[i]); } +static struct list_lru_per_memcg *memcg_list_lru_alloc(gfp_t gfp) +{ + int nid; + struct list_lru_per_memcg *lru; + + lru = kmalloc(struct_size(lru, nodes, nr_node_ids), gfp); + if (!lru) + return NULL; + + for_each_node(nid) + init_one_lru(&lru->nodes[nid]); + + return lru; +} + static int memcg_init_list_lru_range(struct list_lru_memcg *memcg_lrus, int begin, int end) { int i; for (i = begin; i < end; i++) { - int nid; - struct list_lru_per_memcg *lru; - - lru = kmalloc(struct_size(lru, nodes, nr_node_ids), GFP_KERNEL); - if (!lru) + memcg_lrus->lrus[i] = memcg_list_lru_alloc(GFP_KERNEL); + if (!memcg_lrus->lrus[i]) goto fail; - - for_each_node(nid) - init_one_lru(&lru->nodes[nid]); - memcg_lrus->lrus[i] = lru; } return 0; fail: @@ -372,6 +380,8 @@ static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) return 0; } + spin_lock_init(&lru->lock); + memcg_lrus = kvmalloc(sizeof(*memcg_lrus) + size * sizeof(memcg_lrus->lrus[0]), GFP_KERNEL); if (!memcg_lrus) @@ -419,9 +429,11 @@ static int memcg_update_list_lru(struct list_lru *lru, int old_size, int new_siz return -ENOMEM; } + spin_lock_irq(&lru->lock); memcpy(&new->lrus, &old->lrus, old_size * sizeof(new->lrus[0])); - rcu_assign_pointer(lru->memcg_lrus, new); + spin_unlock_irq(&lru->lock); + kvfree_rcu(old, rcu); return 0; @@ -507,6 +519,89 @@ void memcg_drain_all_list_lrus(int src_idx, struct mem_cgroup *dst_memcg) memcg_drain_list_lru(lru, src_idx, dst_memcg); mutex_unlock(&list_lrus_mutex); } + +static bool memcg_list_lru_skip_alloc(struct list_lru *lru, + struct mem_cgroup *memcg) +{ + struct list_lru_memcg *memcg_lrus; + int idx = memcg_cache_id(memcg); + + if (unlikely(idx < 0)) + return true; + + rcu_read_lock(); + memcg_lrus = rcu_dereference(lru->memcg_lrus); + if (memcg_lrus->lrus[idx]) { + rcu_read_unlock(); + return true; + } + rcu_read_unlock(); + + return false; +} + +int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t gfp) +{ + unsigned long flags; + struct list_lru_memcg *memcg_lrus; + int i; + + struct list_lru_memcg { + struct list_lru_per_memcg *mlru; + struct mem_cgroup *memcg; + } *table; + + if (!list_lru_memcg_aware(lru)) + return 0; + + if (memcg_list_lru_skip_alloc(lru, memcg)) + return 0; + + /* + * The allocated list_lru_per_memcg array is not accounted directly. + * Moreover, it should not come from DMA buffer and is not readily + * reclaimable. So those GFP bits should be masked off. + */ + gfp &= ~(__GFP_DMA | __GFP_RECLAIMABLE | __GFP_ACCOUNT | __GFP_ZERO); + table = kmalloc_array(memcg->css.cgroup->level, sizeof(*table), gfp); + if (!table) + return -ENOMEM; + + /* + * Because the list_lru can be reparented to the parent cgroup's + * list_lru, we should make sure that this cgroup and all its + * ancestors have allocated list_lru_per_memcg. + */ + for (i = 0; memcg; memcg = parent_mem_cgroup(memcg), i++) { + if (memcg_list_lru_skip_alloc(lru, memcg)) + break; + + table[i].memcg = memcg; + table[i].mlru = memcg_list_lru_alloc(gfp); + if (!table[i].mlru) { + while (i--) + kfree(table[i].mlru); + kfree(table); + return -ENOMEM; + } + } + + spin_lock_irqsave(&lru->lock, flags); + memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, true); + while (i--) { + int index = memcg_cache_id(table[i].memcg); + + if (memcg_lrus->lrus[index]) + kfree(table[i].mlru); + else + memcg_lrus->lrus[index] = table[i].mlru; + } + spin_unlock_irqrestore(&lru->lock, flags); + + kfree(table); + + return 0; +} #else static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 377ec9847179..09bafa82781f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2787,20 +2787,6 @@ static void commit_charge(struct page *page, struct mem_cgroup *memcg) page->memcg_data = (unsigned long)memcg; } -static struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *objcg) -{ - struct mem_cgroup *memcg; - - rcu_read_lock(); -retry: - memcg = obj_cgroup_memcg(objcg); - if (unlikely(!css_tryget(&memcg->css))) - goto retry; - rcu_read_unlock(); - - return memcg; -} - #ifdef CONFIG_MEMCG_KMEM /* * The allocated objcg pointers array is not accounted directly. diff --git a/mm/slab.c b/mm/slab.c index d0f725637663..9a001aabc77b 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3219,7 +3219,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_ bool init = false; flags &= gfp_allowed_mask; - cachep = slab_pre_alloc_hook(cachep, &objcg, 1, flags); + cachep = slab_pre_alloc_hook(cachep, NULL, &objcg, 1, flags); if (unlikely(!cachep)) return NULL; @@ -3295,7 +3295,8 @@ __do_cache_alloc(struct kmem_cache *cachep, gfp_t flags) #endif /* CONFIG_NUMA */ static __always_inline void * -slab_alloc(struct kmem_cache *cachep, gfp_t flags, size_t orig_size, unsigned long caller) +slab_alloc(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags, + size_t orig_size, unsigned long caller) { unsigned long save_flags; void *objp; @@ -3303,7 +3304,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, size_t orig_size, unsigned lo bool init = false; flags &= gfp_allowed_mask; - cachep = slab_pre_alloc_hook(cachep, &objcg, 1, flags); + cachep = slab_pre_alloc_hook(cachep, lru, &objcg, 1, flags); if (unlikely(!cachep)) return NULL; @@ -3492,6 +3493,18 @@ void ___cache_free(struct kmem_cache *cachep, void *objp, __free_one(ac, objp); } +static __always_inline +void *__kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, + gfp_t flags) +{ + void *ret = slab_alloc(cachep, lru, flags, cachep->object_size, _RET_IP_); + + trace_kmem_cache_alloc(_RET_IP_, ret, + cachep->object_size, cachep->size, flags); + + return ret; +} + /** * kmem_cache_alloc - Allocate an object * @cachep: The cache to allocate from. @@ -3504,15 +3517,17 @@ void ___cache_free(struct kmem_cache *cachep, void *objp, */ void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) { - void *ret = slab_alloc(cachep, flags, cachep->object_size, _RET_IP_); - - trace_kmem_cache_alloc(_RET_IP_, ret, - cachep->object_size, cachep->size, flags); - - return ret; + return __kmem_cache_alloc_lru(cachep, NULL, flags); } EXPORT_SYMBOL(kmem_cache_alloc); +void *kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, + gfp_t flags) +{ + return __kmem_cache_alloc_lru(cachep, lru, flags); +} +EXPORT_SYMBOL(kmem_cache_alloc_lru); + static __always_inline void cache_alloc_debugcheck_after_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p, unsigned long caller) @@ -3529,7 +3544,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, size_t i; struct obj_cgroup *objcg = NULL; - s = slab_pre_alloc_hook(s, &objcg, size, flags); + s = slab_pre_alloc_hook(s, NULL, &objcg, size, flags); if (!s) return 0; @@ -3570,7 +3585,7 @@ kmem_cache_alloc_trace(struct kmem_cache *cachep, gfp_t flags, size_t size) { void *ret; - ret = slab_alloc(cachep, flags, size, _RET_IP_); + ret = slab_alloc(cachep, NULL, flags, size, _RET_IP_); ret = kasan_kmalloc(cachep, ret, size, flags); trace_kmalloc(_RET_IP_, ret, @@ -3697,7 +3712,7 @@ static __always_inline void *__do_kmalloc(size_t size, gfp_t flags, cachep = kmalloc_slab(size, flags); if (unlikely(ZERO_OR_NULL_PTR(cachep))) return cachep; - ret = slab_alloc(cachep, flags, size, caller); + ret = slab_alloc(cachep, NULL, flags, size, caller); ret = kasan_kmalloc(cachep, ret, size, flags); trace_kmalloc(caller, ret, diff --git a/mm/slab.h b/mm/slab.h index b4cdf8687120..473b75777a4d 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -46,6 +46,7 @@ struct kmem_cache { #include #include #include +#include /* * State of the slab allocator. @@ -262,6 +263,7 @@ static inline size_t obj_full_size(struct kmem_cache *s) * Returns false if the allocation should fail. */ static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, + struct list_lru *lru, struct obj_cgroup **objcgp, size_t objects, gfp_t flags) { @@ -277,6 +279,17 @@ static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, if (!objcg) return true; + if (lru) { + struct mem_cgroup *memcg = get_mem_cgroup_from_objcg(objcg); + + if (list_lru_memcg_alloc(lru, memcg, flags)) { + css_put(&memcg->css); + obj_cgroup_put(objcg); + return false; + } + css_put(&memcg->css); + } + if (obj_cgroup_charge(objcg, flags, objects * obj_full_size(s))) { obj_cgroup_put(objcg); return false; @@ -379,6 +392,7 @@ static inline void memcg_free_page_obj_cgroups(struct page *page) } static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, + struct list_lru *lru, struct obj_cgroup **objcgp, size_t objects, gfp_t flags) { @@ -477,6 +491,7 @@ static inline size_t slab_ksize(const struct kmem_cache *s) } static inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, + struct list_lru *lru, struct obj_cgroup **objcgp, size_t size, gfp_t flags) { @@ -487,7 +502,7 @@ static inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, if (should_failslab(s, flags)) return NULL; - if (!memcg_slab_pre_alloc_hook(s, objcgp, size, flags)) + if (!memcg_slab_pre_alloc_hook(s, lru, objcgp, size, flags)) return NULL; return s; diff --git a/mm/slub.c b/mm/slub.c index 6b896b8c36f0..d49c139d6dce 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2862,7 +2862,7 @@ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s, * * Otherwise we can simply pick the next object from the lockless free list. */ -static __always_inline void *slab_alloc_node(struct kmem_cache *s, +static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags, int node, unsigned long addr, size_t orig_size) { void *object; @@ -2872,7 +2872,7 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, struct obj_cgroup *objcg = NULL; bool init = false; - s = slab_pre_alloc_hook(s, &objcg, 1, gfpflags); + s = slab_pre_alloc_hook(s, lru, &objcg, 1, gfpflags); if (!s) return NULL; @@ -2956,27 +2956,41 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, return object; } -static __always_inline void *slab_alloc(struct kmem_cache *s, +static __always_inline void *slab_alloc(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags, unsigned long addr, size_t orig_size) { - return slab_alloc_node(s, gfpflags, NUMA_NO_NODE, addr, orig_size); + return slab_alloc_node(s, lru, gfpflags, NUMA_NO_NODE, addr, orig_size); } -void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) +static __always_inline +void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, + gfp_t gfpflags) { - void *ret = slab_alloc(s, gfpflags, _RET_IP_, s->object_size); + void *ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size); trace_kmem_cache_alloc(_RET_IP_, ret, s->object_size, s->size, gfpflags); return ret; } + +void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) +{ + return __kmem_cache_alloc_lru(s, NULL, gfpflags); +} EXPORT_SYMBOL(kmem_cache_alloc); +void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, + gfp_t gfpflags) +{ + return __kmem_cache_alloc_lru(s, lru, gfpflags); +} +EXPORT_SYMBOL(kmem_cache_alloc_lru); + #ifdef CONFIG_TRACING void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) { - void *ret = slab_alloc(s, gfpflags, _RET_IP_, size); + void *ret = slab_alloc(s, NULL, gfpflags, _RET_IP_, size); trace_kmalloc(_RET_IP_, ret, size, s->size, gfpflags); ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; @@ -2987,7 +3001,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_trace); #ifdef CONFIG_NUMA void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) { - void *ret = slab_alloc_node(s, gfpflags, node, _RET_IP_, s->object_size); + void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size); trace_kmem_cache_alloc_node(_RET_IP_, ret, s->object_size, s->size, gfpflags, node); @@ -3001,7 +3015,7 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, int node, size_t size) { - void *ret = slab_alloc_node(s, gfpflags, node, _RET_IP_, size); + void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, size); trace_kmalloc_node(_RET_IP_, ret, size, s->size, gfpflags, node); @@ -3352,7 +3366,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, struct obj_cgroup *objcg = NULL; /* memcg and kmem_cache debug support */ - s = slab_pre_alloc_hook(s, &objcg, size, flags); + s = slab_pre_alloc_hook(s, NULL, &objcg, size, flags); if (unlikely(!s)) return false; /* @@ -4109,7 +4123,7 @@ void *__kmalloc(size_t size, gfp_t flags) if (unlikely(ZERO_OR_NULL_PTR(s))) return s; - ret = slab_alloc(s, flags, _RET_IP_, size); + ret = slab_alloc(s, NULL, flags, _RET_IP_, size); trace_kmalloc(_RET_IP_, ret, size, s->size, flags); @@ -4157,7 +4171,7 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node) if (unlikely(ZERO_OR_NULL_PTR(s))) return s; - ret = slab_alloc_node(s, flags, node, _RET_IP_, size); + ret = slab_alloc_node(s, NULL, flags, node, _RET_IP_, size); trace_kmalloc_node(_RET_IP_, ret, size, s->size, flags, node); @@ -4619,7 +4633,7 @@ void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller) if (unlikely(ZERO_OR_NULL_PTR(s))) return s; - ret = slab_alloc(s, gfpflags, caller, size); + ret = slab_alloc(s, NULL, gfpflags, caller, size); /* Honor the call site pointer we received. */ trace_kmalloc(caller, ret, size, s->size, gfpflags); @@ -4650,7 +4664,7 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags, if (unlikely(ZERO_OR_NULL_PTR(s))) return s; - ret = slab_alloc_node(s, gfpflags, node, caller, size); + ret = slab_alloc_node(s, NULL, gfpflags, node, caller, size); /* Honor the call site pointer we received. */ trace_kmalloc_node(caller, ret, size, s->size, gfpflags, node); From patchwork Thu May 27 06:21:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3D63C4708A for ; Thu, 27 May 2021 06:25:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 280B8610CC for ; Thu, 27 May 2021 06:25:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 280B8610CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C54F96B006E; Thu, 27 May 2021 02:25:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C2AA98D0001; Thu, 27 May 2021 02:25:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A7F176B0080; Thu, 27 May 2021 02:25:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0121.hostedemail.com [216.40.44.121]) by kanga.kvack.org (Postfix) with ESMTP id 620056B006E for ; Thu, 27 May 2021 02:25:30 -0400 (EDT) Received: from smtpin35.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 007B68249980 for ; Thu, 27 May 2021 06:25:29 +0000 (UTC) X-FDA: 78186024420.35.D33E371 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf15.hostedemail.com (Postfix) with ESMTP id 202C2A0001C8 for ; Thu, 27 May 2021 06:25:25 +0000 (UTC) Received: by mail-pf1-f178.google.com with SMTP id x188so2852576pfd.7 for ; Wed, 26 May 2021 23:25:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=W+XDpXtgnloFQ+TzXm2Lkz4gcpdlW86il6bXZrzroV8=; b=uNhA97Kd/IL4zUzrfFXFYa5MMoYSnh6t02n75mOdOxqaXgn2DfEckubCYm8WlzpFZk isNL+QIlnR6JpoGxLRwChZVF2eIu0zwI9WXAZ0Ya9iKX4kPRlk6qx1VddfpGZerc66mk b6Oa68Y4cSE94Ca+AwwBS7dVMh/AMeG7jQvTvdbhOKmfljAhjl4/W/Aa5hKxRE2pYU55 Z3gGMCnXwYV1Axy8lNY4ds4Y6EHOM/VQqitXN+BN0J2dcDZb71cbeKdu9Fw/jTq7F4jg lp+MDINnI5ihJuVlUoQnkjIYO0TrAG4VUJWe96B9HLxvncH+sJPl77Z4bq+SUSqWkXJu MH7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=W+XDpXtgnloFQ+TzXm2Lkz4gcpdlW86il6bXZrzroV8=; b=slQr+ZPko408hCoqJvbuUuF64uvZFFSvNzjLWpEhn4dDJ+aX5FP9bl8Q/7jSp6bBpo qMI+SqJwCSA+aR4X4vbMRxoWh1YauYM4nwoDTV98T9+3Ep+89F56elOvEM17z+8Oqwq9 WF0wFYzt9XjWVc8zDlS+B+ec2Vtm+a84ThxBMgiEtAag4OFzXD+I7imlitHe/yZ4tax/ SIBBASa1HO2AxW1pNjnj4XdCsBr37NFSLkyE+sMUfalaW8wjz77rYDsC6AJcmmDXMaKB vONGBB3rItCuG/cCzhaK3MmhRBneUouqZpc+HHxjQEMAAaIkN0W1cCvt25/3bdh1P1zH sH+Q== X-Gm-Message-State: AOAM530XySaGgXMoJcBn53A6CBBpyIKFuM12Nr0pTSpxdSHU5LhASWLd qt1JRF0ASIKR5X0LKh4RSiZQcg== X-Google-Smtp-Source: ABdhPJyO06J1i7YbgNg0XJqhzH4oKhkepT7GeUCPZRS0GcxAK9FpBS9Dsge7s97JpPwmfZH2eEkWYg== X-Received: by 2002:a63:e058:: with SMTP id n24mr2272816pgj.91.1622096728657; Wed, 26 May 2021 23:25:28 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.25.21 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:25:28 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 10/21] fs: introduce alloc_inode_sb() to allocate filesystems specific inode Date: Thu, 27 May 2021 14:21:37 +0800 Message-Id: <20210527062148.9361-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=uNhA97Kd; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf15.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: yne4ocqq7nz6s9ywfokf5wugmb51ruzh X-Rspamd-Queue-Id: 202C2A0001C8 X-Rspamd-Server: rspam02 X-HE-Tag: 1622096725-886292 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The allocated inode cache will be added to its memcg list_lru, we should allocate its list_lru_one structure in advance. Introduce alloc_inode_sb which is used for allocating filesystems specific inodes to set up the inode reclaim context correctly. Signed-off-by: Muchun Song --- Documentation/filesystems/porting.rst | 5 +++++ drivers/dax/super.c | 2 +- fs/9p/vfs_inode.c | 2 +- fs/adfs/super.c | 2 +- fs/affs/super.c | 2 +- fs/afs/super.c | 2 +- fs/befs/linuxvfs.c | 2 +- fs/bfs/inode.c | 2 +- fs/block_dev.c | 2 +- fs/btrfs/inode.c | 2 +- fs/ceph/inode.c | 2 +- fs/cifs/cifsfs.c | 2 +- fs/coda/inode.c | 2 +- fs/ecryptfs/super.c | 2 +- fs/efs/super.c | 2 +- fs/erofs/super.c | 2 +- fs/exfat/super.c | 2 +- fs/ext2/super.c | 2 +- fs/ext4/super.c | 2 +- fs/f2fs/super.c | 2 +- fs/fat/inode.c | 2 +- fs/freevxfs/vxfs_super.c | 2 +- fs/fuse/inode.c | 2 +- fs/gfs2/super.c | 2 +- fs/hfs/super.c | 2 +- fs/hfsplus/super.c | 2 +- fs/hostfs/hostfs_kern.c | 2 +- fs/hpfs/super.c | 2 +- fs/hugetlbfs/inode.c | 2 +- fs/inode.c | 2 +- fs/isofs/inode.c | 2 +- fs/jffs2/super.c | 2 +- fs/jfs/super.c | 2 +- fs/minix/inode.c | 2 +- fs/nfs/inode.c | 2 +- fs/nilfs2/super.c | 2 +- fs/ntfs/inode.c | 2 +- fs/ocfs2/dlmfs/dlmfs.c | 2 +- fs/ocfs2/super.c | 2 +- fs/openpromfs/inode.c | 2 +- fs/orangefs/super.c | 2 +- fs/overlayfs/super.c | 2 +- fs/proc/inode.c | 2 +- fs/qnx4/inode.c | 2 +- fs/qnx6/inode.c | 2 +- fs/reiserfs/super.c | 2 +- fs/romfs/super.c | 2 +- fs/squashfs/super.c | 2 +- fs/sysv/inode.c | 2 +- fs/ubifs/super.c | 2 +- fs/udf/super.c | 2 +- fs/ufs/super.c | 2 +- fs/vboxsf/super.c | 2 +- fs/xfs/xfs_icache.c | 3 ++- fs/zonefs/super.c | 2 +- include/linux/fs.h | 11 +++++++++++ ipc/mqueue.c | 2 +- mm/shmem.c | 2 +- net/socket.c | 2 +- net/sunrpc/rpc_pipe.c | 2 +- 60 files changed, 75 insertions(+), 58 deletions(-) diff --git a/Documentation/filesystems/porting.rst b/Documentation/filesystems/porting.rst index 0302035781be..7db1c462cfcc 100644 --- a/Documentation/filesystems/porting.rst +++ b/Documentation/filesystems/porting.rst @@ -45,6 +45,11 @@ typically between calling iget_locked() and unlocking the inode. At some point that will become mandatory. +**mandatory** + +The foo_inode_info should always be allocated through alloc_inode_sb() rather +than kmem_cache_alloc(). + --- **mandatory** diff --git a/drivers/dax/super.c b/drivers/dax/super.c index 5fa6ae9dbc8b..30474ba337c8 100644 --- a/drivers/dax/super.c +++ b/drivers/dax/super.c @@ -460,7 +460,7 @@ static struct inode *dax_alloc_inode(struct super_block *sb) struct dax_device *dax_dev; struct inode *inode; - dax_dev = kmem_cache_alloc(dax_cache, GFP_KERNEL); + dax_dev = alloc_inode_sb(sb, dax_cache, GFP_KERNEL); if (!dax_dev) return NULL; diff --git a/fs/9p/vfs_inode.c b/fs/9p/vfs_inode.c index 795706520b5e..5311f35accf5 100644 --- a/fs/9p/vfs_inode.c +++ b/fs/9p/vfs_inode.c @@ -223,7 +223,7 @@ v9fs_blank_wstat(struct p9_wstat *wstat) struct inode *v9fs_alloc_inode(struct super_block *sb) { struct v9fs_inode *v9inode; - v9inode = kmem_cache_alloc(v9fs_inode_cache, GFP_KERNEL); + v9inode = alloc_inode_sb(sb, v9fs_inode_cache, GFP_KERNEL); if (!v9inode) return NULL; #ifdef CONFIG_9P_FSCACHE diff --git a/fs/adfs/super.c b/fs/adfs/super.c index bdbd26e571ed..e8bfc38239cd 100644 --- a/fs/adfs/super.c +++ b/fs/adfs/super.c @@ -220,7 +220,7 @@ static struct kmem_cache *adfs_inode_cachep; static struct inode *adfs_alloc_inode(struct super_block *sb) { struct adfs_inode_info *ei; - ei = kmem_cache_alloc(adfs_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, adfs_inode_cachep, GFP_KERNEL); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/fs/affs/super.c b/fs/affs/super.c index c6c2a513ec92..67458f63a1ab 100644 --- a/fs/affs/super.c +++ b/fs/affs/super.c @@ -100,7 +100,7 @@ static struct inode *affs_alloc_inode(struct super_block *sb) { struct affs_inode_info *i; - i = kmem_cache_alloc(affs_inode_cachep, GFP_KERNEL); + i = alloc_inode_sb(sb, affs_inode_cachep, GFP_KERNEL); if (!i) return NULL; diff --git a/fs/afs/super.c b/fs/afs/super.c index e38bb1e7a4d2..0ecea5a94af9 100644 --- a/fs/afs/super.c +++ b/fs/afs/super.c @@ -677,7 +677,7 @@ static struct inode *afs_alloc_inode(struct super_block *sb) { struct afs_vnode *vnode; - vnode = kmem_cache_alloc(afs_inode_cachep, GFP_KERNEL); + vnode = alloc_inode_sb(sb, afs_inode_cachep, GFP_KERNEL); if (!vnode) return NULL; diff --git a/fs/befs/linuxvfs.c b/fs/befs/linuxvfs.c index c1ba13d19024..b4b3567ac655 100644 --- a/fs/befs/linuxvfs.c +++ b/fs/befs/linuxvfs.c @@ -277,7 +277,7 @@ befs_alloc_inode(struct super_block *sb) { struct befs_inode_info *bi; - bi = kmem_cache_alloc(befs_inode_cachep, GFP_KERNEL); + bi = alloc_inode_sb(sb, befs_inode_cachep, GFP_KERNEL); if (!bi) return NULL; return &bi->vfs_inode; diff --git a/fs/bfs/inode.c b/fs/bfs/inode.c index fd691e4815c5..1926bec2c850 100644 --- a/fs/bfs/inode.c +++ b/fs/bfs/inode.c @@ -239,7 +239,7 @@ static struct kmem_cache *bfs_inode_cachep; static struct inode *bfs_alloc_inode(struct super_block *sb) { struct bfs_inode_info *bi; - bi = kmem_cache_alloc(bfs_inode_cachep, GFP_KERNEL); + bi = alloc_inode_sb(sb, bfs_inode_cachep, GFP_KERNEL); if (!bi) return NULL; return &bi->vfs_inode; diff --git a/fs/block_dev.c b/fs/block_dev.c index b8abccd03e5d..14a0ef54ae50 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -796,7 +796,7 @@ static struct kmem_cache * bdev_cachep __read_mostly; static struct inode *bdev_alloc_inode(struct super_block *sb) { - struct bdev_inode *ei = kmem_cache_alloc(bdev_cachep, GFP_KERNEL); + struct bdev_inode *ei = alloc_inode_sb(sb, bdev_cachep, GFP_KERNEL); if (!ei) return NULL; diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index b44c28455d4f..75decdbdbd45 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -8793,7 +8793,7 @@ struct inode *btrfs_alloc_inode(struct super_block *sb) struct btrfs_inode *ei; struct inode *inode; - ei = kmem_cache_alloc(btrfs_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, btrfs_inode_cachep, GFP_KERNEL); if (!ei) return NULL; diff --git a/fs/ceph/inode.c b/fs/ceph/inode.c index e1c63adb196d..9fe9db45a6ab 100644 --- a/fs/ceph/inode.c +++ b/fs/ceph/inode.c @@ -447,7 +447,7 @@ struct inode *ceph_alloc_inode(struct super_block *sb) struct ceph_inode_info *ci; int i; - ci = kmem_cache_alloc(ceph_inode_cachep, GFP_NOFS); + ci = alloc_inode_sb(sb, ceph_inode_cachep, GFP_NOFS); if (!ci) return NULL; diff --git a/fs/cifs/cifsfs.c b/fs/cifs/cifsfs.c index 2ffcb29d5c8f..d68cb0ddb85a 100644 --- a/fs/cifs/cifsfs.c +++ b/fs/cifs/cifsfs.c @@ -363,7 +363,7 @@ static struct inode * cifs_alloc_inode(struct super_block *sb) { struct cifsInodeInfo *cifs_inode; - cifs_inode = kmem_cache_alloc(cifs_inode_cachep, GFP_KERNEL); + cifs_inode = alloc_inode_sb(sb, cifs_inode_cachep, GFP_KERNEL); if (!cifs_inode) return NULL; cifs_inode->cifsAttrs = 0x20; /* default */ diff --git a/fs/coda/inode.c b/fs/coda/inode.c index d9f1bd7153df..2185328b65c7 100644 --- a/fs/coda/inode.c +++ b/fs/coda/inode.c @@ -43,7 +43,7 @@ static struct kmem_cache * coda_inode_cachep; static struct inode *coda_alloc_inode(struct super_block *sb) { struct coda_inode_info *ei; - ei = kmem_cache_alloc(coda_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, coda_inode_cachep, GFP_KERNEL); if (!ei) return NULL; memset(&ei->c_fid, 0, sizeof(struct CodaFid)); diff --git a/fs/ecryptfs/super.c b/fs/ecryptfs/super.c index 39116af0390f..0b1c878317ab 100644 --- a/fs/ecryptfs/super.c +++ b/fs/ecryptfs/super.c @@ -38,7 +38,7 @@ static struct inode *ecryptfs_alloc_inode(struct super_block *sb) struct ecryptfs_inode_info *inode_info; struct inode *inode = NULL; - inode_info = kmem_cache_alloc(ecryptfs_inode_info_cache, GFP_KERNEL); + inode_info = alloc_inode_sb(sb, ecryptfs_inode_info_cache, GFP_KERNEL); if (unlikely(!inode_info)) goto out; if (ecryptfs_init_crypt_stat(&inode_info->crypt_stat)) { diff --git a/fs/efs/super.c b/fs/efs/super.c index 62b155b9366b..b287f47c165b 100644 --- a/fs/efs/super.c +++ b/fs/efs/super.c @@ -69,7 +69,7 @@ static struct kmem_cache * efs_inode_cachep; static struct inode *efs_alloc_inode(struct super_block *sb) { struct efs_inode_info *ei; - ei = kmem_cache_alloc(efs_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, efs_inode_cachep, GFP_KERNEL); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/fs/erofs/super.c b/fs/erofs/super.c index bbf3bbd908e0..03f2dd684f8d 100644 --- a/fs/erofs/super.c +++ b/fs/erofs/super.c @@ -83,7 +83,7 @@ static void erofs_inode_init_once(void *ptr) static struct inode *erofs_alloc_inode(struct super_block *sb) { struct erofs_inode *vi = - kmem_cache_alloc(erofs_inode_cachep, GFP_KERNEL); + alloc_inode_sb(sb, erofs_inode_cachep, GFP_KERNEL); if (!vi) return NULL; diff --git a/fs/exfat/super.c b/fs/exfat/super.c index d38d17a77e76..c88e0f218df0 100644 --- a/fs/exfat/super.c +++ b/fs/exfat/super.c @@ -182,7 +182,7 @@ static struct inode *exfat_alloc_inode(struct super_block *sb) { struct exfat_inode_info *ei; - ei = kmem_cache_alloc(exfat_inode_cachep, GFP_NOFS); + ei = alloc_inode_sb(sb, exfat_inode_cachep, GFP_NOFS); if (!ei) return NULL; diff --git a/fs/ext2/super.c b/fs/ext2/super.c index 21e09fbaa46f..7eba9d6fbdc6 100644 --- a/fs/ext2/super.c +++ b/fs/ext2/super.c @@ -180,7 +180,7 @@ static struct kmem_cache * ext2_inode_cachep; static struct inode *ext2_alloc_inode(struct super_block *sb) { struct ext2_inode_info *ei; - ei = kmem_cache_alloc(ext2_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, ext2_inode_cachep, GFP_KERNEL); if (!ei) return NULL; ei->i_block_alloc_info = NULL; diff --git a/fs/ext4/super.c b/fs/ext4/super.c index 7dc94f3e18e6..b2c486d224b4 100644 --- a/fs/ext4/super.c +++ b/fs/ext4/super.c @@ -1275,7 +1275,7 @@ static struct inode *ext4_alloc_inode(struct super_block *sb) { struct ext4_inode_info *ei; - ei = kmem_cache_alloc(ext4_inode_cachep, GFP_NOFS); + ei = alloc_inode_sb(sb, ext4_inode_cachep, GFP_NOFS); if (!ei) return NULL; diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index 7d325bfaf65a..444134950ae4 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -1169,7 +1169,7 @@ static struct inode *f2fs_alloc_inode(struct super_block *sb) { struct f2fs_inode_info *fi; - fi = kmem_cache_alloc(f2fs_inode_cachep, GFP_F2FS_ZERO); + fi = alloc_inode_sb(sb, f2fs_inode_cachep, GFP_F2FS_ZERO); if (!fi) return NULL; diff --git a/fs/fat/inode.c b/fs/fat/inode.c index bab9b202b496..adc8a525553a 100644 --- a/fs/fat/inode.c +++ b/fs/fat/inode.c @@ -744,7 +744,7 @@ static struct kmem_cache *fat_inode_cachep; static struct inode *fat_alloc_inode(struct super_block *sb) { struct msdos_inode_info *ei; - ei = kmem_cache_alloc(fat_inode_cachep, GFP_NOFS); + ei = alloc_inode_sb(sb, fat_inode_cachep, GFP_NOFS); if (!ei) return NULL; diff --git a/fs/freevxfs/vxfs_super.c b/fs/freevxfs/vxfs_super.c index 578a5062706e..22eed5a73ac2 100644 --- a/fs/freevxfs/vxfs_super.c +++ b/fs/freevxfs/vxfs_super.c @@ -124,7 +124,7 @@ static struct inode *vxfs_alloc_inode(struct super_block *sb) { struct vxfs_inode_info *vi; - vi = kmem_cache_alloc(vxfs_inode_cachep, GFP_KERNEL); + vi = alloc_inode_sb(sb, vxfs_inode_cachep, GFP_KERNEL); if (!vi) return NULL; inode_init_once(&vi->vfs_inode); diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 393e36b74dc4..3fe7d1484ae6 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -73,7 +73,7 @@ static struct inode *fuse_alloc_inode(struct super_block *sb) { struct fuse_inode *fi; - fi = kmem_cache_alloc(fuse_inode_cachep, GFP_KERNEL); + fi = alloc_inode_sb(sb, fuse_inode_cachep, GFP_KERNEL); if (!fi) return NULL; diff --git a/fs/gfs2/super.c b/fs/gfs2/super.c index 4d4ceb0b6903..e9e8ad6f0c66 100644 --- a/fs/gfs2/super.c +++ b/fs/gfs2/super.c @@ -1496,7 +1496,7 @@ static struct inode *gfs2_alloc_inode(struct super_block *sb) { struct gfs2_inode *ip; - ip = kmem_cache_alloc(gfs2_inode_cachep, GFP_KERNEL); + ip = alloc_inode_sb(sb, gfs2_inode_cachep, GFP_KERNEL); if (!ip) return NULL; ip->i_flags = 0; diff --git a/fs/hfs/super.c b/fs/hfs/super.c index 44d07c9e3a7f..e40334c93183 100644 --- a/fs/hfs/super.c +++ b/fs/hfs/super.c @@ -162,7 +162,7 @@ static struct inode *hfs_alloc_inode(struct super_block *sb) { struct hfs_inode_info *i; - i = kmem_cache_alloc(hfs_inode_cachep, GFP_KERNEL); + i = alloc_inode_sb(sb, hfs_inode_cachep, GFP_KERNEL); return i ? &i->vfs_inode : NULL; } diff --git a/fs/hfsplus/super.c b/fs/hfsplus/super.c index b9e3db3f855f..8479add998b5 100644 --- a/fs/hfsplus/super.c +++ b/fs/hfsplus/super.c @@ -624,7 +624,7 @@ static struct inode *hfsplus_alloc_inode(struct super_block *sb) { struct hfsplus_inode_info *i; - i = kmem_cache_alloc(hfsplus_inode_cachep, GFP_KERNEL); + i = alloc_inode_sb(sb, hfsplus_inode_cachep, GFP_KERNEL); return i ? &i->vfs_inode : NULL; } diff --git a/fs/hostfs/hostfs_kern.c b/fs/hostfs/hostfs_kern.c index 7d0c3dbb2898..96dc97ae48e4 100644 --- a/fs/hostfs/hostfs_kern.c +++ b/fs/hostfs/hostfs_kern.c @@ -222,7 +222,7 @@ static struct inode *hostfs_alloc_inode(struct super_block *sb) { struct hostfs_inode_info *hi; - hi = kmem_cache_alloc(hostfs_inode_cache, GFP_KERNEL_ACCOUNT); + hi = alloc_inode_sb(sb, hostfs_inode_cache, GFP_KERNEL_ACCOUNT); if (hi == NULL) return NULL; hi->fd = -1; diff --git a/fs/hpfs/super.c b/fs/hpfs/super.c index a7dbfc892022..1cb89595b875 100644 --- a/fs/hpfs/super.c +++ b/fs/hpfs/super.c @@ -232,7 +232,7 @@ static struct kmem_cache * hpfs_inode_cachep; static struct inode *hpfs_alloc_inode(struct super_block *sb) { struct hpfs_inode_info *ei; - ei = kmem_cache_alloc(hpfs_inode_cachep, GFP_NOFS); + ei = alloc_inode_sb(sb, hpfs_inode_cachep, GFP_NOFS); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 9d9e0097c1d3..7a3095f786ec 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -1108,7 +1108,7 @@ static struct inode *hugetlbfs_alloc_inode(struct super_block *sb) if (unlikely(!hugetlbfs_dec_free_inodes(sbinfo))) return NULL; - p = kmem_cache_alloc(hugetlbfs_inode_cachep, GFP_KERNEL); + p = alloc_inode_sb(sb, hugetlbfs_inode_cachep, GFP_KERNEL); if (unlikely(!p)) { hugetlbfs_inc_free_inodes(sbinfo); return NULL; diff --git a/fs/inode.c b/fs/inode.c index c93500d84264..60ab51416a16 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -232,7 +232,7 @@ static struct inode *alloc_inode(struct super_block *sb) if (ops->alloc_inode) inode = ops->alloc_inode(sb); else - inode = kmem_cache_alloc(inode_cachep, GFP_KERNEL); + inode = alloc_inode_sb(sb, inode_cachep, GFP_KERNEL); if (!inode) return NULL; diff --git a/fs/isofs/inode.c b/fs/isofs/inode.c index 21edc423b79f..ef6c29501702 100644 --- a/fs/isofs/inode.c +++ b/fs/isofs/inode.c @@ -70,7 +70,7 @@ static struct kmem_cache *isofs_inode_cachep; static struct inode *isofs_alloc_inode(struct super_block *sb) { struct iso_inode_info *ei; - ei = kmem_cache_alloc(isofs_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, isofs_inode_cachep, GFP_KERNEL); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/fs/jffs2/super.c b/fs/jffs2/super.c index 81ca58c10b72..7ea37f49f1e1 100644 --- a/fs/jffs2/super.c +++ b/fs/jffs2/super.c @@ -39,7 +39,7 @@ static struct inode *jffs2_alloc_inode(struct super_block *sb) { struct jffs2_inode_info *f; - f = kmem_cache_alloc(jffs2_inode_cachep, GFP_KERNEL); + f = alloc_inode_sb(sb, jffs2_inode_cachep, GFP_KERNEL); if (!f) return NULL; return &f->vfs_inode; diff --git a/fs/jfs/super.c b/fs/jfs/super.c index 1f0ffabbde56..65255541d44e 100644 --- a/fs/jfs/super.c +++ b/fs/jfs/super.c @@ -102,7 +102,7 @@ static struct inode *jfs_alloc_inode(struct super_block *sb) { struct jfs_inode_info *jfs_inode; - jfs_inode = kmem_cache_alloc(jfs_inode_cachep, GFP_NOFS); + jfs_inode = alloc_inode_sb(sb, jfs_inode_cachep, GFP_NOFS); if (!jfs_inode) return NULL; #ifdef CONFIG_QUOTA diff --git a/fs/minix/inode.c b/fs/minix/inode.c index a532a99bbe81..f30b1daac386 100644 --- a/fs/minix/inode.c +++ b/fs/minix/inode.c @@ -63,7 +63,7 @@ static struct kmem_cache * minix_inode_cachep; static struct inode *minix_alloc_inode(struct super_block *sb) { struct minix_inode_info *ei; - ei = kmem_cache_alloc(minix_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, minix_inode_cachep, GFP_KERNEL); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c index 529c4099f482..fa3d8e8b695f 100644 --- a/fs/nfs/inode.c +++ b/fs/nfs/inode.c @@ -2240,7 +2240,7 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr) struct inode *nfs_alloc_inode(struct super_block *sb) { struct nfs_inode *nfsi; - nfsi = kmem_cache_alloc(nfs_inode_cachep, GFP_KERNEL); + nfsi = alloc_inode_sb(sb, nfs_inode_cachep, GFP_KERNEL); if (!nfsi) return NULL; nfsi->flags = 0UL; diff --git a/fs/nilfs2/super.c b/fs/nilfs2/super.c index 4abd928b0bc8..e7d7ad7a1e50 100644 --- a/fs/nilfs2/super.c +++ b/fs/nilfs2/super.c @@ -151,7 +151,7 @@ struct inode *nilfs_alloc_inode(struct super_block *sb) { struct nilfs_inode_info *ii; - ii = kmem_cache_alloc(nilfs_inode_cachep, GFP_NOFS); + ii = alloc_inode_sb(sb, nilfs_inode_cachep, GFP_NOFS); if (!ii) return NULL; ii->i_bh = NULL; diff --git a/fs/ntfs/inode.c b/fs/ntfs/inode.c index f5c058b3192c..1e053bb58ead 100644 --- a/fs/ntfs/inode.c +++ b/fs/ntfs/inode.c @@ -310,7 +310,7 @@ struct inode *ntfs_alloc_big_inode(struct super_block *sb) ntfs_inode *ni; ntfs_debug("Entering."); - ni = kmem_cache_alloc(ntfs_big_inode_cache, GFP_NOFS); + ni = alloc_inode_sb(sb, ntfs_big_inode_cache, GFP_NOFS); if (likely(ni != NULL)) { ni->state = 0; return VFS_I(ni); diff --git a/fs/ocfs2/dlmfs/dlmfs.c b/fs/ocfs2/dlmfs/dlmfs.c index fa0a14f199eb..e360543ad7e7 100644 --- a/fs/ocfs2/dlmfs/dlmfs.c +++ b/fs/ocfs2/dlmfs/dlmfs.c @@ -280,7 +280,7 @@ static struct inode *dlmfs_alloc_inode(struct super_block *sb) { struct dlmfs_inode_private *ip; - ip = kmem_cache_alloc(dlmfs_inode_cache, GFP_NOFS); + ip = alloc_inode_sb(sb, dlmfs_inode_cache, GFP_NOFS); if (!ip) return NULL; diff --git a/fs/ocfs2/super.c b/fs/ocfs2/super.c index c86bd4e60e20..cf044448130f 100644 --- a/fs/ocfs2/super.c +++ b/fs/ocfs2/super.c @@ -549,7 +549,7 @@ static struct inode *ocfs2_alloc_inode(struct super_block *sb) { struct ocfs2_inode_info *oi; - oi = kmem_cache_alloc(ocfs2_inode_cachep, GFP_NOFS); + oi = alloc_inode_sb(sb, ocfs2_inode_cachep, GFP_NOFS); if (!oi) return NULL; diff --git a/fs/openpromfs/inode.c b/fs/openpromfs/inode.c index f825176ff4ed..f0b7f4d51a17 100644 --- a/fs/openpromfs/inode.c +++ b/fs/openpromfs/inode.c @@ -335,7 +335,7 @@ static struct inode *openprom_alloc_inode(struct super_block *sb) { struct op_inode_info *oi; - oi = kmem_cache_alloc(op_inode_cachep, GFP_KERNEL); + oi = alloc_inode_sb(sb, op_inode_cachep, GFP_KERNEL); if (!oi) return NULL; diff --git a/fs/orangefs/super.c b/fs/orangefs/super.c index ee5efdc35cc1..5731ce285c98 100644 --- a/fs/orangefs/super.c +++ b/fs/orangefs/super.c @@ -106,7 +106,7 @@ static struct inode *orangefs_alloc_inode(struct super_block *sb) { struct orangefs_inode_s *orangefs_inode; - orangefs_inode = kmem_cache_alloc(orangefs_inode_cache, GFP_KERNEL); + orangefs_inode = alloc_inode_sb(sb, orangefs_inode_cache, GFP_KERNEL); if (!orangefs_inode) return NULL; diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c index b01d4147520d..6d8d16bf4b57 100644 --- a/fs/overlayfs/super.c +++ b/fs/overlayfs/super.c @@ -174,7 +174,7 @@ static struct kmem_cache *ovl_inode_cachep; static struct inode *ovl_alloc_inode(struct super_block *sb) { - struct ovl_inode *oi = kmem_cache_alloc(ovl_inode_cachep, GFP_KERNEL); + struct ovl_inode *oi = alloc_inode_sb(sb, ovl_inode_cachep, GFP_KERNEL); if (!oi) return NULL; diff --git a/fs/proc/inode.c b/fs/proc/inode.c index 599eb724ff2d..cc0a406d3a19 100644 --- a/fs/proc/inode.c +++ b/fs/proc/inode.c @@ -66,7 +66,7 @@ static struct inode *proc_alloc_inode(struct super_block *sb) { struct proc_inode *ei; - ei = kmem_cache_alloc(proc_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, proc_inode_cachep, GFP_KERNEL); if (!ei) return NULL; ei->pid = NULL; diff --git a/fs/qnx4/inode.c b/fs/qnx4/inode.c index 3fb7fc819b4f..a635bb6615e9 100644 --- a/fs/qnx4/inode.c +++ b/fs/qnx4/inode.c @@ -338,7 +338,7 @@ static struct kmem_cache *qnx4_inode_cachep; static struct inode *qnx4_alloc_inode(struct super_block *sb) { struct qnx4_inode_info *ei; - ei = kmem_cache_alloc(qnx4_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, qnx4_inode_cachep, GFP_KERNEL); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/fs/qnx6/inode.c b/fs/qnx6/inode.c index 61191f7bdf62..9d8e7e9788a1 100644 --- a/fs/qnx6/inode.c +++ b/fs/qnx6/inode.c @@ -597,7 +597,7 @@ static struct kmem_cache *qnx6_inode_cachep; static struct inode *qnx6_alloc_inode(struct super_block *sb) { struct qnx6_inode_info *ei; - ei = kmem_cache_alloc(qnx6_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, qnx6_inode_cachep, GFP_KERNEL); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/fs/reiserfs/super.c b/fs/reiserfs/super.c index 3ffafc73acf0..d9e8b0368ce9 100644 --- a/fs/reiserfs/super.c +++ b/fs/reiserfs/super.c @@ -639,7 +639,7 @@ static struct kmem_cache *reiserfs_inode_cachep; static struct inode *reiserfs_alloc_inode(struct super_block *sb) { struct reiserfs_inode_info *ei; - ei = kmem_cache_alloc(reiserfs_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, reiserfs_inode_cachep, GFP_KERNEL); if (!ei) return NULL; atomic_set(&ei->openers, 0); diff --git a/fs/romfs/super.c b/fs/romfs/super.c index 259f684d9236..9e6bbb4219de 100644 --- a/fs/romfs/super.c +++ b/fs/romfs/super.c @@ -375,7 +375,7 @@ static struct inode *romfs_alloc_inode(struct super_block *sb) { struct romfs_inode_info *inode; - inode = kmem_cache_alloc(romfs_inode_cachep, GFP_KERNEL); + inode = alloc_inode_sb(sb, romfs_inode_cachep, GFP_KERNEL); return inode ? &inode->vfs_inode : NULL; } diff --git a/fs/squashfs/super.c b/fs/squashfs/super.c index 88cc94be1076..16b785cb78c4 100644 --- a/fs/squashfs/super.c +++ b/fs/squashfs/super.c @@ -466,7 +466,7 @@ static void __exit exit_squashfs_fs(void) static struct inode *squashfs_alloc_inode(struct super_block *sb) { struct squashfs_inode_info *ei = - kmem_cache_alloc(squashfs_inode_cachep, GFP_KERNEL); + alloc_inode_sb(sb, squashfs_inode_cachep, GFP_KERNEL); return ei ? &ei->vfs_inode : NULL; } diff --git a/fs/sysv/inode.c b/fs/sysv/inode.c index be47263b8605..9e8d4a6fb2f3 100644 --- a/fs/sysv/inode.c +++ b/fs/sysv/inode.c @@ -306,7 +306,7 @@ static struct inode *sysv_alloc_inode(struct super_block *sb) { struct sysv_inode_info *si; - si = kmem_cache_alloc(sysv_inode_cachep, GFP_KERNEL); + si = alloc_inode_sb(sb, sysv_inode_cachep, GFP_KERNEL); if (!si) return NULL; return &si->vfs_inode; diff --git a/fs/ubifs/super.c b/fs/ubifs/super.c index 7b572e1414ba..6851040c598e 100644 --- a/fs/ubifs/super.c +++ b/fs/ubifs/super.c @@ -268,7 +268,7 @@ static struct inode *ubifs_alloc_inode(struct super_block *sb) { struct ubifs_inode *ui; - ui = kmem_cache_alloc(ubifs_inode_slab, GFP_NOFS); + ui = alloc_inode_sb(sb, ubifs_inode_slab, GFP_NOFS); if (!ui) return NULL; diff --git a/fs/udf/super.c b/fs/udf/super.c index 2f83c1204e20..bdeaa5d88a9e 100644 --- a/fs/udf/super.c +++ b/fs/udf/super.c @@ -141,7 +141,7 @@ static struct kmem_cache *udf_inode_cachep; static struct inode *udf_alloc_inode(struct super_block *sb) { struct udf_inode_info *ei; - ei = kmem_cache_alloc(udf_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, udf_inode_cachep, GFP_KERNEL); if (!ei) return NULL; diff --git a/fs/ufs/super.c b/fs/ufs/super.c index 74028b5a7b0a..ba83ad6e9d39 100644 --- a/fs/ufs/super.c +++ b/fs/ufs/super.c @@ -1443,7 +1443,7 @@ static struct inode *ufs_alloc_inode(struct super_block *sb) { struct ufs_inode_info *ei; - ei = kmem_cache_alloc(ufs_inode_cachep, GFP_NOFS); + ei = alloc_inode_sb(sb, ufs_inode_cachep, GFP_NOFS); if (!ei) return NULL; diff --git a/fs/vboxsf/super.c b/fs/vboxsf/super.c index 4f5e59f06284..050ef855158b 100644 --- a/fs/vboxsf/super.c +++ b/fs/vboxsf/super.c @@ -244,7 +244,7 @@ static struct inode *vboxsf_alloc_inode(struct super_block *sb) { struct vboxsf_inode *sf_i; - sf_i = kmem_cache_alloc(vboxsf_inode_cachep, GFP_NOFS); + sf_i = alloc_inode_sb(sb, vboxsf_inode_cachep, GFP_NOFS); if (!sf_i) return NULL; diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c index 3c81daca0e9a..8c8b059e1e26 100644 --- a/fs/xfs/xfs_icache.c +++ b/fs/xfs/xfs_icache.c @@ -40,7 +40,8 @@ xfs_inode_alloc( * XXX: If this didn't occur in transactions, we could drop GFP_NOFAIL * and return NULL here on ENOMEM. */ - ip = kmem_cache_alloc(xfs_inode_zone, GFP_KERNEL | __GFP_NOFAIL); + ip = alloc_inode_sb(mp->m_super, xfs_inode_zone, + GFP_KERNEL | __GFP_NOFAIL); if (inode_init_always(mp->m_super, VFS_I(ip))) { kmem_cache_free(xfs_inode_zone, ip); diff --git a/fs/zonefs/super.c b/fs/zonefs/super.c index cd145d318b17..335b58251b52 100644 --- a/fs/zonefs/super.c +++ b/fs/zonefs/super.c @@ -1152,7 +1152,7 @@ static struct inode *zonefs_alloc_inode(struct super_block *sb) { struct zonefs_inode_info *zi; - zi = kmem_cache_alloc(zonefs_inode_cachep, GFP_KERNEL); + zi = alloc_inode_sb(sb, zonefs_inode_cachep, GFP_KERNEL); if (!zi) return NULL; diff --git a/include/linux/fs.h b/include/linux/fs.h index c3c88fdb9b2a..cb016debb46e 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -41,6 +41,7 @@ #include #include #include +#include #include #include @@ -3200,6 +3201,16 @@ extern void free_inode_nonrcu(struct inode *inode); extern int should_remove_suid(struct dentry *); extern int file_remove_privs(struct file *); +/* + * This must be used for allocating filesystems specific inodes to set + * up the inode reclaim context correctly. + */ +static inline void * +alloc_inode_sb(struct super_block *sb, struct kmem_cache *cache, gfp_t gfp) +{ + return kmem_cache_alloc_lru(cache, &sb->s_inode_lru, gfp); +} + extern void __insert_inode_hash(struct inode *, unsigned long hashval); static inline void insert_inode_hash(struct inode *inode) { diff --git a/ipc/mqueue.c b/ipc/mqueue.c index 4e4e61111500..310178dc5480 100644 --- a/ipc/mqueue.c +++ b/ipc/mqueue.c @@ -483,7 +483,7 @@ static struct inode *mqueue_alloc_inode(struct super_block *sb) { struct mqueue_inode_info *ei; - ei = kmem_cache_alloc(mqueue_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, mqueue_inode_cachep, GFP_KERNEL); if (!ei) return NULL; return &ei->vfs_inode; diff --git a/mm/shmem.c b/mm/shmem.c index ccce139d4e5c..67c8db781727 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3736,7 +3736,7 @@ static struct kmem_cache *shmem_inode_cachep; static struct inode *shmem_alloc_inode(struct super_block *sb) { struct shmem_inode_info *info; - info = kmem_cache_alloc(shmem_inode_cachep, GFP_KERNEL); + info = alloc_inode_sb(sb, shmem_inode_cachep, GFP_KERNEL); if (!info) return NULL; return &info->vfs_inode; diff --git a/net/socket.c b/net/socket.c index 27e3e7d53f8e..d59844b02a2d 100644 --- a/net/socket.c +++ b/net/socket.c @@ -250,7 +250,7 @@ static struct inode *sock_alloc_inode(struct super_block *sb) { struct socket_alloc *ei; - ei = kmem_cache_alloc(sock_inode_cachep, GFP_KERNEL); + ei = alloc_inode_sb(sb, sock_inode_cachep, GFP_KERNEL); if (!ei) return NULL; init_waitqueue_head(&ei->socket.wq.wait); diff --git a/net/sunrpc/rpc_pipe.c b/net/sunrpc/rpc_pipe.c index 09c000d490a1..30707f39694e 100644 --- a/net/sunrpc/rpc_pipe.c +++ b/net/sunrpc/rpc_pipe.c @@ -197,7 +197,7 @@ static struct inode * rpc_alloc_inode(struct super_block *sb) { struct rpc_inode *rpci; - rpci = kmem_cache_alloc(rpc_inode_cachep, GFP_KERNEL); + rpci = alloc_inode_sb(sb, rpc_inode_cachep, GFP_KERNEL); if (!rpci) return NULL; return &rpci->vfs_inode; From patchwork Thu May 27 06:21:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283489 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 695C5C47089 for ; Thu, 27 May 2021 06:25:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0E63D613C9 for ; Thu, 27 May 2021 06:25:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0E63D613C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A7F6D6B007E; Thu, 27 May 2021 02:25:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A55D16B0080; Thu, 27 May 2021 02:25:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F77C6B0081; Thu, 27 May 2021 02:25:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0145.hostedemail.com [216.40.44.145]) by kanga.kvack.org (Postfix) with ESMTP id 60BC66B007E for ; Thu, 27 May 2021 02:25:37 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 0CC146C2D for ; Thu, 27 May 2021 06:25:37 +0000 (UTC) X-FDA: 78186024714.36.5FC1ACA Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf18.hostedemail.com (Postfix) with ESMTP id BFE73200098C for ; Thu, 27 May 2021 06:25:29 +0000 (UTC) Received: by mail-pj1-f50.google.com with SMTP id pi6-20020a17090b1e46b029015cec51d7cdso1749265pjb.5 for ; Wed, 26 May 2021 23:25:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=y++panFifneasGq960GJtHjLst8uLUglPFxOcyVBRks=; b=jgeKoWpk2GjyMfypyzxlbbnac3TWTwOgzJt96jJqAvQI95liqLVhJNfUx8EOTKfhGF 4JR0WlMdklz4TbN/7KMayxK4+sRAHemy4e3o6vpu33f0sRKCaHXUkHVByGhkconcpC5P LVZbgz7ILWTP2+tCT7TKnGNrFHgNmlEnja61hYro+SvTwKieHVACeLNmwwsV0asUCd0M HSoxbz9PvDa2LP22/s6Zhv7I3z8MYyKLC/z1+kvna2RnFttogyHW5fjMhYEC3T41dxS1 gsv1Z2Um5suWkCM0QPtsCPBSJ6VDZasnKWJg78xlOePEKtoJTMvVLbpln17g+eDbIZY6 lgUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=y++panFifneasGq960GJtHjLst8uLUglPFxOcyVBRks=; b=CVFygA438teGHkzEnMNDNG68VMZa974pFZvMZtgu20oHRPyBptqNtXeGqgUPknSkIS OurDvjj1HB3RYNG8XDKiv21nYMWQ64f70EEf7dNhFUbTR2jWE5887upmH2MS+0tkjJww Amux0sRXjM/3RaGFoITXRCUxGF7WBoXyDi2CwGSed//fUJ9cJbGS2iEwJFhnKgNCDQVV yzny2V8L76UBEmlEYN2sxZmEz34hKiYbwVurMirf7Y4fYJ1N4GSzx2v2Hmh2ACTIwpKy R1uSmmshDHA+HgfyWSQ5uio9Zgp2pu12KPsPTqmKh9uacoEe/zvCvSW9XOpn3CxBLNhG RpTQ== X-Gm-Message-State: AOAM532muTRZlbDkbb7WwcN8ZJz5Cx7cPa2TKdGN8T3sW5qUpPS/1AdG /Xkw9mFktSk1gBx2/hyMnT4z+g== X-Google-Smtp-Source: ABdhPJxcGuF5DQLh4LjJiXw5aeyBoGXy1Hjz00Eh2PAxD5BdIel1s/tP58hZR4Q7aTL9uSZUuaTZsQ== X-Received: by 2002:a17:90a:df0c:: with SMTP id gp12mr366298pjb.80.1622096736011; Wed, 26 May 2021 23:25:36 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.25.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:25:35 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 11/21] mm: dcache: use kmem_cache_alloc_lru() to allocate dentry Date: Thu, 27 May 2021 14:21:38 +0800 Message-Id: <20210527062148.9361-12-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BFE73200098C Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=jgeKoWpk; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf18.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.50 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam03 X-Stat-Signature: pzywbqye1migsws41k8kq4cf35ewzeti X-HE-Tag: 1622096729-821774 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like inode cache, the dentry will also be added to its memcg list_lru. So replace kmem_cache_alloc() with kmem_cache_alloc_lru() to allocate dentry. Signed-off-by: Muchun Song --- fs/dcache.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/dcache.c b/fs/dcache.c index cf871a81f4fd..36d4806d7284 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -1741,7 +1741,8 @@ static struct dentry *__d_alloc(struct super_block *sb, const struct qstr *name) char *dname; int err; - dentry = kmem_cache_alloc(dentry_cache, GFP_KERNEL); + dentry = kmem_cache_alloc_lru(dentry_cache, &sb->s_dentry_lru, + GFP_KERNEL); if (!dentry) return NULL; From patchwork Thu May 27 06:21:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283491 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0413AC4707F for ; Thu, 27 May 2021 06:25:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A7477610CC for ; Thu, 27 May 2021 06:25:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A7477610CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 48D796B0080; Thu, 27 May 2021 02:25:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 462726B0081; Thu, 27 May 2021 02:25:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B5ED6B0082; Thu, 27 May 2021 02:25:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0041.hostedemail.com [216.40.44.41]) by kanga.kvack.org (Postfix) with ESMTP id EFCCD6B0080 for ; Thu, 27 May 2021 02:25:44 -0400 (EDT) Received: from smtpin39.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 939F86C16 for ; Thu, 27 May 2021 06:25:44 +0000 (UTC) X-FDA: 78186025008.39.185330B Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf12.hostedemail.com (Postfix) with ESMTP id C7F992C0 for ; Thu, 27 May 2021 06:25:33 +0000 (UTC) Received: by mail-pj1-f49.google.com with SMTP id f8so97284pjh.0 for ; Wed, 26 May 2021 23:25:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=O9BJLsBD6jksCciTZGdmyHq2qkXc8II2TIDnrT2cyiU=; b=FJZgs1UL1z0my49Azq2hr0n8sscr/I33Tt7ZJ7/5pTqF5PlVpBytz21l20UwTaaemz Cd6gjsbyJXnE+z24F+l394YMOknHJU5R6Y+0cOJDFyj5Ke5qYdZN2D1h7Bw0EIehRYXM jk0UzBbqZKDtPyY7z3heDw5ZjzO6v48H5oHEZEolFWJOHBodu60/IpFdEgb9/Yfs8/fS g+aP04+BW6pXtvUTWQTF5eC/9hij+e/aN/5dOfYQKWFX4eqOktY2ufGeFIUm7MdDPF55 9VR3qK8PckMO5KDm2CkCvtwgNvlVKOH4hTYMiuusDXg7pQns30ASKdicpuxO1E7i65b+ 1/Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=O9BJLsBD6jksCciTZGdmyHq2qkXc8II2TIDnrT2cyiU=; b=ti8VvfoqiPEM8DeJijVeQGYyCuGYdINiDCvZBM6z+sWx4eP5zsK0O/KXX2/N2/+R99 oY3mus29h1/r7XFkts+lEElGngflIwmKXb4XQccnCA4023dvKyBlm5V0frRCdU26yH7w +RIg81PNX67TX5/ovTafeOBLrE0Xx9VdeQt1t9D+wrQI+5Y8whLTP9F0FsUM/kEAduMX s2wW7b/wBDxK4vNS9mPboMyb9w0DZBuJyvGt/TKtROqJYJEowXroFsfcwp6CsnWXmgzR GWDjOvFlIVNhhFarqPrRNGdXa2HyKSWhNj/QDfMSdMUpqsQfqDaG+CGoYDSX8av4RiJ8 9z0w== X-Gm-Message-State: AOAM531QSY2/1XsdNZzwDraqn0QMI+M6wNEWfr/KArmYexf6cbcTrSxo QO67jvoCuz3sTXOMVxSGr/vRbQ== X-Google-Smtp-Source: ABdhPJyITgWwJf3ArkmGB8EQB+3oU5oRfWiZaxjGohk/rm3HWIHqMNYrh4zk1ZJe4Tz/8GzmJpP71g== X-Received: by 2002:a17:90a:bd01:: with SMTP id y1mr7650173pjr.166.1622096743377; Wed, 26 May 2021 23:25:43 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.25.36 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:25:43 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 12/21] xarray: use kmem_cache_alloc_lru to allocate xa_node Date: Thu, 27 May 2021 14:21:39 +0800 Message-Id: <20210527062148.9361-13-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C7F992C0 Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=FJZgs1UL; spf=pass (imf12.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam04 X-Stat-Signature: 1gkdkg5ioy3ru74d86q9hsd9tjkeixt9 X-HE-Tag: 1622096733-415838 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The workingset will add the xa_node to the shadow_nodes list. So the allocation of xa_node should be done by kmem_cache_alloc_lru(). The workingset can use xas_set_lru() to pass the list_lru. Signed-off-by: Muchun Song --- include/linux/xarray.h | 9 ++++++++- lib/xarray.c | 10 +++++----- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index a91e3d90df8a..31f3e5ef3c7b 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -1317,6 +1317,7 @@ struct xa_state { struct xa_node *xa_node; struct xa_node *xa_alloc; xa_update_node_t xa_update; + struct list_lru *xa_lru; }; /* @@ -1336,7 +1337,8 @@ struct xa_state { .xa_pad = 0, \ .xa_node = XAS_RESTART, \ .xa_alloc = NULL, \ - .xa_update = NULL \ + .xa_update = NULL, \ + .xa_lru = NULL, \ } /** @@ -1613,6 +1615,11 @@ static inline void xas_set_update(struct xa_state *xas, xa_update_node_t update) xas->xa_update = update; } +static inline void xas_set_lru(struct xa_state *xas, struct list_lru *lru) +{ + xas->xa_lru = lru; +} + /** * xas_next_entry() - Advance iterator to next present entry. * @xas: XArray operation state. diff --git a/lib/xarray.c b/lib/xarray.c index f5d8f54907b4..e9b818abc823 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -302,7 +302,7 @@ bool xas_nomem(struct xa_state *xas, gfp_t gfp) } if (xas->xa->xa_flags & XA_FLAGS_ACCOUNT) gfp |= __GFP_ACCOUNT; - xas->xa_alloc = kmem_cache_alloc(radix_tree_node_cachep, gfp); + xas->xa_alloc = kmem_cache_alloc_lru(radix_tree_node_cachep, xas->xa_lru, gfp); if (!xas->xa_alloc) return false; xas->xa_alloc->parent = NULL; @@ -334,10 +334,10 @@ static bool __xas_nomem(struct xa_state *xas, gfp_t gfp) gfp |= __GFP_ACCOUNT; if (gfpflags_allow_blocking(gfp)) { xas_unlock_type(xas, lock_type); - xas->xa_alloc = kmem_cache_alloc(radix_tree_node_cachep, gfp); + xas->xa_alloc = kmem_cache_alloc_lru(radix_tree_node_cachep, xas->xa_lru, gfp); xas_lock_type(xas, lock_type); } else { - xas->xa_alloc = kmem_cache_alloc(radix_tree_node_cachep, gfp); + xas->xa_alloc = kmem_cache_alloc_lru(radix_tree_node_cachep, xas->xa_lru, gfp); } if (!xas->xa_alloc) return false; @@ -371,7 +371,7 @@ static void *xas_alloc(struct xa_state *xas, unsigned int shift) if (xas->xa->xa_flags & XA_FLAGS_ACCOUNT) gfp |= __GFP_ACCOUNT; - node = kmem_cache_alloc(radix_tree_node_cachep, gfp); + node = kmem_cache_alloc_lru(radix_tree_node_cachep, xas->xa_lru, gfp); if (!node) { xas_set_err(xas, -ENOMEM); return NULL; @@ -1014,7 +1014,7 @@ void xas_split_alloc(struct xa_state *xas, void *entry, unsigned int order, void *sibling = NULL; struct xa_node *node; - node = kmem_cache_alloc(radix_tree_node_cachep, gfp); + node = kmem_cache_alloc_lru(radix_tree_node_cachep, xas->xa_lru, gfp); if (!node) goto nomem; node->array = xas->xa; From patchwork Thu May 27 06:21:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1286DC4707F for ; Thu, 27 May 2021 06:25:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ABE6F6135F for ; Thu, 27 May 2021 06:25:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ABE6F6135F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4B5506B0072; Thu, 27 May 2021 02:25:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4899F6B0082; Thu, 27 May 2021 02:25:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3794B6B0083; Thu, 27 May 2021 02:25:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0199.hostedemail.com [216.40.44.199]) by kanga.kvack.org (Postfix) with ESMTP id 0800C6B0072 for ; Thu, 27 May 2021 02:25:52 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 9D23F8249980 for ; Thu, 27 May 2021 06:25:52 +0000 (UTC) X-FDA: 78186025344.22.5E15BF2 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf15.hostedemail.com (Postfix) with ESMTP id D8DCAA0001D7 for ; Thu, 27 May 2021 06:25:47 +0000 (UTC) Received: by mail-pl1-f170.google.com with SMTP id t21so1834889plo.2 for ; Wed, 26 May 2021 23:25:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=esJzMd1gfpQ9fKrHNcdHQMirHFw6zXQFsOGdVn4/sAs=; b=lg7fglwo870iqNRshvVXpvKjYWsyy0gdHArvvqoq9VYfn+EdTnRqUg7WMOQWjdPQE6 eZ0wlnDPIul1IsIbXIyv9OaDHgh2r7ZAsx0HfZ0dZHWXaBBEp6dr/GLKXlQBU3DmAnSh x6gtjGTnKimDBrG31MHDks0RemPb+3PpCBEbNliGZbN1YUdDLHnfg7cmuwdFyngP2jL5 yoW1mBXFX1bp074oOHquuzkEPGVqzHGbFrw6xJCXt5j+in5tvYnmgPYrfhMtwiR72uix dpN+CB+XL+pdYAsTGIzG8AAhxdlrp9hlpMiombqEn+bdkO7f0lyemvTTnaM9jN/A1vnx d8og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=esJzMd1gfpQ9fKrHNcdHQMirHFw6zXQFsOGdVn4/sAs=; b=QS7yNOMfS4EN/RrHjhZyZUkU/+LinhEQBQm92GCx+lJJBnFZiWmQGKSePDDPS8QaZy g8TWiOFANl0LLwh4uk5fb+ij7zhcj0dP9Kx2/hTIy0kScG/j1GTcE+HFIG6xVC2StIIn o/47NR0bBMpuOutNMbLvt1V7N+taKTcrzrvJ0hWajdA2j/b3ri3pIi9jdKr732HPw695 3JDQ6Gxfo0VPNUWVlh6L0bliuJJq1WOP3erG4PXcPt8y2mQ/BYUYLz14JOWClMm/4raa TK17ivtaLCQXs130ZT/cihvmtg0AfEue0/K/ZVcsmkAxcDkD7wTxNyiNSbHW8fEopaWg zvew== X-Gm-Message-State: AOAM532krwyp1n7ihTVzKCKLSFFJPO7ahqlkoyYDObm9Tn7COuorLPIS wNuipM5em8/8FIIzR4rz+Aw6zw== X-Google-Smtp-Source: ABdhPJwE3j/uDdrFkGioyn232Py8LrD+rF3I2uZIA4806ZDgCQOieiNs1D8mhZLhedMe5GVStbUYTg== X-Received: by 2002:a17:902:c94a:b029:f4:ad23:7e59 with SMTP id i10-20020a170902c94ab02900f4ad237e59mr1757618pla.73.1622096751538; Wed, 26 May 2021 23:25:51 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.25.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:25:51 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 13/21] mm: workingset: use xas_set_lru() to pass shadow_nodes Date: Thu, 27 May 2021 14:21:40 +0800 Message-Id: <20210527062148.9361-14-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D8DCAA0001D7 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=lg7fglwo; spf=pass (imf15.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam04 X-Stat-Signature: ofa89q4q6aoxiad7nbi47d93buu4iku4 X-HE-Tag: 1622096747-356988 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The workingset will add the xa_node to shadow_nodes, so we should use xas_set_lru() to pass the list_lru which we want to insert xa_node into to set up the xa_node reclaim context correctly. Signed-off-by: Muchun Song --- include/linux/swap.h | 5 ++++- mm/workingset.c | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 46d51d058d05..a2e1363b1509 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -323,9 +323,12 @@ void workingset_activation(struct page *page); /* Only track the nodes of mappings with shadow entries */ void workingset_update_node(struct xa_node *node); +extern struct list_lru shadow_nodes; #define mapping_set_update(xas, mapping) do { \ - if (!dax_mapping(mapping) && !shmem_mapping(mapping)) \ + if (!dax_mapping(mapping) && !shmem_mapping(mapping)) { \ xas_set_update(xas, workingset_update_node); \ + xas_set_lru(xas, &shadow_nodes); \ + } \ } while (0) /* linux/mm/page_alloc.c */ diff --git a/mm/workingset.c b/mm/workingset.c index 4f7a306ce75a..e50827fc3994 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -426,7 +426,7 @@ void workingset_activation(struct page *page) * point where they would still be useful. */ -static struct list_lru shadow_nodes; +struct list_lru shadow_nodes; void workingset_update_node(struct xa_node *node) { From patchwork Thu May 27 06:21:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 774FCC4708A for ; Thu, 27 May 2021 06:26:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 21F3A6135F for ; Thu, 27 May 2021 06:26:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 21F3A6135F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B90086B0071; Thu, 27 May 2021 02:26:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B67336B0082; Thu, 27 May 2021 02:26:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A090D6B0083; Thu, 27 May 2021 02:26:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0083.hostedemail.com [216.40.44.83]) by kanga.kvack.org (Postfix) with ESMTP id 6E8E46B0071 for ; Thu, 27 May 2021 02:26:00 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 0B558181AF5C6 for ; Thu, 27 May 2021 06:26:00 +0000 (UTC) X-FDA: 78186025680.23.1650C81 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf21.hostedemail.com (Postfix) with ESMTP id D0653E0001AC for ; Thu, 27 May 2021 06:25:49 +0000 (UTC) Received: by mail-pj1-f53.google.com with SMTP id q6so2153529pjj.2 for ; Wed, 26 May 2021 23:25:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=W7N2ct9tvwpvEjrUOuEEAnTDqNZXNpAiEvvqkroYr+g=; b=ytjArmjrCynH8+BHYQhJn3l2xFHC/kzH5JV7q3S09vFyI8q2M7bQ7H62vKgZUJzl3j BsD8YXyssOEb2ChO4D+qbmQ2b4dPnvcel+FgDdQbmUnbWOkODiQ1QRpam/cuzMraDjWL dsZGm36R0hAdHaZ/jo3aDsKoOLz6/P1r8u4bG24ikZr9037EoG2id02RD5n0te9o1yMI 8Qgts2qu7IjeiOOlf5lDi5QmTN6HSHOkOewNVSh/L+zt5mH3WOP3IRGvOXo/F5neNqJu GE9SuvApfitnMvcJpO0BfrfVE1nkfD++1HL4nEkRxuqhVGcmnVjoJ5uhRy/LgVDgyE3p CTag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=W7N2ct9tvwpvEjrUOuEEAnTDqNZXNpAiEvvqkroYr+g=; b=WkTR1OKHPkVCh3mRTrQrdUPV1nSOrZTUC+LE6KAfWy1HXLcOnVQulHOSKT2kTANssw tIDDNMMz0jZ6x15wQqNybWnGxLHSm7T3aNSHIRmVp/LfHMMKKCnpAu4HiVGzVnTbEsQP oHD45ILzdW3mbZsKNaFugQaJ136d8KIbPcklLde9YNwQCSeu9UqbLz6WAqw9Ulz9G3OG JlSc7/h6JDXPO9Nxz5AP6rHHqdowsDi/BqapTCwz7SQRbTeSnzisnxlA4K5nhXsPSQm1 94KOwXqYE3EaXG7MI6swxBEM+J4jz0TwmX7fbgJeFL6LsPcF5BVtU40UV1nx6n0Di5T6 V0FQ== X-Gm-Message-State: AOAM5310JZYgxsOSuN4aS8TAm56HflY3ljgZxS7kCNk4uDyFtIwSwzxq dfrI7sXjyZvfZVJ1ydTQ2NBsag== X-Google-Smtp-Source: ABdhPJyjKtx6LhNjABgtOZQr2Qu+kUPjO8341ZuPAPYwgXBiRvhc/qzEB6KLPEC0xwN4HImLEvQ84A== X-Received: by 2002:a17:902:8ec4:b029:fb:33ad:e86f with SMTP id x4-20020a1709028ec4b02900fb33ade86fmr1972930plo.4.1622096758917; Wed, 26 May 2021 23:25:58 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.25.51 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:25:58 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 14/21] nfs42: use a specific kmem_cache to allocate nfs4_xattr_entry Date: Thu, 27 May 2021 14:21:41 +0800 Message-Id: <20210527062148.9361-15-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=ytjArmjr; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf21.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: qj1xzyjufntjqdurhtbnq6dt5ukm1cq3 X-Rspamd-Queue-Id: D0653E0001AC X-Rspamd-Server: rspam02 X-HE-Tag: 1622096749-460930 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we want to add the allocated objects to its list_lru, we should use kmem_cache_alloc_lru() to allocate objects. So intruduce nfs4_xattr_entry_cachep which is used to allocate nfs4_xattr_entry. Signed-off-by: Muchun Song --- fs/nfs/nfs42xattr.c | 95 ++++++++++++++++++++++++++--------------------------- 1 file changed, 47 insertions(+), 48 deletions(-) diff --git a/fs/nfs/nfs42xattr.c b/fs/nfs/nfs42xattr.c index 1c4d2a05b401..5b7af9080db0 100644 --- a/fs/nfs/nfs42xattr.c +++ b/fs/nfs/nfs42xattr.c @@ -81,7 +81,7 @@ struct nfs4_xattr_entry { struct hlist_node hnode; struct list_head lru; struct list_head dispose; - char *xattr_name; + const char *xattr_name; void *xattr_value; size_t xattr_size; struct nfs4_xattr_bucket *bucket; @@ -98,6 +98,7 @@ static struct list_lru nfs4_xattr_entry_lru; static struct list_lru nfs4_xattr_large_entry_lru; static struct kmem_cache *nfs4_xattr_cache_cachep; +static struct kmem_cache *nfs4_xattr_entry_cachep; /* * Hashing helper functions. @@ -177,49 +178,27 @@ nfs4_xattr_alloc_entry(const char *name, const void *value, { struct nfs4_xattr_entry *entry; void *valp; - char *namep; - size_t alloclen, slen; - char *buf; - uint32_t flags; + const char *namep; + uint32_t flags = len > PAGE_SIZE ? NFS4_XATTR_ENTRY_EXTVAL : 0; + gfp_t gfp = GFP_KERNEL_ACCOUNT | GFP_NOFS; + struct list_lru *lru; BUILD_BUG_ON(sizeof(struct nfs4_xattr_entry) + XATTR_NAME_MAX + 1 > PAGE_SIZE); - alloclen = sizeof(struct nfs4_xattr_entry); - if (name != NULL) { - slen = strlen(name) + 1; - alloclen += slen; - } else - slen = 0; - - if (alloclen + len <= PAGE_SIZE) { - alloclen += len; - flags = 0; - } else { - flags = NFS4_XATTR_ENTRY_EXTVAL; - } - - buf = kmalloc(alloclen, GFP_KERNEL_ACCOUNT | GFP_NOFS); - if (buf == NULL) + lru = flags & NFS4_XATTR_ENTRY_EXTVAL ? &nfs4_xattr_large_entry_lru : + &nfs4_xattr_entry_lru; + entry = kmem_cache_alloc_lru(nfs4_xattr_entry_cachep, lru, gfp); + if (!entry) return NULL; - entry = (struct nfs4_xattr_entry *)buf; - - if (name != NULL) { - namep = buf + sizeof(struct nfs4_xattr_entry); - memcpy(namep, name, slen); - } else { - namep = NULL; - } - - - if (flags & NFS4_XATTR_ENTRY_EXTVAL) { - valp = kvmalloc(len, GFP_KERNEL_ACCOUNT | GFP_NOFS); - if (valp == NULL) { - kfree(buf); - return NULL; - } - } else if (len != 0) { - valp = buf + sizeof(struct nfs4_xattr_entry) + slen; + namep = kstrdup_const(name, gfp); + if (!namep && name) + goto free_buf; + + if (len != 0) { + valp = kvmalloc(len, gfp); + if (!valp) + goto free_name; } else valp = NULL; @@ -232,23 +211,23 @@ nfs4_xattr_alloc_entry(const char *name, const void *value, entry->flags = flags; entry->xattr_value = valp; - kref_init(&entry->ref); entry->xattr_name = namep; entry->xattr_size = len; - entry->bucket = NULL; - INIT_LIST_HEAD(&entry->lru); - INIT_LIST_HEAD(&entry->dispose); - INIT_HLIST_NODE(&entry->hnode); return entry; +free_name: + kfree_const(namep); +free_buf: + kmem_cache_free(nfs4_xattr_entry_cachep, entry); + return NULL; } static void nfs4_xattr_free_entry(struct nfs4_xattr_entry *entry) { - if (entry->flags & NFS4_XATTR_ENTRY_EXTVAL) - kvfree(entry->xattr_value); - kfree(entry); + kvfree(entry->xattr_value); + kfree_const(entry->xattr_name); + kmem_cache_free(nfs4_xattr_entry_cachep, entry); } static void @@ -289,7 +268,7 @@ nfs4_xattr_alloc_cache(void) { struct nfs4_xattr_cache *cache; - cache = kmem_cache_alloc(nfs4_xattr_cache_cachep, + cache = kmem_cache_alloc_lru(nfs4_xattr_cache_cachep, &nfs4_xattr_cache_lru, GFP_KERNEL_ACCOUNT | GFP_NOFS); if (cache == NULL) return NULL; @@ -992,6 +971,17 @@ static void nfs4_xattr_cache_init_once(void *p) INIT_LIST_HEAD(&cache->dispose); } +static void nfs4_xattr_entry_init_once(void *p) +{ + struct nfs4_xattr_entry *entry = p; + + kref_init(&entry->ref); + entry->bucket = NULL; + INIT_LIST_HEAD(&entry->lru); + INIT_LIST_HEAD(&entry->dispose); + INIT_HLIST_NODE(&entry->hnode); +} + int __init nfs4_xattr_cache_init(void) { int ret = 0; @@ -1003,6 +993,13 @@ int __init nfs4_xattr_cache_init(void) if (nfs4_xattr_cache_cachep == NULL) return -ENOMEM; + nfs4_xattr_entry_cachep = kmem_cache_create("nfs4_xattr_entry", + sizeof(struct nfs4_xattr_entry), 0, + (SLAB_RECLAIM_ACCOUNT | SLAB_MEM_SPREAD | SLAB_ACCOUNT), + nfs4_xattr_entry_init_once); + if (!nfs4_xattr_entry_cachep) + goto out5; + ret = list_lru_init_memcg(&nfs4_xattr_large_entry_lru, &nfs4_xattr_large_entry_shrinker); if (ret) @@ -1040,6 +1037,8 @@ int __init nfs4_xattr_cache_init(void) out3: list_lru_destroy(&nfs4_xattr_large_entry_lru); out4: + kmem_cache_destroy(nfs4_xattr_entry_cachep); +out5: kmem_cache_destroy(nfs4_xattr_cache_cachep); return ret; From patchwork Thu May 27 06:21:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9913C4707F for ; Thu, 27 May 2021 06:26:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 854AC613D8 for ; Thu, 27 May 2021 06:26:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 854AC613D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 267CD6B0082; Thu, 27 May 2021 02:26:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 23FB76B0083; Thu, 27 May 2021 02:26:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 10BD06B0085; Thu, 27 May 2021 02:26:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0171.hostedemail.com [216.40.44.171]) by kanga.kvack.org (Postfix) with ESMTP id D3CA46B0082 for ; Thu, 27 May 2021 02:26:07 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 78080180AF420 for ; Thu, 27 May 2021 06:26:07 +0000 (UTC) X-FDA: 78186025974.06.A6E97FE Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf21.hostedemail.com (Postfix) with ESMTP id 4242BE0001AC for ; Thu, 27 May 2021 06:25:57 +0000 (UTC) Received: by mail-pj1-f41.google.com with SMTP id ot16so2149419pjb.3 for ; Wed, 26 May 2021 23:26:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oxzrjvjXV2Eb7kWOVBsK4sUU1R7f70qUcGgMHnDVcHI=; b=t4hkMdYNvrvfTjfTlG4PkoB+1Y72l29crVEA4B0v9iJzdHGAeR7stiwSa9UlZhvTKv BrPAqyck16KYdqqW1zhyGFF9NRlSY3TlHhpTZz34YNZPZg9kaJRCIbhXMRZ1zVM52mcz 0fJTaCqA1XgvhKynVVYi3qbYRbFZK1pHmZ7n55qPOauFpIOVFHVyd5fN2YBWS3XMbWqo beud+Ewd9d81BNQpBv9GViXrbb4dwTG3u8ZGJzSoU9tUM3tJX2p8WXsUS23pCEfAvc1H O259gMTGuy/TEsH8cJ/lUgmzyL2akCG0RjTUdyjzNpKiFozXI0Z5glni2U5T5BScwLvO PviQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oxzrjvjXV2Eb7kWOVBsK4sUU1R7f70qUcGgMHnDVcHI=; b=AFRdtz9nFECT0igb8aFCmZzjdvwD8VZ9Cykpvjrsti69c004X3/uMtTjJYb7EMJFJp hNKPv9mikGIDt2ZCbO78YU6eChw0snt69v1671WjLofEOAoT3raGv+1GnR3/ZBKMB4ro B52Z8M/khymr5NcykrFaWfeI/sXodZ4AY+GNuWRQ2oG61Fu+hc8wZ96fzZDvqxYtCsQs 1lHM84pdfswmonJpWaO1xSFXFf/8TXdOEZlpBAiEAUv4ZnxRj2zDy+25iuubLoOJlX16 5tujusfKK4KZP13JYtKPdxZDLreEJFWog27CA9n6zeqDvbzacqcB65hvrWfW9oJITBty ChlA== X-Gm-Message-State: AOAM530bBp/KAlT2SSRIG9DF3I0VfG/LOzUVO3w0lEGM68zBUDpvy6Cm 2ftv2kqVWgCOD1kKnE1z/W+kKw== X-Google-Smtp-Source: ABdhPJzb1VGLxwaOmIEI3Kq0EsIx65fHzy4pZUgLiumgkQjJ8LkIodccqpolnl9x+a/RgJ1oO/TpfA== X-Received: by 2002:a17:90a:590d:: with SMTP id k13mr2203021pji.68.1622096766339; Wed, 26 May 2021 23:26:06 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.25.59 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:26:06 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 15/21] mm: list_lru: allocate list_lru_one only when needed Date: Thu, 27 May 2021 14:21:42 +0800 Message-Id: <20210527062148.9361-16-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=t4hkMdYN; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf21.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: 8sokk9ef9wgewpaaxdj317dky984kksp X-Rspamd-Queue-Id: 4242BE0001AC X-Rspamd-Server: rspam02 X-HE-Tag: 1622096757-15955 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In our server, we found a suspected memory leak problem. The kmalloc-32 consumes more than 6GB of memory. Other kmem_caches consume less than 2GB memory. After our in-depth analysis, the memory consumption of kmalloc-32 slab cache is the cause of list_lru_one allocation. crash> p memcg_nr_cache_ids memcg_nr_cache_ids = $2 = 24574 memcg_nr_cache_ids is very large and memory consumption of each list_lru can be calculated with the following formula. num_numa_node * memcg_nr_cache_ids * 32 (kmalloc-32) There are 4 numa nodes in our system, so each list_lru consumes ~3MB. crash> list super_blocks | wc -l 952 Every mount will register 2 list lrus, one is for inode, another is for dentry. There are 952 super_blocks. So the total memory is 952 * 2 * 3 MB (~5.6GB). But the number of memory cgroup is less than 500. So I guess more than 12286 containers have been deployed on this machine (I do not know why there are so many containers, it may be a user's bug or the user really want to do that). And memcg_nr_cache_ids has not been reduced to a suitable value. This can waste a lot of memory. Now the infrastructure for dynamic list_lru_one allocation is ready, so remove statically allocated memory code to save memory. Signed-off-by: Muchun Song --- include/linux/list_lru.h | 8 +-- mm/list_lru.c | 125 +++++++++++++++++++++++++++-------------------- mm/memcontrol.c | 7 ++- 3 files changed, 81 insertions(+), 59 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 2083f4f2701f..00c15dc1ede3 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -32,14 +32,15 @@ struct list_lru_one { }; struct list_lru_per_memcg { + struct rcu_head rcu; /* array of per cgroup per node lists, indexed by node id */ - struct list_lru_one nodes[0]; + struct list_lru_one nodes[]; }; struct list_lru_memcg { struct rcu_head rcu; /* array of per cgroup lists, indexed by memcg_cache_id */ - struct list_lru_per_memcg *lrus[]; + struct list_lru_per_memcg __rcu *lrus[]; }; struct list_lru_node { @@ -75,7 +76,8 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, __list_lru_init((lru), true, NULL, shrinker) int memcg_update_all_list_lrus(int num_memcgs); -void memcg_drain_all_list_lrus(int src_idx, struct mem_cgroup *dst_memcg); +void memcg_drain_all_list_lrus(struct mem_cgroup *src_memcg, + struct mem_cgroup *dst_memcg); /** * list_lru_add: add an element to the lru list's tail diff --git a/mm/list_lru.c b/mm/list_lru.c index 4ba1db6d4409..43eadb0834e1 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -60,8 +60,12 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) */ memcg_lrus = rcu_dereference_check(lru->memcg_lrus, lockdep_is_held(&nlru->lock)); - if (memcg_lrus && idx >= 0) - return &memcg_lrus->lrus[idx]->nodes[nid]; + if (memcg_lrus && idx >= 0) { + struct list_lru_per_memcg *mlru; + + mlru = rcu_dereference_check(memcg_lrus->lrus[idx], true); + return mlru ? &mlru->nodes[nid] : NULL; + } return &nlru->lru; } @@ -184,11 +188,12 @@ unsigned long list_lru_count_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg) { struct list_lru_one *l; - long count; + long count = 0; rcu_read_lock(); l = list_lru_from_memcg_idx(lru, nid, memcg_cache_id(memcg)); - count = READ_ONCE(l->nr_items); + if (l) + count = READ_ONCE(l->nr_items); rcu_read_unlock(); if (unlikely(count < 0)) @@ -217,8 +222,11 @@ __list_lru_walk_one(struct list_lru *lru, int nid, int memcg_idx, struct list_head *item, *n; unsigned long isolated = 0; - l = list_lru_from_memcg_idx(lru, nid, memcg_idx); restart: + l = list_lru_from_memcg_idx(lru, nid, memcg_idx); + if (!l) + goto out; + list_for_each_safe(item, n, &l->list) { enum lru_status ret; @@ -262,6 +270,7 @@ __list_lru_walk_one(struct list_lru *lru, int nid, int memcg_idx, BUG(); } } +out: return isolated; } @@ -354,20 +363,26 @@ static struct list_lru_per_memcg *memcg_list_lru_alloc(gfp_t gfp) return lru; } -static int memcg_init_list_lru_range(struct list_lru_memcg *memcg_lrus, - int begin, int end) +static void memcg_list_lru_free(struct list_lru *lru, int src_idx) { - int i; + struct list_lru_memcg *memcg_lrus; + struct list_lru_per_memcg *mlru; - for (i = begin; i < end; i++) { - memcg_lrus->lrus[i] = memcg_list_lru_alloc(GFP_KERNEL); - if (!memcg_lrus->lrus[i]) - goto fail; - } - return 0; -fail: - memcg_destroy_list_lru_range(memcg_lrus, begin, i); - return -ENOMEM; + spin_lock_irq(&lru->lock); + memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, true); + mlru = rcu_dereference_protected(memcg_lrus->lrus[src_idx], true); + if (mlru) + rcu_assign_pointer(memcg_lrus->lrus[src_idx], NULL); + spin_unlock_irq(&lru->lock); + + /* + * The __list_lru_walk_one() can walk the list of this node. + * We need kvfree_rcu() here. And the walking of the list + * is under lru->node[nid]->lock, which can serve as a RCU + * read-side critical section. + */ + if (mlru) + kvfree_rcu(mlru, rcu); } static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) @@ -382,15 +397,11 @@ static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) spin_lock_init(&lru->lock); - memcg_lrus = kvmalloc(sizeof(*memcg_lrus) + + memcg_lrus = kvzalloc(sizeof(*memcg_lrus) + size * sizeof(memcg_lrus->lrus[0]), GFP_KERNEL); if (!memcg_lrus) return -ENOMEM; - if (memcg_init_list_lru_range(memcg_lrus, 0, size)) { - kvfree(memcg_lrus); - return -ENOMEM; - } RCU_INIT_POINTER(lru->memcg_lrus, memcg_lrus); return 0; @@ -424,13 +435,9 @@ static int memcg_update_list_lru(struct list_lru *lru, int old_size, int new_siz if (!new) return -ENOMEM; - if (memcg_init_list_lru_range(new, old_size, new_size)) { - kvfree(new); - return -ENOMEM; - } - spin_lock_irq(&lru->lock); memcpy(&new->lrus, &old->lrus, old_size * sizeof(new->lrus[0])); + memset(&new->lrus[old_size], 0, (new_size - old_size) * sizeof(new->lrus[0])); rcu_assign_pointer(lru->memcg_lrus, new); spin_unlock_irq(&lru->lock); @@ -439,20 +446,6 @@ static int memcg_update_list_lru(struct list_lru *lru, int old_size, int new_siz return 0; } -static void memcg_cancel_update_list_lru(struct list_lru *lru, - int old_size, int new_size) -{ - struct list_lru_memcg *memcg_lrus; - - memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, - lockdep_is_held(&list_lrus_mutex)); - /* - * Do not bother shrinking the array back to the old size, because we - * cannot handle allocation failures here. - */ - memcg_destroy_list_lru_range(memcg_lrus, old_size, new_size); -} - int memcg_update_all_list_lrus(int new_size) { int ret = 0; @@ -463,15 +456,10 @@ int memcg_update_all_list_lrus(int new_size) list_for_each_entry(lru, &list_lrus, list) { ret = memcg_update_list_lru(lru, old_size, new_size); if (ret) - goto fail; + break; } -out: mutex_unlock(&list_lrus_mutex); return ret; -fail: - list_for_each_entry_continue_reverse(lru, &list_lrus, list) - memcg_cancel_update_list_lru(lru, old_size, new_size); - goto out; } static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, @@ -488,6 +476,8 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, spin_lock_irq(&nlru->lock); src = list_lru_from_memcg_idx(lru, nid, src_idx); + if (!src) + goto out; dst = list_lru_from_memcg_idx(lru, nid, dst_idx); list_splice_init(&src->list, &dst->list); @@ -497,7 +487,7 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0; } - +out: spin_unlock_irq(&nlru->lock); } @@ -508,11 +498,37 @@ static void memcg_drain_list_lru(struct list_lru *lru, for_each_node(i) memcg_drain_list_lru_node(lru, i, src_idx, dst_memcg); + + memcg_list_lru_free(lru, src_idx); } -void memcg_drain_all_list_lrus(int src_idx, struct mem_cgroup *dst_memcg) +void memcg_drain_all_list_lrus(struct mem_cgroup *src_memcg, + struct mem_cgroup *dst_memcg) { + struct cgroup_subsys_state *css; struct list_lru *lru; + int src_idx = src_memcg->kmemcg_id; + + /* + * Change kmemcg_id of this cgroup and all its descendants to the + * parent's id, and then move all entries from this cgroup's list_lrus + * to ones of the parent. + * + * After we have finished, all list_lrus corresponding to this cgroup + * are guaranteed to remain empty. So we can safely free this cgroup's + * list lrus which is freed in memcg_list_lru_free(). + * Changing ->kmemcg_id to the parent can prevent list_lru_memcg_alloc() + * from allocating list lrus for this cgroup after calling + * memcg_list_lru_free(). + */ + rcu_read_lock(); + css_for_each_descendant_pre(css, &src_memcg->css) { + struct mem_cgroup *memcg; + + memcg = mem_cgroup_from_css(css); + memcg->kmemcg_id = dst_memcg->kmemcg_id; + } + rcu_read_unlock(); mutex_lock(&list_lrus_mutex); list_for_each_entry(lru, &list_lrus, list) @@ -531,7 +547,7 @@ static bool memcg_list_lru_skip_alloc(struct list_lru *lru, rcu_read_lock(); memcg_lrus = rcu_dereference(lru->memcg_lrus); - if (memcg_lrus->lrus[idx]) { + if (rcu_access_pointer(memcg_lrus->lrus[idx])) { rcu_read_unlock(); return true; } @@ -546,7 +562,7 @@ int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t g struct list_lru_memcg *memcg_lrus; int i; - struct list_lru_memcg { + struct list_lru_memcg_table { struct list_lru_per_memcg *mlru; struct mem_cgroup *memcg; } *table; @@ -590,11 +606,12 @@ int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t g memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, true); while (i--) { int index = memcg_cache_id(table[i].memcg); + struct list_lru_per_memcg *mlru = table[i].mlru; - if (memcg_lrus->lrus[index]) - kfree(table[i].mlru); + if (index < 0 || rcu_dereference_protected(memcg_lrus->lrus[index], true)) + kfree(mlru); else - memcg_lrus->lrus[index] = table[i].mlru; + rcu_assign_pointer(memcg_lrus->lrus[index], mlru); } spin_unlock_irqrestore(&lru->lock, flags); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 09bafa82781f..1994513a33d4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3654,10 +3654,13 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) memcg_reparent_objcgs(memcg, parent); + /* + * memcg_drain_all_list_lrus() can change memcg->kmemcg_id. + * Cache it to @kmemcg_id. + */ kmemcg_id = memcg->kmemcg_id; - /* memcg_reparent_objcgs() must be called before this. */ - memcg_drain_all_list_lrus(kmemcg_id, parent); + memcg_drain_all_list_lrus(memcg, parent); memcg_free_cache_id(kmemcg_id); } From patchwork Thu May 27 06:21:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283499 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFF0EC47089 for ; Thu, 27 May 2021 06:26:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5D2C66135F for ; Thu, 27 May 2021 06:26:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5D2C66135F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F3A766B0083; Thu, 27 May 2021 02:26:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F11466B0085; Thu, 27 May 2021 02:26:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB2508D0001; Thu, 27 May 2021 02:26:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0159.hostedemail.com [216.40.44.159]) by kanga.kvack.org (Postfix) with ESMTP id AC6CA6B0083 for ; Thu, 27 May 2021 02:26:15 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 4B1AD52AD for ; Thu, 27 May 2021 06:26:15 +0000 (UTC) X-FDA: 78186026310.27.29BBED4 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf25.hostedemail.com (Postfix) with ESMTP id 154A16000250 for ; Thu, 27 May 2021 06:26:06 +0000 (UTC) Received: by mail-pj1-f41.google.com with SMTP id h20-20020a17090aa894b029015db8f3969eso1757507pjq.3 for ; Wed, 26 May 2021 23:26:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Pnkdwc8KsT1GR9l5lzjPujWtU4Fy6ow0RMQ7lQzc3wo=; b=gLcNOB+p56/1OyW0KDnFNXJnhY1wq5sR/LUtKbPM+6AIV1MAhLgP0CbU0JWSE7RZS8 8trQWtZPBkUOu5Z9rExHocprXBJSQptaR17TjUhihZR/N8TeKodMIdEH6osF99IgiNSX sWYYGQPW8HyF/53eTiHoJC1r9YMik7tIh5LmQgUhgbOkJ24ucutk+CyX5oaA6bOAxcxK 2IYhWdHcInlaymo6ez7CvCJCsxLJQi51k2sDd7pf+57wHZ9QvNv9x1hos9L7X8sZzXKG Ugji1MktpwaEqvl9nredGmJiv9OFyYzJlQW/gs8ySASY0o3oE6Y4pYznYfg1fBXciJJG pGig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Pnkdwc8KsT1GR9l5lzjPujWtU4Fy6ow0RMQ7lQzc3wo=; b=ZJJxLSx66Fbrz9J6uRS3+LkGn7zKH0zuwtU4QwEKkOH6R+iPIPIdOB3JSQLgYY1ujp 7Tisob+w5M3lbMhVL/j0CT3vICdm2KLw8T6geTursUFnhcrs1GwSINdQj2lHVyRNGpm9 0bIFclIa76Z0J5b5sgiYvKtf1Ge8TPJLRHCcAGiLYzFkTuqE3o6Fp3lFGw6k4DsLC5dD tAC2Nps+PN+WgGhHj8hG7mK2USMohoP52gN488iovJD1Q1j+mTyIvLdGFirEb/CbQdil r8TUyGz1nvHBwNcpqH+g/mB3iEIYfYtQPyIyU4rdHL0gC3ncSb7c4FXQ/oklQVCoBwai JZSg== X-Gm-Message-State: AOAM530Wapan8hf9PLGKBXA4KgZFuvKZW/PGNkUlJCvyLrnmLkOgW32n f6xpNq3I+nfRn5WjQTCJu9IkEg== X-Google-Smtp-Source: ABdhPJxiIU2q4U//tQXOFSyBTMchzcY7zKsgW3oycyLxDKvIr+hJxeAKcfNys0Oh43G5MEPm7F+WHw== X-Received: by 2002:a17:90a:eb0f:: with SMTP id j15mr2009156pjz.149.1622096774257; Wed, 26 May 2021 23:26:14 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.26.06 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:26:13 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 16/21] mm: list_lru: rename memcg_drain_all_list_lrus to memcg_reparent_list_lrus Date: Thu, 27 May 2021 14:21:43 +0800 Message-Id: <20210527062148.9361-17-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=gLcNOB+p; spf=pass (imf25.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 154A16000250 X-Stat-Signature: ckxmp5mzjo43tffic6o8bex6rcjsattd X-HE-Tag: 1622096766-221687 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The purpose of the memcg_drain_all_list_lrus() is list_lrus reparenting. It is very similar to memcg_reparent_objcgs(). Rename it to memcg_reparent_list_lrus() so that the name can more consistent with memcg_reparent_objcgs(). Signed-off-by: Muchun Song --- include/linux/list_lru.h | 3 +-- mm/list_lru.c | 25 ++++++++++++------------- mm/memcontrol.c | 4 ++-- 3 files changed, 15 insertions(+), 17 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 00c15dc1ede3..60970ea48c29 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -76,8 +76,7 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, __list_lru_init((lru), true, NULL, shrinker) int memcg_update_all_list_lrus(int num_memcgs); -void memcg_drain_all_list_lrus(struct mem_cgroup *src_memcg, - struct mem_cgroup *dst_memcg); +void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent); /** * list_lru_add: add an element to the lru list's tail diff --git a/mm/list_lru.c b/mm/list_lru.c index 43eadb0834e1..dec116a12629 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -462,8 +462,8 @@ int memcg_update_all_list_lrus(int new_size) return ret; } -static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, - int src_idx, struct mem_cgroup *dst_memcg) +static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, + int src_idx, struct mem_cgroup *dst_memcg) { struct list_lru_node *nlru = &lru->node[nid]; int dst_idx = dst_memcg->kmemcg_id; @@ -491,23 +491,22 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, spin_unlock_irq(&nlru->lock); } -static void memcg_drain_list_lru(struct list_lru *lru, - int src_idx, struct mem_cgroup *dst_memcg) +static void memcg_reparent_list_lru(struct list_lru *lru, + int src_idx, struct mem_cgroup *dst_memcg) { int i; for_each_node(i) - memcg_drain_list_lru_node(lru, i, src_idx, dst_memcg); + memcg_reparent_list_lru_node(lru, i, src_idx, dst_memcg); memcg_list_lru_free(lru, src_idx); } -void memcg_drain_all_list_lrus(struct mem_cgroup *src_memcg, - struct mem_cgroup *dst_memcg) +void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent) { struct cgroup_subsys_state *css; struct list_lru *lru; - int src_idx = src_memcg->kmemcg_id; + int src_idx = memcg->kmemcg_id; /* * Change kmemcg_id of this cgroup and all its descendants to the @@ -522,17 +521,17 @@ void memcg_drain_all_list_lrus(struct mem_cgroup *src_memcg, * memcg_list_lru_free(). */ rcu_read_lock(); - css_for_each_descendant_pre(css, &src_memcg->css) { - struct mem_cgroup *memcg; + css_for_each_descendant_pre(css, &memcg->css) { + struct mem_cgroup *child; - memcg = mem_cgroup_from_css(css); - memcg->kmemcg_id = dst_memcg->kmemcg_id; + child = mem_cgroup_from_css(css); + child->kmemcg_id = parent->kmemcg_id; } rcu_read_unlock(); mutex_lock(&list_lrus_mutex); list_for_each_entry(lru, &list_lrus, list) - memcg_drain_list_lru(lru, src_idx, dst_memcg); + memcg_reparent_list_lru(lru, src_idx, parent); mutex_unlock(&list_lrus_mutex); } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1994513a33d4..205bf710486c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3655,12 +3655,12 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) memcg_reparent_objcgs(memcg, parent); /* - * memcg_drain_all_list_lrus() can change memcg->kmemcg_id. + * memcg_reparent_list_lrus() can change memcg->kmemcg_id. * Cache it to @kmemcg_id. */ kmemcg_id = memcg->kmemcg_id; - memcg_drain_all_list_lrus(memcg, parent); + memcg_reparent_list_lrus(memcg, parent); memcg_free_cache_id(kmemcg_id); } From patchwork Thu May 27 06:21:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283501 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1908DC4708A for ; Thu, 27 May 2021 06:26:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A788A610CC for ; Thu, 27 May 2021 06:26:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A788A610CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 22C256B0085; Thu, 27 May 2021 02:26:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D7E46B0087; Thu, 27 May 2021 02:26:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ADA688D0001; Thu, 27 May 2021 02:26:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0186.hostedemail.com [216.40.44.186]) by kanga.kvack.org (Postfix) with ESMTP id 5A62A6B0085 for ; Thu, 27 May 2021 02:26:23 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id F2907181AF5C6 for ; Thu, 27 May 2021 06:26:22 +0000 (UTC) X-FDA: 78186026604.24.FC98E96 Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) by imf05.hostedemail.com (Postfix) with ESMTP id 8292DE000805 for ; Thu, 27 May 2021 06:26:12 +0000 (UTC) Received: by mail-pg1-f182.google.com with SMTP id e22so2913419pgv.10 for ; Wed, 26 May 2021 23:26:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=euRdI3+gePz0YL3AmMLDmFSwJVE1/9POfNyNrwNdGts=; b=u5udodkivP1/r7m/y55ITUBRvgZ5nw+e01yxKI0hvDN+FAtUtZ1t9dCyPpr0cGKK0n PlUy5uGLHuOO7wDrKNzoDGBBQT4yiLttmkGTKSq6f2ubAR2oOhCczhLaEhzx8tBI3lX1 fXhtRt6U4VeqfTH+Rkbd5ipQ26rKTUjX4Dr56Om8YhMaSWAS+b5PpQptKJrSMUYeEvQe lFZJNoXpdw0NkgfTDMP8GoPPm0Go7Tbg5Q6tCCE/PQkigjDczcdSrLYjB9jVqV4oVdqz 0KBe5eB98dDswVYHkZ/6MkErV8tuuffBshnLcyABC1L5GJ2CWcG8WUmvsm9qvBmUgMbm jehQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=euRdI3+gePz0YL3AmMLDmFSwJVE1/9POfNyNrwNdGts=; b=H3u4WFJi5pt2hmXvtn0pHiQwsCsQyh3EXM868M/48mS4OgM+WHbEYg8CS2cmII769D 0wFk+TBN5D14Eo/8H2fhySt3UPdxpXt5CZ/JG2vingUGiDjzjVvsMaeTddH3UGBRno2F J8BcUk4fO5sOcghmZ0TUgNYwwvxsPKg/WtY/dRDBEKEaeEGNWPXoD12IZjVV6Lf9vV0c EkqIAipTFfOz46z5zUia6Vq4CbwnUNmW/duDppRlCSkw5/+HDlQRWJ9ompNuf9t/WyuI Dy+U2S/EB2s4pFe3l+RTCQOqlS8kaBpFZaIkOJiuIRZwiSODs7AQiVptbamk5ZK3/I9y KZ0Q== X-Gm-Message-State: AOAM531V8VGhI04jC+BtOD0ppTqM0gOes/1fYcBc5m7DRtOQRlt7ahbN 9G71eXj7PajzLrd10ZvTaaJNmQ== X-Google-Smtp-Source: ABdhPJyRYvAYd1RhR0zZeEllPWvhZpSW6Y8Srt/81mS8UrOongws4wuYEIDPpD+2ZeOzlZx8xBViEA== X-Received: by 2002:a62:3344:0:b029:28c:6f0f:cb90 with SMTP id z65-20020a6233440000b029028c6f0fcb90mr2245523pfz.58.1622096781592; Wed, 26 May 2021 23:26:21 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.26.14 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:26:21 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 17/21] mm: list_lru: replace linear array with xarray Date: Thu, 27 May 2021 14:21:44 +0800 Message-Id: <20210527062148.9361-18-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=u5udodki; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf05.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.182 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8292DE000805 X-Stat-Signature: bjcqiquddrqd6u1z9grfoupm8w4bj4m4 X-HE-Tag: 1622096772-244672 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we run 10k containers in the system, the size of the list_lru_memcg->lrus can be ~96KB per list_lru. When we decrease the number containers, the size of the array will not be shrinked. It is not scalable. The xarray is a good choice for this case. We can save a lot of memory when there are tens of thousands continers in the system. If we use xarray, we also can remove the logic code of resizing array, which can simplify the code. Signed-off-by: Muchun Song --- include/linux/list_lru.h | 12 +--- include/linux/memcontrol.h | 23 ------ mm/list_lru.c | 173 +++++++++++++++------------------------------ mm/memcontrol.c | 77 ++------------------ 4 files changed, 63 insertions(+), 222 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 60970ea48c29..d7c9bd29e836 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -11,6 +11,7 @@ #include #include #include +#include struct mem_cgroup; @@ -37,12 +38,6 @@ struct list_lru_per_memcg { struct list_lru_one nodes[]; }; -struct list_lru_memcg { - struct rcu_head rcu; - /* array of per cgroup lists, indexed by memcg_cache_id */ - struct list_lru_per_memcg __rcu *lrus[]; -}; - struct list_lru_node { /* protects all lists on the node, including per cgroup */ spinlock_t lock; @@ -56,10 +51,8 @@ struct list_lru { #ifdef CONFIG_MEMCG_KMEM struct list_head list; int shrinker_id; - /* protects ->memcg_lrus->lrus[i] */ - spinlock_t lock; /* for cgroup aware lrus points to per cgroup lists, otherwise NULL */ - struct list_lru_memcg __rcu *memcg_lrus; + struct xarray *xa; #endif }; @@ -75,7 +68,6 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, #define list_lru_init_memcg(lru, shrinker) \ __list_lru_init((lru), true, NULL, shrinker) -int memcg_update_all_list_lrus(int num_memcgs); void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent); /** diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 84a70c219e85..e129c7067f63 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1631,18 +1631,6 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size); extern struct static_key_false memcg_kmem_enabled_key; -extern int memcg_nr_cache_ids; -void memcg_get_cache_ids(void); -void memcg_put_cache_ids(void); - -/* - * Helper macro to loop through all memcg-specific caches. Callers must still - * check if the cache is valid (it is either valid or NULL). - * the slab_mutex must be held when looping through those caches - */ -#define for_each_memcg_cache_index(_idx) \ - for ((_idx) = 0; (_idx) < memcg_nr_cache_ids; (_idx)++) - static inline bool memcg_kmem_enabled(void) { return static_branch_likely(&memcg_kmem_enabled_key); @@ -1695,9 +1683,6 @@ static inline void __memcg_kmem_uncharge_page(struct page *page, int order) { } -#define for_each_memcg_cache_index(_idx) \ - for (; NULL; ) - static inline bool memcg_kmem_enabled(void) { return false; @@ -1708,14 +1693,6 @@ static inline int memcg_cache_id(struct mem_cgroup *memcg) return -1; } -static inline void memcg_get_cache_ids(void) -{ -} - -static inline void memcg_put_cache_ids(void) -{ -} - static inline struct mem_cgroup *mem_cgroup_from_obj(void *p) { return NULL; diff --git a/mm/list_lru.c b/mm/list_lru.c index dec116a12629..37052864bf78 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -20,7 +20,7 @@ static DEFINE_MUTEX(list_lrus_mutex); static inline bool list_lru_memcg_aware(struct list_lru *lru) { - return !!rcu_access_pointer(lru->memcg_lrus); + return !!lru->xa; } static void list_lru_register(struct list_lru *lru) @@ -51,22 +51,12 @@ static int lru_shrinker_id(struct list_lru *lru) static inline struct list_lru_one * list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) { - struct list_lru_memcg *memcg_lrus; - struct list_lru_node *nlru = &lru->node[nid]; + if (list_lru_memcg_aware(lru) && idx >= 0) { + struct list_lru_per_memcg *mlru = xa_load(lru->xa, idx); - /* - * Either lock or RCU protects the array of per cgroup lists - * from relocation (see memcg_update_list_lru). - */ - memcg_lrus = rcu_dereference_check(lru->memcg_lrus, - lockdep_is_held(&nlru->lock)); - if (memcg_lrus && idx >= 0) { - struct list_lru_per_memcg *mlru; - - mlru = rcu_dereference_check(memcg_lrus->lrus[idx], true); return mlru ? &mlru->nodes[nid] : NULL; } - return &nlru->lru; + return &lru->node[nid].lru; } static inline struct list_lru_one * @@ -310,16 +300,18 @@ unsigned long list_lru_walk_node(struct list_lru *lru, int nid, unsigned long *nr_to_walk) { long isolated = 0; - int memcg_idx; isolated += list_lru_walk_one(lru, nid, NULL, isolate, cb_arg, nr_to_walk); if (*nr_to_walk > 0 && list_lru_memcg_aware(lru)) { - for_each_memcg_cache_index(memcg_idx) { + struct list_lru_per_memcg *mlru; + unsigned long index; + + xa_for_each(lru->xa, index, mlru) { struct list_lru_node *nlru = &lru->node[nid]; spin_lock(&nlru->lock); - isolated += __list_lru_walk_one(lru, nid, memcg_idx, + isolated += __list_lru_walk_one(lru, nid, index, isolate, cb_arg, nr_to_walk); spin_unlock(&nlru->lock); @@ -339,15 +331,6 @@ static void init_one_lru(struct list_lru_one *l) } #ifdef CONFIG_MEMCG_KMEM -static void memcg_destroy_list_lru_range(struct list_lru_memcg *memcg_lrus, - int begin, int end) -{ - int i; - - for (i = begin; i < end; i++) - kfree(memcg_lrus->lrus[i]); -} - static struct list_lru_per_memcg *memcg_list_lru_alloc(gfp_t gfp) { int nid; @@ -365,15 +348,7 @@ static struct list_lru_per_memcg *memcg_list_lru_alloc(gfp_t gfp) static void memcg_list_lru_free(struct list_lru *lru, int src_idx) { - struct list_lru_memcg *memcg_lrus; - struct list_lru_per_memcg *mlru; - - spin_lock_irq(&lru->lock); - memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, true); - mlru = rcu_dereference_protected(memcg_lrus->lrus[src_idx], true); - if (mlru) - rcu_assign_pointer(memcg_lrus->lrus[src_idx], NULL); - spin_unlock_irq(&lru->lock); + struct list_lru_per_memcg *mlru = xa_erase_irq(lru->xa, src_idx); /* * The __list_lru_walk_one() can walk the list of this node. @@ -387,79 +362,35 @@ static void memcg_list_lru_free(struct list_lru *lru, int src_idx) static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) { - struct list_lru_memcg *memcg_lrus; - int size = memcg_nr_cache_ids; - if (!memcg_aware) { - lru->memcg_lrus = NULL; + lru->xa = NULL; return 0; } - spin_lock_init(&lru->lock); - - memcg_lrus = kvzalloc(sizeof(*memcg_lrus) + - size * sizeof(memcg_lrus->lrus[0]), GFP_KERNEL); - if (!memcg_lrus) + lru->xa = kmalloc(sizeof(*lru->xa), GFP_KERNEL); + if (!lru->xa) return -ENOMEM; - - RCU_INIT_POINTER(lru->memcg_lrus, memcg_lrus); + xa_init_flags(lru->xa, XA_FLAGS_LOCK_IRQ); return 0; } static void memcg_destroy_list_lru(struct list_lru *lru) { - struct list_lru_memcg *memcg_lrus; + XA_STATE(xas, lru->xa, 0); + struct list_lru_per_memcg *mlru; if (!list_lru_memcg_aware(lru)) return; - /* - * This is called when shrinker has already been unregistered, - * and nobody can use it. So, there is no need to use kvfree_rcu(). - */ - memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, true); - memcg_destroy_list_lru_range(memcg_lrus, 0, memcg_nr_cache_ids); - kvfree(memcg_lrus); -} - -static int memcg_update_list_lru(struct list_lru *lru, int old_size, int new_size) -{ - struct list_lru_memcg *old, *new; - - BUG_ON(old_size > new_size); - - old = rcu_dereference_protected(lru->memcg_lrus, - lockdep_is_held(&list_lrus_mutex)); - new = kvmalloc(sizeof(*new) + new_size * sizeof(new->lrus[0]), GFP_KERNEL); - if (!new) - return -ENOMEM; - - spin_lock_irq(&lru->lock); - memcpy(&new->lrus, &old->lrus, old_size * sizeof(new->lrus[0])); - memset(&new->lrus[old_size], 0, (new_size - old_size) * sizeof(new->lrus[0])); - rcu_assign_pointer(lru->memcg_lrus, new); - spin_unlock_irq(&lru->lock); - - kvfree_rcu(old, rcu); - - return 0; -} - -int memcg_update_all_list_lrus(int new_size) -{ - int ret = 0; - struct list_lru *lru; - int old_size = memcg_nr_cache_ids; - - mutex_lock(&list_lrus_mutex); - list_for_each_entry(lru, &list_lrus, list) { - ret = memcg_update_list_lru(lru, old_size, new_size); - if (ret) - break; + xas_lock_irq(&xas); + xas_for_each(&xas, mlru, ULONG_MAX) { + kfree(mlru); + xas_store(&xas, NULL); } - mutex_unlock(&list_lrus_mutex); - return ret; + xas_unlock_irq(&xas); + + kfree(lru->xa); } static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, @@ -538,28 +469,18 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren static bool memcg_list_lru_skip_alloc(struct list_lru *lru, struct mem_cgroup *memcg) { - struct list_lru_memcg *memcg_lrus; int idx = memcg_cache_id(memcg); - if (unlikely(idx < 0)) + if (unlikely(idx < 0) || xa_load(lru->xa, idx)) return true; - - rcu_read_lock(); - memcg_lrus = rcu_dereference(lru->memcg_lrus); - if (rcu_access_pointer(memcg_lrus->lrus[idx])) { - rcu_read_unlock(); - return true; - } - rcu_read_unlock(); - return false; } int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t gfp) { + XA_STATE(xas, lru->xa, 0); unsigned long flags; - struct list_lru_memcg *memcg_lrus; - int i; + int i, ret = 0; struct list_lru_memcg_table { struct list_lru_per_memcg *mlru; @@ -601,22 +522,45 @@ int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t g } } - spin_lock_irqsave(&lru->lock, flags); - memcg_lrus = rcu_dereference_protected(lru->memcg_lrus, true); + xas_lock_irqsave(&xas, flags); while (i--) { int index = memcg_cache_id(table[i].memcg); struct list_lru_per_memcg *mlru = table[i].mlru; - if (index < 0 || rcu_dereference_protected(memcg_lrus->lrus[index], true)) + xas_set(&xas, index); +retry: + if (unlikely(index < 0 || ret || xas_load(&xas))) { kfree(mlru); - else - rcu_assign_pointer(memcg_lrus->lrus[index], mlru); + } else { + ret = xa_err(xas_store(&xas, mlru)); + if (ret == -ENOMEM) { + xas_unlock_irqrestore(&xas, flags); + if (xas_nomem(&xas, gfp)) + ret = 0; + xas_lock_irqsave(&xas, flags); + /* + * The xas lock has been released, this memcg + * can be reparented before us. So reload + * memcg id. More details see the comments + * in memcg_reparent_list_lrus(). + */ + index = memcg_cache_id(table[i].memcg); + if (index < 0) + ret = 0; + else if (!ret && index != xas.xa_index) + xas_set(&xas, index); + goto retry; + } + } } - spin_unlock_irqrestore(&lru->lock, flags); + /* xas_nomem() is used to free memory instead of memory allocation. */ + if (xas.xa_alloc) + xas_nomem(&xas, gfp); + xas_unlock_irqrestore(&xas, flags); kfree(table); - return 0; + return ret; } #else static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) @@ -641,7 +585,6 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, else lru->shrinker_id = -1; #endif - memcg_get_cache_ids(); lru->node = kcalloc(nr_node_ids, sizeof(*lru->node), GFP_KERNEL); if (!lru->node) @@ -664,7 +607,6 @@ int __list_lru_init(struct list_lru *lru, bool memcg_aware, list_lru_register(lru); out: - memcg_put_cache_ids(); return err; } EXPORT_SYMBOL_GPL(__list_lru_init); @@ -675,8 +617,6 @@ void list_lru_destroy(struct list_lru *lru) if (!lru->node) return; - memcg_get_cache_ids(); - list_lru_unregister(lru); memcg_destroy_list_lru(lru); @@ -686,6 +626,5 @@ void list_lru_destroy(struct list_lru *lru) #ifdef CONFIG_MEMCG_KMEM lru->shrinker_id = -1; #endif - memcg_put_cache_ids(); } EXPORT_SYMBOL_GPL(list_lru_destroy); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 205bf710486c..82b147597138 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -345,42 +345,17 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg, * This will be used as a shrinker list's index. * The main reason for not using cgroup id for this: * this works better in sparse environments, where we have a lot of memcgs, - * but only a few kmem-limited. Or also, if we have, for instance, 200 - * memcgs, and none but the 200th is kmem-limited, we'd have to have a - * 200 entry array for that. - * - * The current size of the caches array is stored in memcg_nr_cache_ids. It - * will double each time we have to increase it. + * but only a few kmem-limited. */ static DEFINE_IDA(memcg_cache_ida); -int memcg_nr_cache_ids; - -/* Protects memcg_nr_cache_ids */ -static DECLARE_RWSEM(memcg_cache_ids_sem); - -void memcg_get_cache_ids(void) -{ - down_read(&memcg_cache_ids_sem); -} - -void memcg_put_cache_ids(void) -{ - up_read(&memcg_cache_ids_sem); -} /* - * MIN_SIZE is different than 1, because we would like to avoid going through - * the alloc/free process all the time. In a small machine, 4 kmem-limited - * cgroups is a reasonable guess. In the future, it could be a parameter or - * tunable, but that is strictly not necessary. - * * MAX_SIZE should be as large as the number of cgrp_ids. Ideally, we could get * this constant directly from cgroup, but it is understandable that this is * better kept as an internal representation in cgroup.c. In any case, the * cgrp_id space is not getting any smaller, and we don't have to necessarily * increase ours as well if it increases. */ -#define MEMCG_CACHES_MIN_SIZE 4 #define MEMCG_CACHES_MAX_SIZE MEM_CGROUP_ID_MAX /* @@ -2903,49 +2878,6 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) return objcg; } -static int memcg_alloc_cache_id(void) -{ - int id, size; - int err; - - id = ida_simple_get(&memcg_cache_ida, - 0, MEMCG_CACHES_MAX_SIZE, GFP_KERNEL); - if (id < 0) - return id; - - if (id < memcg_nr_cache_ids) - return id; - - /* - * There's no space for the new id in memcg_caches arrays, - * so we have to grow them. - */ - down_write(&memcg_cache_ids_sem); - - size = 2 * (id + 1); - if (size < MEMCG_CACHES_MIN_SIZE) - size = MEMCG_CACHES_MIN_SIZE; - else if (size > MEMCG_CACHES_MAX_SIZE) - size = MEMCG_CACHES_MAX_SIZE; - - err = memcg_update_all_list_lrus(size); - if (!err) - memcg_nr_cache_ids = size; - - up_write(&memcg_cache_ids_sem); - - if (err) { - ida_simple_remove(&memcg_cache_ida, id); - return err; - } - return id; -} - -static void memcg_free_cache_id(int id) -{ - ida_simple_remove(&memcg_cache_ida, id); -} - /* * obj_cgroup_uncharge_pages: uncharge a number of kernel pages from a objcg * @objcg: object cgroup to uncharge @@ -3618,13 +3550,14 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) if (unlikely(mem_cgroup_is_root(memcg))) return 0; - memcg_id = memcg_alloc_cache_id(); + memcg_id = ida_simple_get(&memcg_cache_ida, 0, MEMCG_CACHES_MAX_SIZE, + GFP_KERNEL); if (memcg_id < 0) return memcg_id; objcg = obj_cgroup_alloc(); if (!objcg) { - memcg_free_cache_id(memcg_id); + ida_simple_remove(&memcg_cache_ida, memcg_id); return -ENOMEM; } objcg->memcg = memcg; @@ -3662,7 +3595,7 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) memcg_reparent_list_lrus(memcg, parent); - memcg_free_cache_id(kmemcg_id); + ida_simple_remove(&memcg_cache_ida, kmemcg_id); } #else static int memcg_online_kmem(struct mem_cgroup *memcg) From patchwork Thu May 27 06:21:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECE5DC4708B for ; Thu, 27 May 2021 06:26:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 925746135F for ; Thu, 27 May 2021 06:26:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 925746135F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 379636B007B; Thu, 27 May 2021 02:26:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3519D6B0087; Thu, 27 May 2021 02:26:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F1DC8D0001; Thu, 27 May 2021 02:26:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0243.hostedemail.com [216.40.44.243]) by kanga.kvack.org (Postfix) with ESMTP id E28806B007B for ; Thu, 27 May 2021 02:26:30 -0400 (EDT) Received: from smtpin32.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 34F49815E for ; Thu, 27 May 2021 06:26:30 +0000 (UTC) X-FDA: 78186026940.32.CA6541B Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by imf09.hostedemail.com (Postfix) with ESMTP id 467FC600024C for ; Thu, 27 May 2021 06:26:25 +0000 (UTC) Received: by mail-pg1-f170.google.com with SMTP id f22so2919386pgb.9 for ; Wed, 26 May 2021 23:26:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IDeG++/blhLntk2pJHHgm+mSW8cVt7IZBOEblp/zC60=; b=DtpEH0I9Yks3SIBK6zIWD8R8VgT9fGU33bCNhHyJmktNP98gHzu6h5t1RIT/meam5f awAT2WP+76Z51QkQ60dtOvamBUPPUXfM7qdhq15wSMBcEDBXnUhMqIOPyw73jnG2RYBB aCIRzaW/jpP+oXABiXaym8ZbosFDdfpKQwlrOCOJ6OjqFW0pZpufqP59sJuI0dV5/5IU aP3QoANGK74pYhNxTnZNbTKuT9o1FUosMaEJiCF6tg9c1bXjVR+v6bBNZpgvu1qzAIdu WpnQvxRgE6K5THPZVmMwQ04qmWOaphT0yyP6G83/Af5J6y4eGMr/xB7qUiozd2IdKRvV OKug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IDeG++/blhLntk2pJHHgm+mSW8cVt7IZBOEblp/zC60=; b=Pfyuh3WEK7QOheLf6U1GAR2j0I/IUR5U8tyqqTGGqtycLlJ9141mHtnEzSenv+oc75 kaWSF6xZm7AnSeZlpx+Jqc88JNKrUTqjGkcvZhbNm/O7AmXZMEJYovOPke9IliFZyeRB D60v2kWpKfB2QycHm80Q3H1DFVXS1En7Yt/Ph9Vi4n5oJFKwbJ4Re8LyYCqjrs3qjWmO u3wp8oGzwq0uhR88n5B7NYlF/eAKNRbQNtfP/P4fgyuwkdp215ebenmCC+QOUVOflejP +u+w601oobSeZkbdu5l9+LTNoS9+ntbxQbmdL4kkD1BIFwXmARYWOjyXO8nKGJ6+tw16 JkFw== X-Gm-Message-State: AOAM53341zSz4szOjwzpPUs4VkNr1j7lBXVyQmfSvtICQKW52MBQQ++N sxEziVNi2HKZnn3kyoTQ4iuZxQ== X-Google-Smtp-Source: ABdhPJy6LpMfHvxJKRmRoM7o+N0+95VtxvCbt414eASgexHfTHjbHGFW4pU6bsnvfRmroLKxBTlUXg== X-Received: by 2002:a65:64cc:: with SMTP id t12mr2334997pgv.64.1622096789011; Wed, 26 May 2021 23:26:29 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.26.21 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:26:28 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 18/21] mm: memcontrol: reuse memory cgroup ID for kmem ID Date: Thu, 27 May 2021 14:21:45 +0800 Message-Id: <20210527062148.9361-19-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=DtpEH0I9; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf09.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 467FC600024C X-Stat-Signature: 46nz3b1h3cdcrxa8cxa38yscug3cuop9 X-HE-Tag: 1622096785-125818 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There are two idrs being used by memory cgroup, one is for kmem ID, another is for memory cgroup ID. The maximum ID of both is 64Ki. Both of them can limit the total number of memory cgroups. Actually, we can reuse memory cgroup ID for kmem ID to simplify the code. Signed-off-by: Muchun Song --- mm/memcontrol.c | 40 +++------------------------------------- 1 file changed, 3 insertions(+), 37 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 82b147597138..ae3ad1001824 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -342,23 +342,6 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg, } /* - * This will be used as a shrinker list's index. - * The main reason for not using cgroup id for this: - * this works better in sparse environments, where we have a lot of memcgs, - * but only a few kmem-limited. - */ -static DEFINE_IDA(memcg_cache_ida); - -/* - * MAX_SIZE should be as large as the number of cgrp_ids. Ideally, we could get - * this constant directly from cgroup, but it is understandable that this is - * better kept as an internal representation in cgroup.c. In any case, the - * cgrp_id space is not getting any smaller, and we don't have to necessarily - * increase ours as well if it increases. - */ -#define MEMCG_CACHES_MAX_SIZE MEM_CGROUP_ID_MAX - -/* * A lot of the calls to the cache allocation functions are expected to be * inlined by the compiler. Since the calls to memcg_slab_pre_alloc_hook() are * conditional to this static branch, we'll have to allow modules that does @@ -3542,7 +3525,6 @@ static u64 mem_cgroup_read_u64(struct cgroup_subsys_state *css, static int memcg_online_kmem(struct mem_cgroup *memcg) { struct obj_cgroup *objcg; - int memcg_id; if (cgroup_memory_nokmem) return 0; @@ -3550,22 +3532,16 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) if (unlikely(mem_cgroup_is_root(memcg))) return 0; - memcg_id = ida_simple_get(&memcg_cache_ida, 0, MEMCG_CACHES_MAX_SIZE, - GFP_KERNEL); - if (memcg_id < 0) - return memcg_id; - objcg = obj_cgroup_alloc(); - if (!objcg) { - ida_simple_remove(&memcg_cache_ida, memcg_id); + if (!objcg) return -ENOMEM; - } + objcg->memcg = memcg; rcu_assign_pointer(memcg->objcg, objcg); static_branch_enable(&memcg_kmem_enabled_key); - memcg->kmemcg_id = memcg_id; + memcg->kmemcg_id = memcg->id.id; return 0; } @@ -3573,7 +3549,6 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) static void memcg_offline_kmem(struct mem_cgroup *memcg) { struct mem_cgroup *parent; - int kmemcg_id; if (cgroup_memory_nokmem) return; @@ -3586,16 +3561,7 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) parent = root_mem_cgroup; memcg_reparent_objcgs(memcg, parent); - - /* - * memcg_reparent_list_lrus() can change memcg->kmemcg_id. - * Cache it to @kmemcg_id. - */ - kmemcg_id = memcg->kmemcg_id; - memcg_reparent_list_lrus(memcg, parent); - - ida_simple_remove(&memcg_cache_ida, kmemcg_id); } #else static int memcg_online_kmem(struct mem_cgroup *memcg) From patchwork Thu May 27 06:21:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283505 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92C5CC47089 for ; Thu, 27 May 2021 06:26:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 40E606135F for ; Thu, 27 May 2021 06:26:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 40E606135F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CF0556B0075; Thu, 27 May 2021 02:26:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CC6936B0087; Thu, 27 May 2021 02:26:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B68108D0001; Thu, 27 May 2021 02:26:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0089.hostedemail.com [216.40.44.89]) by kanga.kvack.org (Postfix) with ESMTP id 845D96B0075 for ; Thu, 27 May 2021 02:26:38 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6BF31689A for ; Thu, 27 May 2021 06:26:37 +0000 (UTC) X-FDA: 78186027234.27.9DCB934 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf08.hostedemail.com (Postfix) with ESMTP id A25AD801912B for ; Thu, 27 May 2021 06:26:29 +0000 (UTC) Received: by mail-pj1-f47.google.com with SMTP id h20-20020a17090aa894b029015db8f3969eso1757950pjq.3 for ; Wed, 26 May 2021 23:26:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UxSxre2wDGLXpYpNahDKZAu3VAo2pfHZzOxhvD0wk18=; b=ZKubg9WZzxZUy30dowed69brRlvqEItapcPwLPgZswfpMdJ3cT/WDnRIkB4sOtsBvJ RF7PS1NNnnnkDwfJ3P+IMjR8tD0QZW8/88ZGllOgEBMnS1fbZYKJHaulTBoU99agPnhZ b7VcpA0nZy7pOhH05kUxqa4Rmfkw/fB2ALHIUiAxdfA17MOhOPuYft4X6x8pVDer+vG4 ojkmOtmZTMf9QhdmT0rqC8Xtx0DA1msBALyhiqnWNwMWBSYkuIsziIMZ80EwhJkKlu6T mAHy106uSOoAlzLgtccHS2Nfo2XjJSHM4jMD4f+S9fyOqzocjIqw4YphBEEcJT8RrEJu cyKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UxSxre2wDGLXpYpNahDKZAu3VAo2pfHZzOxhvD0wk18=; b=VAb/rXZQuoCHmdDj9dUhrR69rOA9DNvOQoDwBV9CCrbZxYjcK/y1Uh+Z2t+W3cV8P1 LMHHSbGouKh5et/pi50qrPP6Zo644LnwO3qafM0tFhZrLVgJE6JFOw9Z0UZFrQY6eagx 7Q7I9pCAHSsdoHArFUldM7hDqXrp/yefbaFve7BdfOJ4P7+coLsjNVAmmVAP5gGAKOal APa78z+8ztHsHmTR5u+NFTXBKh4biyyjiZYM7k7z38Zqd7gOevvIDCAVJIg5Ly6CZ1LI IdyimhAfajEavwiTcPpZJPL9ei2sqSS9x/MFVs5zlHwZTVrlJKpGYrMRTgmGFdKa717v BvVQ== X-Gm-Message-State: AOAM533FtwcMOJw3Zxz70NUksL3AfSp7LuqUaULLetmmqv9tdw/FQxHR T8AV22NKrbBQlf60G9Ji/R6+Yg== X-Google-Smtp-Source: ABdhPJyLAUuIAkmyYhUp7SVzVwB+y2sVQYdILO46NBcSQ91Ugrj375USQf1NBRX88v2oz8WRagoiDA== X-Received: by 2002:a17:90a:4a89:: with SMTP id f9mr7889513pjh.50.1622096796281; Wed, 26 May 2021 23:26:36 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.26.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:26:35 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 19/21] mm: memcontrol: fix cannot alloc the maximum memcg ID Date: Thu, 27 May 2021 14:21:46 +0800 Message-Id: <20210527062148.9361-20-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=ZKubg9WZ; spf=pass (imf08.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: A25AD801912B X-Stat-Signature: s6wz3oekrxdz83u34ijpc4oy3qbjb3rg X-HE-Tag: 1622096789-767846 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The idr_alloc() does not include @max ID. So in the current implementation, the maximum memcg ID is 65534 instead of 65535. It seems a bug. So fix this. Signed-off-by: Muchun Song --- mm/memcontrol.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ae3ad1001824..a1c8ec858593 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5044,7 +5044,7 @@ static struct mem_cgroup *mem_cgroup_alloc(void) return ERR_PTR(error); memcg->id.id = idr_alloc(&mem_cgroup_idr, NULL, - 1, MEM_CGROUP_ID_MAX, + 1, MEM_CGROUP_ID_MAX + 1, GFP_KERNEL); if (memcg->id.id < 0) { error = memcg->id.id; From patchwork Thu May 27 06:21:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85C5EC4708B for ; Thu, 27 May 2021 06:26:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 33F8A613F0 for ; Thu, 27 May 2021 06:26:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 33F8A613F0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CC8526B0087; Thu, 27 May 2021 02:26:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C78428D0001; Thu, 27 May 2021 02:26:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA3E86B0089; Thu, 27 May 2021 02:26:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0038.hostedemail.com [216.40.44.38]) by kanga.kvack.org (Postfix) with ESMTP id 6FC3A6B0087 for ; Thu, 27 May 2021 02:26:44 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 196728249980 for ; Thu, 27 May 2021 06:26:44 +0000 (UTC) X-FDA: 78186027528.36.9B02A96 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf20.hostedemail.com (Postfix) with ESMTP id 03F282C0 for ; Thu, 27 May 2021 06:26:33 +0000 (UTC) Received: by mail-pj1-f53.google.com with SMTP id q6so2154486pjj.2 for ; Wed, 26 May 2021 23:26:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=93dDqyeWkKnQFRCldXYX+fDzer/GYrGyPOg9jpoyqVw=; b=Nk9bXw59X2fysUcGQbiJMRbXMwzRloCfTMmf3h085vLvhUOQeTEj2ksEGX73TEk4bK Kz08lppd8dGTEaRfD8bBIiUOOK66s4TnKXwycowV3NGBegFdDfHQrPGDzdG+Coqgbgwq lRd+c+DWMv6Y8nEwvBB+32DzLORgDxngetMIIjI0HnbSpPzlVs1plFXdOKSCZj/wvx+h xXNth7QnMMfQgMkd3Te6l1EaPrWTDQmxAQn90dqYPGiocbYkCkS9rco3A3K+7iPQd/Eo 1s9b+z8z7cY/PdTna6AhelSJ9qtnpNHvhvFJD+L9PQxlrFFzKzoz0ICM0wEQ3D+Uk2dM yc9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=93dDqyeWkKnQFRCldXYX+fDzer/GYrGyPOg9jpoyqVw=; b=kehLmdvdxef4tBFUKm7CRim0N/NLCSauNPteqcq+hT+JDZtbRhPT+QnKvIs1SZ3MVd eNqWNxIK81yzChj5W9dwY7NT2KTfouYSgzGcJksE8aRX4F2H06RDNZuai+9+iigN8R/S 9S8M3YAh0buFrLADzQExg18ZRXS5UMGlaI3vojq154YS3b/FfgQds8CMcwTo00z1KoX0 bnk+KJomZLbDfuFdWddbqzTdGqFPhEcUZLZJ9xnZWFIOzH7XBp8vViHAwCj8t+oeFawK Na9ZA1qOos1UuIv2FgPsFWdAHq5nWF6InwZr5oEeMxCguhd5VaLS6OK1EZO4H2epYPsw 9fLw== X-Gm-Message-State: AOAM5324hg7ZBn5DIIdNvIM+DiZPL0VwtfbFRvz/rOYCGY7z6UZQTAtQ 13v06hvQrUhmkGyzpwuaS9XeSQ== X-Google-Smtp-Source: ABdhPJw5QU0YZIs70xkXlCIuCRFp/pljdG9zXUyMmIfOzkQiMS+SEzkpZQ4ufkNhu7bF4YXppazbQQ== X-Received: by 2002:a17:903:1cd:b029:f0:c1c2:9e75 with SMTP id e13-20020a17090301cdb02900f0c1c29e75mr1808657plh.54.1622096802991; Wed, 26 May 2021 23:26:42 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.26.36 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:26:42 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 20/21] mm: list_lru: rename list_lru_per_memcg to list_lru_memcg Date: Thu, 27 May 2021 14:21:47 +0800 Message-Id: <20210527062148.9361-21-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 03F282C0 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=Nk9bXw59; spf=pass (imf20.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam04 X-Stat-Signature: t7wkeuxu747g6kzzyk8ab4cdsfsrhxrz X-HE-Tag: 1622096793-948363 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Before now, the name of list_lru_memcg was occupied. Since previous patch, the name is free. So rename list_lru_per_memcg to list_lru_memcg, it is more brief. Signed-off-by: Muchun Song --- include/linux/list_lru.h | 2 +- mm/list_lru.c | 20 ++++++++++---------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index d7c9bd29e836..7497719ec71c 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -32,7 +32,7 @@ struct list_lru_one { long nr_items; }; -struct list_lru_per_memcg { +struct list_lru_memcg { struct rcu_head rcu; /* array of per cgroup per node lists, indexed by node id */ struct list_lru_one nodes[]; diff --git a/mm/list_lru.c b/mm/list_lru.c index 37052864bf78..77efdd0c8b24 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -52,7 +52,7 @@ static inline struct list_lru_one * list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) { if (list_lru_memcg_aware(lru) && idx >= 0) { - struct list_lru_per_memcg *mlru = xa_load(lru->xa, idx); + struct list_lru_memcg *mlru = xa_load(lru->xa, idx); return mlru ? &mlru->nodes[nid] : NULL; } @@ -304,7 +304,7 @@ unsigned long list_lru_walk_node(struct list_lru *lru, int nid, isolated += list_lru_walk_one(lru, nid, NULL, isolate, cb_arg, nr_to_walk); if (*nr_to_walk > 0 && list_lru_memcg_aware(lru)) { - struct list_lru_per_memcg *mlru; + struct list_lru_memcg *mlru; unsigned long index; xa_for_each(lru->xa, index, mlru) { @@ -331,10 +331,10 @@ static void init_one_lru(struct list_lru_one *l) } #ifdef CONFIG_MEMCG_KMEM -static struct list_lru_per_memcg *memcg_list_lru_alloc(gfp_t gfp) +static struct list_lru_memcg *memcg_list_lru_alloc(gfp_t gfp) { int nid; - struct list_lru_per_memcg *lru; + struct list_lru_memcg *lru; lru = kmalloc(struct_size(lru, nodes, nr_node_ids), gfp); if (!lru) @@ -348,7 +348,7 @@ static struct list_lru_per_memcg *memcg_list_lru_alloc(gfp_t gfp) static void memcg_list_lru_free(struct list_lru *lru, int src_idx) { - struct list_lru_per_memcg *mlru = xa_erase_irq(lru->xa, src_idx); + struct list_lru_memcg *mlru = xa_erase_irq(lru->xa, src_idx); /* * The __list_lru_walk_one() can walk the list of this node. @@ -378,7 +378,7 @@ static int memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) static void memcg_destroy_list_lru(struct list_lru *lru) { XA_STATE(xas, lru->xa, 0); - struct list_lru_per_memcg *mlru; + struct list_lru_memcg *mlru; if (!list_lru_memcg_aware(lru)) return; @@ -483,7 +483,7 @@ int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t g int i, ret = 0; struct list_lru_memcg_table { - struct list_lru_per_memcg *mlru; + struct list_lru_memcg *mlru; struct mem_cgroup *memcg; } *table; @@ -494,7 +494,7 @@ int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t g return 0; /* - * The allocated list_lru_per_memcg array is not accounted directly. + * The allocated list_lru_memcg array is not accounted directly. * Moreover, it should not come from DMA buffer and is not readily * reclaimable. So those GFP bits should be masked off. */ @@ -506,7 +506,7 @@ int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t g /* * Because the list_lru can be reparented to the parent cgroup's * list_lru, we should make sure that this cgroup and all its - * ancestors have allocated list_lru_per_memcg. + * ancestors have allocated list_lru_memcg. */ for (i = 0; memcg; memcg = parent_mem_cgroup(memcg), i++) { if (memcg_list_lru_skip_alloc(lru, memcg)) @@ -525,7 +525,7 @@ int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t g xas_lock_irqsave(&xas, flags); while (i--) { int index = memcg_cache_id(table[i].memcg); - struct list_lru_per_memcg *mlru = table[i].mlru; + struct list_lru_memcg *mlru = table[i].mlru; xas_set(&xas, index); retry: From patchwork Thu May 27 06:21:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12283509 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD33CC4708C for ; Thu, 27 May 2021 06:26:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 76444610A6 for ; Thu, 27 May 2021 06:26:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 76444610A6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1F27B6B0080; Thu, 27 May 2021 02:26:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A3368D0001; Thu, 27 May 2021 02:26:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F10956B0089; Thu, 27 May 2021 02:26:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0041.hostedemail.com [216.40.44.41]) by kanga.kvack.org (Postfix) with ESMTP id B953E6B0080 for ; Thu, 27 May 2021 02:26:52 -0400 (EDT) Received: from smtpin37.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6C4DBA2BE for ; Thu, 27 May 2021 06:26:52 +0000 (UTC) X-FDA: 78186027864.37.F83E1BA Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf13.hostedemail.com (Postfix) with ESMTP id 60E6AE00080F for ; Thu, 27 May 2021 06:26:44 +0000 (UTC) Received: by mail-pf1-f181.google.com with SMTP id g18so2875275pfr.2 for ; Wed, 26 May 2021 23:26:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=saeNxjM08r5E+Sxs4m/QIhTTsKS/EbXdDHTVWv65coY=; b=e477haKznXStV5QieRFLlp0oO6Mf7tJUPQD/akahq0QEMUAs9WhgOa8q4TAUL0mNbp tCbPP9HrJFaOCgJMFNXRdTj2N+slXNojfhxZ9+KsGhkbA57CcSwqoF9fSxEYlm3NpMtf b2KAO/RQjm/XUHC8jhaCDihIBYEGU57r/yVkCp3Pbri6GKVGfn0avw1etj1Ek1htHp3M pc0wYQtUG6MUlJwFlS+nKjMeE14vx21zzCA1R8wRZQkSvAEkyjpFYU2fotx8UefC8WSG JXj21Ts/X5GhBs0I032lldbxlfC2xKl/4YpZUHk10bi1NYIHI8k/9fcmyigK4tjnhkyT 23Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=saeNxjM08r5E+Sxs4m/QIhTTsKS/EbXdDHTVWv65coY=; b=D6IFA4kTRdazXrsYACSJdsLvwUM6xiVDb16KsMbzuAR2mIJL3Cu6OZE5inFGBVnEnB F2RsS+6VOHkAqdzCfhmS+JN+48xsTAm/uYEY+pDfqBsvHWfE5OK7TWYHZs4UlVGEgLFo 4KmQAE7XIGRgtaZfqlxcvLlSZsaB3Ts7xeJEdDDPSsd+uF9GfiWTnEvGPC9yscugWxm/ pJSDRjZ2TlHFuQkrj9RFW3DNEnvNFxUjAkTw8U4fdCi3ZiI+TD7IM870POed/V/9qGi8 N1DSFf+jg6MEchcpAt6uIuhxVK2OiOxM9nT9vacAEJbkv60M97WceG3azqBzsoe3gKxZ f5pg== X-Gm-Message-State: AOAM533OYyx6cIZ2Z9l3DSBznmgFoRpZpLxBDWebCOtI3yy4nEM2K2dY TRm3eKZnFRdFrga2SdAx/VWMWw== X-Google-Smtp-Source: ABdhPJyqz+2+MPfbYrdMkC+0d1NyNM7mJV+9i/WYx14HfOOaqe+ZuDWugDDPw7OgbcsrX7OPn62MqQ== X-Received: by 2002:a63:752:: with SMTP id 79mr2314672pgh.10.1622096811343; Wed, 26 May 2021 23:26:51 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id m5sm882971pgl.75.2021.05.26.23.26.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 May 2021 23:26:51 -0700 (PDT) From: Muchun Song To: willy@infradead.org, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, shakeelb@google.com, guro@fb.com, shy828301@gmail.com, alexs@kernel.org, richard.weiyang@gmail.com, david@fromorbit.com, trond.myklebust@hammerspace.com, anna.schumaker@netapp.com Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, zhengqi.arch@bytedance.com, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, Muchun Song Subject: [PATCH v2 21/21] mm: memcontrol: rename memcg_cache_id to memcg_kmem_id Date: Thu, 27 May 2021 14:21:48 +0800 Message-Id: <20210527062148.9361-22-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210527062148.9361-1-songmuchun@bytedance.com> References: <20210527062148.9361-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=e477haKz; spf=pass (imf13.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 60E6AE00080F X-Stat-Signature: aymqme6r6j4ud7k3p5i4qs9eu3dubaz6 X-HE-Tag: 1622096804-842801 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The memcg_cache_id is introduced by commit 2633d7a02823 ("slab/slub: consider a memcg parameter in kmem_create_cache"). It is used to index in the kmem_cache->memcg_params->memcg_caches array. Since kmem_cache->memcg_params.memcg_caches has been removed by commit 9855609bde03 ("mm: memcg/slab: use a single set of kmem_caches for all accounted allocations"). So the name does not need to reflect cache related. Just rename it to memcg_kmem_id. And it can reflect kmem related. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 4 ++-- mm/list_lru.c | 14 +++++++------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e129c7067f63..637d5854fdda 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1654,7 +1654,7 @@ static inline void memcg_kmem_uncharge_page(struct page *page, int order) * A helper for accessing memcg's kmem_id, used for getting * corresponding LRU lists. */ -static inline int memcg_cache_id(struct mem_cgroup *memcg) +static inline int memcg_kmem_id(struct mem_cgroup *memcg) { return memcg ? memcg->kmemcg_id : -1; } @@ -1688,7 +1688,7 @@ static inline bool memcg_kmem_enabled(void) return false; } -static inline int memcg_cache_id(struct mem_cgroup *memcg) +static inline int memcg_kmem_id(struct mem_cgroup *memcg) { return -1; } diff --git a/mm/list_lru.c b/mm/list_lru.c index 77efdd0c8b24..1ee2b28ded7d 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -74,7 +74,7 @@ list_lru_from_kmem(struct list_lru *lru, int nid, void *ptr, if (!memcg) goto out; - l = list_lru_from_memcg_idx(lru, nid, memcg_cache_id(memcg)); + l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); out: if (memcg_ptr) *memcg_ptr = memcg; @@ -181,7 +181,7 @@ unsigned long list_lru_count_one(struct list_lru *lru, long count = 0; rcu_read_lock(); - l = list_lru_from_memcg_idx(lru, nid, memcg_cache_id(memcg)); + l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); if (l) count = READ_ONCE(l->nr_items); rcu_read_unlock(); @@ -273,7 +273,7 @@ list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg, unsigned long ret; spin_lock(&nlru->lock); - ret = __list_lru_walk_one(lru, nid, memcg_cache_id(memcg), isolate, + ret = __list_lru_walk_one(lru, nid, memcg_kmem_id(memcg), isolate, cb_arg, nr_to_walk); spin_unlock(&nlru->lock); return ret; @@ -289,7 +289,7 @@ list_lru_walk_one_irq(struct list_lru *lru, int nid, struct mem_cgroup *memcg, unsigned long ret; spin_lock_irq(&nlru->lock); - ret = __list_lru_walk_one(lru, nid, memcg_cache_id(memcg), isolate, + ret = __list_lru_walk_one(lru, nid, memcg_kmem_id(memcg), isolate, cb_arg, nr_to_walk); spin_unlock_irq(&nlru->lock); return ret; @@ -469,7 +469,7 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren static bool memcg_list_lru_skip_alloc(struct list_lru *lru, struct mem_cgroup *memcg) { - int idx = memcg_cache_id(memcg); + int idx = memcg_kmem_id(memcg); if (unlikely(idx < 0) || xa_load(lru->xa, idx)) return true; @@ -524,7 +524,7 @@ int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t g xas_lock_irqsave(&xas, flags); while (i--) { - int index = memcg_cache_id(table[i].memcg); + int index = memcg_kmem_id(table[i].memcg); struct list_lru_memcg *mlru = table[i].mlru; xas_set(&xas, index); @@ -544,7 +544,7 @@ int list_lru_memcg_alloc(struct list_lru *lru, struct mem_cgroup *memcg, gfp_t g * memcg id. More details see the comments * in memcg_reparent_list_lrus(). */ - index = memcg_cache_id(table[i].memcg); + index = memcg_kmem_id(table[i].memcg); if (index < 0) ret = 0; else if (!ret && index != xas.xa_index)