From patchwork Wed Feb 3 17:20:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12064881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90BA9C433E0 for ; Wed, 3 Feb 2021 17:21:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0281E64F92 for ; Wed, 3 Feb 2021 17:21:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0281E64F92 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7CE1F6B0006; Wed, 3 Feb 2021 12:21:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 759776B006E; Wed, 3 Feb 2021 12:21:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61EC56B0070; Wed, 3 Feb 2021 12:21:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0175.hostedemail.com [216.40.44.175]) by kanga.kvack.org (Postfix) with ESMTP id 4BEC76B0006 for ; Wed, 3 Feb 2021 12:21:02 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id F36101EE6 for ; Wed, 3 Feb 2021 17:21:01 +0000 (UTC) X-FDA: 77777621964.20.base88_2113353275d5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id C3BE5180C07AB for ; Wed, 3 Feb 2021 17:21:01 +0000 (UTC) X-HE-Tag: base88_2113353275d5 X-Filterd-Recvd-Size: 4235 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf45.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Feb 2021 17:21:01 +0000 (UTC) Received: by mail-pl1-f175.google.com with SMTP id a16so225666plh.8 for ; Wed, 03 Feb 2021 09:21:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=e8sqXu20RyBiU7ffjtxKZyn1hplKw3RF/btW7z5M9C0=; b=uiAqYe9zRIwao70xWEqsolcMy5qOp5XT1CkCzEkGSrfTkNeETfjPePiwhrqoiX/Hn2 UOavcXyTgrdZfH48yY//YhXa5yEuXE7fd/4uUxVCzoNS0hSbSnum4Dc9CZYu2ujgEGWM xTqof9l1yC280ftthAa9tYTgOGkpQMYHbonO4fPIOIGQqzJ3mPgWlghGQTnP3SfVfDv1 BXEz9UDM8syq0QsG5aDoiYTyq8fdeA0/N46yfQhGd2QfTnRYuXNLCSFsxJkjMFeXEiHg qkdln4MmVoJveVwH3Ah+EtrY6o7JQqjzSr7tM/mKUG+Y4WMFPUiXmFCJiiPXPmasVw3u oeEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=e8sqXu20RyBiU7ffjtxKZyn1hplKw3RF/btW7z5M9C0=; b=t+e/iFczHsQftwHYcSRgwkNS0uBmZx/0oS0KkADkitUYjN+f4Kc7P5Eg2YsMb1w18Y FW0o6hTpNDrvzmezUG7FP0Gbi5ioxbijoRVXrzMcH6NFBfs9hknwPhcQH42qgcuUbupk sfskA1CbNUmT+h8JvLSga3sKL2M/iy06WAVPxFO5EbGKV4sy2k9ssMXBfF7jRhtHgQYS QSxyF4zwcYSmyYMhD7sPpLY7UB6RODL1fZzuY/eq/21fvveW5M1Olq0tlIfh8OHxZfwV ubBtYUcdYMYoveWcwi6641Pz2zDik3EuGfVcwihRB9usrvnNk+fJiIaE6MMIm1ye8zEE sZMw== X-Gm-Message-State: AOAM533LhD8GccOewabVYrwKKSmLW9qZu4OVBSZ9SlzlQAPy4r0WSQS3 T9OPeN5ESfYpQn9xJPaDSC4= X-Google-Smtp-Source: ABdhPJzNg/Qk4hSC89YhvrVOZOzMTxGazWcO34MDK2tdtuW03bSsS4K8i6kUaDz/ixoBfWCHZvDIoA== X-Received: by 2002:a17:902:ab83:b029:e1:6021:dd19 with SMTP id f3-20020a170902ab83b02900e16021dd19mr4092689plr.40.1612372860288; Wed, 03 Feb 2021 09:21:00 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id x21sm2368636pfn.100.2021.02.03.09.20.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 09:20:58 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v6 PATCH 01/11] mm: vmscan: use nid from shrink_control for tracepoint Date: Wed, 3 Feb 2021 09:20:32 -0800 Message-Id: <20210203172042.800474-2-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210203172042.800474-1-shy828301@gmail.com> References: <20210203172042.800474-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The tracepoint's nid should show what node the shrink happens on, the start tracepoint uses nid from shrinkctl, but the nid might be set to 0 before end tracepoint if the shrinker is not NUMA aware, so the traceing log may show the shrink happens on one node but end up on the other node. It seems confusing. And the following patch will remove using nid directly in do_shrink_slab(), this patch also helps cleanup the code. Acked-by: Vlastimil Babka Signed-off-by: Yang Shi Acked-by: Kirill Tkhai --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b1b574ad199d..b512dd5e3a1c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -535,7 +535,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, else new_nr = atomic_long_read(&shrinker->nr_deferred[nid]); - trace_mm_shrink_slab_end(shrinker, nid, freed, nr, new_nr, total_scan); + trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); return freed; } From patchwork Wed Feb 3 17:20:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12064883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADD40C433E0 for ; Wed, 3 Feb 2021 17:21:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0A97964F8C for ; Wed, 3 Feb 2021 17:21:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0A97964F8C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 66CEB6B006E; Wed, 3 Feb 2021 12:21:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F5596B0070; Wed, 3 Feb 2021 12:21:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 49A2C6B0071; Wed, 3 Feb 2021 12:21:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0063.hostedemail.com [216.40.44.63]) by kanga.kvack.org (Postfix) with ESMTP id 322126B006E for ; Wed, 3 Feb 2021 12:21:08 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 9DB1B362A for ; Wed, 3 Feb 2021 17:21:07 +0000 (UTC) X-FDA: 77777622174.02.trade45_19136c2275d5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 7962110097AA4 for ; Wed, 3 Feb 2021 17:21:07 +0000 (UTC) X-HE-Tag: trade45_19136c2275d5 X-Filterd-Recvd-Size: 15354 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Feb 2021 17:21:06 +0000 (UTC) Received: by mail-pf1-f180.google.com with SMTP id i63so248731pfg.7 for ; Wed, 03 Feb 2021 09:21:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6ldQWvBJKlx4rkKLWQr3pD3ycbwbpPH8qTpQmWvHnx4=; b=UgFaHBYyAqZKj4ZsMSH698hAXuuc6vFzuqjn2TmypyK8VHmsxGD6Gl+dEULWCj+7Sc LTvy2aG/V92qdFYcDlsZwGu8LPZIym7MvSWehtfST5Ej+yIUzaTInIPstjdIR+/Z5c0h K7NNIRwX0LRhi/o+6WrHX5RUEr9xRw7kVgKbdjdIATxeq5SYoZUY3sTjm1z23+bnHnoQ nEvBSRCecgZiwgSuLd4fiTQKqHUMvGlbAiuvOF+qbHuBdwqYyf1nnZlnzpvlulbo3YuB R+74Gpa3IZaPueo5eWCbDiv+POrK/gwPZOnQR2dCWvxa+8UAwCAlxheDkCGMD0etB9OK fImQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6ldQWvBJKlx4rkKLWQr3pD3ycbwbpPH8qTpQmWvHnx4=; b=BUrJ+zOAPoNK9KG3F8tsSCYv+T8IKFpZvsh2JHREILvxRRy4NP159UKm/Ut0SOFpnj USZjjZmHC5xZsvwM1vmFBRW8oSwF494+jp550DG8AE/4l+pLCLqG9Xgpi1boZxvVMJZV ZoISQVmvGnUTCw6sFBWXkTK/P2aTLN3/rC3S3s9Ps0/6EMm2raX0D1TtgIpWErU5OFTr GGVRFBWTDwWcuMeFkoXy+pQIZi30F3PEIw2j+bK6tfmxSuOgg5QgPWbtOO1KNEKvowHf c7nv9DpQGQF69SNy1i7Z7JRKMr5gki/DRRa945YelGFgZ7xlBS2HwFjvJdbwqsK6WC2C EQXw== X-Gm-Message-State: AOAM531unuCyduUU2HlytrtBtX3WbeclrMX1xPqi+NPBtgphGF0RVyWi 3W+XeN2reTy5ZTCbhsrYEFU= X-Google-Smtp-Source: ABdhPJx2T/6LDduDFKWMjrrPkIwNdG+Z5OIi8reJfD4ZBeMEFNRpHhxUVbayZVYwpGWA1mXGnp7okg== X-Received: by 2002:aa7:99db:0:b029:1ba:5263:63c4 with SMTP id v27-20020aa799db0000b02901ba526363c4mr3901319pfi.53.1612372865989; Wed, 03 Feb 2021 09:21:05 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id x21sm2368636pfn.100.2021.02.03.09.21.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 09:21:02 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v6 PATCH 02/11] mm: vmscan: consolidate shrinker_maps handling code Date: Wed, 3 Feb 2021 09:20:33 -0800 Message-Id: <20210203172042.800474-3-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210203172042.800474-1-shy828301@gmail.com> References: <20210203172042.800474-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shrinker map management is not purely memcg specific, it is at the intersection between memory cgroup and shrinkers. It's allocation and assignment of a structure, and the only memcg bit is the map is being stored in a memcg structure. So move the shrinker_maps handling code into vmscan.c for tighter integration with shrinker code, and remove the "memcg_" prefix. There is no functional change. Acked-by: Vlastimil Babka Signed-off-by: Yang Shi Acked-by: Kirill Tkhai --- include/linux/memcontrol.h | 11 ++-- mm/huge_memory.c | 4 +- mm/list_lru.c | 6 +- mm/memcontrol.c | 129 +----------------------------------- mm/vmscan.c | 131 ++++++++++++++++++++++++++++++++++++- 5 files changed, 141 insertions(+), 140 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index eeb0b52203e9..1739f17e0939 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1581,10 +1581,9 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -extern int memcg_expand_shrinker_maps(int new_id); - -extern void memcg_set_shrinker_bit(struct mem_cgroup *memcg, - int nid, int shrinker_id); +int alloc_shrinker_maps(struct mem_cgroup *memcg); +void free_shrinker_maps(struct mem_cgroup *memcg); +void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; @@ -1594,8 +1593,8 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -static inline void memcg_set_shrinker_bit(struct mem_cgroup *memcg, - int nid, int shrinker_id) +static inline void set_shrinker_bit(struct mem_cgroup *memcg, + int nid, int shrinker_id) { } #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9237976abe72..05190d7f32ae 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2823,8 +2823,8 @@ void deferred_split_huge_page(struct page *page) ds_queue->split_queue_len++; #ifdef CONFIG_MEMCG if (memcg) - memcg_set_shrinker_bit(memcg, page_to_nid(page), - deferred_split_shrinker.id); + set_shrinker_bit(memcg, page_to_nid(page), + deferred_split_shrinker.id); #endif } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); diff --git a/mm/list_lru.c b/mm/list_lru.c index fe230081690b..628030fa5f69 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -125,8 +125,8 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item) list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) - memcg_set_shrinker_bit(memcg, nid, - lru_shrinker_id(lru)); + set_shrinker_bit(memcg, nid, + lru_shrinker_id(lru)); nlru->nr_items++; spin_unlock(&nlru->lock); return true; @@ -548,7 +548,7 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, if (src->nr_items) { dst->nr_items += src->nr_items; - memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); + set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1bdb93ee8e72..f5c9a0d2160b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -397,129 +397,6 @@ DEFINE_STATIC_KEY_FALSE(memcg_kmem_enabled_key); EXPORT_SYMBOL(memcg_kmem_enabled_key); #endif -static int memcg_shrinker_map_size; -static DEFINE_MUTEX(memcg_shrinker_map_mutex); - -static void memcg_free_shrinker_map_rcu(struct rcu_head *head) -{ - kvfree(container_of(head, struct memcg_shrinker_map, rcu)); -} - -static int memcg_expand_one_shrinker_map(struct mem_cgroup *memcg, - int size, int old_size) -{ - struct memcg_shrinker_map *new, *old; - int nid; - - lockdep_assert_held(&memcg_shrinker_map_mutex); - - for_each_node(nid) { - old = rcu_dereference_protected( - mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); - /* Not yet online memcg */ - if (!old) - return 0; - - new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); - if (!new) - return -ENOMEM; - - /* Set all old bits, clear all new bits */ - memset(new->map, (int)0xff, old_size); - memset((void *)new->map + old_size, 0, size - old_size); - - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, new); - call_rcu(&old->rcu, memcg_free_shrinker_map_rcu); - } - - return 0; -} - -static void memcg_free_shrinker_maps(struct mem_cgroup *memcg) -{ - struct mem_cgroup_per_node *pn; - struct memcg_shrinker_map *map; - int nid; - - if (mem_cgroup_is_root(memcg)) - return; - - for_each_node(nid) { - pn = mem_cgroup_nodeinfo(memcg, nid); - map = rcu_dereference_protected(pn->shrinker_map, true); - kvfree(map); - rcu_assign_pointer(pn->shrinker_map, NULL); - } -} - -static int memcg_alloc_shrinker_maps(struct mem_cgroup *memcg) -{ - struct memcg_shrinker_map *map; - int nid, size, ret = 0; - - if (mem_cgroup_is_root(memcg)) - return 0; - - mutex_lock(&memcg_shrinker_map_mutex); - size = memcg_shrinker_map_size; - for_each_node(nid) { - map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); - if (!map) { - memcg_free_shrinker_maps(memcg); - ret = -ENOMEM; - break; - } - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); - } - mutex_unlock(&memcg_shrinker_map_mutex); - - return ret; -} - -int memcg_expand_shrinker_maps(int new_id) -{ - int size, old_size, ret = 0; - struct mem_cgroup *memcg; - - size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); - old_size = memcg_shrinker_map_size; - if (size <= old_size) - return 0; - - mutex_lock(&memcg_shrinker_map_mutex); - if (!root_mem_cgroup) - goto unlock; - - for_each_mem_cgroup(memcg) { - if (mem_cgroup_is_root(memcg)) - continue; - ret = memcg_expand_one_shrinker_map(memcg, size, old_size); - if (ret) { - mem_cgroup_iter_break(NULL, memcg); - goto unlock; - } - } -unlock: - if (!ret) - memcg_shrinker_map_size = size; - mutex_unlock(&memcg_shrinker_map_mutex); - return ret; -} - -void memcg_set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) -{ - if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { - struct memcg_shrinker_map *map; - - rcu_read_lock(); - map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, map->map); - rcu_read_unlock(); - } -} - /** * mem_cgroup_css_from_page - css of the memcg associated with a page * @page: page of interest @@ -5369,11 +5246,11 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) struct mem_cgroup *memcg = mem_cgroup_from_css(css); /* - * A memcg must be visible for memcg_expand_shrinker_maps() + * A memcg must be visible for expand_shrinker_maps() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (memcg_alloc_shrinker_maps(memcg)) { + if (alloc_shrinker_maps(memcg)) { mem_cgroup_id_remove(memcg); return -ENOMEM; } @@ -5437,7 +5314,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) vmpressure_cleanup(&memcg->vmpressure); cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); - memcg_free_shrinker_maps(memcg); + free_shrinker_maps(memcg); memcg_free_kmem(memcg); mem_cgroup_free(memcg); } diff --git a/mm/vmscan.c b/mm/vmscan.c index b512dd5e3a1c..96b08c79f18d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -185,6 +185,131 @@ static LIST_HEAD(shrinker_list); static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG + +static int memcg_shrinker_map_size; +static DEFINE_MUTEX(memcg_shrinker_map_mutex); + +static void free_shrinker_map_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct memcg_shrinker_map, rcu)); +} + +static int expand_one_shrinker_map(struct mem_cgroup *memcg, + int size, int old_size) +{ + struct memcg_shrinker_map *new, *old; + int nid; + + lockdep_assert_held(&memcg_shrinker_map_mutex); + + for_each_node(nid) { + old = rcu_dereference_protected( + mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); + /* Not yet online memcg */ + if (!old) + return 0; + + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); + if (!new) + return -ENOMEM; + + /* Set all old bits, clear all new bits */ + memset(new->map, (int)0xff, old_size); + memset((void *)new->map + old_size, 0, size - old_size); + + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, new); + call_rcu(&old->rcu, free_shrinker_map_rcu); + } + + return 0; +} + +void free_shrinker_maps(struct mem_cgroup *memcg) +{ + struct mem_cgroup_per_node *pn; + struct memcg_shrinker_map *map; + int nid; + + if (mem_cgroup_is_root(memcg)) + return; + + for_each_node(nid) { + pn = mem_cgroup_nodeinfo(memcg, nid); + map = rcu_dereference_protected(pn->shrinker_map, true); + kvfree(map); + rcu_assign_pointer(pn->shrinker_map, NULL); + } +} + +int alloc_shrinker_maps(struct mem_cgroup *memcg) +{ + struct memcg_shrinker_map *map; + int nid, size, ret = 0; + + if (mem_cgroup_is_root(memcg)) + return 0; + + mutex_lock(&memcg_shrinker_map_mutex); + size = memcg_shrinker_map_size; + for_each_node(nid) { + map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); + if (!map) { + free_shrinker_maps(memcg); + ret = -ENOMEM; + break; + } + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); + } + mutex_unlock(&memcg_shrinker_map_mutex); + + return ret; +} + +static int expand_shrinker_maps(int new_id) +{ + int size, old_size, ret = 0; + struct mem_cgroup *memcg; + + size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); + old_size = memcg_shrinker_map_size; + if (size <= old_size) + return 0; + + mutex_lock(&memcg_shrinker_map_mutex); + if (!root_mem_cgroup) + goto unlock; + + memcg = mem_cgroup_iter(NULL, NULL, NULL); + do { + if (mem_cgroup_is_root(memcg)) + continue; + ret = expand_one_shrinker_map(memcg, size, old_size); + if (ret) { + mem_cgroup_iter_break(NULL, memcg); + goto unlock; + } + } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); +unlock: + if (!ret) + memcg_shrinker_map_size = size; + mutex_unlock(&memcg_shrinker_map_mutex); + return ret; +} + +void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) +{ + if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { + struct memcg_shrinker_map *map; + + rcu_read_lock(); + map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, map->map); + rcu_read_unlock(); + } +} + /* * We allow subsystems to populate their shrinker-related * LRU lists before register_shrinker_prepared() is called @@ -212,7 +337,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) goto unlock; if (id >= shrinker_nr_max) { - if (memcg_expand_shrinker_maps(id)) { + if (expand_shrinker_maps(id)) { idr_remove(&shrinker_idr, id); goto unlock; } @@ -589,7 +714,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, * case, we invoke the shrinker one more time and reset * the bit if it reports that it is not empty anymore. * The memory barrier here pairs with the barrier in - * memcg_set_shrinker_bit(): + * set_shrinker_bit(): * * list_lru_add() shrink_slab_memcg() * list_add_tail() clear_bit() @@ -601,7 +726,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; else - memcg_set_shrinker_bit(memcg, nid, i); + set_shrinker_bit(memcg, nid, i); } freed += ret; From patchwork Wed Feb 3 17:20:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12064885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57CE7C433DB for ; Wed, 3 Feb 2021 17:21:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B5B0564F92 for ; Wed, 3 Feb 2021 17:21:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B5B0564F92 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3395F6B0070; Wed, 3 Feb 2021 12:21:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 29AE86B0071; Wed, 3 Feb 2021 12:21:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 18A966B0072; Wed, 3 Feb 2021 12:21:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0053.hostedemail.com [216.40.44.53]) by kanga.kvack.org (Postfix) with ESMTP id F42106B0070 for ; Wed, 3 Feb 2021 12:21:10 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id B8451180AD822 for ; Wed, 3 Feb 2021 17:21:10 +0000 (UTC) X-FDA: 77777622300.21.2382ABC Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by imf03.hostedemail.com (Postfix) with ESMTP id 2C4C5C000C4E for ; Wed, 3 Feb 2021 17:21:08 +0000 (UTC) Received: by mail-pf1-f173.google.com with SMTP id o20so272425pfu.0 for ; Wed, 03 Feb 2021 09:21:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0bFxTF/fV4lUiN+s4AG3CM0LDtyH0oMQ5dQsmLRqP1s=; b=cF9bUgmWRIFWkWywVBfWjyk3eYjGnd+C3QQJJQ429Ohbfhz4RMjMXDJYmRQWzPLRxo DP1bvqrQ4E5QQiUdFn8vgOVZJiPdbEEfH5Ejak4tBOPi091dHewFQX2tEE67nKjoKb+i p4pdIj/RQfrCbIQTIfUB9JmtANB/k4OjATh1Tb5KIqiNUIQ6xdiBILHCEnp8gXDMRMNr dF7RiPXNoomwPlUVR0eq8bWVh8LjJhD8aI6w4HARvGZKP5dJCrGdxEV9W8iUaVwCA38T FxUCb1UywDlLDwvrsD11hFtOXXVr7pTEfXpvgSWqiUtjTZYaA6gvuu2iB5s0EtaGV/ve voaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0bFxTF/fV4lUiN+s4AG3CM0LDtyH0oMQ5dQsmLRqP1s=; b=UDaplwhVDXqXnt7DuM0jfJFc6SHbcXz7HLp9Bx4I/YtBUCy0wpotqleD1I2LumzGJl mAf3n8w+8zJKQUsz9eEUTqXKMyVbwrx6af+Xf/1psrWV7h12ZpWWGCy5IGAYuDmqEjFh KIowJvMMmv6lJoYfV5zVJfwkDjvEclU6OSr18LdgS+uWUq15JaywMOABR0HYRMf1Q+nA Qv/OwaJBIY5T/Qhs6PBgvJ64IpKSqgfNW53kuaT3Xzj2EBGhgtcRmRxHSgs+9RhGvlMf fpqAH7DdhuuDeF8qgTr5veLAUNHcljwA9LWLwrgSDpP/wrMiPZQbBrtDcPc/nh+B9PU2 ustQ== X-Gm-Message-State: AOAM5337yd+ycZ1fAj3ASYYf9qWiEKTfc/yge0d/do2ocq9HAtyMybYp 1EJT8pML/EkAMQP27OeQXh8= X-Google-Smtp-Source: ABdhPJyFn1IB5V7R9cYm0jM5l0K5a2g7aBSrAi+GhAnFcb0PnNL5kNoBuaHVMv7Ok8iEy1DWiX5QzA== X-Received: by 2002:a63:1f45:: with SMTP id q5mr4657962pgm.414.1612372868943; Wed, 03 Feb 2021 09:21:08 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id x21sm2368636pfn.100.2021.02.03.09.21.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 09:21:07 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v6 PATCH 03/11] mm: vmscan: use shrinker_rwsem to protect shrinker_maps allocation Date: Wed, 3 Feb 2021 09:20:34 -0800 Message-Id: <20210203172042.800474-4-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210203172042.800474-1-shy828301@gmail.com> References: <20210203172042.800474-1-shy828301@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 2C4C5C000C4E X-Stat-Signature: 4x84nm9zt1zx19zxu41f5xcmg7xuu4e4 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf03; identity=mailfrom; envelope-from=""; helo=mail-pf1-f173.google.com; client-ip=209.85.210.173 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1612372868-10989 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since memcg_shrinker_map_size just can be changed under holding shrinker_rwsem exclusively, the read side can be protected by holding read lock, so it sounds superfluous to have a dedicated mutex. Kirill Tkhai suggested use write lock since: * We want the assignment to shrinker_maps is visible for shrink_slab_memcg(). * The rcu_dereference_protected() dereferrencing in shrink_slab_memcg(), but in case of we use READ lock in alloc_shrinker_maps(), the dereferrencing is not actually protected. * READ lock makes alloc_shrinker_info() racy against memory allocation fail. alloc_shrinker_info()->free_shrinker_info() may free memory right after shrink_slab_memcg() dereferenced it. You may say shrink_slab_memcg()->mem_cgroup_online() protects us from it? Yes, sure, but this is not the thing we want to remember in the future, since this spreads modularity. And a test with heavy paging workload didn't show write lock makes things worse. Acked-by: Vlastimil Babka Signed-off-by: Yang Shi Acked-by: Kirill Tkhai --- mm/vmscan.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 96b08c79f18d..e4ddaaaeffe2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -187,7 +187,6 @@ static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG static int memcg_shrinker_map_size; -static DEFINE_MUTEX(memcg_shrinker_map_mutex); static void free_shrinker_map_rcu(struct rcu_head *head) { @@ -200,8 +199,6 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, struct memcg_shrinker_map *new, *old; int nid; - lockdep_assert_held(&memcg_shrinker_map_mutex); - for_each_node(nid) { old = rcu_dereference_protected( mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); @@ -249,7 +246,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) if (mem_cgroup_is_root(memcg)) return 0; - mutex_lock(&memcg_shrinker_map_mutex); + down_write(&shrinker_rwsem); size = memcg_shrinker_map_size; for_each_node(nid) { map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); @@ -260,7 +257,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) } rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); } - mutex_unlock(&memcg_shrinker_map_mutex); + up_write(&shrinker_rwsem); return ret; } @@ -275,9 +272,8 @@ static int expand_shrinker_maps(int new_id) if (size <= old_size) return 0; - mutex_lock(&memcg_shrinker_map_mutex); if (!root_mem_cgroup) - goto unlock; + goto out; memcg = mem_cgroup_iter(NULL, NULL, NULL); do { @@ -286,13 +282,13 @@ static int expand_shrinker_maps(int new_id) ret = expand_one_shrinker_map(memcg, size, old_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); - goto unlock; + goto out; } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); -unlock: +out: if (!ret) memcg_shrinker_map_size = size; - mutex_unlock(&memcg_shrinker_map_mutex); + return ret; } From patchwork Wed Feb 3 17:20:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12064887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 191B9C433E0 for ; Wed, 3 Feb 2021 17:21:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8E54C64DA1 for ; Wed, 3 Feb 2021 17:21:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8E54C64DA1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 163466B0071; Wed, 3 Feb 2021 12:21:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0ED636B0072; Wed, 3 Feb 2021 12:21:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1D506B0073; Wed, 3 Feb 2021 12:21:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0061.hostedemail.com [216.40.44.61]) by kanga.kvack.org (Postfix) with ESMTP id DAD886B0071 for ; Wed, 3 Feb 2021 12:21:14 -0500 (EST) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 23655362A for ; Wed, 3 Feb 2021 17:21:14 +0000 (UTC) X-FDA: 77777622468.28.bells01_380383c275d5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 6242A6C2C for ; Wed, 3 Feb 2021 17:21:13 +0000 (UTC) X-HE-Tag: bells01_380383c275d5 X-Filterd-Recvd-Size: 5636 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by imf43.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Feb 2021 17:21:12 +0000 (UTC) Received: by mail-pf1-f176.google.com with SMTP id f63so228476pfa.13 for ; Wed, 03 Feb 2021 09:21:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jGo1wZIt/fCsKVzxBYv264UFJPxmYGbYaCbSnPwe2l4=; b=vh7uA7u67PDDrYGZSYLwqb7USds8nfHkUzW09yhUIYOG1uHAEAoqqzzvBTMxaNrGJp BHm4T1Yd5hHzpIreZVDVorSzUu0BK/i/JMCoVChOz1Aii8fX2P7gycxDwf2YJOn6IvwM J1TyoHF/7lEDhBAtvAVpzepcaP+E9TD+NzP6OTYM43LnEpSPGmtmP1iduHYTHxzrhKzo a2XVhUy9WazceAk/apQK0ZAhVdhm+QJQdHH1Skd1n2XPKwxAOEkv34+y5Y31DZtMlgcf SmhDvmz44toTKe+o6sGY5cGmf7FMAvFEoytv1hlsQT8Wki5mG+8UfNdwcdLvUMEVEB+J wMAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jGo1wZIt/fCsKVzxBYv264UFJPxmYGbYaCbSnPwe2l4=; b=We4B0Q5jw6GU5jTszPTVqsIQQKqVNdupHSN23uXjG+RQUy2AfMdpJDaOJMqWHArioa M2SOqEJwb5b+KB0IowNbu9q0VT1AixlwzRzJUqWxp1Gm2GsiTCOGIAXi0WFAh/zGXxnn rH3vqbp8gwGHyZCH8XSd8bLEfgvw8RJhMOv9dS80VyIJuI8xd42CLGICE8KfoQdUeGdh 7m7gOuVzQzbM3gTEpx6j9xKwSBow6DiIwbayEJizj5AYU0U6F9uePbo0iZXaP+DvviuN PJqgpsyxyGbF4im1ElPAmoSG1VGxwEjBaPgk0ShBDNHuP5p7gPGzAlWzopEwRxu+fNd7 t2aw== X-Gm-Message-State: AOAM531rqFo5JBeCsUdiUCAJzc6BLCidQkm5gZxvJ8nRRpfbFFPb+E6I t8H4x3awLlL7y70luU02DkI= X-Google-Smtp-Source: ABdhPJzVaNXDkSSog60kMfbrB19ALynz5XcuMkn6xrdACF5Gn2BkoSfmP0yMQI/oQwzIph4vEgw6BQ== X-Received: by 2002:a62:528c:0:b029:19e:4a39:d9ea with SMTP id g134-20020a62528c0000b029019e4a39d9eamr4036033pfb.20.1612372872042; Wed, 03 Feb 2021 09:21:12 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id x21sm2368636pfn.100.2021.02.03.09.21.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 09:21:10 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v6 PATCH 04/11] mm: vmscan: remove memcg_shrinker_map_size Date: Wed, 3 Feb 2021 09:20:35 -0800 Message-Id: <20210203172042.800474-5-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210203172042.800474-1-shy828301@gmail.com> References: <20210203172042.800474-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Both memcg_shrinker_map_size and shrinker_nr_max is maintained, but actually the map size can be calculated via shrinker_nr_max, so it seems unnecessary to keep both. Remove memcg_shrinker_map_size since shrinker_nr_max is also used by iterating the bit map. Signed-off-by: Yang Shi Acked-by: Kirill Tkhai --- mm/vmscan.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index e4ddaaaeffe2..641077b09e5d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -185,8 +185,10 @@ static LIST_HEAD(shrinker_list); static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG +static int shrinker_nr_max; -static int memcg_shrinker_map_size; +#define NR_MAX_TO_SHR_MAP_SIZE(nr_max) \ + (DIV_ROUND_UP(nr_max, BITS_PER_LONG) * sizeof(unsigned long)) static void free_shrinker_map_rcu(struct rcu_head *head) { @@ -247,7 +249,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) return 0; down_write(&shrinker_rwsem); - size = memcg_shrinker_map_size; + size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); for_each_node(nid) { map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); if (!map) { @@ -265,12 +267,13 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) static int expand_shrinker_maps(int new_id) { int size, old_size, ret = 0; + int new_nr_max = new_id + 1; struct mem_cgroup *memcg; - size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); - old_size = memcg_shrinker_map_size; + size = NR_MAX_TO_SHR_MAP_SIZE(new_nr_max); + old_size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); if (size <= old_size) - return 0; + goto out; if (!root_mem_cgroup) goto out; @@ -287,7 +290,7 @@ static int expand_shrinker_maps(int new_id) } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); out: if (!ret) - memcg_shrinker_map_size = size; + shrinker_nr_max = new_nr_max; return ret; } @@ -320,7 +323,6 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) #define SHRINKER_REGISTERING ((struct shrinker *)~0UL) static DEFINE_IDR(shrinker_idr); -static int shrinker_nr_max; static int prealloc_memcg_shrinker(struct shrinker *shrinker) { @@ -337,8 +339,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) idr_remove(&shrinker_idr, id); goto unlock; } - - shrinker_nr_max = id + 1; } shrinker->id = id; ret = 0; From patchwork Wed Feb 3 17:20:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12064889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1E33C433E6 for ; Wed, 3 Feb 2021 17:21:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 404DA64DA5 for ; Wed, 3 Feb 2021 17:21:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 404DA64DA5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C3AB56B0072; Wed, 3 Feb 2021 12:21:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BEB7E6B0073; Wed, 3 Feb 2021 12:21:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9CA736B0074; Wed, 3 Feb 2021 12:21:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0046.hostedemail.com [216.40.44.46]) by kanga.kvack.org (Postfix) with ESMTP id 8201B6B0072 for ; Wed, 3 Feb 2021 12:21:16 -0500 (EST) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 38DBB180AD820 for ; Wed, 3 Feb 2021 17:21:16 +0000 (UTC) X-FDA: 77777622552.30.1CDA2CB Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf08.hostedemail.com (Postfix) with ESMTP id ACCC280191FD for ; Wed, 3 Feb 2021 17:21:13 +0000 (UTC) Received: by mail-pf1-f178.google.com with SMTP id w14so266224pfi.2 for ; Wed, 03 Feb 2021 09:21:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AXtPEywIbSFfu0wKh0tRPmppW5eyLPez/dQsEA4k5Mk=; b=iYaAM05Oa/JGSauSHx/gM2aYGkUdHiRvExStSYA19xz4gqFK4yAfIL+AIP9i2Jwh5P 2f6kvPmnLn8rbifhyj4NNs1j4kzLZD8/tY6wQLJ+qLEOXTed3zwnB726auz+CsFMMES3 gK6M6if3+T3eneJe8gfRGxNPUUiB1YGuAysZ9A2hwo432mkiCrSRuAvFWvhKCbhzxvWg sZcVEAhNJfpWmlGupWkGHeNmazl+wTv7iW8EKIyLsPYwnZlZpFh8OuzK6NKoVEkSv8F8 KW0SW+2nXJP9UxmdFaxPZ9ugNVDcEPjeYl1qb9bYK/JXNnxDs8XWDiehXHPWlAskARem qIvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AXtPEywIbSFfu0wKh0tRPmppW5eyLPez/dQsEA4k5Mk=; b=UsEb1JoIxDeCUfdz8Bdy3lQ7o2rPYfClh5cUjRr3g6TtFyG3F5ZjrmyxPWnYRGqoHG LOQnAduyTkiXeutmEE2naEUJm5LctNfis5qXLuShiwTJ98meNx3RDBBs+O20FlrquF00 I5bmxYqwQa1evLm2p/ksATIthApaqAoe11+oqh/DXfakB25/MMHAZ27HynBMdToJ5iNs OX4ggoQPQgej0EtzlNAyakC1m4fqZUPr9RqEKgv27ML8+PvqyFezDxyEEd7dqmI1AJcW kSQx2z+0ZQzv67/dBccQm8V80L64gd6gYc2D/VpOH8ODo1c70fD3/h5z9CLZgFX1C9xX Wlxg== X-Gm-Message-State: AOAM530R9eJ1pm2TtUCjGyYcbKXXbqLidxlPjbIMS04y+ggpAQv/Xz+y N70t3Pdzq2HkNJTXAXFkvys= X-Google-Smtp-Source: ABdhPJznzEslG9nHRE7Ytmf1OboCKCEf9yXdAenfA36ZxnUA5qVj5+NnnSPF7LWT22GsDgdoK01/DQ== X-Received: by 2002:a05:6a00:138f:b029:1b8:b9d5:3a2c with SMTP id t15-20020a056a00138fb02901b8b9d53a2cmr3970345pfg.10.1612372874625; Wed, 03 Feb 2021 09:21:14 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id x21sm2368636pfn.100.2021.02.03.09.21.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 09:21:13 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v6 PATCH 05/11] mm: memcontrol: rename shrinker_map to shrinker_info Date: Wed, 3 Feb 2021 09:20:36 -0800 Message-Id: <20210203172042.800474-6-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210203172042.800474-1-shy828301@gmail.com> References: <20210203172042.800474-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: i1oi4pm39btkkxqmjawww8mz1hshxduq X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: ACCC280191FD Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf08; identity=mailfrom; envelope-from=""; helo=mail-pf1-f178.google.com; client-ip=209.85.210.178 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1612372873-623108 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The following patch is going to add nr_deferred into shrinker_map, the change will make shrinker_map not only include map anymore, so rename it to "memcg_shrinker_info". And this should make the patch adding nr_deferred cleaner and readable and make review easier. Also remove the "memcg_" prefix. Acked-by: Vlastimil Babka Signed-off-by: Yang Shi Acked-by: Kirill Tkhai --- include/linux/memcontrol.h | 8 ++--- mm/memcontrol.c | 6 ++-- mm/vmscan.c | 62 +++++++++++++++++++------------------- 3 files changed, 38 insertions(+), 38 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 1739f17e0939..4c9253896e25 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -96,7 +96,7 @@ struct lruvec_stat { * Bitmap of shrinker::id corresponding to memcg-aware shrinkers, * which have elements charged to this memcg. */ -struct memcg_shrinker_map { +struct shrinker_info { struct rcu_head rcu; unsigned long map[]; }; @@ -118,7 +118,7 @@ struct mem_cgroup_per_node { struct mem_cgroup_reclaim_iter iter; - struct memcg_shrinker_map __rcu *shrinker_map; + struct shrinker_info __rcu *shrinker_info; struct rb_node tree_node; /* RB tree node */ unsigned long usage_in_excess;/* Set to the value by which */ @@ -1581,8 +1581,8 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -int alloc_shrinker_maps(struct mem_cgroup *memcg); -void free_shrinker_maps(struct mem_cgroup *memcg); +int alloc_shrinker_info(struct mem_cgroup *memcg); +void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); #else #define mem_cgroup_sockets_enabled 0 diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f5c9a0d2160b..f64ad0d044d9 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5246,11 +5246,11 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) struct mem_cgroup *memcg = mem_cgroup_from_css(css); /* - * A memcg must be visible for expand_shrinker_maps() + * A memcg must be visible for expand_shrinker_info() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (alloc_shrinker_maps(memcg)) { + if (alloc_shrinker_info(memcg)) { mem_cgroup_id_remove(memcg); return -ENOMEM; } @@ -5314,7 +5314,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) vmpressure_cleanup(&memcg->vmpressure); cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); - free_shrinker_maps(memcg); + free_shrinker_info(memcg); memcg_free_kmem(memcg); mem_cgroup_free(memcg); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 641077b09e5d..9436f9246d32 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -190,20 +190,20 @@ static int shrinker_nr_max; #define NR_MAX_TO_SHR_MAP_SIZE(nr_max) \ (DIV_ROUND_UP(nr_max, BITS_PER_LONG) * sizeof(unsigned long)) -static void free_shrinker_map_rcu(struct rcu_head *head) +static void free_shrinker_info_rcu(struct rcu_head *head) { - kvfree(container_of(head, struct memcg_shrinker_map, rcu)); + kvfree(container_of(head, struct shrinker_info, rcu)); } -static int expand_one_shrinker_map(struct mem_cgroup *memcg, +static int expand_one_shrinker_info(struct mem_cgroup *memcg, int size, int old_size) { - struct memcg_shrinker_map *new, *old; + struct shrinker_info *new, *old; int nid; for_each_node(nid) { old = rcu_dereference_protected( - mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); + mem_cgroup_nodeinfo(memcg, nid)->shrinker_info, true); /* Not yet online memcg */ if (!old) return 0; @@ -216,17 +216,17 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, memset(new->map, (int)0xff, old_size); memset((void *)new->map + old_size, 0, size - old_size); - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, new); - call_rcu(&old->rcu, free_shrinker_map_rcu); + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, new); + call_rcu(&old->rcu, free_shrinker_info_rcu); } return 0; } -void free_shrinker_maps(struct mem_cgroup *memcg) +void free_shrinker_info(struct mem_cgroup *memcg) { struct mem_cgroup_per_node *pn; - struct memcg_shrinker_map *map; + struct shrinker_info *info; int nid; if (mem_cgroup_is_root(memcg)) @@ -234,15 +234,15 @@ void free_shrinker_maps(struct mem_cgroup *memcg) for_each_node(nid) { pn = mem_cgroup_nodeinfo(memcg, nid); - map = rcu_dereference_protected(pn->shrinker_map, true); - kvfree(map); - rcu_assign_pointer(pn->shrinker_map, NULL); + info = rcu_dereference_protected(pn->shrinker_info, true); + kvfree(info); + rcu_assign_pointer(pn->shrinker_info, NULL); } } -int alloc_shrinker_maps(struct mem_cgroup *memcg) +int alloc_shrinker_info(struct mem_cgroup *memcg) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; int nid, size, ret = 0; if (mem_cgroup_is_root(memcg)) @@ -251,20 +251,20 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) down_write(&shrinker_rwsem); size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); for_each_node(nid) { - map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); - if (!map) { - free_shrinker_maps(memcg); + info = kvzalloc_node(sizeof(*info) + size, GFP_KERNEL, nid); + if (!info) { + free_shrinker_info(memcg); ret = -ENOMEM; break; } - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); return ret; } -static int expand_shrinker_maps(int new_id) +static int expand_shrinker_info(int new_id) { int size, old_size, ret = 0; int new_nr_max = new_id + 1; @@ -282,7 +282,7 @@ static int expand_shrinker_maps(int new_id) do { if (mem_cgroup_is_root(memcg)) continue; - ret = expand_one_shrinker_map(memcg, size, old_size); + ret = expand_one_shrinker_info(memcg, size, old_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -298,13 +298,13 @@ static int expand_shrinker_maps(int new_id) void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; rcu_read_lock(); - map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); + info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); - set_bit(shrinker_id, map->map); + set_bit(shrinker_id, info->map); rcu_read_unlock(); } } @@ -335,7 +335,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) goto unlock; if (id >= shrinker_nr_max) { - if (expand_shrinker_maps(id)) { + if (expand_shrinker_info(id)) { idr_remove(&shrinker_idr, id); goto unlock; } @@ -664,7 +664,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, int priority) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; unsigned long ret, freed = 0; int i; @@ -674,12 +674,12 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (!down_read_trylock(&shrinker_rwsem)) return 0; - map = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_map, - true); - if (unlikely(!map)) + info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + true); + if (unlikely(!info)) goto unlock; - for_each_set_bit(i, map->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, shrinker_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -690,7 +690,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, shrinker = idr_find(&shrinker_idr, i); if (unlikely(!shrinker || shrinker == SHRINKER_REGISTERING)) { if (!shrinker) - clear_bit(i, map->map); + clear_bit(i, info->map); continue; } @@ -701,7 +701,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, ret = do_shrink_slab(&sc, shrinker, priority); if (ret == SHRINK_EMPTY) { - clear_bit(i, map->map); + clear_bit(i, info->map); /* * After the shrinker reported that it had no objects to * free, but before we cleared the corresponding bit in From patchwork Wed Feb 3 17:20:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12064891 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D447C433DB for ; Wed, 3 Feb 2021 17:21:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E1A4764DA5 for ; Wed, 3 Feb 2021 17:21:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E1A4764DA5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 699AF6B0073; Wed, 3 Feb 2021 12:21:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F6A76B0074; Wed, 3 Feb 2021 12:21:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E81B6B0075; Wed, 3 Feb 2021 12:21:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0178.hostedemail.com [216.40.44.178]) by kanga.kvack.org (Postfix) with ESMTP id 38B046B0073 for ; Wed, 3 Feb 2021 12:21:19 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id EDEA0824999B for ; Wed, 3 Feb 2021 17:21:18 +0000 (UTC) X-FDA: 77777622636.25.pest66_4f0082d275d5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id C73671804E3A0 for ; Wed, 3 Feb 2021 17:21:18 +0000 (UTC) X-HE-Tag: pest66_4f0082d275d5 X-Filterd-Recvd-Size: 7255 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by imf35.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Feb 2021 17:21:18 +0000 (UTC) Received: by mail-pg1-f169.google.com with SMTP id c132so194370pga.3 for ; Wed, 03 Feb 2021 09:21:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=b8SjHseMw8krLA2MpYALwqlIDq/DWMFppkEMGt0xsr4=; b=QbZ8PU4NP3odPP1vupCLfUgpwl39qIZ5AuroWXUrRE8osaEdLyjG387k3x3rH+DbXP GTP0LI1osaMkATRAERt9MA6c4VdbwWUOI6E+AGksuil72cgzsrhw7fUGFaLsOhMgVHYQ 8iz0EJ/UMaFHhLepa6PF9wlVSfwscIxMUZOBoEkinY+di8/EvzfwKmdMx4amjEP9gaRt IRTuWCW4NbYgt9JsF/Ffxt8iCKnIIC7jwEVE1zjRvn7kAl0X5NSTbW29KNI6Lkpw7f+D k1KgVrcIsIdJfmRxb7Sdyg5zQRGryI6Sb9QRmOHzo7SvpKNXwW0ujbQav9RmpGxsrogb DcgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=b8SjHseMw8krLA2MpYALwqlIDq/DWMFppkEMGt0xsr4=; b=Fp+WiD76MvIQiMan+6lqOPxI7MHaHp8mzseiDYGabBl4mSFlg2pHBHfRnCEUfTrkdF DjFnu2F5Nysy/lliCLDoIGTSvlOrzrqLXph7mrPoFpNeYOP86/y07WZhgtleh3hX9DN+ FRUoMvShNzsuOz4kpCeBFI74tQU9rD5qGDQNqDn3DgQapEJQOlOzlEnHljaSbK9zU3NT dBpSLLO9YJiFwAdg67F/o+kW7NKe/sodj7rSzsV8SdnyAIu8qUSRy9P9aUrT6cv7gbqM wh8feY/N5NaLGOSdFI/BWcXizVINmQbCA+I+jxJdaXAik8NMCAQCHBV0Y2tMVAJp3JkS eMhQ== X-Gm-Message-State: AOAM5314Sg4VqwnInh70knIselfMrULi3f3+MzLnssSNOs/6drfIlz5q lyhlOHEmjxMU4yd34/qqYlQ= X-Google-Smtp-Source: ABdhPJyn+hb44RpH/HX9dquVyuyoP5juqu0UjxrgA0FVOZZ1q8tQfy9vwahs/USZbbQTwfk32MxwnQ== X-Received: by 2002:aa7:8a99:0:b029:1a6:c8b8:1414 with SMTP id a25-20020aa78a990000b02901a6c8b81414mr3867071pfc.66.1612372877527; Wed, 03 Feb 2021 09:21:17 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id x21sm2368636pfn.100.2021.02.03.09.21.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 09:21:16 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v6 PATCH 06/11] mm: vmscan: use a new flag to indicate shrinker is registered Date: Wed, 3 Feb 2021 09:20:37 -0800 Message-Id: <20210203172042.800474-7-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210203172042.800474-1-shy828301@gmail.com> References: <20210203172042.800474-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently registered shrinker is indicated by non-NULL shrinker->nr_deferred. This approach is fine with nr_deferred at the shrinker level, but the following patches will move MEMCG_AWARE shrinkers' nr_deferred to memcg level, so their shrinker->nr_deferred would always be NULL. This would prevent the shrinkers from unregistering correctly. Remove SHRINKER_REGISTERING since we could check if shrinker is registered successfully by the new flag. Signed-off-by: Yang Shi Acked-by: Kirill Tkhai --- include/linux/shrinker.h | 7 ++++--- mm/vmscan.c | 31 +++++++++---------------------- 2 files changed, 13 insertions(+), 25 deletions(-) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 0f80123650e2..1eac79ce57d4 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -79,13 +79,14 @@ struct shrinker { #define DEFAULT_SEEKS 2 /* A good number if you don't know better. */ /* Flags */ -#define SHRINKER_NUMA_AWARE (1 << 0) -#define SHRINKER_MEMCG_AWARE (1 << 1) +#define SHRINKER_REGISTERED (1 << 0) +#define SHRINKER_NUMA_AWARE (1 << 1) +#define SHRINKER_MEMCG_AWARE (1 << 2) /* * It just makes sense when the shrinker is also MEMCG_AWARE for now, * non-MEMCG_AWARE shrinker should not have this flag set. */ -#define SHRINKER_NONSLAB (1 << 2) +#define SHRINKER_NONSLAB (1 << 3) extern int prealloc_shrinker(struct shrinker *shrinker); extern void register_shrinker_prepared(struct shrinker *shrinker); diff --git a/mm/vmscan.c b/mm/vmscan.c index 9436f9246d32..dc0d69e081b0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -309,19 +309,6 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) } } -/* - * We allow subsystems to populate their shrinker-related - * LRU lists before register_shrinker_prepared() is called - * for the shrinker, since we don't want to impose - * restrictions on their internal registration order. - * In this case shrink_slab_memcg() may find corresponding - * bit is set in the shrinkers map. - * - * This value is used by the function to detect registering - * shrinkers and to skip do_shrink_slab() calls for them. - */ -#define SHRINKER_REGISTERING ((struct shrinker *)~0UL) - static DEFINE_IDR(shrinker_idr); static int prealloc_memcg_shrinker(struct shrinker *shrinker) @@ -330,7 +317,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) down_write(&shrinker_rwsem); /* This may call shrinker, so it must use down_read_trylock() */ - id = idr_alloc(&shrinker_idr, SHRINKER_REGISTERING, 0, 0, GFP_KERNEL); + id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -493,10 +480,7 @@ void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); list_add_tail(&shrinker->list, &shrinker_list); -#ifdef CONFIG_MEMCG - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - idr_replace(&shrinker_idr, shrinker, shrinker->id); -#endif + shrinker->flags |= SHRINKER_REGISTERED; up_write(&shrinker_rwsem); } @@ -516,13 +500,16 @@ EXPORT_SYMBOL(register_shrinker); */ void unregister_shrinker(struct shrinker *shrinker) { - if (!shrinker->nr_deferred) + if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - unregister_memcg_shrinker(shrinker); + down_write(&shrinker_rwsem); list_del(&shrinker->list); + shrinker->flags &= ~SHRINKER_REGISTERED; up_write(&shrinker_rwsem); + + if (shrinker->flags & SHRINKER_MEMCG_AWARE) + unregister_memcg_shrinker(shrinker); kfree(shrinker->nr_deferred); shrinker->nr_deferred = NULL; } @@ -688,7 +675,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct shrinker *shrinker; shrinker = idr_find(&shrinker_idr, i); - if (unlikely(!shrinker || shrinker == SHRINKER_REGISTERING)) { + if (unlikely(!shrinker || !(shrinker->flags & SHRINKER_REGISTERED))) { if (!shrinker) clear_bit(i, info->map); continue; From patchwork Wed Feb 3 17:20:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12064893 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4F36C433E0 for ; Wed, 3 Feb 2021 17:21:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 03BEC64DA5 for ; Wed, 3 Feb 2021 17:21:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 03BEC64DA5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7F3E76B0074; Wed, 3 Feb 2021 12:21:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 77BAE6B0075; Wed, 3 Feb 2021 12:21:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6451F6B0078; Wed, 3 Feb 2021 12:21:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0089.hostedemail.com [216.40.44.89]) by kanga.kvack.org (Postfix) with ESMTP id 4F85D6B0074 for ; Wed, 3 Feb 2021 12:21:23 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id F2F2B181AC9C6 for ; Wed, 3 Feb 2021 17:21:22 +0000 (UTC) X-FDA: 77777622846.06.plant11_3210e60275d5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id B360D1003C318 for ; Wed, 3 Feb 2021 17:21:22 +0000 (UTC) X-HE-Tag: plant11_3210e60275d5 X-Filterd-Recvd-Size: 9400 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by imf20.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Feb 2021 17:21:22 +0000 (UTC) Received: by mail-pg1-f181.google.com with SMTP id o63so185038pgo.6 for ; Wed, 03 Feb 2021 09:21:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=J6v171T8udhMDGyBshwnSqs/Lfk2C4H0HVeOa7wJUcc=; b=Qwxsh2wrKevFmc2hiX6IlcrcssrpasIOnE5m67edh6AQz1mclxn2sKGTZLUcZ/9YZP 3LPLb9HscvMSn73XPYzZH/tHFeQsBEW4fiBAYoX3n+ayp95uhqBvdpno28pdeYtqkhfL 7ph1Nx90VwSE4x738JtXiQ7vpjBZeTPI6+EicF8Q0WtpxC4Nz/ycKuRM2nRol15Y2Tn1 helX0lJInJm7yilVAPsryMXn2w7EMLQRm1n2RGRJRyw/9Nfx7zM91a6ivAc6K7GqTUXk mMQ2XF8FFYrFusumPBPe59VRM/KqLoNxdxYkD+GIwgdhPtofwRajh12lAXvwB8oprwZP Hnuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=J6v171T8udhMDGyBshwnSqs/Lfk2C4H0HVeOa7wJUcc=; b=mlS41kxzbbiiQesIhnmsp7uvJEWoGujhAQv6TSPwYbX5ch4MN0W5WinajPyvj/QLe2 e+jGGHrIvYvha1cRropTh6PG383MHJTeaPEsFKsqWaJnyCWBibrEw8RjLMLkLE4vwLXf aDw3J+R3CL7OHQTlnd9r3Y+OkiyvhaxR+O7pYqf3U1lRCyCTKQsJH8S7N6OUeR6IzSHM WEG080W/dOrgUROV3BgbWaqX0buaqgrn8Tr+LFJrxOCk8k6oxrTdhxO5Ko5KM53Ko4tz 8I0RaVqPRYhc+lnPCFoyrzOPHIwpKNqlOZ+onMR80Lq53nz+48308FiWw4gx8MuispoZ oPPA== X-Gm-Message-State: AOAM531wXEsJxB7s3yfhWDtMOubGDbQLWtyslO6IWZn7PWep0QGntlmO 11vhH9iGdsJS7UH6Cw5vX9OYVHZcDCVmlg== X-Google-Smtp-Source: ABdhPJyZ2yrggSL0vGz19HHhV8i3cZjX9S9HdYmi52Jd/Z/1q9XktcXrNgYsKj1YZ720SRjlGAkNOA== X-Received: by 2002:a62:4e0f:0:b029:1c0:d61e:8543 with SMTP id c15-20020a624e0f0000b02901c0d61e8543mr3906057pfb.77.1612372881473; Wed, 03 Feb 2021 09:21:21 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id x21sm2368636pfn.100.2021.02.03.09.21.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 09:21:19 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v6 PATCH 07/11] mm: vmscan: add per memcg shrinker nr_deferred Date: Wed, 3 Feb 2021 09:20:38 -0800 Message-Id: <20210203172042.800474-8-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210203172042.800474-1-shy828301@gmail.com> References: <20210203172042.800474-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently the number of deferred objects are per shrinker, but some slabs, for example, vfs inode/dentry cache are per memcg, this would result in poor isolation among memcgs. The deferred objects typically are generated by __GFP_NOFS allocations, one memcg with excessive __GFP_NOFS allocations may blow up deferred objects, then other innocent memcgs may suffer from over shrink, excessive reclaim latency, etc. For example, two workloads run in memcgA and memcgB respectively, workload in B is vfs heavy workload. Workload in A generates excessive deferred objects, then B's vfs cache might be hit heavily (drop half of caches) by B's limit reclaim or global reclaim. We observed this hit in our production environment which was running vfs heavy workload shown as the below tracing log: <...>-409454 [016] .... 28286961.747146: mm_shrink_slab_start: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 1 objects to shrink 3641681686040 gfp_flags GFP_HIGHUSER_MOVABLE|__GFP_ZERO pgs_scanned 1 lru_pgs 15721 cache items 246404277 delta 31345 total_scan 123202138 <...>-409454 [022] .... 28287105.928018: mm_shrink_slab_end: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 1 unused scan count 3641681686040 new scan count 3641798379189 total_scan 602 last shrinker return val 123186855 The vfs cache and page cache ration was 10:1 on this machine, and half of caches were dropped. This also resulted in significant amount of page caches were dropped due to inodes eviction. Make nr_deferred per memcg for memcg aware shrinkers would solve the unfairness and bring better isolation. When memcg is not enabled (!CONFIG_MEMCG or memcg disabled), the shrinker's nr_deferred would be used. And non memcg aware shrinkers use shrinker's nr_deferred all the time. Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 7 +++--- mm/vmscan.c | 45 ++++++++++++++++++++++++-------------- 2 files changed, 33 insertions(+), 19 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4c9253896e25..c457fc7bc631 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -93,12 +93,13 @@ struct lruvec_stat { }; /* - * Bitmap of shrinker::id corresponding to memcg-aware shrinkers, - * which have elements charged to this memcg. + * Bitmap and deferred work of shrinker::id corresponding to memcg-aware + * shrinkers, which have elements charged to this memcg. */ struct shrinker_info { struct rcu_head rcu; - unsigned long map[]; + atomic_long_t *nr_deferred; + unsigned long *map; }; /* diff --git a/mm/vmscan.c b/mm/vmscan.c index dc0d69e081b0..d9126f12890f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -196,10 +196,12 @@ static void free_shrinker_info_rcu(struct rcu_head *head) } static int expand_one_shrinker_info(struct mem_cgroup *memcg, - int size, int old_size) + int m_size, int d_size, + int old_m_size, int old_d_size) { struct shrinker_info *new, *old; int nid; + int size = m_size + d_size; for_each_node(nid) { old = rcu_dereference_protected( @@ -212,9 +214,15 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!new) return -ENOMEM; - /* Set all old bits, clear all new bits */ - memset(new->map, (int)0xff, old_size); - memset((void *)new->map + old_size, 0, size - old_size); + new->nr_deferred = (atomic_long_t *)(new + 1); + new->map = (void *)new->nr_deferred + d_size; + + /* map: set all old bits, clear all new bits */ + memset(new->map, (int)0xff, old_m_size); + memset((void *)new->map + old_m_size, 0, m_size - old_m_size); + /* nr_deferred: copy old values, clear all new values */ + memcpy(new->nr_deferred, old->nr_deferred, old_d_size); + memset((void *)new->nr_deferred + old_d_size, 0, d_size - old_d_size); rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, new); call_rcu(&old->rcu, free_shrinker_info_rcu); @@ -229,9 +237,6 @@ void free_shrinker_info(struct mem_cgroup *memcg) struct shrinker_info *info; int nid; - if (mem_cgroup_is_root(memcg)) - return; - for_each_node(nid) { pn = mem_cgroup_nodeinfo(memcg, nid); info = rcu_dereference_protected(pn->shrinker_info, true); @@ -244,12 +249,13 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) { struct shrinker_info *info; int nid, size, ret = 0; - - if (mem_cgroup_is_root(memcg)) - return 0; + int m_size, d_size = 0; down_write(&shrinker_rwsem); - size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); + m_size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); + d_size = shrinker_nr_max * sizeof(atomic_long_t); + size = m_size + d_size; + for_each_node(nid) { info = kvzalloc_node(sizeof(*info) + size, GFP_KERNEL, nid); if (!info) { @@ -257,6 +263,8 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) ret = -ENOMEM; break; } + info->nr_deferred = (atomic_long_t *)(info + 1); + info->map = (void *)info->nr_deferred + d_size; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -268,10 +276,16 @@ static int expand_shrinker_info(int new_id) { int size, old_size, ret = 0; int new_nr_max = new_id + 1; + int m_size, d_size = 0; + int old_m_size, old_d_size = 0; struct mem_cgroup *memcg; - size = NR_MAX_TO_SHR_MAP_SIZE(new_nr_max); - old_size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); + m_size = NR_MAX_TO_SHR_MAP_SIZE(new_nr_max); + d_size = new_nr_max * sizeof(atomic_long_t); + size = m_size + d_size; + old_m_size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); + old_d_size = shrinker_nr_max * sizeof(atomic_long_t); + old_size = old_m_size + old_d_size; if (size <= old_size) goto out; @@ -280,9 +294,8 @@ static int expand_shrinker_info(int new_id) memcg = mem_cgroup_iter(NULL, NULL, NULL); do { - if (mem_cgroup_is_root(memcg)) - continue; - ret = expand_one_shrinker_info(memcg, size, old_size); + ret = expand_one_shrinker_info(memcg, m_size, d_size, + old_m_size, old_d_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; From patchwork Wed Feb 3 17:20:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12064895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7D07C433E0 for ; Wed, 3 Feb 2021 17:21:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2956664DDB for ; Wed, 3 Feb 2021 17:21:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2956664DDB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AB4B66B0075; Wed, 3 Feb 2021 12:21:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A61AB6B0078; Wed, 3 Feb 2021 12:21:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 951C26B007B; Wed, 3 Feb 2021 12:21:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0202.hostedemail.com [216.40.44.202]) by kanga.kvack.org (Postfix) with ESMTP id 7C04E6B0075 for ; Wed, 3 Feb 2021 12:21:27 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 3FFDD181AC9C6 for ; Wed, 3 Feb 2021 17:21:27 +0000 (UTC) X-FDA: 77777623014.19.toys83_4e117ed275d5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id 1D4CA1AD1AD for ; Wed, 3 Feb 2021 17:21:27 +0000 (UTC) X-HE-Tag: toys83_4e117ed275d5 X-Filterd-Recvd-Size: 9029 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by imf49.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Feb 2021 17:21:26 +0000 (UTC) Received: by mail-pf1-f173.google.com with SMTP id y205so256609pfc.5 for ; Wed, 03 Feb 2021 09:21:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6ne9c28jAeNdinyG89P4oWkn90QuBd4v0cKiR/cg2+U=; b=orZSCV6SHryTbhEuW2L5pMDNGTnkvPbjkzPIslnHCrX8y7dM+IHRgsQC7hetZxHcH9 s1jEQuYyZaeyxfG0l/QBgW41BLgZqfwtsIXzWZDL/H9N0pE9TdyD1VGD+OcRmJHoPE6R lvLGCmdZLEdXmQz89rED6ElQ1i7IV/FhmdXJzWZNRUcHmprcn8JHklhLvFxTSRIiWS37 aivC59/7hrfd16QOLI2SRDmsmog3iAGEbVdGAtnWWQycXbNhAaLkpqz/zb/QrZVp93Kw v8pSWU0c24/FNFn+04rdxNSmw9EvUvHRHKPJ0EVpDFNp5Z9WGwV3lbaM7m1TLitZHB7W RQNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6ne9c28jAeNdinyG89P4oWkn90QuBd4v0cKiR/cg2+U=; b=Id5j4YLnGuUZECyRelGmXQrN9//CpIJRZg96VU2PtE3VYm4oSmcCyEtNcEPbdGlIdb LNDYCek0CzDZeNcl3A9MkM09e422mhm0a8GuNhY63kwlmdTfe1lf4yUpvX/4pS2YATp2 A5Rd5rH4MXSXLr1CYwDR2md3HOC9GkRXBN2qpEEJoKG2DnzWWK64oxyEruws7USrU4uB 5f5Dm2B2hI04YoZYSzCltC8iQvucYwUIn4S+fUN5pXKY7nehGRXWcCwkTxG5ro9sT3kq 7s8n93OQ8KvT4JU+DRYCPQbt5yrEK5Hr9eCVn4l2R/A2aoysSqwKn3UdvZg5kTmSynkd CjNw== X-Gm-Message-State: AOAM533khb+oVbOauMwjVXmkPohhhQXT1UG0qUV0zJR80OeIDbmMveA9 MbxNxKZzIXNg10IoT+0+Rq0= X-Google-Smtp-Source: ABdhPJweS1ZsAvxHk9E6TASUPG1UDfzIqhh+bhCZU+sXXpV+OultKCvbpaaw1AfBEFMLbFaCmGylnA== X-Received: by 2002:a62:1ec5:0:b029:1ce:b354:b25f with SMTP id e188-20020a621ec50000b02901ceb354b25fmr3901139pfe.56.1612372885959; Wed, 03 Feb 2021 09:21:25 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id x21sm2368636pfn.100.2021.02.03.09.21.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 09:21:23 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v6 PATCH 08/11] mm: vmscan: use per memcg nr_deferred of shrinker Date: Wed, 3 Feb 2021 09:20:39 -0800 Message-Id: <20210203172042.800474-9-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210203172042.800474-1-shy828301@gmail.com> References: <20210203172042.800474-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use per memcg's nr_deferred for memcg aware shrinkers. The shrinker's nr_deferred will be used in the following cases: 1. Non memcg aware shrinkers 2. !CONFIG_MEMCG 3. memcg is disabled by boot parameter Signed-off-by: Yang Shi Reported-by: kernel test robot --- mm/vmscan.c | 94 +++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 77 insertions(+), 17 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d9126f12890f..545422d2aeec 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -190,6 +190,13 @@ static int shrinker_nr_max; #define NR_MAX_TO_SHR_MAP_SIZE(nr_max) \ (DIV_ROUND_UP(nr_max, BITS_PER_LONG) * sizeof(unsigned long)) +static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, + int nid) +{ + return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + lockdep_is_held(&shrinker_rwsem)); +} + static void free_shrinker_info_rcu(struct rcu_head *head) { kvfree(container_of(head, struct shrinker_info, rcu)); @@ -204,8 +211,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, int size = m_size + d_size; for_each_node(nid) { - old = rcu_dereference_protected( - mem_cgroup_nodeinfo(memcg, nid)->shrinker_info, true); + old = shrinker_info_protected(memcg, nid); /* Not yet online memcg */ if (!old) return 0; @@ -239,7 +245,7 @@ void free_shrinker_info(struct mem_cgroup *memcg) for_each_node(nid) { pn = mem_cgroup_nodeinfo(memcg, nid); - info = rcu_dereference_protected(pn->shrinker_info, true); + info = shrinker_info_protected(memcg, nid); kvfree(info); rcu_assign_pointer(pn->shrinker_info, NULL); } @@ -358,6 +364,25 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) up_write(&shrinker_rwsem); } + +static long count_nr_deferred_memcg(int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + + info = shrinker_info_protected(memcg, nid); + return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); +} + +static long set_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + + info = shrinker_info_protected(memcg, nid); + return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); +} + static bool cgroup_reclaim(struct scan_control *sc) { return sc->target_mem_cgroup; @@ -396,6 +421,18 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) { } +static long count_nr_deferred_memcg(int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + return 0; +} + +static long set_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + return 0; +} + static bool cgroup_reclaim(struct scan_control *sc) { return false; @@ -407,6 +444,39 @@ static bool writeback_throttling_sane(struct scan_control *sc) } #endif +static long count_nr_deferred(struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (sc->memcg && + (shrinker->flags & SHRINKER_MEMCG_AWARE)) + return count_nr_deferred_memcg(nid, shrinker, + sc->memcg); + + return atomic_long_xchg(&shrinker->nr_deferred[nid], 0); +} + + +static long set_nr_deferred(long nr, struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (sc->memcg && + (shrinker->flags & SHRINKER_MEMCG_AWARE)) + return set_nr_deferred_memcg(nr, nid, shrinker, + sc->memcg); + + return atomic_long_add_return(nr, &shrinker->nr_deferred[nid]); +} + /* * This misses isolated pages which are not accounted for to save counters. * As the data only determines if reclaim or compaction continues, it is @@ -539,14 +609,10 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, long freeable; long nr; long new_nr; - int nid = shrinkctl->nid; long batch_size = shrinker->batch ? shrinker->batch : SHRINK_BATCH; long scanned = 0, next_deferred; - if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) - nid = 0; - freeable = shrinker->count_objects(shrinker, shrinkctl); if (freeable == 0 || freeable == SHRINK_EMPTY) return freeable; @@ -556,7 +622,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, * and zero it so that other concurrent shrinker invocations * don't also do this scanning work. */ - nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0); + nr = count_nr_deferred(shrinker, shrinkctl); total_scan = nr; if (shrinker->seeks) { @@ -647,14 +713,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, next_deferred = 0; /* * move the unused scan count back into the shrinker in a - * manner that handles concurrent updates. If we exhausted the - * scan, there is no need to do an update. + * manner that handles concurrent updates. */ - if (next_deferred > 0) - new_nr = atomic_long_add_return(next_deferred, - &shrinker->nr_deferred[nid]); - else - new_nr = atomic_long_read(&shrinker->nr_deferred[nid]); + new_nr = set_nr_deferred(next_deferred, shrinker, shrinkctl); trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); return freed; @@ -674,8 +735,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (!down_read_trylock(&shrinker_rwsem)) return 0; - info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - true); + info = shrinker_info_protected(memcg, nid); if (unlikely(!info)) goto unlock; From patchwork Wed Feb 3 17:20:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12064897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13835C433E0 for ; Wed, 3 Feb 2021 17:21:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 78BBD64DDB for ; Wed, 3 Feb 2021 17:21:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 78BBD64DDB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 09E076B0078; Wed, 3 Feb 2021 12:21:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 02BEF6B007B; Wed, 3 Feb 2021 12:21:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E343B6B007D; Wed, 3 Feb 2021 12:21:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0020.hostedemail.com [216.40.44.20]) by kanga.kvack.org (Postfix) with ESMTP id C55A26B0078 for ; Wed, 3 Feb 2021 12:21:30 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 466BF362A for ; Wed, 3 Feb 2021 17:21:30 +0000 (UTC) X-FDA: 77777623140.17.hen38_5108245275d5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 24DEC180D0180 for ; Wed, 3 Feb 2021 17:21:30 +0000 (UTC) X-HE-Tag: hen38_5108245275d5 X-Filterd-Recvd-Size: 5748 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Feb 2021 17:21:29 +0000 (UTC) Received: by mail-pg1-f176.google.com with SMTP id c132so194786pga.3 for ; Wed, 03 Feb 2021 09:21:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZyoHuzV1k2TIojfPlHgQ9uQzq96PQTgKepbMyYmNmc0=; b=sItIFo8+Tv2+14wSPbPAt5ataYj6jUfSFcSvfEdBqVF6eJH7vgm6Lq5sXA5/68u4dn sw118c/7ai8dwWAaGxIGo1pCnH2NswhKt7TNxF3DwfQgwlynTIDa0vgaUsVH8MA6NKIJ CrdP7aCjHuKt1uXcBPzdPgey/+8QE89OpT82AAiGh/KDMBxyV3ohfEpTuIMhTYBlYOtI 099vjIo9ZpsbV/EXg2FCX+krFVDctzo1RpPu9OnpPFsSiaJgJTk0JbzlrsjHvzEnH4yw wA35sVtLpS2Y43o+oomX7WICvQlBpciz1XGF59guja7MqQqjQCzIz56uRSqwnpN8ngEb iO6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZyoHuzV1k2TIojfPlHgQ9uQzq96PQTgKepbMyYmNmc0=; b=tJLKm0Kf8oUwfTcf/Bw8/kbnvpX5Jpgw75EW+B6dJS0UE1wO45sCErWDJYwurviDlH uBSnALg6NAd+1POKW1+IQxY8Xbohbhwm7IsMecDkd9UuDCSoRMm1lt8hA2lAoX7X5YFv HkkH2a71XJUdmh+CKpSpOIVzP77bJLaQ264VkmXO1kn2P1Bwno9beD45RFWyaK8zB+fv 6WTX4lWFNPa744WlA8aJ4Qu/jUH5mNdZtAioC6D9LOcd92A49H5U3aNKKQhSeCeMYW03 vtghv/3RfUpj7h+1uhjCWQtHKTCbBkUUP8bjw9F9F/UxYbOp2HVK2QVrkvkA/3r6zbIV d3Tw== X-Gm-Message-State: AOAM530TnJZYf41fqt9jnA7EqiIo0r9fmR/LU193sfcy5jZtb1ugyw5k CR0G3hqq7Qa7jlJvgPVMBrA= X-Google-Smtp-Source: ABdhPJz+Zs59uEkfLCtuU40slq96EYyTOTHEgvjkTOIZezz5jTc8D2BmsjoMjeIQUTzuvj0ddSrIzQ== X-Received: by 2002:a62:ae0c:0:b029:1bc:a0b9:f0aa with SMTP id q12-20020a62ae0c0000b02901bca0b9f0aamr3901720pff.78.1612372888904; Wed, 03 Feb 2021 09:21:28 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id x21sm2368636pfn.100.2021.02.03.09.21.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 09:21:27 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v6 PATCH 09/11] mm: vmscan: don't need allocate shrinker->nr_deferred for memcg aware shrinkers Date: Wed, 3 Feb 2021 09:20:40 -0800 Message-Id: <20210203172042.800474-10-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210203172042.800474-1-shy828301@gmail.com> References: <20210203172042.800474-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now nr_deferred is available on per memcg level for memcg aware shrinkers, so don't need allocate shrinker->nr_deferred for such shrinkers anymore. The prealloc_memcg_shrinker() would return -ENOSYS if !CONFIG_MEMCG or memcg is disabled by kernel command line, then shrinker's SHRINKER_MEMCG_AWARE flag would be cleared. This makes the implementation of this patch simpler. Acked-by: Vlastimil Babka Signed-off-by: Yang Shi Reviewed-by: Kirill Tkhai --- mm/vmscan.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 545422d2aeec..20a35d26ae12 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -334,6 +334,9 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) { int id, ret = -ENOMEM; + if (mem_cgroup_disabled()) + return -ENOSYS; + down_write(&shrinker_rwsem); /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); @@ -414,7 +417,7 @@ static bool writeback_throttling_sane(struct scan_control *sc) #else static int prealloc_memcg_shrinker(struct shrinker *shrinker) { - return 0; + return -ENOSYS; } static void unregister_memcg_shrinker(struct shrinker *shrinker) @@ -525,8 +528,18 @@ unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru, int zone */ int prealloc_shrinker(struct shrinker *shrinker) { - unsigned int size = sizeof(*shrinker->nr_deferred); + unsigned int size; + int err; + + if (shrinker->flags & SHRINKER_MEMCG_AWARE) { + err = prealloc_memcg_shrinker(shrinker); + if (err != -ENOSYS) + return err; + shrinker->flags &= ~SHRINKER_MEMCG_AWARE; + } + + size = sizeof(*shrinker->nr_deferred); if (shrinker->flags & SHRINKER_NUMA_AWARE) size *= nr_node_ids; @@ -534,26 +547,14 @@ int prealloc_shrinker(struct shrinker *shrinker) if (!shrinker->nr_deferred) return -ENOMEM; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - if (prealloc_memcg_shrinker(shrinker)) - goto free_deferred; - } return 0; - -free_deferred: - kfree(shrinker->nr_deferred); - shrinker->nr_deferred = NULL; - return -ENOMEM; } void free_prealloced_shrinker(struct shrinker *shrinker) { - if (!shrinker->nr_deferred) - return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - unregister_memcg_shrinker(shrinker); + return unregister_memcg_shrinker(shrinker); kfree(shrinker->nr_deferred); shrinker->nr_deferred = NULL; From patchwork Wed Feb 3 17:20:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12064899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38EE0C433DB for ; Wed, 3 Feb 2021 17:21:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D56AD64DDB for ; Wed, 3 Feb 2021 17:21:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D56AD64DDB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6443A6B007B; Wed, 3 Feb 2021 12:21:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D0836B007D; Wed, 3 Feb 2021 12:21:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 46EDB6B007E; Wed, 3 Feb 2021 12:21:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0226.hostedemail.com [216.40.44.226]) by kanga.kvack.org (Postfix) with ESMTP id 309786B007B for ; Wed, 3 Feb 2021 12:21:33 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id E93E1180AD81F for ; Wed, 3 Feb 2021 17:21:32 +0000 (UTC) X-FDA: 77777623224.18.fly80_3912560275d5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id BA6D3100ED3B2 for ; Wed, 3 Feb 2021 17:21:32 +0000 (UTC) X-HE-Tag: fly80_3912560275d5 X-Filterd-Recvd-Size: 5675 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Feb 2021 17:21:32 +0000 (UTC) Received: by mail-pg1-f169.google.com with SMTP id s23so162083pgh.11 for ; Wed, 03 Feb 2021 09:21:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nUMzA/3BdE993wfDsCQXdvkjPYfyPVaNNJORKwDz80c=; b=LfePu6UejxvQdOh2GoD/fAj3ONicQbkZc2mxXNU+hyyYQdjy+n01l4dwHIL3TDJNHO AdoGTQBnvO8yWixQGBdGasySYE0ZZ3RDXxzE7Mej7z0noxcd+JARknsNgIQnql7xvO85 VH6SJ/HLz90ywXZizbp/ooQSNn+GF27tDVTEhVwcPnwp6a3CuVS8QdSPFULH10IXLefO BxNVxluFNk8His5xm6J3gzqJQ9Uw+kheGFOarhrreg5auDg37mVW2H7ahS4n5W8J8N/E mspkFr/a4UqQxVP+K2DmWtfNLKcri14wm/mD+aAZPFOZDWMO28TObbEeRckub47dLOJF W2Ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nUMzA/3BdE993wfDsCQXdvkjPYfyPVaNNJORKwDz80c=; b=LtOnEFXjpiTmM1xDLAi4sSd66G38j6UeMIQydV720FpKraLCjcTP0izZN0yLQEVpS7 rRg7flevBQs2l92tDxE0rwCNO16OPAAozIgIMzrwjrpVkJbw7ND6AxcuGiP9iTS1nqhk 9RhTgKAvlWRTGGjjoR+M3xxi7dmDu4mCDe49BflvS1x2h9I68KG48kainEb+a/0jRmPK WyJiEv8NF5Jd+lEwjbPRzWZAk/gBTK1CrcgHBOSFGPWS4jZmKK4YA0D+aH62XYdpsMvq 5LaGq4s2YI3Qfjdn8gKCzIy6ttuVgWrB5LmZcnjR/Vjm652VNubCiCHNN1smnWqD5aaF ZoTw== X-Gm-Message-State: AOAM532uWFP2My9xr2vDXi/5432bgcIkkyugdekNRzBU03zNS0HVrB/7 yqVaereAv4j9CTriRV47yLc= X-Google-Smtp-Source: ABdhPJxZ85kIgUNuaFI1hCRoW+MvELqUQ5WPGXjcXFhMMHfHj4lg+FZdDFrWcWpXQWIVRJnKqH+SuQ== X-Received: by 2002:aa7:8881:0:b029:1b4:5b52:b00b with SMTP id z1-20020aa788810000b02901b45b52b00bmr4015162pfe.47.1612372891539; Wed, 03 Feb 2021 09:21:31 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id x21sm2368636pfn.100.2021.02.03.09.21.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 09:21:30 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v6 PATCH 10/11] mm: memcontrol: reparent nr_deferred when memcg offline Date: Wed, 3 Feb 2021 09:20:41 -0800 Message-Id: <20210203172042.800474-11-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210203172042.800474-1-shy828301@gmail.com> References: <20210203172042.800474-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now shrinker's nr_deferred is per memcg for memcg aware shrinkers, add to parent's corresponding nr_deferred when memcg offline. Acked-by: Vlastimil Babka Signed-off-by: Yang Shi Acked-by: Kirill Tkhai --- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 1 + mm/vmscan.c | 24 ++++++++++++++++++++++++ 3 files changed, 26 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index c457fc7bc631..e1c4b93889ad 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1585,6 +1585,7 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) int alloc_shrinker_info(struct mem_cgroup *memcg); void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); +void reparent_shrinker_deferred(struct mem_cgroup *memcg); #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f64ad0d044d9..21f36b73f36a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5282,6 +5282,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) page_counter_set_low(&memcg->memory, 0); memcg_offline_kmem(memcg); + reparent_shrinker_deferred(memcg); wb_memcg_offline(memcg); drain_all_stock(memcg); diff --git a/mm/vmscan.c b/mm/vmscan.c index 20a35d26ae12..574d920c4cab 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -386,6 +386,30 @@ static long set_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } +void reparent_shrinker_deferred(struct mem_cgroup *memcg) +{ + int i, nid; + long nr; + struct mem_cgroup *parent; + struct shrinker_info *child_info, *parent_info; + + parent = parent_mem_cgroup(memcg); + if (!parent) + parent = root_mem_cgroup; + + /* Prevent from concurrent shrinker_info expand */ + down_read(&shrinker_rwsem); + for_each_node(nid) { + child_info = shrinker_info_protected(memcg, nid); + parent_info = shrinker_info_protected(parent, nid); + for (i = 0; i < shrinker_nr_max; i++) { + nr = atomic_long_read(&child_info->nr_deferred[i]); + atomic_long_add(nr, &parent_info->nr_deferred[i]); + } + } + up_read(&shrinker_rwsem); +} + static bool cgroup_reclaim(struct scan_control *sc) { return sc->target_mem_cgroup; From patchwork Wed Feb 3 17:20:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12064901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AED2C433E0 for ; Wed, 3 Feb 2021 17:21:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B204564DDB for ; Wed, 3 Feb 2021 17:21:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B204564DDB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3FCF16B007D; Wed, 3 Feb 2021 12:21:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 35F2E6B007E; Wed, 3 Feb 2021 12:21:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 166A46B0080; Wed, 3 Feb 2021 12:21:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0169.hostedemail.com [216.40.44.169]) by kanga.kvack.org (Postfix) with ESMTP id F05146B007D for ; Wed, 3 Feb 2021 12:21:35 -0500 (EST) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B7B07824999B for ; Wed, 3 Feb 2021 17:21:35 +0000 (UTC) X-FDA: 77777623350.28.brake63_1d0ece5275d5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 7B7E76C21 for ; Wed, 3 Feb 2021 17:21:35 +0000 (UTC) X-HE-Tag: brake63_1d0ece5275d5 X-Filterd-Recvd-Size: 6090 Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Feb 2021 17:21:34 +0000 (UTC) Received: by mail-pg1-f182.google.com with SMTP id z21so192093pgj.4 for ; Wed, 03 Feb 2021 09:21:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WjCWNWtT0pxxhAlETV2zIWjv0Gl4p3lxrm3/RhLgkFI=; b=WOTROWby4+O9mii6I9VKEbRf1e0JneRRZWuOzB+sxeZyY83rimGnXwHvmaiAJl950J UC1zbSHRcY3mTA7B7y5OXXYbolcIsJvey4DCBh5Rx9c8WPmcnqZ8EoMX7SoBo1/dBquC tR1gRLP/wJVJIFcmMJeEliFw3zJ+c6urFzhFbDBrToGBAXbPykkH8hZXQzmV8T5PwuOV ZTYQ445OhWUCveVh+i5qqcAGuex8IQZuctk9F/8j9ZQF2vUTFLPiSFOGi51pyiuqTTet 1cWjwGBAO9noeYyjswJyg79dR8Ck0twVz5IdcPGwpVR02yX8Xg1W8CbhBOQe0PhHS1mv Occg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WjCWNWtT0pxxhAlETV2zIWjv0Gl4p3lxrm3/RhLgkFI=; b=Kj1w8WhXtNXYcG9dqVWVN5IDLZChZ0umdoiQgs65dklQ5GMU+U/h/KyOdwBx8FWY/n ysNJNPWcI2x0qQGKGHXssxqs5B5dWOMuenh1krlKR4UT9450aEZNQqRNe+PNg5u/NvW3 MOyWfuNpQgcpJrk8tprY5gxaeyKkaSq2N1xLt9aNYynIpoOJhrpzLw7d9BGhb+yyXen4 Y/okZvOUCQb5IBqJ0P5wHEz+eTtxzXqP3fs7x0F8Xz8yBLbqj4J6YEj2yBYrtzdHr19m Rc2x9O6MAYAhNM7XBFraTJIGuumZjDzWToHmg35Crm6mXT5HwmShe6Lf0jYJY6pDp9L4 YUbg== X-Gm-Message-State: AOAM530R5G7kGY/uy+luW1xVTFsM+lWaF9UfMwRG7na5gCREBnK4nc29 798ORNbnMYF8zlHZbtlTtiVvOuS5iVUoBw== X-Google-Smtp-Source: ABdhPJzqcaYYDKlp3sUAGpXDR3SSEmVgbvV1MQeGqx85pwWP5dg6r1K9s1IWZLx1A1HO79RYXirMpQ== X-Received: by 2002:a62:760b:0:b029:1b6:3897:3f86 with SMTP id r11-20020a62760b0000b02901b638973f86mr3803216pfc.24.1612372894244; Wed, 03 Feb 2021 09:21:34 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id x21sm2368636pfn.100.2021.02.03.09.21.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 09:21:32 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v6 PATCH 11/11] mm: vmscan: shrink deferred objects proportional to priority Date: Wed, 3 Feb 2021 09:20:42 -0800 Message-Id: <20210203172042.800474-12-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210203172042.800474-1-shy828301@gmail.com> References: <20210203172042.800474-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The number of deferred objects might get windup to an absurd number, and it results in clamp of slab objects. It is undesirable for sustaining workingset. So shrink deferred objects proportional to priority and cap nr_deferred to twice of cache items. The idea is borrowed fron Dave Chinner's patch: https://lore.kernel.org/linux-xfs/20191031234618.15403-13-david@fromorbit.com/ Tested with kernel build and vfs metadata heavy workload in our production environment, no regression is spotted so far. Signed-off-by: Yang Shi --- mm/vmscan.c | 40 +++++----------------------------------- 1 file changed, 5 insertions(+), 35 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 574d920c4cab..d0a86170854b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -649,7 +649,6 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, */ nr = count_nr_deferred(shrinker, shrinkctl); - total_scan = nr; if (shrinker->seeks) { delta = freeable >> priority; delta *= 4; @@ -663,37 +662,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, delta = freeable / 2; } + total_scan = nr >> priority; total_scan += delta; - if (total_scan < 0) { - pr_err("shrink_slab: %pS negative objects to delete nr=%ld\n", - shrinker->scan_objects, total_scan); - total_scan = freeable; - next_deferred = nr; - } else - next_deferred = total_scan; - - /* - * We need to avoid excessive windup on filesystem shrinkers - * due to large numbers of GFP_NOFS allocations causing the - * shrinkers to return -1 all the time. This results in a large - * nr being built up so when a shrink that can do some work - * comes along it empties the entire cache due to nr >>> - * freeable. This is bad for sustaining a working set in - * memory. - * - * Hence only allow the shrinker to scan the entire cache when - * a large delta change is calculated directly. - */ - if (delta < freeable / 4) - total_scan = min(total_scan, freeable / 2); - - /* - * Avoid risking looping forever due to too large nr value: - * never try to free more than twice the estimate number of - * freeable entries. - */ - if (total_scan > freeable * 2) - total_scan = freeable * 2; + total_scan = min(total_scan, (2 * freeable)); trace_mm_shrink_slab_start(shrinker, shrinkctl, nr, freeable, delta, total_scan, priority); @@ -732,10 +703,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, cond_resched(); } - if (next_deferred >= scanned) - next_deferred -= scanned; - else - next_deferred = 0; + next_deferred = max_t(long, (nr - scanned), 0) + total_scan; + next_deferred = min(next_deferred, (2 * freeable)); + /* * move the unused scan count back into the shrinker in a * manner that handles concurrent updates.