From patchwork Thu Mar 11 19:08:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 198B9C433E6 for ; Thu, 11 Mar 2021 19:09:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 94A3464EBB for ; Thu, 11 Mar 2021 19:09:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 94A3464EBB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C93528D02E5; Thu, 11 Mar 2021 14:09:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BF74C8D02B2; Thu, 11 Mar 2021 14:09:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A20C28D02E5; Thu, 11 Mar 2021 14:09:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0073.hostedemail.com [216.40.44.73]) by kanga.kvack.org (Postfix) with ESMTP id 79BE08D02B2 for ; Thu, 11 Mar 2021 14:09:03 -0500 (EST) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 32AB9180ACF9A for ; Thu, 11 Mar 2021 19:09:03 +0000 (UTC) X-FDA: 77908530966.10.19EBCA8 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf13.hostedemail.com (Postfix) with ESMTP id 6A9B3E005F2E for ; Thu, 11 Mar 2021 19:09:00 +0000 (UTC) Received: by mail-pf1-f177.google.com with SMTP id y67so204916pfb.2 for ; Thu, 11 Mar 2021 11:09:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ad6rFdrjF9qkUCg0uueBGRPkzrqRpMOaGc3NS4qxCRQ=; b=vKEaDb0pRFGw1ac2fSVbByJ/bryyv9m8N66XhwfI1mxAIXCGvTtnVNSRknaDGOpu3r mF3ky3Se8SjadopZ5vbnZFBEAtxE+6nQMUWUQsIiqZYLtnIoCW76v532jRs1CJv6MJW0 UXSaDvB/qYPV0hTsi4Ofsshg6au5Kmv82jo+dQ9t7h1iXScyhgymtcd1OkWkMN1oO7SX fzGehcR1dmA2UtgSb0aaSuzWHC9mBk6khdIF8MdEBsDZVrYgl43eE3Y1jm2kG4jfVMzH /1raUVFgjAKf1VVMxt/sL6d7NmzzBfCWRM/JIwTbu05KOem0EBBq+foWfz4PzbZ1OoV6 z09w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ad6rFdrjF9qkUCg0uueBGRPkzrqRpMOaGc3NS4qxCRQ=; b=YxqTSqZoKqyUqIRL4pRk4k+oqoS5V7ccZXE9szfWy1jSOe9fYhuP/YsPk0PVWGjUDj lvMqd45wFWj/6HupSe35i7dC1W9kE1an1QJtI+Ee7jOvxRve1XQkkoOCIFAmD7Z3kgGA 1Zp1J7jPJ1co8S5Rwuj0ggXBcOsPykOGSGtFeiOG0uiUr1qpuVxreSe9oHTBYEvSioR2 4nbUeMfs8nTO5dSBIs7dABXhBUvXAoBWJSlwIQllpYYCVMG/Q7tH20jIe3lOFTAv7eAO +Ia+aJBpVrs6v/SN2Acue5FL/pQ15XiHF4UsFoqLZsaX8zkZI9+0t27ZFHrGZjP4B2Br gRaA== X-Gm-Message-State: AOAM532R0LEoedf/h/GTz1ZkdM+vXaZ3aJaHGT7w1sdXmnDHAawVQBse 8SKgiDR6WUkI+4+1qzJjod8= X-Google-Smtp-Source: ABdhPJzkCuz0y/v1wgMPEFCdMw3j4bSNGMz2Ido7wOOTliNLwGqxRrstshQGOD0XEXWudM+Ej+Xhfg== X-Received: by 2002:a63:d601:: with SMTP id q1mr8342915pgg.417.1615489741567; Thu, 11 Mar 2021 11:09:01 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.08.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:00 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 01/13] mm: vmscan: use nid from shrink_control for tracepoint Date: Thu, 11 Mar 2021 11:08:33 -0800 Message-Id: <20210311190845.9708-2-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 3dkstfq15rrojnhxwgudoqwe5iajhguc X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 6A9B3E005F2E Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf13; identity=mailfrom; envelope-from=""; helo=mail-pf1-f177.google.com; client-ip=209.85.210.177 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615489740-32724 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The tracepoint's nid should show what node the shrink happens on, the start tracepoint uses nid from shrinkctl, but the nid might be set to 0 before end tracepoint if the shrinker is not NUMA aware, so the tracing log may show the shrink happens on one node but end up on the other node. It seems confusing. And the following patch will remove using nid directly in do_shrink_slab(), this patch also helps cleanup the code. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Reviewed-by: Shakeel Butt Acked-by: Roman Gushchin Signed-off-by: Yang Shi --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 407051ebe869..bdc32c803c66 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -536,7 +536,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, else new_nr = atomic_long_read(&shrinker->nr_deferred[nid]); - trace_mm_shrink_slab_end(shrinker, nid, freed, nr, new_nr, total_scan); + trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); return freed; } From patchwork Thu Mar 11 19:08:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132519 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69519C4332D for ; Thu, 11 Mar 2021 19:09:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C3AFA64EBB for ; Thu, 11 Mar 2021 19:09:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C3AFA64EBB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 566CC8D02E6; Thu, 11 Mar 2021 14:09:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A1378D02B2; Thu, 11 Mar 2021 14:09:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 281018D02E6; Thu, 11 Mar 2021 14:09:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0177.hostedemail.com [216.40.44.177]) by kanga.kvack.org (Postfix) with ESMTP id 0248D8D02B2 for ; Thu, 11 Mar 2021 14:09:05 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id A21D2181AF5CC for ; Thu, 11 Mar 2021 19:09:05 +0000 (UTC) X-FDA: 77908531050.01.085AE39 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf19.hostedemail.com (Postfix) with ESMTP id ECB549001441 for ; Thu, 11 Mar 2021 19:09:02 +0000 (UTC) Received: by mail-pg1-f177.google.com with SMTP id t37so3486588pga.11 for ; Thu, 11 Mar 2021 11:09:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SXVD1HfpEWEnPEpQNQZ2lptmmkG469kO8T607Rced4s=; b=EUuo4QwlFAcOak5Bd6ymMS/yjW6KIitnzDmweZ+6TMFTs28kzMP153UsCSwNHtB8GN ugv5lyBUhhNlU83j1V7JpEQoliqop1Q/rAAYy83B4rbWQf0A+Gn3m8NEWbY0op5Av6zB 4w0trsxiK0JWamO+YOPbZt7TEopBi9efsgdQL4bnapW6XkuA9B+YkpFG9H+oMr+kBEEI /lWkv23t+88xGcpnNJ9+rAYmEhY3qa9LvPx7MKG0EGzga6YZDoYnmrFEndeiBeXzbyOJ k1tM+ljirBNAuBvPLvi/eJbFzKWfBZXrdqzYGN7KSOuqooAyI/M4+mhgNzCXSSmZuVOS WyaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SXVD1HfpEWEnPEpQNQZ2lptmmkG469kO8T607Rced4s=; b=iNR77VduIz+1KQVkHN2ZSc77lmXtTHy1aWsKadic/3z7uxSYqKeCB1elqK229UYRp9 B8g+7DZ0I4/K1bXvPpzjRqCVhOEwQ6RSc4V4XXFnhEG867BJeqYdxQ5B7UxQ2FBRihsz 5xzzX37OLZ5GRe02KtKgqLKEpcsvrm+obZQDI8Ch10kv6gppsYTkiriUVSZ2HpTgH3tC RjW5p1AHjKGpeS0zHCA8BX6VgoyHCPsPi4vcGOEmVxvJEtk3ISINFuxKhwv0O0DlBFFy blI8b1Fy1kOsyeCZd2U1KRaLe/eNx98bkUTfzK7wCzejajPrH73DIzaC9FqZOi+JINmz 48lg== X-Gm-Message-State: AOAM531q6SuGiYk6tr6NQQozaaxVtEXVW/xnhgE4rTvM/ljqECqkNCjJ JTD7KnBQwjbSTcyqQwf8m5CvCk0IOP/xVg== X-Google-Smtp-Source: ABdhPJx98HcHkFsDZbrn39hydWs6DZ5rXIRgUHXoKJBLC01b9UOlPPllqcquB2rNWDllsSt1a3aSEw== X-Received: by 2002:aa7:8498:0:b029:1f6:8a25:7ade with SMTP id u24-20020aa784980000b02901f68a257ademr8837086pfn.76.1615489743876; Thu, 11 Mar 2021 11:09:03 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:02 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 02/13] mm: vmscan: consolidate shrinker_maps handling code Date: Thu, 11 Mar 2021 11:08:34 -0800 Message-Id: <20210311190845.9708-3-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: y3ejkg4u6r1ud1yis88cht4heweb5bxr X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: ECB549001441 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf19; identity=mailfrom; envelope-from=""; helo=mail-pg1-f177.google.com; client-ip=209.85.215.177 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615489742-443040 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shrinker map management is not purely memcg specific, it is at the intersection between memory cgroup and shrinkers. It's allocation and assignment of a structure, and the only memcg bit is the map is being stored in a memcg structure. So move the shrinker_maps handling code into vmscan.c for tighter integration with shrinker code, and remove the "memcg_" prefix. There is no functional change. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 11 ++-- mm/huge_memory.c | 4 +- mm/list_lru.c | 6 +- mm/memcontrol.c | 130 +----------------------------------- mm/vmscan.c | 132 ++++++++++++++++++++++++++++++++++++- 5 files changed, 142 insertions(+), 141 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e946c96daa32..b07dc2d5014d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1533,10 +1533,9 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -extern int memcg_expand_shrinker_maps(int new_id); - -extern void memcg_set_shrinker_bit(struct mem_cgroup *memcg, - int nid, int shrinker_id); +int alloc_shrinker_maps(struct mem_cgroup *memcg); +void free_shrinker_maps(struct mem_cgroup *memcg); +void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; @@ -1546,8 +1545,8 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -static inline void memcg_set_shrinker_bit(struct mem_cgroup *memcg, - int nid, int shrinker_id) +static inline void set_shrinker_bit(struct mem_cgroup *memcg, + int nid, int shrinker_id) { } #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index bff92dea5ab3..92a1227be029 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2836,8 +2836,8 @@ void deferred_split_huge_page(struct page *page) ds_queue->split_queue_len++; #ifdef CONFIG_MEMCG if (memcg) - memcg_set_shrinker_bit(memcg, page_to_nid(page), - deferred_split_shrinker.id); + set_shrinker_bit(memcg, page_to_nid(page), + deferred_split_shrinker.id); #endif } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); diff --git a/mm/list_lru.c b/mm/list_lru.c index 6f067b6b935f..cd58790d0fb3 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -125,8 +125,8 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item) list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) - memcg_set_shrinker_bit(memcg, nid, - lru_shrinker_id(lru)); + set_shrinker_bit(memcg, nid, + lru_shrinker_id(lru)); nlru->nr_items++; spin_unlock(&nlru->lock); return true; @@ -540,7 +540,7 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, if (src->nr_items) { dst->nr_items += src->nr_items; - memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); + set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index da6fded852df..aeb1847f159c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -402,130 +402,6 @@ DEFINE_STATIC_KEY_FALSE(memcg_kmem_enabled_key); EXPORT_SYMBOL(memcg_kmem_enabled_key); #endif -static int memcg_shrinker_map_size; -static DEFINE_MUTEX(memcg_shrinker_map_mutex); - -static void memcg_free_shrinker_map_rcu(struct rcu_head *head) -{ - kvfree(container_of(head, struct memcg_shrinker_map, rcu)); -} - -static int memcg_expand_one_shrinker_map(struct mem_cgroup *memcg, - int size, int old_size) -{ - struct memcg_shrinker_map *new, *old; - struct mem_cgroup_per_node *pn; - int nid; - - lockdep_assert_held(&memcg_shrinker_map_mutex); - - for_each_node(nid) { - pn = memcg->nodeinfo[nid]; - old = rcu_dereference_protected(pn->shrinker_map, true); - /* Not yet online memcg */ - if (!old) - return 0; - - new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); - if (!new) - return -ENOMEM; - - /* Set all old bits, clear all new bits */ - memset(new->map, (int)0xff, old_size); - memset((void *)new->map + old_size, 0, size - old_size); - - rcu_assign_pointer(pn->shrinker_map, new); - call_rcu(&old->rcu, memcg_free_shrinker_map_rcu); - } - - return 0; -} - -static void memcg_free_shrinker_maps(struct mem_cgroup *memcg) -{ - struct mem_cgroup_per_node *pn; - struct memcg_shrinker_map *map; - int nid; - - if (mem_cgroup_is_root(memcg)) - return; - - for_each_node(nid) { - pn = memcg->nodeinfo[nid]; - map = rcu_dereference_protected(pn->shrinker_map, true); - kvfree(map); - rcu_assign_pointer(pn->shrinker_map, NULL); - } -} - -static int memcg_alloc_shrinker_maps(struct mem_cgroup *memcg) -{ - struct memcg_shrinker_map *map; - int nid, size, ret = 0; - - if (mem_cgroup_is_root(memcg)) - return 0; - - mutex_lock(&memcg_shrinker_map_mutex); - size = memcg_shrinker_map_size; - for_each_node(nid) { - map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); - if (!map) { - memcg_free_shrinker_maps(memcg); - ret = -ENOMEM; - break; - } - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); - } - mutex_unlock(&memcg_shrinker_map_mutex); - - return ret; -} - -int memcg_expand_shrinker_maps(int new_id) -{ - int size, old_size, ret = 0; - struct mem_cgroup *memcg; - - size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); - old_size = memcg_shrinker_map_size; - if (size <= old_size) - return 0; - - mutex_lock(&memcg_shrinker_map_mutex); - if (!root_mem_cgroup) - goto unlock; - - for_each_mem_cgroup(memcg) { - if (mem_cgroup_is_root(memcg)) - continue; - ret = memcg_expand_one_shrinker_map(memcg, size, old_size); - if (ret) { - mem_cgroup_iter_break(NULL, memcg); - goto unlock; - } - } -unlock: - if (!ret) - memcg_shrinker_map_size = size; - mutex_unlock(&memcg_shrinker_map_mutex); - return ret; -} - -void memcg_set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) -{ - if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { - struct memcg_shrinker_map *map; - - rcu_read_lock(); - map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, map->map); - rcu_read_unlock(); - } -} - /** * mem_cgroup_css_from_page - css of the memcg associated with a page * @page: page of interest @@ -5255,11 +5131,11 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) struct mem_cgroup *memcg = mem_cgroup_from_css(css); /* - * A memcg must be visible for memcg_expand_shrinker_maps() + * A memcg must be visible for expand_shrinker_maps() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (memcg_alloc_shrinker_maps(memcg)) { + if (alloc_shrinker_maps(memcg)) { mem_cgroup_id_remove(memcg); return -ENOMEM; } @@ -5323,7 +5199,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) vmpressure_cleanup(&memcg->vmpressure); cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); - memcg_free_shrinker_maps(memcg); + free_shrinker_maps(memcg); memcg_free_kmem(memcg); mem_cgroup_free(memcg); } diff --git a/mm/vmscan.c b/mm/vmscan.c index bdc32c803c66..98a44fb81f8a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -185,6 +185,132 @@ static LIST_HEAD(shrinker_list); static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG + +static int memcg_shrinker_map_size; +static DEFINE_MUTEX(memcg_shrinker_map_mutex); + +static void free_shrinker_map_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct memcg_shrinker_map, rcu)); +} + +static int expand_one_shrinker_map(struct mem_cgroup *memcg, + int size, int old_size) +{ + struct memcg_shrinker_map *new, *old; + struct mem_cgroup_per_node *pn; + int nid; + + lockdep_assert_held(&memcg_shrinker_map_mutex); + + for_each_node(nid) { + pn = memcg->nodeinfo[nid]; + old = rcu_dereference_protected(pn->shrinker_map, true); + /* Not yet online memcg */ + if (!old) + return 0; + + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); + if (!new) + return -ENOMEM; + + /* Set all old bits, clear all new bits */ + memset(new->map, (int)0xff, old_size); + memset((void *)new->map + old_size, 0, size - old_size); + + rcu_assign_pointer(pn->shrinker_map, new); + call_rcu(&old->rcu, free_shrinker_map_rcu); + } + + return 0; +} + +void free_shrinker_maps(struct mem_cgroup *memcg) +{ + struct mem_cgroup_per_node *pn; + struct memcg_shrinker_map *map; + int nid; + + if (mem_cgroup_is_root(memcg)) + return; + + for_each_node(nid) { + pn = memcg->nodeinfo[nid]; + map = rcu_dereference_protected(pn->shrinker_map, true); + kvfree(map); + rcu_assign_pointer(pn->shrinker_map, NULL); + } +} + +int alloc_shrinker_maps(struct mem_cgroup *memcg) +{ + struct memcg_shrinker_map *map; + int nid, size, ret = 0; + + if (mem_cgroup_is_root(memcg)) + return 0; + + mutex_lock(&memcg_shrinker_map_mutex); + size = memcg_shrinker_map_size; + for_each_node(nid) { + map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); + if (!map) { + free_shrinker_maps(memcg); + ret = -ENOMEM; + break; + } + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); + } + mutex_unlock(&memcg_shrinker_map_mutex); + + return ret; +} + +static int expand_shrinker_maps(int new_id) +{ + int size, old_size, ret = 0; + struct mem_cgroup *memcg; + + size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); + old_size = memcg_shrinker_map_size; + if (size <= old_size) + return 0; + + mutex_lock(&memcg_shrinker_map_mutex); + if (!root_mem_cgroup) + goto unlock; + + memcg = mem_cgroup_iter(NULL, NULL, NULL); + do { + if (mem_cgroup_is_root(memcg)) + continue; + ret = expand_one_shrinker_map(memcg, size, old_size); + if (ret) { + mem_cgroup_iter_break(NULL, memcg); + goto unlock; + } + } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); +unlock: + if (!ret) + memcg_shrinker_map_size = size; + mutex_unlock(&memcg_shrinker_map_mutex); + return ret; +} + +void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) +{ + if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { + struct memcg_shrinker_map *map; + + rcu_read_lock(); + map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, map->map); + rcu_read_unlock(); + } +} + /* * We allow subsystems to populate their shrinker-related * LRU lists before register_shrinker_prepared() is called @@ -212,7 +338,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) goto unlock; if (id >= shrinker_nr_max) { - if (memcg_expand_shrinker_maps(id)) { + if (expand_shrinker_maps(id)) { idr_remove(&shrinker_idr, id); goto unlock; } @@ -590,7 +716,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, * case, we invoke the shrinker one more time and reset * the bit if it reports that it is not empty anymore. * The memory barrier here pairs with the barrier in - * memcg_set_shrinker_bit(): + * set_shrinker_bit(): * * list_lru_add() shrink_slab_memcg() * list_add_tail() clear_bit() @@ -602,7 +728,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; else - memcg_set_shrinker_bit(memcg, nid, i); + set_shrinker_bit(memcg, nid, i); } freed += ret; From patchwork Thu Mar 11 19:08:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED302C433E0 for ; Thu, 11 Mar 2021 19:09:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7B51564EC0 for ; Thu, 11 Mar 2021 19:09:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7B51564EC0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2A9C48D02E7; Thu, 11 Mar 2021 14:09:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2592E8D02B2; Thu, 11 Mar 2021 14:09:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0AFBD8D02E7; Thu, 11 Mar 2021 14:09:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0128.hostedemail.com [216.40.44.128]) by kanga.kvack.org (Postfix) with ESMTP id D3C928D02B2 for ; Thu, 11 Mar 2021 14:09:07 -0500 (EST) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 813551730873 for ; Thu, 11 Mar 2021 19:09:07 +0000 (UTC) X-FDA: 77908531134.04.184134C Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by imf09.hostedemail.com (Postfix) with ESMTP id B2DC86000110 for ; Thu, 11 Mar 2021 19:09:04 +0000 (UTC) Received: by mail-pg1-f169.google.com with SMTP id v14so7357552pgq.2 for ; Thu, 11 Mar 2021 11:09:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=i2D+FuQfaS2ZRj4Qwe7N96xecJSI2gtUN3o6WXvG0vk=; b=q1nH2RikMTupiiFtmscgQ+W70fe3sDorZ4fRkLKO0XRwx+fzc24172LPMv2hziIcIF Sj0Gme3WOUqsQaMT129Zo5hCy7cm+AqdnoYV2Cuh9FuQW3Y2CCm0po075aPxccdcP7MH JQtHynViMEbBUKTFxHGjdWkTdjffgcUZdS9/R5dL+UTxEWcdWqze3uwdwYU8l/dGDHKq aJVp+6mo8dpTwNLt9NTxXBqI6czJdjPZbxuZ/IW1dEL8UGxEfeQs6DOWGB9+U/ModK9y xS9MFPQFUcmim0HU0A0XnS0WKcK5jKf7wSEN3pe5Dq2HVn+Qn8xBGNHIO3XjWMEIbKDG IwiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=i2D+FuQfaS2ZRj4Qwe7N96xecJSI2gtUN3o6WXvG0vk=; b=ayOCDDCe7cmjio5lxK2eAF2NXjIStyH62g6x5+20ZvyMU3lWNiLET32O6Nbwg7Jc+t cB2UlAj3l6yWNTO8TpOCkGIaGfXQOE/Fl0PzwybiEGeFYmEdgQ9xDiVFbrOSls27ClaB RRujCoirUJPFbIXgfjbPqRhTUjj4xVTld4ASyD9HOx24a6t0V6rc648bcLu6aPY+tbT+ EV/8OAIs+59EQw7MR69Khtp495mfCZzfHIEy6V93UWBAz3qU815bkKCNpZqgWISRI06M PvWyw/yNKxW9BMwF9XcMp7B6OZWquiq0vhDkgmGyw3PqCApWx/ZWBvV0HyYGfCoYqb5E MqJQ== X-Gm-Message-State: AOAM53384TMHkLJTw5q3v8XfF53jFtm82BgGQ/OyPmLc52Ga/dwxAf0+ AyetWc5fxaoLj8jB98BxPgY= X-Google-Smtp-Source: ABdhPJxrXzkIllUs4SsPxfmE/PuCtdbdraco1NMsd2Am1G7MNDDRxNRwRr7Wjx+o2aWZ6a4lBHN1eg== X-Received: by 2002:aa7:9431:0:b029:1f1:52fd:5444 with SMTP id y17-20020aa794310000b02901f152fd5444mr8525393pfo.47.1615489745932; Thu, 11 Mar 2021 11:09:05 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:05 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 03/13] mm: vmscan: use shrinker_rwsem to protect shrinker_maps allocation Date: Thu, 11 Mar 2021 11:08:35 -0800 Message-Id: <20210311190845.9708-4-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 157ekzy47nkn753znybarjsdow34hceu X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: B2DC86000110 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf09; identity=mailfrom; envelope-from=""; helo=mail-pg1-f169.google.com; client-ip=209.85.215.169 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615489744-329913 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since memcg_shrinker_map_size just can be changed under holding shrinker_rwsem exclusively, the read side can be protected by holding read lock, so it sounds superfluous to have a dedicated mutex. Kirill Tkhai suggested use write lock since: * We want the assignment to shrinker_maps is visible for shrink_slab_memcg(). * The rcu_dereference_protected() dereferrencing in shrink_slab_memcg(), but in case of we use READ lock in alloc_shrinker_maps(), the dereferrencing is not actually protected. * READ lock makes alloc_shrinker_info() racy against memory allocation fail. alloc_shrinker_info()->free_shrinker_info() may free memory right after shrink_slab_memcg() dereferenced it. You may say shrink_slab_memcg()->mem_cgroup_online() protects us from it? Yes, sure, but this is not the thing we want to remember in the future, since this spreads modularity. And a test with heavy paging workload didn't show write lock makes things worse. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 98a44fb81f8a..b08c8d9055ae 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -187,7 +187,6 @@ static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG static int memcg_shrinker_map_size; -static DEFINE_MUTEX(memcg_shrinker_map_mutex); static void free_shrinker_map_rcu(struct rcu_head *head) { @@ -201,8 +200,6 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, struct mem_cgroup_per_node *pn; int nid; - lockdep_assert_held(&memcg_shrinker_map_mutex); - for_each_node(nid) { pn = memcg->nodeinfo[nid]; old = rcu_dereference_protected(pn->shrinker_map, true); @@ -250,7 +247,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) if (mem_cgroup_is_root(memcg)) return 0; - mutex_lock(&memcg_shrinker_map_mutex); + down_write(&shrinker_rwsem); size = memcg_shrinker_map_size; for_each_node(nid) { map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); @@ -261,7 +258,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) } rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); } - mutex_unlock(&memcg_shrinker_map_mutex); + up_write(&shrinker_rwsem); return ret; } @@ -276,9 +273,10 @@ static int expand_shrinker_maps(int new_id) if (size <= old_size) return 0; - mutex_lock(&memcg_shrinker_map_mutex); if (!root_mem_cgroup) - goto unlock; + goto out; + + lockdep_assert_held(&shrinker_rwsem); memcg = mem_cgroup_iter(NULL, NULL, NULL); do { @@ -287,13 +285,13 @@ static int expand_shrinker_maps(int new_id) ret = expand_one_shrinker_map(memcg, size, old_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); - goto unlock; + goto out; } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); -unlock: +out: if (!ret) memcg_shrinker_map_size = size; - mutex_unlock(&memcg_shrinker_map_mutex); + return ret; } From patchwork Thu Mar 11 19:08:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76A6CC433E0 for ; Thu, 11 Mar 2021 19:09:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F3A4164EFE for ; Thu, 11 Mar 2021 19:09:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F3A4164EFE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2E1738D02E8; Thu, 11 Mar 2021 14:09:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 27EB08D02B2; Thu, 11 Mar 2021 14:09:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05B7E8D02E8; Thu, 11 Mar 2021 14:09:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0181.hostedemail.com [216.40.44.181]) by kanga.kvack.org (Postfix) with ESMTP id CE3528D02B2 for ; Thu, 11 Mar 2021 14:09:10 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 79A768249980 for ; Thu, 11 Mar 2021 19:09:10 +0000 (UTC) X-FDA: 77908531260.18.C593500 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf19.hostedemail.com (Postfix) with ESMTP id EDB3B90009ED for ; Thu, 11 Mar 2021 19:09:06 +0000 (UTC) Received: by mail-pl1-f170.google.com with SMTP id a8so4178535plp.13 for ; Thu, 11 Mar 2021 11:09:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=x5MgFJmx+qCDHDBM5m9Z4h2MfyiNvh6Fg7+MEJeG5Hc=; b=hXphMEKp+eXWbBLvMNGFTgWa2pZEr+COjecriAAxFhNEeooa/BbyJ/YAUkkA5wytZg 1AOGSEuDCvR6ZGXii+4K3P587qy73WiAQZneeJ7C0uD7Z0mWY+a/BVm80vwEktHPLVTL h0JA6lJ4dixnOPuJ+vtVuF6X3lHth7AamKG6/i7xTYqmvTrxQhTaXeydTKIhQQ8uyErC cARoNTMaCQIoEqXH/PxRgv7SeHSbS16T9+VEylrr+clF2S7LDC5E05N64ru7HbzhA4pX 0jtq6H5/L4hmEqQHSIWq/c/kIuUfTpqG0Y+azFNdJvwnwW6cfCDMFHKjSJgtsN4MUgSl ZsCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x5MgFJmx+qCDHDBM5m9Z4h2MfyiNvh6Fg7+MEJeG5Hc=; b=sSEk98vA32Oqrt3SFcV8CVNfZiERNMYZS0oe9NkYezSZiy1KzC9EmtpDJcYHIO9Pk5 Om+eZXG3fYhrr7IlQqUoIF+4chAlaG52Ak4E0Mu17mdQpExuIe8j6/Wzw/EvsWKDoiZR jhf9GUkRICUmvw4K+3eDTjr4GQNh39AuMD27FONZ1yuw/QS31Eo4Rk2G/QVQeWoEUiij QQhlTLwqQU4oaP8mai4zUgjYGrKs9xJk5qRj2oaqXJP7u9SFKM745X1OfV8h61YoQeSL ULCwPmAfW1NMBHzGjVV5Tg8QUPimImMUyQ7ez5euwtU468bvAX56LPWLmsmN5tuf36lM YuWQ== X-Gm-Message-State: AOAM531RSNAIEerQYMN5BubEE8BRs/QV8/tVisZPQUWBFaxqGRNqGirU 1oIsXsjczXZ56rhtvAK8Be0= X-Google-Smtp-Source: ABdhPJxjyKooM+K/uNd8RQ5pj9o2KzidC1fcsHNXVoUBXa8795yLlGRklotc+LNSeYmurtCbsO8ZbQ== X-Received: by 2002:a17:90b:3553:: with SMTP id lt19mr4840931pjb.222.1615489747922; Thu, 11 Mar 2021 11:09:07 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:07 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 04/13] mm: vmscan: remove memcg_shrinker_map_size Date: Thu, 11 Mar 2021 11:08:36 -0800 Message-Id: <20210311190845.9708-5-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: EDB3B90009ED X-Stat-Signature: fecfp7zuu18z715iys5nhz88yocadgp6 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf19; identity=mailfrom; envelope-from=""; helo=mail-pl1-f170.google.com; client-ip=209.85.214.170 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615489746-623925 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Both memcg_shrinker_map_size and shrinker_nr_max is maintained, but actually the map size can be calculated via shrinker_nr_max, so it seems unnecessary to keep both. Remove memcg_shrinker_map_size since shrinker_nr_max is also used by iterating the bit map. Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Acked-by: Vlastimil Babka Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b08c8d9055ae..641a0b8b4ea9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -185,8 +185,12 @@ static LIST_HEAD(shrinker_list); static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG +static int shrinker_nr_max; -static int memcg_shrinker_map_size; +static inline int shrinker_map_size(int nr_items) +{ + return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); +} static void free_shrinker_map_rcu(struct rcu_head *head) { @@ -248,7 +252,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) return 0; down_write(&shrinker_rwsem); - size = memcg_shrinker_map_size; + size = shrinker_map_size(shrinker_nr_max); for_each_node(nid) { map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); if (!map) { @@ -266,12 +270,13 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) static int expand_shrinker_maps(int new_id) { int size, old_size, ret = 0; + int new_nr_max = new_id + 1; struct mem_cgroup *memcg; - size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); - old_size = memcg_shrinker_map_size; + size = shrinker_map_size(new_nr_max); + old_size = shrinker_map_size(shrinker_nr_max); if (size <= old_size) - return 0; + goto out; if (!root_mem_cgroup) goto out; @@ -290,7 +295,7 @@ static int expand_shrinker_maps(int new_id) } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); out: if (!ret) - memcg_shrinker_map_size = size; + shrinker_nr_max = new_nr_max; return ret; } @@ -323,7 +328,6 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) #define SHRINKER_REGISTERING ((struct shrinker *)~0UL) static DEFINE_IDR(shrinker_idr); -static int shrinker_nr_max; static int prealloc_memcg_shrinker(struct shrinker *shrinker) { @@ -340,8 +344,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) idr_remove(&shrinker_idr, id); goto unlock; } - - shrinker_nr_max = id + 1; } shrinker->id = id; ret = 0; From patchwork Thu Mar 11 19:08:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 843FBC433DB for ; Thu, 11 Mar 2021 19:09:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1493064EBB for ; Thu, 11 Mar 2021 19:09:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1493064EBB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1E4108D02EB; Thu, 11 Mar 2021 14:09:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 172EC8D02B2; Thu, 11 Mar 2021 14:09:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB4E28D02EB; Thu, 11 Mar 2021 14:09:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0213.hostedemail.com [216.40.44.213]) by kanga.kvack.org (Postfix) with ESMTP id C545C8D02B2 for ; Thu, 11 Mar 2021 14:09:16 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 8A4162C1E for ; Thu, 11 Mar 2021 19:09:16 +0000 (UTC) X-FDA: 77908531512.16.B6C08FE Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf03.hostedemail.com (Postfix) with ESMTP id B30C5C0007D2 for ; Thu, 11 Mar 2021 19:09:08 +0000 (UTC) Received: by mail-pj1-f51.google.com with SMTP id mz6-20020a17090b3786b02900c16cb41d63so9831522pjb.2 for ; Thu, 11 Mar 2021 11:09:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uVGBdCD7D46KwnzDeHFbJFDsSaALpjrd7Gd+gGd1rXE=; b=LeRQjwnw2LeCRF7BgliFeq65y5FqMfUxpySNgLPB8dqZdAbhQXJ40f0eFMxW9ZtgGG E0PyIWo7ZNLhtfEW2BvloPIVyyCGRi+eJE5NS1W9uxh8vnnaGPDGH3tWoMIsh5z4AQTm hxSr5rUUO7S4wWeYBcp826RkRVHHQk6oSA3g706vUAtBpGUQVwgQoWexRVWjLYArFBiI OginCWSvf/ZVJjz1oYD6mFaQp/CuFxViPEO1jemusmUawvrw12sNUWHgWyw46IV8UU7q x08rY9hN+i8bF6OlEQF1ItrcygFnRzH2Q0KW/19CbdnCl0MlSTw+tZVmt9AczdOCx6xl KY3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uVGBdCD7D46KwnzDeHFbJFDsSaALpjrd7Gd+gGd1rXE=; b=Y+XyIxw1f/mwlq+LZxZBOVMwYV357u6o/8mTv7qNPybZ5xHw3joBD7DRDuZKra2sgK jTmkQKsw6W0uXk+ej2DMD9XF3UcPw+XezhkJhn+438ZYTvpN3f8z9rAR5es5R5YJ7pFR jByjKj4vSwMWnn7J/lO0JfgDFyQjgZRxwb+mp6DWaFAKPsA1QaE/ursKT/9IVgQMpG+9 SpdfcDYTgu5H7Miw2VDSreRY7lfWLDI5ey0bK6pY9NQm1j7VCB5CSYailvSMFBT5DubC BX7m9DvBpZlDJYR5deDlIS0o76Vn0FeIBZT1ja/N+WXbZ6i/KeS7WPNQ7IvSny3DfxeG rJjw== X-Gm-Message-State: AOAM533Ryuk1SYgm5CwjI+IgsMFuMDaycTk8+65mp1UePuedIR77jle4 qQedm3Pze/iIEcPs0JZvM4c= X-Google-Smtp-Source: ABdhPJzDjGg0JWVU4rrN0/JBFOTry3IxyIysSN54EZUh/VsTJfCbQjf4QVKvcXFG8sZ5SEY6RKN2Pg== X-Received: by 2002:a17:90a:540c:: with SMTP id z12mr9963872pjh.163.1615489750107; Thu, 11 Mar 2021 11:09:10 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:09 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 05/13] mm: vmscan: use kvfree_rcu instead of call_rcu Date: Thu, 11 Mar 2021 11:08:37 -0800 Message-Id: <20210311190845.9708-6-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: ezz9wzu6zpnfh9t7catrnchzb6efhsd4 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: B30C5C0007D2 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf03; identity=mailfrom; envelope-from=""; helo=mail-pj1-f51.google.com; client-ip=209.85.216.51 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615489748-780628 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Using kvfree_rcu() to free the old shrinker_maps instead of call_rcu(). We don't have to define a dedicated callback for call_rcu() anymore. Acked-by: Roman Gushchin Acked-by: Kirill Tkhai Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 641a0b8b4ea9..bbe13985ae05 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -192,11 +192,6 @@ static inline int shrinker_map_size(int nr_items) return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); } -static void free_shrinker_map_rcu(struct rcu_head *head) -{ - kvfree(container_of(head, struct memcg_shrinker_map, rcu)); -} - static int expand_one_shrinker_map(struct mem_cgroup *memcg, int size, int old_size) { @@ -220,7 +215,7 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, memset((void *)new->map + old_size, 0, size - old_size); rcu_assign_pointer(pn->shrinker_map, new); - call_rcu(&old->rcu, free_shrinker_map_rcu); + kvfree_rcu(old, rcu); } return 0; From patchwork Thu Mar 11 19:08:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F33EDC433E6 for ; Thu, 11 Mar 2021 19:09:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7140E64EF2 for ; Thu, 11 Mar 2021 19:09:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7140E64EF2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EDCE48D02E9; Thu, 11 Mar 2021 14:09:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E652B8D02B2; Thu, 11 Mar 2021 14:09:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE1D78D02E9; Thu, 11 Mar 2021 14:09:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0170.hostedemail.com [216.40.44.170]) by kanga.kvack.org (Postfix) with ESMTP id AB16F8D02B2 for ; Thu, 11 Mar 2021 14:09:14 -0500 (EST) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 665FF52A8 for ; Thu, 11 Mar 2021 19:09:14 +0000 (UTC) X-FDA: 77908531428.15.DEAF0DE Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf26.hostedemail.com (Postfix) with ESMTP id 5CEB24080F45 for ; Thu, 11 Mar 2021 19:09:10 +0000 (UTC) Received: by mail-pj1-f42.google.com with SMTP id a22-20020a17090aa516b02900c1215e9b33so9542268pjq.5 for ; Thu, 11 Mar 2021 11:09:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yTJ/H30mQn6R1SM8EhChDB9qQz/lBTytGLmfD9l2+II=; b=bKs5nQfvV9VNRtxx31OVFCDRjVIUA1MqSq0Qlg7NDlpuCeL1wAHmHtmIMAixLQz84t gPqu8nQa8Yo6lewYSj7868vMjMp7E13B28hlaGRZecAESSK19pvSf5/3tEEFZ3I5gnqN 83SHWyHA4YlXuoBJshYoOzb3nB8xQ0OG4UkP05L+mAPXiz8u+476k26xYw9lNzRiUO1u VHj5ADPOHJxzusnnpD0P5BPOKwkgBSyqYfgITS2LK6oTpYeiNZobE2yjjQwzYmqlOF1/ yCweqr+zHrLlL/Wv0Fi/rODH+MBcPWLvKCjaMJtgskIdVpAjdTfRvg7WrTBiFK0fSnvN 5X+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yTJ/H30mQn6R1SM8EhChDB9qQz/lBTytGLmfD9l2+II=; b=ojsYb45rlGOO70v1qjk5NTolZNImhHU4XeiKevlAl+nI70i/qO85sT3URVdtlalwwZ +3+XqKmQ8JuyDVK8BlUfyzAu1IXqv/D6j2DEOUdrpY2rucH4m9V5ARxemRFAnT9oytvY HampHG5A7hi5Y7WdNeWbuH55CcElh6dJHc3wzEUmnBVF+0SLICYpJIqpi6b7RLIgzZ4h acWdUioPl5T0FGsj/ygM+POdbdzVlZGJ43Z5IyHmvJ24ON7QLiwCRrzMJF9OL63vE/ld 9D3LAwC+V4uAT+7lNIyDHlpu5KEW5ebaRWZpuEO7ccCNAR1uQiOhzz7fHJLc1DKv3ybl cPKg== X-Gm-Message-State: AOAM531LAFs51iyrmMmoxrtLlwdJ5ovK3CKDjnt5KRm/SIITn/R52a6l HwPPpmqBx8oEnA69Vo0fq8Z+XPr0OVv0Zg== X-Google-Smtp-Source: ABdhPJwOllGQof/4c+nVC7dRgHvATP7IsRB2CncLFsjLly9r/Xyx9wDnDSbkHWVV8oVmalZuJixtNQ== X-Received: by 2002:a17:90a:a4cb:: with SMTP id l11mr10054163pjw.144.1615489752274; Thu, 11 Mar 2021 11:09:12 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:11 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 06/13] mm: memcontrol: rename shrinker_map to shrinker_info Date: Thu, 11 Mar 2021 11:08:38 -0800 Message-Id: <20210311190845.9708-7-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: gaicxpni5hfo5dyiarpwgyyzwawc8tir X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5CEB24080F45 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=mail-pj1-f42.google.com; client-ip=209.85.216.42 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615489750-853760 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The following patch is going to add nr_deferred into shrinker_map, the change will make shrinker_map not only include map anymore, so rename it to "memcg_shrinker_info". And this should make the patch adding nr_deferred cleaner and readable and make review easier. Also remove the "memcg_" prefix. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 8 +++--- mm/memcontrol.c | 6 ++-- mm/vmscan.c | 58 +++++++++++++++++++------------------- 3 files changed, 36 insertions(+), 36 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b07dc2d5014d..dc7d0e2cb3ad 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -117,7 +117,7 @@ struct batched_lruvec_stat { * Bitmap of shrinker::id corresponding to memcg-aware shrinkers, * which have elements charged to this memcg. */ -struct memcg_shrinker_map { +struct shrinker_info { struct rcu_head rcu; unsigned long map[]; }; @@ -145,7 +145,7 @@ struct mem_cgroup_per_node { struct mem_cgroup_reclaim_iter iter; - struct memcg_shrinker_map __rcu *shrinker_map; + struct shrinker_info __rcu *shrinker_info; struct rb_node tree_node; /* RB tree node */ unsigned long usage_in_excess;/* Set to the value by which */ @@ -1533,8 +1533,8 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -int alloc_shrinker_maps(struct mem_cgroup *memcg); -void free_shrinker_maps(struct mem_cgroup *memcg); +int alloc_shrinker_info(struct mem_cgroup *memcg); +void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); #else #define mem_cgroup_sockets_enabled 0 diff --git a/mm/memcontrol.c b/mm/memcontrol.c index aeb1847f159c..35d44afdd9fc 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5131,11 +5131,11 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) struct mem_cgroup *memcg = mem_cgroup_from_css(css); /* - * A memcg must be visible for expand_shrinker_maps() + * A memcg must be visible for expand_shrinker_info() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (alloc_shrinker_maps(memcg)) { + if (alloc_shrinker_info(memcg)) { mem_cgroup_id_remove(memcg); return -ENOMEM; } @@ -5199,7 +5199,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) vmpressure_cleanup(&memcg->vmpressure); cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); - free_shrinker_maps(memcg); + free_shrinker_info(memcg); memcg_free_kmem(memcg); mem_cgroup_free(memcg); } diff --git a/mm/vmscan.c b/mm/vmscan.c index bbe13985ae05..7fdfdacf9a1f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -192,16 +192,16 @@ static inline int shrinker_map_size(int nr_items) return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); } -static int expand_one_shrinker_map(struct mem_cgroup *memcg, - int size, int old_size) +static int expand_one_shrinker_info(struct mem_cgroup *memcg, + int size, int old_size) { - struct memcg_shrinker_map *new, *old; + struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; int nid; for_each_node(nid) { pn = memcg->nodeinfo[nid]; - old = rcu_dereference_protected(pn->shrinker_map, true); + old = rcu_dereference_protected(pn->shrinker_info, true); /* Not yet online memcg */ if (!old) return 0; @@ -214,17 +214,17 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, memset(new->map, (int)0xff, old_size); memset((void *)new->map + old_size, 0, size - old_size); - rcu_assign_pointer(pn->shrinker_map, new); + rcu_assign_pointer(pn->shrinker_info, new); kvfree_rcu(old, rcu); } return 0; } -void free_shrinker_maps(struct mem_cgroup *memcg) +void free_shrinker_info(struct mem_cgroup *memcg) { struct mem_cgroup_per_node *pn; - struct memcg_shrinker_map *map; + struct shrinker_info *info; int nid; if (mem_cgroup_is_root(memcg)) @@ -232,15 +232,15 @@ void free_shrinker_maps(struct mem_cgroup *memcg) for_each_node(nid) { pn = memcg->nodeinfo[nid]; - map = rcu_dereference_protected(pn->shrinker_map, true); - kvfree(map); - rcu_assign_pointer(pn->shrinker_map, NULL); + info = rcu_dereference_protected(pn->shrinker_info, true); + kvfree(info); + rcu_assign_pointer(pn->shrinker_info, NULL); } } -int alloc_shrinker_maps(struct mem_cgroup *memcg) +int alloc_shrinker_info(struct mem_cgroup *memcg) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; int nid, size, ret = 0; if (mem_cgroup_is_root(memcg)) @@ -249,20 +249,20 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) down_write(&shrinker_rwsem); size = shrinker_map_size(shrinker_nr_max); for_each_node(nid) { - map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); - if (!map) { - free_shrinker_maps(memcg); + info = kvzalloc_node(sizeof(*info) + size, GFP_KERNEL, nid); + if (!info) { + free_shrinker_info(memcg); ret = -ENOMEM; break; } - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); return ret; } -static int expand_shrinker_maps(int new_id) +static int expand_shrinker_info(int new_id) { int size, old_size, ret = 0; int new_nr_max = new_id + 1; @@ -282,7 +282,7 @@ static int expand_shrinker_maps(int new_id) do { if (mem_cgroup_is_root(memcg)) continue; - ret = expand_one_shrinker_map(memcg, size, old_size); + ret = expand_one_shrinker_info(memcg, size, old_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -298,13 +298,13 @@ static int expand_shrinker_maps(int new_id) void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; rcu_read_lock(); - map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); + info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); - set_bit(shrinker_id, map->map); + set_bit(shrinker_id, info->map); rcu_read_unlock(); } } @@ -335,7 +335,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) goto unlock; if (id >= shrinker_nr_max) { - if (expand_shrinker_maps(id)) { + if (expand_shrinker_info(id)) { idr_remove(&shrinker_idr, id); goto unlock; } @@ -665,7 +665,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, int priority) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; unsigned long ret, freed = 0; int i; @@ -675,12 +675,12 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (!down_read_trylock(&shrinker_rwsem)) return 0; - map = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_map, - true); - if (unlikely(!map)) + info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + true); + if (unlikely(!info)) goto unlock; - for_each_set_bit(i, map->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, shrinker_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -691,7 +691,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, shrinker = idr_find(&shrinker_idr, i); if (unlikely(!shrinker || shrinker == SHRINKER_REGISTERING)) { if (!shrinker) - clear_bit(i, map->map); + clear_bit(i, info->map); continue; } @@ -702,7 +702,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, ret = do_shrink_slab(&sc, shrinker, priority); if (ret == SHRINK_EMPTY) { - clear_bit(i, map->map); + clear_bit(i, info->map); /* * After the shrinker reported that it had no objects to * free, but before we cleared the corresponding bit in From patchwork Thu Mar 11 19:08:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87C5DC433E0 for ; Thu, 11 Mar 2021 19:09:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 06B9364E60 for ; Thu, 11 Mar 2021 19:09:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 06B9364E60 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5DC968D02EA; Thu, 11 Mar 2021 14:09:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 58D638D02B2; Thu, 11 Mar 2021 14:09:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 391E68D02EA; Thu, 11 Mar 2021 14:09:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0129.hostedemail.com [216.40.44.129]) by kanga.kvack.org (Postfix) with ESMTP id 13EA98D02B2 for ; Thu, 11 Mar 2021 14:09:16 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C42B55DF8 for ; Thu, 11 Mar 2021 19:09:15 +0000 (UTC) X-FDA: 77908531470.05.DC310B6 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf24.hostedemail.com (Postfix) with ESMTP id 2E84EA0009FD for ; Thu, 11 Mar 2021 19:09:11 +0000 (UTC) Received: by mail-pl1-f170.google.com with SMTP id q12so225541plr.1 for ; Thu, 11 Mar 2021 11:09:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=m2bLEDF9fLJ0ubUMCl5gxJ4ENWjURCD7GlteN0I+Pn0=; b=Mi6pTvW8VJpVzzmhCHBU6uqews5NbK1wW8i7GWTUtcQJm40tQfsU3q/4i1dMSxHChA fk6MOedbmi4Dz8b7j7f1wh094bz8Tx5NrY7UsuzP2ASxI72pn0IuoBBVuMk43qFKPe8C 1cZA14u5QtwgFBzSURdke1P3WrCzExGbRVN1esGKANq6X5K9sbVyb3V7XoqeNexexucz FPYL/JPcPEexxog48emnyY+vMOIt9v1CyZKrbKjjc4BFkKUTaiNbT3hhy29DxT0AdsAK Rzk+DWX1+6urZv54JXCIQfJ7IxTYAkD9RnK4LQFq1LcW0b8nwp6SP+d25idwc3fprVC+ Iv3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=m2bLEDF9fLJ0ubUMCl5gxJ4ENWjURCD7GlteN0I+Pn0=; b=djRazStONl0XHkCswZIsV94/DA9iNhdHSeHj08IrA5RwDboJ0rceSKOotLGsdHRu66 /xX7Mz/bnXik1AZn2yxwljC4L0ZZytguGzja4PQTzjVCi2/AgFTs8aDMGG+mVDXSQKvi FFhiZ5uAIW8gUDavCBa64IvXJ7vvPQylJgGG5P0jqC0V/slGmwPGzNwxfa11WeD5cw50 PuCakYKxTTyPkt251t2eB1NpRnDRyHlRyQ7qdltL4qwjT8Ic5sNEyh4nKimhV2BigZp0 F+8MpGLg+3h75ThS0xVujrCJxQ/nu9scHbKMLlttioaxcSNPEZE/2e/mrV2jBhnaRM+B YYkw== X-Gm-Message-State: AOAM530TPL3JzUrbmViXBpge8WWffK90di76/X/5Tpoyh5g7sYP+ESk4 uO4Ll7vWo1aUrQQDLkgNRgvwxefcVXDmFg== X-Google-Smtp-Source: ABdhPJy6nmhaP70hLjuP7UUkSI2vRg+R6cLVH31ME6nZFnyfjO/2L99AjBImPmBY4M11zbaB7ha1TA== X-Received: by 2002:a17:90a:fd0b:: with SMTP id cv11mr10319170pjb.183.1615489754458; Thu, 11 Mar 2021 11:09:14 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:13 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 07/13] mm: vmscan: add shrinker_info_protected() helper Date: Thu, 11 Mar 2021 11:08:39 -0800 Message-Id: <20210311190845.9708-8-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: unaywezdodzm8m9rb967drynw14ui9rw X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 2E84EA0009FD Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf24; identity=mailfrom; envelope-from=""; helo=mail-pl1-f170.google.com; client-ip=209.85.214.170 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615489751-821130 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shrinker_info is dereferenced in a couple of places via rcu_dereference_protected with different calling conventions, for example, using mem_cgroup_nodeinfo helper or dereferencing memcg->nodeinfo[nid]->shrinker_info. And the later patch will add more dereference places. So extract the dereference into a helper to make the code more readable. No functional change. Acked-by: Roman Gushchin Acked-by: Kirill Tkhai Acked-by: Vlastimil Babka Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 7fdfdacf9a1f..ef9f1531a6ee 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -192,6 +192,13 @@ static inline int shrinker_map_size(int nr_items) return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); } +static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, + int nid) +{ + return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + lockdep_is_held(&shrinker_rwsem)); +} + static int expand_one_shrinker_info(struct mem_cgroup *memcg, int size, int old_size) { @@ -201,7 +208,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, for_each_node(nid) { pn = memcg->nodeinfo[nid]; - old = rcu_dereference_protected(pn->shrinker_info, true); + old = shrinker_info_protected(memcg, nid); /* Not yet online memcg */ if (!old) return 0; @@ -232,7 +239,7 @@ void free_shrinker_info(struct mem_cgroup *memcg) for_each_node(nid) { pn = memcg->nodeinfo[nid]; - info = rcu_dereference_protected(pn->shrinker_info, true); + info = shrinker_info_protected(memcg, nid); kvfree(info); rcu_assign_pointer(pn->shrinker_info, NULL); } @@ -675,8 +682,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (!down_read_trylock(&shrinker_rwsem)) return 0; - info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - true); + info = shrinker_info_protected(memcg, nid); if (unlikely(!info)) goto unlock; From patchwork Thu Mar 11 19:08:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2E9CC433E0 for ; Thu, 11 Mar 2021 19:09:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 56EDB64EBB for ; Thu, 11 Mar 2021 19:09:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 56EDB64EBB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E8F3A8D02EC; Thu, 11 Mar 2021 14:09:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E42168D02B2; Thu, 11 Mar 2021 14:09:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C45738D02EC; Thu, 11 Mar 2021 14:09:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0068.hostedemail.com [216.40.44.68]) by kanga.kvack.org (Postfix) with ESMTP id 981AE8D02B2 for ; Thu, 11 Mar 2021 14:09:18 -0500 (EST) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 59A478249980 for ; Thu, 11 Mar 2021 19:09:18 +0000 (UTC) X-FDA: 77908531596.13.4A2BA3C Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf20.hostedemail.com (Postfix) with ESMTP id D6DE3FE for ; Thu, 11 Mar 2021 19:09:13 +0000 (UTC) Received: by mail-pj1-f42.google.com with SMTP id nh23-20020a17090b3657b02900c0d5e235a8so9827539pjb.0 for ; Thu, 11 Mar 2021 11:09:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YP+QHK4if14P2cPRJw16+qngvdLDRS9w75Y8NgWmP/4=; b=UMjzDnKIdce/WKJadmRH1OB/wiqQKKr+f1T+PfUDbCDkayv4q0SdCmFnxCvebi1rmK N0X+YvJKnGtGn/u5SmEzUuZuWZmPf0Xzb5cMXwth9/bWEi+Km1Ro1LMWQIXQlLNX7aMt FJ9LDWfDcvXOLwBsKG0KkLK2O2YsOgAJHSzPaKzCXI0TXt/KR9uBTseGJj0o6y3u5RmK ezEVLLxogRYTP8XAjxzK0ifp7zhKzVu1/9ebNHs6DsCuhHe9muaqwY4VKeFF/N4xl+gP eDWje8rFc72MqW0JxBm6Htkeu4oAwzwSgf5GCerc0BydB0mrdHuunXPOwuMSa2wqjCr5 O/nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YP+QHK4if14P2cPRJw16+qngvdLDRS9w75Y8NgWmP/4=; b=LNl05joNvIzaHD3o4m6jj39en3n+x6iovH0G04SV+JaG6vEYRDMrqBlqHObgK/Y9Gv jAz0nJR5XadGrq770HTfH1TgFkmf5PbPcGoeVipRxdW3jSa86oYNwaSvLlUlX0QxSfcd EjfOpSWbMlFMp8zBRDwIIVtrDiE4+0gSXJgefJDCLJosjcDJih3kg51UpdpQ4cIPhDbb +oU9jMJ8S85A5dbRQiRRgcmBsfO13qYnQcV1dOEEJ25+OnqaZfjFDQUJXg/O4iVRMZcr gbtyaZavkyvzRXtHketX21b57A0FKZpNtBRurEQFUtExl/6LsPLURG1WGjDpr9vFtluF Bf1Q== X-Gm-Message-State: AOAM533EUBOrlFHRqWpaaDNvHSBvpmE4ZxWvO4AIPf+6OzIqdfxfxvXU yyzWkhtR6Uj4pqc76eTmrumsudPqklGnCg== X-Google-Smtp-Source: ABdhPJwPuyvEh2XO3JP8w7da5IZt5jzq/Fiw92FtaiLp0S9sNPUyursRbVNVjwbuTZ8/7zTGL7GBjA== X-Received: by 2002:a17:902:cec8:b029:e4:a497:da8d with SMTP id d8-20020a170902cec8b02900e4a497da8dmr9643886plg.16.1615489756447; Thu, 11 Mar 2021 11:09:16 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:15 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 08/13] mm: vmscan: use a new flag to indicate shrinker is registered Date: Thu, 11 Mar 2021 11:08:40 -0800 Message-Id: <20210311190845.9708-9-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D6DE3FE X-Stat-Signature: uzkmwrg8cu46xjbbsxgioqyjdga8i9gf Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf20; identity=mailfrom; envelope-from=""; helo=mail-pj1-f42.google.com; client-ip=209.85.216.42 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615489753-725173 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently registered shrinker is indicated by non-NULL shrinker->nr_deferred. This approach is fine with nr_deferred at the shrinker level, but the following patches will move MEMCG_AWARE shrinkers' nr_deferred to memcg level, so their shrinker->nr_deferred would always be NULL. This would prevent the shrinkers from unregistering correctly. Remove SHRINKER_REGISTERING since we could check if shrinker is registered successfully by the new flag. Acked-by: Kirill Tkhai Acked-by: Vlastimil Babka Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/shrinker.h | 7 ++++--- mm/vmscan.c | 40 +++++++++++++++------------------------- 2 files changed, 19 insertions(+), 28 deletions(-) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 0f80123650e2..1eac79ce57d4 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -79,13 +79,14 @@ struct shrinker { #define DEFAULT_SEEKS 2 /* A good number if you don't know better. */ /* Flags */ -#define SHRINKER_NUMA_AWARE (1 << 0) -#define SHRINKER_MEMCG_AWARE (1 << 1) +#define SHRINKER_REGISTERED (1 << 0) +#define SHRINKER_NUMA_AWARE (1 << 1) +#define SHRINKER_MEMCG_AWARE (1 << 2) /* * It just makes sense when the shrinker is also MEMCG_AWARE for now, * non-MEMCG_AWARE shrinker should not have this flag set. */ -#define SHRINKER_NONSLAB (1 << 2) +#define SHRINKER_NONSLAB (1 << 3) extern int prealloc_shrinker(struct shrinker *shrinker); extern void register_shrinker_prepared(struct shrinker *shrinker); diff --git a/mm/vmscan.c b/mm/vmscan.c index ef9f1531a6ee..34cf3d84309c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -316,19 +316,6 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) } } -/* - * We allow subsystems to populate their shrinker-related - * LRU lists before register_shrinker_prepared() is called - * for the shrinker, since we don't want to impose - * restrictions on their internal registration order. - * In this case shrink_slab_memcg() may find corresponding - * bit is set in the shrinkers map. - * - * This value is used by the function to detect registering - * shrinkers and to skip do_shrink_slab() calls for them. - */ -#define SHRINKER_REGISTERING ((struct shrinker *)~0UL) - static DEFINE_IDR(shrinker_idr); static int prealloc_memcg_shrinker(struct shrinker *shrinker) @@ -337,7 +324,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) down_write(&shrinker_rwsem); /* This may call shrinker, so it must use down_read_trylock() */ - id = idr_alloc(&shrinker_idr, SHRINKER_REGISTERING, 0, 0, GFP_KERNEL); + id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -360,9 +347,9 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) BUG_ON(id < 0); - down_write(&shrinker_rwsem); + lockdep_assert_held(&shrinker_rwsem); + idr_remove(&shrinker_idr, id); - up_write(&shrinker_rwsem); } static bool cgroup_reclaim(struct scan_control *sc) @@ -490,8 +477,11 @@ void free_prealloced_shrinker(struct shrinker *shrinker) if (!shrinker->nr_deferred) return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) + if (shrinker->flags & SHRINKER_MEMCG_AWARE) { + down_write(&shrinker_rwsem); unregister_memcg_shrinker(shrinker); + up_write(&shrinker_rwsem); + } kfree(shrinker->nr_deferred); shrinker->nr_deferred = NULL; @@ -501,10 +491,7 @@ void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); list_add_tail(&shrinker->list, &shrinker_list); -#ifdef CONFIG_MEMCG - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - idr_replace(&shrinker_idr, shrinker, shrinker->id); -#endif + shrinker->flags |= SHRINKER_REGISTERED; up_write(&shrinker_rwsem); } @@ -524,13 +511,16 @@ EXPORT_SYMBOL(register_shrinker); */ void unregister_shrinker(struct shrinker *shrinker) { - if (!shrinker->nr_deferred) + if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - unregister_memcg_shrinker(shrinker); + down_write(&shrinker_rwsem); list_del(&shrinker->list); + shrinker->flags &= ~SHRINKER_REGISTERED; + if (shrinker->flags & SHRINKER_MEMCG_AWARE) + unregister_memcg_shrinker(shrinker); up_write(&shrinker_rwsem); + kfree(shrinker->nr_deferred); shrinker->nr_deferred = NULL; } @@ -695,7 +685,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct shrinker *shrinker; shrinker = idr_find(&shrinker_idr, i); - if (unlikely(!shrinker || shrinker == SHRINKER_REGISTERING)) { + if (unlikely(!shrinker || !(shrinker->flags & SHRINKER_REGISTERED))) { if (!shrinker) clear_bit(i, info->map); continue; From patchwork Thu Mar 11 19:08:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E176C433E6 for ; Thu, 11 Mar 2021 19:09:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 72EA764EFE for ; Thu, 11 Mar 2021 19:09:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 72EA764EFE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 715E88D02ED; Thu, 11 Mar 2021 14:09:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C5528D02B2; Thu, 11 Mar 2021 14:09:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F0678D02ED; Thu, 11 Mar 2021 14:09:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0122.hostedemail.com [216.40.44.122]) by kanga.kvack.org (Postfix) with ESMTP id 27F608D02B2 for ; Thu, 11 Mar 2021 14:09:21 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id C80EE180ACF9A for ; Thu, 11 Mar 2021 19:09:20 +0000 (UTC) X-FDA: 77908531680.19.A04AE8D Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf02.hostedemail.com (Postfix) with ESMTP id 1E499407F8DC for ; Thu, 11 Mar 2021 19:09:10 +0000 (UTC) Received: by mail-pl1-f174.google.com with SMTP id q12so225669plr.1 for ; Thu, 11 Mar 2021 11:09:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=O2soj3VRJNBvxi9PWCn4DecTfTo8NOmelRWHo0xmQP0=; b=uRBsx+Kax6NkZ9uc+8SG6XZVcl8rkcflBM04WQkKe1hczZa1EixUiP1LbAu/TXH5hg aubdIpn0WNho5jsbSkooNSxqL+0DoAV7/yLncRMbbAPHNrWjm0Gx0FQ7c/1xiSsF0T7w NB1Cnzo8DSLhgxKQt7ME2aPCPXHczsjcj6JDCf4SK1FyAU2ZoLaSanGJLs89l2972BFU ns5j3EUPu5KWJXRwJ83uO2U93T1inYVPo7DB7nLqglpWDOIEky7EVHBnbI73wmB8P8Cp WvhGvR4WOhdI6nKogLkB1VXMSkTfFg/FEdhqY8kvdoWm5K6DD4F0QHchHO42vyknUHq0 2Chw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=O2soj3VRJNBvxi9PWCn4DecTfTo8NOmelRWHo0xmQP0=; b=WgjbSv9C9KKI8VJ6SmcyIoWI+EbzgSbWwRdGh2CwptBhpvy7ydv9aYVEIQ8kp/lo7+ lAKdeO+QASuB6xlwpjXyHoZjb/asBI+vJBK6Olrb0UHiyYvszi+qDcYGMnKYwFpxz6c5 SxXGLRxxxfIYQSJTrOkvG2kTXqw06MlaN1NUn1KQCsZx/XwY5wjQSBvZVVV0KvP+RmQR 6c4iM1nr3AUybfITUUs8cKzA6Utg7T8UR4juPgFggDD3LKe0gBLQbudsj7lEIzrH/po4 cxWa+rBrX98pXf7SGVv3VnjqL7gc+r8tW/kW6q/fCzrLztZkSxjaL0ymUqIlbskBL9K1 FOxA== X-Gm-Message-State: AOAM5338XB/Q02y3iuVQh+g2erVfYIYVgmlDPXfdf/It/xy18UIrFhzo R1FtkQBmudwxrzSbpprCUQ4= X-Google-Smtp-Source: ABdhPJxCLYddJof6Hlc1njrzRqI8meh0rFXR/jtVGiDahS5BvuBn7cw7/b5cJIri8oPuKD+CJDP5KQ== X-Received: by 2002:a17:902:aa0a:b029:e4:c090:ad76 with SMTP id be10-20020a170902aa0ab02900e4c090ad76mr9687150plb.2.1615489759072; Thu, 11 Mar 2021 11:09:19 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:18 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 09/13] mm: vmscan: add per memcg shrinker nr_deferred Date: Thu, 11 Mar 2021 11:08:41 -0800 Message-Id: <20210311190845.9708-10-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1E499407F8DC X-Stat-Signature: pp7pepeo5de4pqbn1gtxnnb9q11doaq3 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf02; identity=mailfrom; envelope-from=""; helo=mail-pl1-f174.google.com; client-ip=209.85.214.174 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615489750-40898 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently the number of deferred objects are per shrinker, but some slabs, for example, vfs inode/dentry cache are per memcg, this would result in poor isolation among memcgs. The deferred objects typically are generated by __GFP_NOFS allocations, one memcg with excessive __GFP_NOFS allocations may blow up deferred objects, then other innocent memcgs may suffer from over shrink, excessive reclaim latency, etc. For example, two workloads run in memcgA and memcgB respectively, workload in B is vfs heavy workload. Workload in A generates excessive deferred objects, then B's vfs cache might be hit heavily (drop half of caches) by B's limit reclaim or global reclaim. We observed this hit in our production environment which was running vfs heavy workload shown as the below tracing log: <...>-409454 [016] .... 28286961.747146: mm_shrink_slab_start: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 1 objects to shrink 3641681686040 gfp_flags GFP_HIGHUSER_MOVABLE|__GFP_ZERO pgs_scanned 1 lru_pgs 15721 cache items 246404277 delta 31345 total_scan 123202138 <...>-409454 [022] .... 28287105.928018: mm_shrink_slab_end: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 1 unused scan count 3641681686040 new scan count 3641798379189 total_scan 602 last shrinker return val 123186855 The vfs cache and page cache ratio was 10:1 on this machine, and half of caches were dropped. This also resulted in significant amount of page caches were dropped due to inodes eviction. Make nr_deferred per memcg for memcg aware shrinkers would solve the unfairness and bring better isolation. The following patch will add nr_deferred to parent memcg when memcg offline. To preserve nr_deferred when reparenting memcgs to root, root memcg needs shrinker_info allocated too. When memcg is not enabled (!CONFIG_MEMCG or memcg disabled), the shrinker's nr_deferred would be used. And non memcg aware shrinkers use shrinker's nr_deferred all the time. Acked-by: Roman Gushchin Acked-by: Kirill Tkhai Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 7 +++-- mm/vmscan.c | 60 ++++++++++++++++++++++++++------------ 2 files changed, 46 insertions(+), 21 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index dc7d0e2cb3ad..24e735434a46 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -114,12 +114,13 @@ struct batched_lruvec_stat { }; /* - * Bitmap of shrinker::id corresponding to memcg-aware shrinkers, - * which have elements charged to this memcg. + * Bitmap and deferred work of shrinker::id corresponding to memcg-aware + * shrinkers, which have elements charged to this memcg. */ struct shrinker_info { struct rcu_head rcu; - unsigned long map[]; + atomic_long_t *nr_deferred; + unsigned long *map; }; /* diff --git a/mm/vmscan.c b/mm/vmscan.c index 34cf3d84309c..397f3b67bad8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -187,11 +187,17 @@ static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG static int shrinker_nr_max; +/* The shrinker_info is expanded in a batch of BITS_PER_LONG */ static inline int shrinker_map_size(int nr_items) { return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); } +static inline int shrinker_defer_size(int nr_items) +{ + return (round_up(nr_items, BITS_PER_LONG) * sizeof(atomic_long_t)); +} + static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { @@ -200,11 +206,13 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, } static int expand_one_shrinker_info(struct mem_cgroup *memcg, - int size, int old_size) + int map_size, int defer_size, + int old_map_size, int old_defer_size) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; int nid; + int size = map_size + defer_size; for_each_node(nid) { pn = memcg->nodeinfo[nid]; @@ -217,9 +225,16 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!new) return -ENOMEM; - /* Set all old bits, clear all new bits */ - memset(new->map, (int)0xff, old_size); - memset((void *)new->map + old_size, 0, size - old_size); + new->nr_deferred = (atomic_long_t *)(new + 1); + new->map = (void *)new->nr_deferred + defer_size; + + /* map: set all old bits, clear all new bits */ + memset(new->map, (int)0xff, old_map_size); + memset((void *)new->map + old_map_size, 0, map_size - old_map_size); + /* nr_deferred: copy old values, clear all new values */ + memcpy(new->nr_deferred, old->nr_deferred, old_defer_size); + memset((void *)new->nr_deferred + old_defer_size, 0, + defer_size - old_defer_size); rcu_assign_pointer(pn->shrinker_info, new); kvfree_rcu(old, rcu); @@ -234,9 +249,6 @@ void free_shrinker_info(struct mem_cgroup *memcg) struct shrinker_info *info; int nid; - if (mem_cgroup_is_root(memcg)) - return; - for_each_node(nid) { pn = memcg->nodeinfo[nid]; info = shrinker_info_protected(memcg, nid); @@ -249,12 +261,12 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) { struct shrinker_info *info; int nid, size, ret = 0; - - if (mem_cgroup_is_root(memcg)) - return 0; + int map_size, defer_size = 0; down_write(&shrinker_rwsem); - size = shrinker_map_size(shrinker_nr_max); + map_size = shrinker_map_size(shrinker_nr_max); + defer_size = shrinker_defer_size(shrinker_nr_max); + size = map_size + defer_size; for_each_node(nid) { info = kvzalloc_node(sizeof(*info) + size, GFP_KERNEL, nid); if (!info) { @@ -262,6 +274,8 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) ret = -ENOMEM; break; } + info->nr_deferred = (atomic_long_t *)(info + 1); + info->map = (void *)info->nr_deferred + defer_size; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -269,15 +283,21 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } +static inline bool need_expand(int nr_max) +{ + return round_up(nr_max, BITS_PER_LONG) > + round_up(shrinker_nr_max, BITS_PER_LONG); +} + static int expand_shrinker_info(int new_id) { - int size, old_size, ret = 0; + int ret = 0; int new_nr_max = new_id + 1; + int map_size, defer_size = 0; + int old_map_size, old_defer_size = 0; struct mem_cgroup *memcg; - size = shrinker_map_size(new_nr_max); - old_size = shrinker_map_size(shrinker_nr_max); - if (size <= old_size) + if (!need_expand(new_nr_max)) goto out; if (!root_mem_cgroup) @@ -285,11 +305,15 @@ static int expand_shrinker_info(int new_id) lockdep_assert_held(&shrinker_rwsem); + map_size = shrinker_map_size(new_nr_max); + defer_size = shrinker_defer_size(new_nr_max); + old_map_size = shrinker_map_size(shrinker_nr_max); + old_defer_size = shrinker_defer_size(shrinker_nr_max); + memcg = mem_cgroup_iter(NULL, NULL, NULL); do { - if (mem_cgroup_is_root(memcg)) - continue; - ret = expand_one_shrinker_info(memcg, size, old_size); + ret = expand_one_shrinker_info(memcg, map_size, defer_size, + old_map_size, old_defer_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; From patchwork Thu Mar 11 19:08:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51943C433DB for ; Thu, 11 Mar 2021 19:09:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CDBB364EC0 for ; Thu, 11 Mar 2021 19:09:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CDBB364EC0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1BEA18D02EE; Thu, 11 Mar 2021 14:09:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 146B78D02B2; Thu, 11 Mar 2021 14:09:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E654B8D02EE; Thu, 11 Mar 2021 14:09:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0170.hostedemail.com [216.40.44.170]) by kanga.kvack.org (Postfix) with ESMTP id C04FD8D02B2 for ; Thu, 11 Mar 2021 14:09:22 -0500 (EST) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 83CD952D9 for ; Thu, 11 Mar 2021 19:09:22 +0000 (UTC) X-FDA: 77908531764.13.35390EC Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf12.hostedemail.com (Postfix) with ESMTP id 25BF2DA for ; Thu, 11 Mar 2021 19:09:18 +0000 (UTC) Received: by mail-pj1-f49.google.com with SMTP id mz6-20020a17090b3786b02900c16cb41d63so9831788pjb.2 for ; Thu, 11 Mar 2021 11:09:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tkw5+G90n94OUjiIXFC/vgHLYhl7hCOYiRf2Qw5jx3o=; b=AZcAO2+Zu7/HIsDOkfadeet7K1FjCBUqPJHcDw10r6Ip/XSXQgclt7/on3IYyaVxzb gVfFraVv9Ztw5I3oKVe3BZ+0bbxXZc9dxi9YjiQKKhZ54f4Mm7t538LK+afS4g9WAmlk ZUI3rjA2+8XWNmrElOpXt/U5+ruaHwy9YwDgDjAvMK8uy0435JphBeu107Q5p1bmAE38 EBMmd2n59UEAmYKPBEd1hi/iFeUyPjyP5NgQd9ECU+/ZD81eV3zRtWvSdYjad3VbncTI drbXs6PpDSyksEWLlBYEYfq+beef6niS2B+C13hy1CNOZ7jK1J/pQ424C5TVHyf6yocq o2Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tkw5+G90n94OUjiIXFC/vgHLYhl7hCOYiRf2Qw5jx3o=; b=hrOnWIFdIkG+Cw2CdDjtIQSRNXgqxBw7ChXokDJtM7EA1NAtoa6r6Kv5R+v17iipYq sYemQ5GIxvRBSdac6TCwaSlWY2elrjV8Bk6izEXCVkpLQEjnA2WGha2xstvHPhXICWGt GZEoPsIKyRp4bWYAQbCpvt8xWJt9pqy7q9v2gfwr1dSF8MNaI3UAWnvC5EzvxAg48dhp HsTJODCEzCd+528Zz9MzhG37Q3XmM8RZC4P9SqE4evZb1AtBtfB20sj2VpV87VznLrOC /16vve7WmUfbrCzFPFnbvuLXDCARvIOWzeZbmBnLt891/wZm17HD3a69eZqzx6ynkE0+ l7Nw== X-Gm-Message-State: AOAM530UEfmDAJWQVOc2JVhg1un7QTVLZ3TFETEAIeCmBRIEJjvWgZcO U2WcXt8X50S7kc8AarFx1jo= X-Google-Smtp-Source: ABdhPJxnH70sPhLOC3ftHwNORmFlXsCr7F56porR1CAZav7poDXAWwFtgfvr0AYbNHuZhf+apfLeCg== X-Received: by 2002:a17:902:7c8e:b029:e5:f70f:b7ab with SMTP id y14-20020a1709027c8eb02900e5f70fb7abmr9749952pll.32.1615489761244; Thu, 11 Mar 2021 11:09:21 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:20 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 10/13] mm: vmscan: use per memcg nr_deferred of shrinker Date: Thu, 11 Mar 2021 11:08:42 -0800 Message-Id: <20210311190845.9708-11-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: wkjogd99ykj3xije3gb8rkoqx3wz88t5 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 25BF2DA Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf12; identity=mailfrom; envelope-from=""; helo=mail-pj1-f49.google.com; client-ip=209.85.216.49 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615489758-486140 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use per memcg's nr_deferred for memcg aware shrinkers. The shrinker's nr_deferred will be used in the following cases: 1. Non memcg aware shrinkers 2. !CONFIG_MEMCG 3. memcg is disabled by boot parameter Acked-by: Roman Gushchin Acked-by: Kirill Tkhai Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 78 ++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 66 insertions(+), 12 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 397f3b67bad8..5bc6975cb635 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -376,6 +376,24 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) idr_remove(&shrinker_idr, id); } +static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + + info = shrinker_info_protected(memcg, nid); + return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); +} + +static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + + info = shrinker_info_protected(memcg, nid); + return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); +} + static bool cgroup_reclaim(struct scan_control *sc) { return sc->target_mem_cgroup; @@ -414,6 +432,18 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) { } +static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + return 0; +} + +static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + return 0; +} + static bool cgroup_reclaim(struct scan_control *sc) { return false; @@ -425,6 +455,39 @@ static bool writeback_throttling_sane(struct scan_control *sc) } #endif +static long xchg_nr_deferred(struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (sc->memcg && + (shrinker->flags & SHRINKER_MEMCG_AWARE)) + return xchg_nr_deferred_memcg(nid, shrinker, + sc->memcg); + + return atomic_long_xchg(&shrinker->nr_deferred[nid], 0); +} + + +static long add_nr_deferred(long nr, struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (sc->memcg && + (shrinker->flags & SHRINKER_MEMCG_AWARE)) + return add_nr_deferred_memcg(nr, nid, shrinker, + sc->memcg); + + return atomic_long_add_return(nr, &shrinker->nr_deferred[nid]); +} + /* * This misses isolated pages which are not accounted for to save counters. * As the data only determines if reclaim or compaction continues, it is @@ -561,14 +624,10 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, long freeable; long nr; long new_nr; - int nid = shrinkctl->nid; long batch_size = shrinker->batch ? shrinker->batch : SHRINK_BATCH; long scanned = 0, next_deferred; - if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) - nid = 0; - freeable = shrinker->count_objects(shrinker, shrinkctl); if (freeable == 0 || freeable == SHRINK_EMPTY) return freeable; @@ -578,7 +637,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, * and zero it so that other concurrent shrinker invocations * don't also do this scanning work. */ - nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0); + nr = xchg_nr_deferred(shrinker, shrinkctl); total_scan = nr; if (shrinker->seeks) { @@ -669,14 +728,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, next_deferred = 0; /* * move the unused scan count back into the shrinker in a - * manner that handles concurrent updates. If we exhausted the - * scan, there is no need to do an update. + * manner that handles concurrent updates. */ - if (next_deferred > 0) - new_nr = atomic_long_add_return(next_deferred, - &shrinker->nr_deferred[nid]); - else - new_nr = atomic_long_read(&shrinker->nr_deferred[nid]); + new_nr = add_nr_deferred(next_deferred, shrinker, shrinkctl); trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); return freed; From patchwork Thu Mar 11 19:08:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47E02C433E0 for ; Thu, 11 Mar 2021 19:09:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CEA3164EF2 for ; Thu, 11 Mar 2021 19:09:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CEA3164EF2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1BD248D02EF; Thu, 11 Mar 2021 14:09:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 170BA8D02B2; Thu, 11 Mar 2021 14:09:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA2FC8D02EF; Thu, 11 Mar 2021 14:09:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0133.hostedemail.com [216.40.44.133]) by kanga.kvack.org (Postfix) with ESMTP id B1F618D02B2 for ; Thu, 11 Mar 2021 14:09:24 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 76DFF12C8 for ; Thu, 11 Mar 2021 19:09:24 +0000 (UTC) X-FDA: 77908531848.03.D3F7079 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf30.hostedemail.com (Postfix) with ESMTP id 2DE93E005F07 for ; Thu, 11 Mar 2021 19:09:17 +0000 (UTC) Received: by mail-pl1-f182.google.com with SMTP id d23so7482440plq.2 for ; Thu, 11 Mar 2021 11:09:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KRjLPceQFZWfN56pzAL4S7TxF614EkVjGd+2AFI1iu8=; b=Ba3D+YLmVCJ+UrsBxNrusvOurFMF0GizvxXy+rbFsNb20EJIN3RNgoNWWh29ly8tw1 nwxuuvd0heENI0w8dr4FLMw7jhR+Okm1bLT2nwfqyGR5ZPmmK4Tf1gybOnD7BHGX+aIr M2As+IkvjxMoc6yjkBTEsv+stBsNyg6GyW1SKRZbakQyo6hqc3SD5r2HAMlia1fTWcKh 6e7PCIuSZPQ8/9NIVLE1ynbs7pFmidWychavvHD9AYqsEvOyXSZWenF+CUQABb240Y1W LMFfTJDUrzJsEsWcTv3baolU9PdD5mTuEmqQiE5MNyqF/zJmb5g6vUzuq8JGitbwnptL W90Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KRjLPceQFZWfN56pzAL4S7TxF614EkVjGd+2AFI1iu8=; b=OYpsxdRl+c7eT1mItx/NjUCMuzKRiBl0cfk7gCaPjienSI4leK/wEfXyuDu6GfQWcO AvdBz01M7ocVKAsl/PgNqEpE5+fa4pSD97L1rC6742C7ObGiT4qEbH+mCakVxXDNKPsc yJ2IzEbQOeoBHDiwxkgfJx0RshDgoSAEy0CTK2By+2diktObQ9/w8PKUXI/1Bu6wu/j3 vKiGWfGAJNyzsPUVvIS7a8+P2f+RVA2U48+y9nqiiF2ZYR5vB3OuAG7zMndnJ+4OSlwu EjbmIoLoAQtSXzqdTMHUMhrg7zCir7QiOECiWUNQNx8CJKMMT4jcpVd2UdQKszvd8ZFi BhGw== X-Gm-Message-State: AOAM533+RA+PTloug8X9RGfiB2kOTfc0hZqaaEIFAz3xzt2pwPzW2Md9 8q+m8WogOtdpswA/moOGO/8= X-Google-Smtp-Source: ABdhPJyi+9KnFTLz7v5ac0t6Oomsbt4Jf4POupwR6yJqXXAM05xu/dsaKrVr9lkiWkjJtFGJ48k1rw== X-Received: by 2002:a17:902:f547:b029:e4:6dbc:6593 with SMTP id h7-20020a170902f547b02900e46dbc6593mr9511144plf.4.1615489763282; Thu, 11 Mar 2021 11:09:23 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:22 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 11/13] mm: vmscan: don't need allocate shrinker->nr_deferred for memcg aware shrinkers Date: Thu, 11 Mar 2021 11:08:43 -0800 Message-Id: <20210311190845.9708-12-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2DE93E005F07 X-Stat-Signature: pp1rp5rmdmhtseege7dzegfykthtosmm Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf30; identity=mailfrom; envelope-from=""; helo=mail-pl1-f182.google.com; client-ip=209.85.214.182 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615489757-192093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now nr_deferred is available on per memcg level for memcg aware shrinkers, so don't need allocate shrinker->nr_deferred for such shrinkers anymore. The prealloc_memcg_shrinker() would return -ENOSYS if !CONFIG_MEMCG or memcg is disabled by kernel command line, then shrinker's SHRINKER_MEMCG_AWARE flag would be cleared. This makes the implementation of this patch simpler. Acked-by: Vlastimil Babka Reviewed-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 5bc6975cb635..324c34c6e5cf 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -346,6 +346,9 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) { int id, ret = -ENOMEM; + if (mem_cgroup_disabled()) + return -ENOSYS; + down_write(&shrinker_rwsem); /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); @@ -425,7 +428,7 @@ static bool writeback_throttling_sane(struct scan_control *sc) #else static int prealloc_memcg_shrinker(struct shrinker *shrinker) { - return 0; + return -ENOSYS; } static void unregister_memcg_shrinker(struct shrinker *shrinker) @@ -537,8 +540,18 @@ static unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru, */ int prealloc_shrinker(struct shrinker *shrinker) { - unsigned int size = sizeof(*shrinker->nr_deferred); + unsigned int size; + int err; + + if (shrinker->flags & SHRINKER_MEMCG_AWARE) { + err = prealloc_memcg_shrinker(shrinker); + if (err != -ENOSYS) + return err; + shrinker->flags &= ~SHRINKER_MEMCG_AWARE; + } + + size = sizeof(*shrinker->nr_deferred); if (shrinker->flags & SHRINKER_NUMA_AWARE) size *= nr_node_ids; @@ -546,28 +559,16 @@ int prealloc_shrinker(struct shrinker *shrinker) if (!shrinker->nr_deferred) return -ENOMEM; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - if (prealloc_memcg_shrinker(shrinker)) - goto free_deferred; - } - return 0; - -free_deferred: - kfree(shrinker->nr_deferred); - shrinker->nr_deferred = NULL; - return -ENOMEM; } void free_prealloced_shrinker(struct shrinker *shrinker) { - if (!shrinker->nr_deferred) - return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) { down_write(&shrinker_rwsem); unregister_memcg_shrinker(shrinker); up_write(&shrinker_rwsem); + return; } kfree(shrinker->nr_deferred); From patchwork Thu Mar 11 19:08:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A782C433DB for ; Thu, 11 Mar 2021 19:09:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0288D64EC0 for ; Thu, 11 Mar 2021 19:09:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0288D64EC0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6F0988D02F0; Thu, 11 Mar 2021 14:09:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 677BC8D02B2; Thu, 11 Mar 2021 14:09:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A4EF8D02F0; Thu, 11 Mar 2021 14:09:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0241.hostedemail.com [216.40.44.241]) by kanga.kvack.org (Postfix) with ESMTP id 20F108D02B2 for ; Thu, 11 Mar 2021 14:09:27 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id C682B181AF5CC for ; Thu, 11 Mar 2021 19:09:26 +0000 (UTC) X-FDA: 77908531932.25.EA5BF36 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf08.hostedemail.com (Postfix) with ESMTP id D8BC78019140 for ; Thu, 11 Mar 2021 19:09:18 +0000 (UTC) Received: by mail-pl1-f174.google.com with SMTP id n17so7135703plc.7 for ; Thu, 11 Mar 2021 11:09:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=387U6EUXBh7mLus/UDVh51X6DyrD4kEMfbx59TkIzlA=; b=ckPBzV+wjhR0jD6bD3xGl+wIqtfYTy22s8zEq9ESXN0mhXp9+zcVvvHlw3gxtFQPKT HRMXK7vIOLygJ54Uifyobx+ZVSjYyKNy3EUYp7N5A10/cn2ZlqM0x+OH6NI47iSMV4Gk 4oOY5NFCM5rH6st4OXi5JWySRl05hCS+SFhxmvxx5l8B4JLWzDBzbIIbNXgy3SHcEtEE cq7hJO5htnHv3DWlJ4wTfFnmUCijyxR70rjXUGMmOLGtWrctfVpkI9sxv25xJGWaF7O8 gpY+b5g+vtqaSyUqWYe3TJ8T90RfZZDRKBWkwBB3zzBgYF9r0WMGR5MI5AP4mEGB6cxk 03jQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=387U6EUXBh7mLus/UDVh51X6DyrD4kEMfbx59TkIzlA=; b=nDsPtr3fq/VaYX5RtrzPLHQpxSodDlbpsg6k9kOowCxfXxgyUynw9NScxxbgn9Aizj HRvNf6LkoyR5Y0orDTISLAWFXqG7YNGBT+8U7VJ0luj0kE7eFqDG8EAU5lXa4LsGBhjy DiuOFp8TzTLIJH2+jKmlTWnQuhvdq3AL7oizicxYeqfrQkHcnVhUgVk5RYfqqC5PEu6q grSc5GP6EA7LNiRz4dYtG+OKmFuITHhQVzxPzDSDOp/WL5u2vTLXDtxRJYTP+1Fj5aYR sTXAfw34oOlBAHRxQWghTHVKoo99+i+IKEMmW14CkaqIkYCKaZVODMHH/BzeJ5ZJwUoC WmQQ== X-Gm-Message-State: AOAM530PPoRPrK4gSuWGS9KBs6famK5RfUiTje+YknEkBYaAKJQCgj4L sMDs3a7S1h7/opx2IX+u1a0= X-Google-Smtp-Source: ABdhPJyQw3LropgZkq0gCBA33nFDKXoIlKKNSZisQrGtPL1ucO8aoBJm/2M9nRLHLxAcIIw7eGjC5g== X-Received: by 2002:a17:902:d64a:b029:e6:30a6:64e3 with SMTP id y10-20020a170902d64ab02900e630a664e3mr9933203plh.28.1615489765521; Thu, 11 Mar 2021 11:09:25 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:24 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 12/13] mm: memcontrol: reparent nr_deferred when memcg offline Date: Thu, 11 Mar 2021 11:08:44 -0800 Message-Id: <20210311190845.9708-13-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D8BC78019140 X-Stat-Signature: yr8zceqa7d1qcjy83oc4soicrt4tpgjf Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf08; identity=mailfrom; envelope-from=""; helo=mail-pl1-f174.google.com; client-ip=209.85.214.174 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615489758-380398 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now shrinker's nr_deferred is per memcg for memcg aware shrinkers, add to parent's corresponding nr_deferred when memcg offline. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 1 + mm/vmscan.c | 24 ++++++++++++++++++++++++ 3 files changed, 26 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 24e735434a46..4064c9dda534 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1537,6 +1537,7 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) int alloc_shrinker_info(struct mem_cgroup *memcg); void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); +void reparent_shrinker_deferred(struct mem_cgroup *memcg); #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 35d44afdd9fc..a945dfc85156 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5167,6 +5167,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) page_counter_set_low(&memcg->memory, 0); memcg_offline_kmem(memcg); + reparent_shrinker_deferred(memcg); wb_memcg_offline(memcg); drain_all_stock(memcg); diff --git a/mm/vmscan.c b/mm/vmscan.c index 324c34c6e5cf..d0791ebd6761 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -397,6 +397,30 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } +void reparent_shrinker_deferred(struct mem_cgroup *memcg) +{ + int i, nid; + long nr; + struct mem_cgroup *parent; + struct shrinker_info *child_info, *parent_info; + + parent = parent_mem_cgroup(memcg); + if (!parent) + parent = root_mem_cgroup; + + /* Prevent from concurrent shrinker_info expand */ + down_read(&shrinker_rwsem); + for_each_node(nid) { + child_info = shrinker_info_protected(memcg, nid); + parent_info = shrinker_info_protected(parent, nid); + for (i = 0; i < shrinker_nr_max; i++) { + nr = atomic_long_read(&child_info->nr_deferred[i]); + atomic_long_add(nr, &parent_info->nr_deferred[i]); + } + } + up_read(&shrinker_rwsem); +} + static bool cgroup_reclaim(struct scan_control *sc) { return sc->target_mem_cgroup; From patchwork Thu Mar 11 19:08:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A986C433E0 for ; Thu, 11 Mar 2021 19:09:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0C65264EBB for ; Thu, 11 Mar 2021 19:09:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0C65264EBB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8EA2F8D02F1; Thu, 11 Mar 2021 14:09:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 872EA8D02B2; Thu, 11 Mar 2021 14:09:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6797A8D02F1; Thu, 11 Mar 2021 14:09:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0222.hostedemail.com [216.40.44.222]) by kanga.kvack.org (Postfix) with ESMTP id 4030A8D02B2 for ; Thu, 11 Mar 2021 14:09:29 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id F4197181AC9CB for ; Thu, 11 Mar 2021 19:09:28 +0000 (UTC) X-FDA: 77908532058.18.CD25F58 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf04.hostedemail.com (Postfix) with ESMTP id 76873130 for ; Thu, 11 Mar 2021 19:09:27 +0000 (UTC) Received: by mail-pg1-f177.google.com with SMTP id q5so1630696pgk.5 for ; Thu, 11 Mar 2021 11:09:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EGK/9wMv8jRkB25hW3W7zHOqmtj2V0f0Nz5E9gUfq7E=; b=ruApEZXuvYHuRSS42zVIXrt9gJn10TsmfS0rhjKeh7yTcyMqOtl+EUi621sWLY3dre JeLTadgUNwsgqpPn2aJrvqDzLnXySmAWzha19/qTRCmT0xHyh1WuhXdQpSIysjiUKTKV F4ShMw4xHCgAc1jmekBhITwEAR07rska7p6IgcsYESGaTX6NpLcQTE708grIJflhLBQk urtp6/waSHfkEb2Pz/w6zSGviQxZc+zw5JPdO9L4+q5QeSUIScgmLd0g65+QkyHuvM/F 8yZpU8GH3N4MLvLnAtlV8qvp7VWvd5rbMz9MtLhgCwr2+TD9hjppVEbgOxLBShEunfr4 F1xQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EGK/9wMv8jRkB25hW3W7zHOqmtj2V0f0Nz5E9gUfq7E=; b=Le/jlAI3b3g3Z73aodjsdo+/SdeZQAXBpsybj6ibYww2vldrmusKmIfenqjcAOqrx2 EOXmhseaCZWE7pcAm3vofAXBPa469UL9cXU9MeaWSkAqOIh2qjTuFIuBBjQqBkat+Hv6 KZ1dJZvWOFgpygFvvvqToHQGHTA87lWDINXHEGwxuQIdylHIg42FmrSrYrRCuqUW0kkv b/d7dNn/+kaeMbVkBzmwPW+WUx57zmd+HvS4N4jC9e7gj0TUIefNihPa9X9F6Cx/lG7x Tw6QPAhZTndixIhPk19mozHyfQntBnruLki306W+4Xzrpke5ECQf+h1ZRM12mUj/r3Fi RJ7A== X-Gm-Message-State: AOAM532bd+EkllMfyduE4/IoOXf/4STmI8fws84xRS6LwVX+V1mEqRhr 04ZptHduoR5UD7tZOHeDO4Q= X-Google-Smtp-Source: ABdhPJzYGwUvbjrr9D3b2YYS9YyC7I1ThLnks5DFJYnKtTpSfEdsyC9fjHwBMOZQXfqjjGMp8cSnLw== X-Received: by 2002:a63:1343:: with SMTP id 3mr8397519pgt.166.1615489767493; Thu, 11 Mar 2021 11:09:27 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:26 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 13/13] mm: vmscan: shrink deferred objects proportional to priority Date: Thu, 11 Mar 2021 11:08:45 -0800 Message-Id: <20210311190845.9708-14-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 76873130 X-Stat-Signature: kizrszct6u4xwyeddjq4zs1xynt1f8mc Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf04; identity=mailfrom; envelope-from=""; helo=mail-pg1-f177.google.com; client-ip=209.85.215.177 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615489767-433673 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The number of deferred objects might get windup to an absurd number, and it results in clamp of slab objects. It is undesirable for sustaining workingset. So shrink deferred objects proportional to priority and cap nr_deferred to twice of cache items. The idea is borrowed from Dave Chinner's patch: https://lore.kernel.org/linux-xfs/20191031234618.15403-13-david@fromorbit.com/ Tested with kernel build and vfs metadata heavy workload in our production environment, no regression is spotted so far. Signed-off-by: Yang Shi --- mm/vmscan.c | 46 +++++++++++----------------------------------- 1 file changed, 11 insertions(+), 35 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d0791ebd6761..163616e78a4e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -664,7 +664,6 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, */ nr = xchg_nr_deferred(shrinker, shrinkctl); - total_scan = nr; if (shrinker->seeks) { delta = freeable >> priority; delta *= 4; @@ -678,37 +677,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, delta = freeable / 2; } + total_scan = nr >> priority; total_scan += delta; - if (total_scan < 0) { - pr_err("shrink_slab: %pS negative objects to delete nr=%ld\n", - shrinker->scan_objects, total_scan); - total_scan = freeable; - next_deferred = nr; - } else - next_deferred = total_scan; - - /* - * We need to avoid excessive windup on filesystem shrinkers - * due to large numbers of GFP_NOFS allocations causing the - * shrinkers to return -1 all the time. This results in a large - * nr being built up so when a shrink that can do some work - * comes along it empties the entire cache due to nr >>> - * freeable. This is bad for sustaining a working set in - * memory. - * - * Hence only allow the shrinker to scan the entire cache when - * a large delta change is calculated directly. - */ - if (delta < freeable / 4) - total_scan = min(total_scan, freeable / 2); - - /* - * Avoid risking looping forever due to too large nr value: - * never try to free more than twice the estimate number of - * freeable entries. - */ - if (total_scan > freeable * 2) - total_scan = freeable * 2; + total_scan = min(total_scan, (2 * freeable)); trace_mm_shrink_slab_start(shrinker, shrinkctl, nr, freeable, delta, total_scan, priority); @@ -747,10 +718,15 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, cond_resched(); } - if (next_deferred >= scanned) - next_deferred -= scanned; - else - next_deferred = 0; + /* + * The deferred work is increased by any new work (delta) that wasn't + * done, decreased by old deferred work that was done now. + * + * And it is capped to two times of the freeable items. + */ + next_deferred = max_t(long, (nr + delta - scanned), 0); + next_deferred = min(next_deferred, (2 * freeable)); + /* * move the unused scan count back into the shrinker in a * manner that handles concurrent updates.