From patchwork Thu Mar 11 19:08:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85404C433DB for ; Thu, 11 Mar 2021 19:09:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4A68164E60 for ; Thu, 11 Mar 2021 19:09:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230033AbhCKTJG (ORCPT ); Thu, 11 Mar 2021 14:09:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35302 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229721AbhCKTJC (ORCPT ); Thu, 11 Mar 2021 14:09:02 -0500 Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EEDB2C061574; Thu, 11 Mar 2021 11:09:01 -0800 (PST) Received: by mail-pf1-x42d.google.com with SMTP id y13so212692pfr.0; Thu, 11 Mar 2021 11:09:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ad6rFdrjF9qkUCg0uueBGRPkzrqRpMOaGc3NS4qxCRQ=; b=vKEaDb0pRFGw1ac2fSVbByJ/bryyv9m8N66XhwfI1mxAIXCGvTtnVNSRknaDGOpu3r mF3ky3Se8SjadopZ5vbnZFBEAtxE+6nQMUWUQsIiqZYLtnIoCW76v532jRs1CJv6MJW0 UXSaDvB/qYPV0hTsi4Ofsshg6au5Kmv82jo+dQ9t7h1iXScyhgymtcd1OkWkMN1oO7SX fzGehcR1dmA2UtgSb0aaSuzWHC9mBk6khdIF8MdEBsDZVrYgl43eE3Y1jm2kG4jfVMzH /1raUVFgjAKf1VVMxt/sL6d7NmzzBfCWRM/JIwTbu05KOem0EBBq+foWfz4PzbZ1OoV6 z09w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ad6rFdrjF9qkUCg0uueBGRPkzrqRpMOaGc3NS4qxCRQ=; b=SHWtG776AkCzUYl9xDmtVGGULwrJjua9p6ZAHee8lM7rjYsmBKKJeUpOPZl8Z3xjcp 6Zy/03Idt98cDs5DzaRF0j4Ic8L5aN9sCv3BNaBd60x8EK74l9WvYwulWIm4adS2B+Pl r5aa/JRDWKR2ucT8orbZvQWqgk4iiRfMy+l2oinu3GeZsUUPZwM6YIsnWM6yQC1ZEK9j MBbkFh/jQ/CR8A48Al1xGPinYvjafqzl0Or6LqyKkgQa3JeDlCPWf6PxOu7R+CzKdIMe u3bOV16mw/SeyJFpCYIO4PbJODj1p+/IV1b1t+pIC9rdVUnsbZoZ5WgQz6L6kc8ruSsK vS7Q== X-Gm-Message-State: AOAM530CM3CXLOLiBwv//Ogs64HZAZsTwnO/V3m3P0+vDYvCdu979YLA GKgEDK57AdzIx8Q2MYhgZLI= X-Google-Smtp-Source: ABdhPJzkCuz0y/v1wgMPEFCdMw3j4bSNGMz2Ido7wOOTliNLwGqxRrstshQGOD0XEXWudM+Ej+Xhfg== X-Received: by 2002:a63:d601:: with SMTP id q1mr8342915pgg.417.1615489741567; Thu, 11 Mar 2021 11:09:01 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.08.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:00 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 01/13] mm: vmscan: use nid from shrink_control for tracepoint Date: Thu, 11 Mar 2021 11:08:33 -0800 Message-Id: <20210311190845.9708-2-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The tracepoint's nid should show what node the shrink happens on, the start tracepoint uses nid from shrinkctl, but the nid might be set to 0 before end tracepoint if the shrinker is not NUMA aware, so the tracing log may show the shrink happens on one node but end up on the other node. It seems confusing. And the following patch will remove using nid directly in do_shrink_slab(), this patch also helps cleanup the code. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Reviewed-by: Shakeel Butt Acked-by: Roman Gushchin Signed-off-by: Yang Shi --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 407051ebe869..bdc32c803c66 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -536,7 +536,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, else new_nr = atomic_long_read(&shrinker->nr_deferred[nid]); - trace_mm_shrink_slab_end(shrinker, nid, freed, nr, new_nr, total_scan); + trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); return freed; } From patchwork Thu Mar 11 19:08:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132489 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD879C433E0 for ; Thu, 11 Mar 2021 19:10:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5BFD364EC0 for ; Thu, 11 Mar 2021 19:10:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230076AbhCKTJm (ORCPT ); Thu, 11 Mar 2021 14:09:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229796AbhCKTJE (ORCPT ); Thu, 11 Mar 2021 14:09:04 -0500 Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FB2BC061574; Thu, 11 Mar 2021 11:09:04 -0800 (PST) Received: by mail-pf1-x42f.google.com with SMTP id q204so174918pfq.10; Thu, 11 Mar 2021 11:09:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SXVD1HfpEWEnPEpQNQZ2lptmmkG469kO8T607Rced4s=; b=EUuo4QwlFAcOak5Bd6ymMS/yjW6KIitnzDmweZ+6TMFTs28kzMP153UsCSwNHtB8GN ugv5lyBUhhNlU83j1V7JpEQoliqop1Q/rAAYy83B4rbWQf0A+Gn3m8NEWbY0op5Av6zB 4w0trsxiK0JWamO+YOPbZt7TEopBi9efsgdQL4bnapW6XkuA9B+YkpFG9H+oMr+kBEEI /lWkv23t+88xGcpnNJ9+rAYmEhY3qa9LvPx7MKG0EGzga6YZDoYnmrFEndeiBeXzbyOJ k1tM+ljirBNAuBvPLvi/eJbFzKWfBZXrdqzYGN7KSOuqooAyI/M4+mhgNzCXSSmZuVOS WyaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SXVD1HfpEWEnPEpQNQZ2lptmmkG469kO8T607Rced4s=; b=L10Y0xncfseiyG1ljI92+KBULnQJa0sCmQVLdMV3apLjDryQmP4BCpAfkZViY02ODq 9k6w6uVsgxg23LbXueZatUBy/cmF7GdQhyqU3gWX8feKyDNU/9lEk/2LaW3K3VhmCqDc GFBeegmGWxAmHEqpXqMIPoaLnV8n30ihPbArRL4DPyIdFPe/j9ZDzHbMKZqtBBZcQqEY gzYgRWSxAOAHrUjbNfBCIr3Rf9574WW2ZbuZAZFtwkJIXbHOsWxaF6cLtKW+tuU/WppH nNX1LUnrBu1X5t0L/NTvHbHkQru0sPr4Ex72MmWWfiF8VnVPpq4MVj1L21Q4FsmjNaFJ x7sA== X-Gm-Message-State: AOAM5308NhofPSEm/1NcVmRXIWxbb9McsnLbRYLFOHmQdT/tylfS4Dxv 7YPxpD5cFznu5xof0nobLf8= X-Google-Smtp-Source: ABdhPJx98HcHkFsDZbrn39hydWs6DZ5rXIRgUHXoKJBLC01b9UOlPPllqcquB2rNWDllsSt1a3aSEw== X-Received: by 2002:aa7:8498:0:b029:1f6:8a25:7ade with SMTP id u24-20020aa784980000b02901f68a257ademr8837086pfn.76.1615489743876; Thu, 11 Mar 2021 11:09:03 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:02 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 02/13] mm: vmscan: consolidate shrinker_maps handling code Date: Thu, 11 Mar 2021 11:08:34 -0800 Message-Id: <20210311190845.9708-3-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The shrinker map management is not purely memcg specific, it is at the intersection between memory cgroup and shrinkers. It's allocation and assignment of a structure, and the only memcg bit is the map is being stored in a memcg structure. So move the shrinker_maps handling code into vmscan.c for tighter integration with shrinker code, and remove the "memcg_" prefix. There is no functional change. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 11 ++-- mm/huge_memory.c | 4 +- mm/list_lru.c | 6 +- mm/memcontrol.c | 130 +----------------------------------- mm/vmscan.c | 132 ++++++++++++++++++++++++++++++++++++- 5 files changed, 142 insertions(+), 141 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e946c96daa32..b07dc2d5014d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1533,10 +1533,9 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -extern int memcg_expand_shrinker_maps(int new_id); - -extern void memcg_set_shrinker_bit(struct mem_cgroup *memcg, - int nid, int shrinker_id); +int alloc_shrinker_maps(struct mem_cgroup *memcg); +void free_shrinker_maps(struct mem_cgroup *memcg); +void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; @@ -1546,8 +1545,8 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -static inline void memcg_set_shrinker_bit(struct mem_cgroup *memcg, - int nid, int shrinker_id) +static inline void set_shrinker_bit(struct mem_cgroup *memcg, + int nid, int shrinker_id) { } #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index bff92dea5ab3..92a1227be029 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2836,8 +2836,8 @@ void deferred_split_huge_page(struct page *page) ds_queue->split_queue_len++; #ifdef CONFIG_MEMCG if (memcg) - memcg_set_shrinker_bit(memcg, page_to_nid(page), - deferred_split_shrinker.id); + set_shrinker_bit(memcg, page_to_nid(page), + deferred_split_shrinker.id); #endif } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); diff --git a/mm/list_lru.c b/mm/list_lru.c index 6f067b6b935f..cd58790d0fb3 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -125,8 +125,8 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item) list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) - memcg_set_shrinker_bit(memcg, nid, - lru_shrinker_id(lru)); + set_shrinker_bit(memcg, nid, + lru_shrinker_id(lru)); nlru->nr_items++; spin_unlock(&nlru->lock); return true; @@ -540,7 +540,7 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, if (src->nr_items) { dst->nr_items += src->nr_items; - memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); + set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index da6fded852df..aeb1847f159c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -402,130 +402,6 @@ DEFINE_STATIC_KEY_FALSE(memcg_kmem_enabled_key); EXPORT_SYMBOL(memcg_kmem_enabled_key); #endif -static int memcg_shrinker_map_size; -static DEFINE_MUTEX(memcg_shrinker_map_mutex); - -static void memcg_free_shrinker_map_rcu(struct rcu_head *head) -{ - kvfree(container_of(head, struct memcg_shrinker_map, rcu)); -} - -static int memcg_expand_one_shrinker_map(struct mem_cgroup *memcg, - int size, int old_size) -{ - struct memcg_shrinker_map *new, *old; - struct mem_cgroup_per_node *pn; - int nid; - - lockdep_assert_held(&memcg_shrinker_map_mutex); - - for_each_node(nid) { - pn = memcg->nodeinfo[nid]; - old = rcu_dereference_protected(pn->shrinker_map, true); - /* Not yet online memcg */ - if (!old) - return 0; - - new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); - if (!new) - return -ENOMEM; - - /* Set all old bits, clear all new bits */ - memset(new->map, (int)0xff, old_size); - memset((void *)new->map + old_size, 0, size - old_size); - - rcu_assign_pointer(pn->shrinker_map, new); - call_rcu(&old->rcu, memcg_free_shrinker_map_rcu); - } - - return 0; -} - -static void memcg_free_shrinker_maps(struct mem_cgroup *memcg) -{ - struct mem_cgroup_per_node *pn; - struct memcg_shrinker_map *map; - int nid; - - if (mem_cgroup_is_root(memcg)) - return; - - for_each_node(nid) { - pn = memcg->nodeinfo[nid]; - map = rcu_dereference_protected(pn->shrinker_map, true); - kvfree(map); - rcu_assign_pointer(pn->shrinker_map, NULL); - } -} - -static int memcg_alloc_shrinker_maps(struct mem_cgroup *memcg) -{ - struct memcg_shrinker_map *map; - int nid, size, ret = 0; - - if (mem_cgroup_is_root(memcg)) - return 0; - - mutex_lock(&memcg_shrinker_map_mutex); - size = memcg_shrinker_map_size; - for_each_node(nid) { - map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); - if (!map) { - memcg_free_shrinker_maps(memcg); - ret = -ENOMEM; - break; - } - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); - } - mutex_unlock(&memcg_shrinker_map_mutex); - - return ret; -} - -int memcg_expand_shrinker_maps(int new_id) -{ - int size, old_size, ret = 0; - struct mem_cgroup *memcg; - - size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); - old_size = memcg_shrinker_map_size; - if (size <= old_size) - return 0; - - mutex_lock(&memcg_shrinker_map_mutex); - if (!root_mem_cgroup) - goto unlock; - - for_each_mem_cgroup(memcg) { - if (mem_cgroup_is_root(memcg)) - continue; - ret = memcg_expand_one_shrinker_map(memcg, size, old_size); - if (ret) { - mem_cgroup_iter_break(NULL, memcg); - goto unlock; - } - } -unlock: - if (!ret) - memcg_shrinker_map_size = size; - mutex_unlock(&memcg_shrinker_map_mutex); - return ret; -} - -void memcg_set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) -{ - if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { - struct memcg_shrinker_map *map; - - rcu_read_lock(); - map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, map->map); - rcu_read_unlock(); - } -} - /** * mem_cgroup_css_from_page - css of the memcg associated with a page * @page: page of interest @@ -5255,11 +5131,11 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) struct mem_cgroup *memcg = mem_cgroup_from_css(css); /* - * A memcg must be visible for memcg_expand_shrinker_maps() + * A memcg must be visible for expand_shrinker_maps() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (memcg_alloc_shrinker_maps(memcg)) { + if (alloc_shrinker_maps(memcg)) { mem_cgroup_id_remove(memcg); return -ENOMEM; } @@ -5323,7 +5199,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) vmpressure_cleanup(&memcg->vmpressure); cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); - memcg_free_shrinker_maps(memcg); + free_shrinker_maps(memcg); memcg_free_kmem(memcg); mem_cgroup_free(memcg); } diff --git a/mm/vmscan.c b/mm/vmscan.c index bdc32c803c66..98a44fb81f8a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -185,6 +185,132 @@ static LIST_HEAD(shrinker_list); static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG + +static int memcg_shrinker_map_size; +static DEFINE_MUTEX(memcg_shrinker_map_mutex); + +static void free_shrinker_map_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct memcg_shrinker_map, rcu)); +} + +static int expand_one_shrinker_map(struct mem_cgroup *memcg, + int size, int old_size) +{ + struct memcg_shrinker_map *new, *old; + struct mem_cgroup_per_node *pn; + int nid; + + lockdep_assert_held(&memcg_shrinker_map_mutex); + + for_each_node(nid) { + pn = memcg->nodeinfo[nid]; + old = rcu_dereference_protected(pn->shrinker_map, true); + /* Not yet online memcg */ + if (!old) + return 0; + + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); + if (!new) + return -ENOMEM; + + /* Set all old bits, clear all new bits */ + memset(new->map, (int)0xff, old_size); + memset((void *)new->map + old_size, 0, size - old_size); + + rcu_assign_pointer(pn->shrinker_map, new); + call_rcu(&old->rcu, free_shrinker_map_rcu); + } + + return 0; +} + +void free_shrinker_maps(struct mem_cgroup *memcg) +{ + struct mem_cgroup_per_node *pn; + struct memcg_shrinker_map *map; + int nid; + + if (mem_cgroup_is_root(memcg)) + return; + + for_each_node(nid) { + pn = memcg->nodeinfo[nid]; + map = rcu_dereference_protected(pn->shrinker_map, true); + kvfree(map); + rcu_assign_pointer(pn->shrinker_map, NULL); + } +} + +int alloc_shrinker_maps(struct mem_cgroup *memcg) +{ + struct memcg_shrinker_map *map; + int nid, size, ret = 0; + + if (mem_cgroup_is_root(memcg)) + return 0; + + mutex_lock(&memcg_shrinker_map_mutex); + size = memcg_shrinker_map_size; + for_each_node(nid) { + map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); + if (!map) { + free_shrinker_maps(memcg); + ret = -ENOMEM; + break; + } + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); + } + mutex_unlock(&memcg_shrinker_map_mutex); + + return ret; +} + +static int expand_shrinker_maps(int new_id) +{ + int size, old_size, ret = 0; + struct mem_cgroup *memcg; + + size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); + old_size = memcg_shrinker_map_size; + if (size <= old_size) + return 0; + + mutex_lock(&memcg_shrinker_map_mutex); + if (!root_mem_cgroup) + goto unlock; + + memcg = mem_cgroup_iter(NULL, NULL, NULL); + do { + if (mem_cgroup_is_root(memcg)) + continue; + ret = expand_one_shrinker_map(memcg, size, old_size); + if (ret) { + mem_cgroup_iter_break(NULL, memcg); + goto unlock; + } + } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); +unlock: + if (!ret) + memcg_shrinker_map_size = size; + mutex_unlock(&memcg_shrinker_map_mutex); + return ret; +} + +void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) +{ + if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { + struct memcg_shrinker_map *map; + + rcu_read_lock(); + map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, map->map); + rcu_read_unlock(); + } +} + /* * We allow subsystems to populate their shrinker-related * LRU lists before register_shrinker_prepared() is called @@ -212,7 +338,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) goto unlock; if (id >= shrinker_nr_max) { - if (memcg_expand_shrinker_maps(id)) { + if (expand_shrinker_maps(id)) { idr_remove(&shrinker_idr, id); goto unlock; } @@ -590,7 +716,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, * case, we invoke the shrinker one more time and reset * the bit if it reports that it is not empty anymore. * The memory barrier here pairs with the barrier in - * memcg_set_shrinker_bit(): + * set_shrinker_bit(): * * list_lru_add() shrink_slab_memcg() * list_add_tail() clear_bit() @@ -602,7 +728,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; else - memcg_set_shrinker_bit(memcg, nid, i); + set_shrinker_bit(memcg, nid, i); } freed += ret; From patchwork Thu Mar 11 19:08:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91AF4C433DB for ; Thu, 11 Mar 2021 19:10:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 42C0864EC0 for ; Thu, 11 Mar 2021 19:10:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230269AbhCKTJn (ORCPT ); Thu, 11 Mar 2021 14:09:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35322 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230016AbhCKTJG (ORCPT ); Thu, 11 Mar 2021 14:09:06 -0500 Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 65944C061574; Thu, 11 Mar 2021 11:09:06 -0800 (PST) Received: by mail-pg1-x52b.google.com with SMTP id o38so14278150pgm.9; Thu, 11 Mar 2021 11:09:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=i2D+FuQfaS2ZRj4Qwe7N96xecJSI2gtUN3o6WXvG0vk=; b=q1nH2RikMTupiiFtmscgQ+W70fe3sDorZ4fRkLKO0XRwx+fzc24172LPMv2hziIcIF Sj0Gme3WOUqsQaMT129Zo5hCy7cm+AqdnoYV2Cuh9FuQW3Y2CCm0po075aPxccdcP7MH JQtHynViMEbBUKTFxHGjdWkTdjffgcUZdS9/R5dL+UTxEWcdWqze3uwdwYU8l/dGDHKq aJVp+6mo8dpTwNLt9NTxXBqI6czJdjPZbxuZ/IW1dEL8UGxEfeQs6DOWGB9+U/ModK9y xS9MFPQFUcmim0HU0A0XnS0WKcK5jKf7wSEN3pe5Dq2HVn+Qn8xBGNHIO3XjWMEIbKDG IwiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=i2D+FuQfaS2ZRj4Qwe7N96xecJSI2gtUN3o6WXvG0vk=; b=FC7wQHYW/Hdrde3KKvpwLWHa5e/O700h6aUMYbF2SLCujQpjHpCfUxT/tSJU4Mx8gd R/2ZMJO+JwL+g6tBJ52xDTd2CzM1oWzzXjopbvmomjTi9fy912AD4S9WOafmHq7HwOJu fe4ZVc+uf1Ih+ox6NfDMioI7cREl0BavI8dXh9phXrZAPz+/x1EvzYcgHRRUdlb2ocE2 L4S/eWnVhW47z0jnVIW3RuIRgXbQcysCKwUj9a9AdT3lA8RwJmQFUG8dUaxshOH0bkLM mpjYSkyTs+FDQ3TPZBX/fKWYXTvtFNZqnHugL5vzHKNfknjvwt1x6vVesZBSqvIsPfyz D2FQ== X-Gm-Message-State: AOAM530QKn1/Y4fbfaEmQMFG8v1VAUlretMTOxZG+P8UxY1wWnRgPLlq G9DyVm09eck1bgCdOnFqXao= X-Google-Smtp-Source: ABdhPJxrXzkIllUs4SsPxfmE/PuCtdbdraco1NMsd2Am1G7MNDDRxNRwRr7Wjx+o2aWZ6a4lBHN1eg== X-Received: by 2002:aa7:9431:0:b029:1f1:52fd:5444 with SMTP id y17-20020aa794310000b02901f152fd5444mr8525393pfo.47.1615489745932; Thu, 11 Mar 2021 11:09:05 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:05 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 03/13] mm: vmscan: use shrinker_rwsem to protect shrinker_maps allocation Date: Thu, 11 Mar 2021 11:08:35 -0800 Message-Id: <20210311190845.9708-4-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Since memcg_shrinker_map_size just can be changed under holding shrinker_rwsem exclusively, the read side can be protected by holding read lock, so it sounds superfluous to have a dedicated mutex. Kirill Tkhai suggested use write lock since: * We want the assignment to shrinker_maps is visible for shrink_slab_memcg(). * The rcu_dereference_protected() dereferrencing in shrink_slab_memcg(), but in case of we use READ lock in alloc_shrinker_maps(), the dereferrencing is not actually protected. * READ lock makes alloc_shrinker_info() racy against memory allocation fail. alloc_shrinker_info()->free_shrinker_info() may free memory right after shrink_slab_memcg() dereferenced it. You may say shrink_slab_memcg()->mem_cgroup_online() protects us from it? Yes, sure, but this is not the thing we want to remember in the future, since this spreads modularity. And a test with heavy paging workload didn't show write lock makes things worse. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 98a44fb81f8a..b08c8d9055ae 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -187,7 +187,6 @@ static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG static int memcg_shrinker_map_size; -static DEFINE_MUTEX(memcg_shrinker_map_mutex); static void free_shrinker_map_rcu(struct rcu_head *head) { @@ -201,8 +200,6 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, struct mem_cgroup_per_node *pn; int nid; - lockdep_assert_held(&memcg_shrinker_map_mutex); - for_each_node(nid) { pn = memcg->nodeinfo[nid]; old = rcu_dereference_protected(pn->shrinker_map, true); @@ -250,7 +247,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) if (mem_cgroup_is_root(memcg)) return 0; - mutex_lock(&memcg_shrinker_map_mutex); + down_write(&shrinker_rwsem); size = memcg_shrinker_map_size; for_each_node(nid) { map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); @@ -261,7 +258,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) } rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); } - mutex_unlock(&memcg_shrinker_map_mutex); + up_write(&shrinker_rwsem); return ret; } @@ -276,9 +273,10 @@ static int expand_shrinker_maps(int new_id) if (size <= old_size) return 0; - mutex_lock(&memcg_shrinker_map_mutex); if (!root_mem_cgroup) - goto unlock; + goto out; + + lockdep_assert_held(&shrinker_rwsem); memcg = mem_cgroup_iter(NULL, NULL, NULL); do { @@ -287,13 +285,13 @@ static int expand_shrinker_maps(int new_id) ret = expand_one_shrinker_map(memcg, size, old_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); - goto unlock; + goto out; } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); -unlock: +out: if (!ret) memcg_shrinker_map_size = size; - mutex_unlock(&memcg_shrinker_map_mutex); + return ret; } From patchwork Thu Mar 11 19:08:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132491 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4679C433E9 for ; Thu, 11 Mar 2021 19:10:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 98B1664F13 for ; Thu, 11 Mar 2021 19:10:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230286AbhCKTJo (ORCPT ); Thu, 11 Mar 2021 14:09:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230131AbhCKTJI (ORCPT ); Thu, 11 Mar 2021 14:09:08 -0500 Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 695CBC061574; Thu, 11 Mar 2021 11:09:08 -0800 (PST) Received: by mail-pl1-x62b.google.com with SMTP id c16so10713098ply.0; Thu, 11 Mar 2021 11:09:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=x5MgFJmx+qCDHDBM5m9Z4h2MfyiNvh6Fg7+MEJeG5Hc=; b=hXphMEKp+eXWbBLvMNGFTgWa2pZEr+COjecriAAxFhNEeooa/BbyJ/YAUkkA5wytZg 1AOGSEuDCvR6ZGXii+4K3P587qy73WiAQZneeJ7C0uD7Z0mWY+a/BVm80vwEktHPLVTL h0JA6lJ4dixnOPuJ+vtVuF6X3lHth7AamKG6/i7xTYqmvTrxQhTaXeydTKIhQQ8uyErC cARoNTMaCQIoEqXH/PxRgv7SeHSbS16T9+VEylrr+clF2S7LDC5E05N64ru7HbzhA4pX 0jtq6H5/L4hmEqQHSIWq/c/kIuUfTpqG0Y+azFNdJvwnwW6cfCDMFHKjSJgtsN4MUgSl ZsCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x5MgFJmx+qCDHDBM5m9Z4h2MfyiNvh6Fg7+MEJeG5Hc=; b=GrBZDOmE8DyTKfwd5fUhl8wFHKx5BTVSVqpnBj4O00Y/zjxdrFUm4bE2qhNbeLhsXe liw1xGd7fDf3PC8q5+4SEjwvz+ycR0UwdXp61IzBLlAC+NOqlnTyfZuIU23Ge4lFKbQz cReCmXkE1jD2I9h1u4wk2HnltkL2rv1GjYzUm94Ktg2glGyzxqMfWjn3jiRvTZf9zQO2 ruE8BO/C3VbiMjTxo2hELGm2wmXk8U5a8wxdeQzZBVa49cYJjhRgliq8EqTVy/vhLji0 dTBJC8QnH5G9eV861WNJIoBHLCp44GQ7zoGP9Ygl5ww9ARfRipntD8rMu8CRbZtwDn3s mvLA== X-Gm-Message-State: AOAM531diJWDZMPRdrW+3ay4qIo6hIX3XUZQ7Lg+1bfzhePd/dO0JDkn /hhNfwAmcULwt3mZG2eE3Qg= X-Google-Smtp-Source: ABdhPJxjyKooM+K/uNd8RQ5pj9o2KzidC1fcsHNXVoUBXa8795yLlGRklotc+LNSeYmurtCbsO8ZbQ== X-Received: by 2002:a17:90b:3553:: with SMTP id lt19mr4840931pjb.222.1615489747922; Thu, 11 Mar 2021 11:09:07 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:07 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 04/13] mm: vmscan: remove memcg_shrinker_map_size Date: Thu, 11 Mar 2021 11:08:36 -0800 Message-Id: <20210311190845.9708-5-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Both memcg_shrinker_map_size and shrinker_nr_max is maintained, but actually the map size can be calculated via shrinker_nr_max, so it seems unnecessary to keep both. Remove memcg_shrinker_map_size since shrinker_nr_max is also used by iterating the bit map. Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Acked-by: Vlastimil Babka Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b08c8d9055ae..641a0b8b4ea9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -185,8 +185,12 @@ static LIST_HEAD(shrinker_list); static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG +static int shrinker_nr_max; -static int memcg_shrinker_map_size; +static inline int shrinker_map_size(int nr_items) +{ + return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); +} static void free_shrinker_map_rcu(struct rcu_head *head) { @@ -248,7 +252,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) return 0; down_write(&shrinker_rwsem); - size = memcg_shrinker_map_size; + size = shrinker_map_size(shrinker_nr_max); for_each_node(nid) { map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); if (!map) { @@ -266,12 +270,13 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) static int expand_shrinker_maps(int new_id) { int size, old_size, ret = 0; + int new_nr_max = new_id + 1; struct mem_cgroup *memcg; - size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); - old_size = memcg_shrinker_map_size; + size = shrinker_map_size(new_nr_max); + old_size = shrinker_map_size(shrinker_nr_max); if (size <= old_size) - return 0; + goto out; if (!root_mem_cgroup) goto out; @@ -290,7 +295,7 @@ static int expand_shrinker_maps(int new_id) } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); out: if (!ret) - memcg_shrinker_map_size = size; + shrinker_nr_max = new_nr_max; return ret; } @@ -323,7 +328,6 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) #define SHRINKER_REGISTERING ((struct shrinker *)~0UL) static DEFINE_IDR(shrinker_idr); -static int shrinker_nr_max; static int prealloc_memcg_shrinker(struct shrinker *shrinker) { @@ -340,8 +344,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) idr_remove(&shrinker_idr, id); goto unlock; } - - shrinker_nr_max = id + 1; } shrinker->id = id; ret = 0; From patchwork Thu Mar 11 19:08:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C152C4332E for ; Thu, 11 Mar 2021 19:10:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1A9A064EFE for ; Thu, 11 Mar 2021 19:10:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230301AbhCKTJp (ORCPT ); Thu, 11 Mar 2021 14:09:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230142AbhCKTJK (ORCPT ); Thu, 11 Mar 2021 14:09:10 -0500 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7BE86C061574; Thu, 11 Mar 2021 11:09:10 -0800 (PST) Received: by mail-pj1-x1036.google.com with SMTP id q2-20020a17090a2e02b02900bee668844dso9543574pjd.3; Thu, 11 Mar 2021 11:09:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uVGBdCD7D46KwnzDeHFbJFDsSaALpjrd7Gd+gGd1rXE=; b=LeRQjwnw2LeCRF7BgliFeq65y5FqMfUxpySNgLPB8dqZdAbhQXJ40f0eFMxW9ZtgGG E0PyIWo7ZNLhtfEW2BvloPIVyyCGRi+eJE5NS1W9uxh8vnnaGPDGH3tWoMIsh5z4AQTm hxSr5rUUO7S4wWeYBcp826RkRVHHQk6oSA3g706vUAtBpGUQVwgQoWexRVWjLYArFBiI OginCWSvf/ZVJjz1oYD6mFaQp/CuFxViPEO1jemusmUawvrw12sNUWHgWyw46IV8UU7q x08rY9hN+i8bF6OlEQF1ItrcygFnRzH2Q0KW/19CbdnCl0MlSTw+tZVmt9AczdOCx6xl KY3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uVGBdCD7D46KwnzDeHFbJFDsSaALpjrd7Gd+gGd1rXE=; b=pqCL1dvwcd8NCBi/mEbSNJ+1onve/jkAh3ANYU8yPGmfuMx3PeqmLNULaK2V+d1n0n h5LzDaGvZV4jSXjtFWYsPQnfzH4qWnlmtncdzLO1+nR2kYQT+ag5dxyZlf+VCzyRFIRZ 7ljC0k/pGXktRW6BUCO8W+qMiWUHt1OSRqfKRgsBhBJ34+lX/mH7sICWW9+t4QJIAlkf x044wt/tHCu0r81Bjd89Z/XtvcuQEmsg/mMylc9NAKNvMWrMmWaYa95TfCGB32x74wzp 8rezRwmX7OYmacm9lquqjU612gdZbFYAh34xRhMfuC+SA/pprD9PjfLWN9Npolsijqse nK1Q== X-Gm-Message-State: AOAM533S1nhcwBLK+CkY6RbiOdfM/560we+Fli0H9380nlYfVNI4rHy7 l4HjLt8WMtWDFuzf86b4IHg= X-Google-Smtp-Source: ABdhPJzDjGg0JWVU4rrN0/JBFOTry3IxyIysSN54EZUh/VsTJfCbQjf4QVKvcXFG8sZ5SEY6RKN2Pg== X-Received: by 2002:a17:90a:540c:: with SMTP id z12mr9963872pjh.163.1615489750107; Thu, 11 Mar 2021 11:09:10 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:09 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 05/13] mm: vmscan: use kvfree_rcu instead of call_rcu Date: Thu, 11 Mar 2021 11:08:37 -0800 Message-Id: <20210311190845.9708-6-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Using kvfree_rcu() to free the old shrinker_maps instead of call_rcu(). We don't have to define a dedicated callback for call_rcu() anymore. Acked-by: Roman Gushchin Acked-by: Kirill Tkhai Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 641a0b8b4ea9..bbe13985ae05 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -192,11 +192,6 @@ static inline int shrinker_map_size(int nr_items) return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); } -static void free_shrinker_map_rcu(struct rcu_head *head) -{ - kvfree(container_of(head, struct memcg_shrinker_map, rcu)); -} - static int expand_one_shrinker_map(struct mem_cgroup *memcg, int size, int old_size) { @@ -220,7 +215,7 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, memset((void *)new->map + old_size, 0, size - old_size); rcu_assign_pointer(pn->shrinker_map, new); - call_rcu(&old->rcu, free_shrinker_map_rcu); + kvfree_rcu(old, rcu); } return 0; From patchwork Thu Mar 11 19:08:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E6C5C4332B for ; Thu, 11 Mar 2021 19:10:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EEC5D64F05 for ; Thu, 11 Mar 2021 19:10:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230353AbhCKTJr (ORCPT ); Thu, 11 Mar 2021 14:09:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230143AbhCKTJM (ORCPT ); Thu, 11 Mar 2021 14:09:12 -0500 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6C6DC061574; Thu, 11 Mar 2021 11:09:12 -0800 (PST) Received: by mail-pj1-x1036.google.com with SMTP id t18so3935481pjs.3; Thu, 11 Mar 2021 11:09:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yTJ/H30mQn6R1SM8EhChDB9qQz/lBTytGLmfD9l2+II=; b=bKs5nQfvV9VNRtxx31OVFCDRjVIUA1MqSq0Qlg7NDlpuCeL1wAHmHtmIMAixLQz84t gPqu8nQa8Yo6lewYSj7868vMjMp7E13B28hlaGRZecAESSK19pvSf5/3tEEFZ3I5gnqN 83SHWyHA4YlXuoBJshYoOzb3nB8xQ0OG4UkP05L+mAPXiz8u+476k26xYw9lNzRiUO1u VHj5ADPOHJxzusnnpD0P5BPOKwkgBSyqYfgITS2LK6oTpYeiNZobE2yjjQwzYmqlOF1/ yCweqr+zHrLlL/Wv0Fi/rODH+MBcPWLvKCjaMJtgskIdVpAjdTfRvg7WrTBiFK0fSnvN 5X+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yTJ/H30mQn6R1SM8EhChDB9qQz/lBTytGLmfD9l2+II=; b=XIvQEMAK4kkGNeDCZFkXpjyX9pGqApq+2GnMWr1JY0qnWZDwazaXUyQ6EiclBK6i0V 0ktO6Czjt+PV2NWBWwNtzG7I+Z+O70IDzOFGa91iLLlv0GvqcxRTPGJtoMe6Kcu+VPpu wYfZQJ7mHJLk6HztopowuiG/k5aulGpSCi8mhZHmvowbJIBbo0U294Uiw77N8gS6syqS b6VxvVc9TGpcJUBTagG0ejk6ZRwbc2VdzCwBOefH9wt4Ft8VBWT3j7iRWnylXSPmxuAq VUBaZ55lSYe0/N+nqZPS5RC3TLcIWlrGvAto2X/ZzRE0jXetRx2k3isuI8yiBJLFinYa iqdg== X-Gm-Message-State: AOAM530ivnHw8Rxjc5hVX2U0tqnTsJqFaT1nYvwnYSTiro4ageKsnCZT ebJ3V7cHMb1NaVPTLCVMr3A= X-Google-Smtp-Source: ABdhPJwOllGQof/4c+nVC7dRgHvATP7IsRB2CncLFsjLly9r/Xyx9wDnDSbkHWVV8oVmalZuJixtNQ== X-Received: by 2002:a17:90a:a4cb:: with SMTP id l11mr10054163pjw.144.1615489752274; Thu, 11 Mar 2021 11:09:12 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:11 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 06/13] mm: memcontrol: rename shrinker_map to shrinker_info Date: Thu, 11 Mar 2021 11:08:38 -0800 Message-Id: <20210311190845.9708-7-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The following patch is going to add nr_deferred into shrinker_map, the change will make shrinker_map not only include map anymore, so rename it to "memcg_shrinker_info". And this should make the patch adding nr_deferred cleaner and readable and make review easier. Also remove the "memcg_" prefix. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 8 +++--- mm/memcontrol.c | 6 ++-- mm/vmscan.c | 58 +++++++++++++++++++------------------- 3 files changed, 36 insertions(+), 36 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b07dc2d5014d..dc7d0e2cb3ad 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -117,7 +117,7 @@ struct batched_lruvec_stat { * Bitmap of shrinker::id corresponding to memcg-aware shrinkers, * which have elements charged to this memcg. */ -struct memcg_shrinker_map { +struct shrinker_info { struct rcu_head rcu; unsigned long map[]; }; @@ -145,7 +145,7 @@ struct mem_cgroup_per_node { struct mem_cgroup_reclaim_iter iter; - struct memcg_shrinker_map __rcu *shrinker_map; + struct shrinker_info __rcu *shrinker_info; struct rb_node tree_node; /* RB tree node */ unsigned long usage_in_excess;/* Set to the value by which */ @@ -1533,8 +1533,8 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -int alloc_shrinker_maps(struct mem_cgroup *memcg); -void free_shrinker_maps(struct mem_cgroup *memcg); +int alloc_shrinker_info(struct mem_cgroup *memcg); +void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); #else #define mem_cgroup_sockets_enabled 0 diff --git a/mm/memcontrol.c b/mm/memcontrol.c index aeb1847f159c..35d44afdd9fc 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5131,11 +5131,11 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) struct mem_cgroup *memcg = mem_cgroup_from_css(css); /* - * A memcg must be visible for expand_shrinker_maps() + * A memcg must be visible for expand_shrinker_info() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (alloc_shrinker_maps(memcg)) { + if (alloc_shrinker_info(memcg)) { mem_cgroup_id_remove(memcg); return -ENOMEM; } @@ -5199,7 +5199,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) vmpressure_cleanup(&memcg->vmpressure); cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); - free_shrinker_maps(memcg); + free_shrinker_info(memcg); memcg_free_kmem(memcg); mem_cgroup_free(memcg); } diff --git a/mm/vmscan.c b/mm/vmscan.c index bbe13985ae05..7fdfdacf9a1f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -192,16 +192,16 @@ static inline int shrinker_map_size(int nr_items) return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); } -static int expand_one_shrinker_map(struct mem_cgroup *memcg, - int size, int old_size) +static int expand_one_shrinker_info(struct mem_cgroup *memcg, + int size, int old_size) { - struct memcg_shrinker_map *new, *old; + struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; int nid; for_each_node(nid) { pn = memcg->nodeinfo[nid]; - old = rcu_dereference_protected(pn->shrinker_map, true); + old = rcu_dereference_protected(pn->shrinker_info, true); /* Not yet online memcg */ if (!old) return 0; @@ -214,17 +214,17 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, memset(new->map, (int)0xff, old_size); memset((void *)new->map + old_size, 0, size - old_size); - rcu_assign_pointer(pn->shrinker_map, new); + rcu_assign_pointer(pn->shrinker_info, new); kvfree_rcu(old, rcu); } return 0; } -void free_shrinker_maps(struct mem_cgroup *memcg) +void free_shrinker_info(struct mem_cgroup *memcg) { struct mem_cgroup_per_node *pn; - struct memcg_shrinker_map *map; + struct shrinker_info *info; int nid; if (mem_cgroup_is_root(memcg)) @@ -232,15 +232,15 @@ void free_shrinker_maps(struct mem_cgroup *memcg) for_each_node(nid) { pn = memcg->nodeinfo[nid]; - map = rcu_dereference_protected(pn->shrinker_map, true); - kvfree(map); - rcu_assign_pointer(pn->shrinker_map, NULL); + info = rcu_dereference_protected(pn->shrinker_info, true); + kvfree(info); + rcu_assign_pointer(pn->shrinker_info, NULL); } } -int alloc_shrinker_maps(struct mem_cgroup *memcg) +int alloc_shrinker_info(struct mem_cgroup *memcg) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; int nid, size, ret = 0; if (mem_cgroup_is_root(memcg)) @@ -249,20 +249,20 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) down_write(&shrinker_rwsem); size = shrinker_map_size(shrinker_nr_max); for_each_node(nid) { - map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); - if (!map) { - free_shrinker_maps(memcg); + info = kvzalloc_node(sizeof(*info) + size, GFP_KERNEL, nid); + if (!info) { + free_shrinker_info(memcg); ret = -ENOMEM; break; } - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); return ret; } -static int expand_shrinker_maps(int new_id) +static int expand_shrinker_info(int new_id) { int size, old_size, ret = 0; int new_nr_max = new_id + 1; @@ -282,7 +282,7 @@ static int expand_shrinker_maps(int new_id) do { if (mem_cgroup_is_root(memcg)) continue; - ret = expand_one_shrinker_map(memcg, size, old_size); + ret = expand_one_shrinker_info(memcg, size, old_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -298,13 +298,13 @@ static int expand_shrinker_maps(int new_id) void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; rcu_read_lock(); - map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); + info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); - set_bit(shrinker_id, map->map); + set_bit(shrinker_id, info->map); rcu_read_unlock(); } } @@ -335,7 +335,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) goto unlock; if (id >= shrinker_nr_max) { - if (expand_shrinker_maps(id)) { + if (expand_shrinker_info(id)) { idr_remove(&shrinker_idr, id); goto unlock; } @@ -665,7 +665,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, int priority) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; unsigned long ret, freed = 0; int i; @@ -675,12 +675,12 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (!down_read_trylock(&shrinker_rwsem)) return 0; - map = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_map, - true); - if (unlikely(!map)) + info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + true); + if (unlikely(!info)) goto unlock; - for_each_set_bit(i, map->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, shrinker_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -691,7 +691,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, shrinker = idr_find(&shrinker_idr, i); if (unlikely(!shrinker || shrinker == SHRINKER_REGISTERING)) { if (!shrinker) - clear_bit(i, map->map); + clear_bit(i, info->map); continue; } @@ -702,7 +702,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, ret = do_shrink_slab(&sc, shrinker, priority); if (ret == SHRINK_EMPTY) { - clear_bit(i, map->map); + clear_bit(i, info->map); /* * After the shrinker reported that it had no objects to * free, but before we cleared the corresponding bit in From patchwork Thu Mar 11 19:08:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132501 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 984CBC43331 for ; Thu, 11 Mar 2021 19:10:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7459A64EC0 for ; Thu, 11 Mar 2021 19:10:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230385AbhCKTJr (ORCPT ); Thu, 11 Mar 2021 14:09:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35366 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230145AbhCKTJP (ORCPT ); Thu, 11 Mar 2021 14:09:15 -0500 Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1120C061574; Thu, 11 Mar 2021 11:09:14 -0800 (PST) Received: by mail-pj1-x1031.google.com with SMTP id gb6so3940110pjb.0; Thu, 11 Mar 2021 11:09:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=m2bLEDF9fLJ0ubUMCl5gxJ4ENWjURCD7GlteN0I+Pn0=; b=Mi6pTvW8VJpVzzmhCHBU6uqews5NbK1wW8i7GWTUtcQJm40tQfsU3q/4i1dMSxHChA fk6MOedbmi4Dz8b7j7f1wh094bz8Tx5NrY7UsuzP2ASxI72pn0IuoBBVuMk43qFKPe8C 1cZA14u5QtwgFBzSURdke1P3WrCzExGbRVN1esGKANq6X5K9sbVyb3V7XoqeNexexucz FPYL/JPcPEexxog48emnyY+vMOIt9v1CyZKrbKjjc4BFkKUTaiNbT3hhy29DxT0AdsAK Rzk+DWX1+6urZv54JXCIQfJ7IxTYAkD9RnK4LQFq1LcW0b8nwp6SP+d25idwc3fprVC+ Iv3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=m2bLEDF9fLJ0ubUMCl5gxJ4ENWjURCD7GlteN0I+Pn0=; b=HaDrr8kRB5AKnmXPvLXWwyc4xkDhWJtwWwwsqRLbah8brOg25HMV69eHpikoplDXwM 2KGBZS0LX5GikzORAdbbVtEonlvIV7rQ77qZTtDXfy5tv1bd88CbYT64HHujyIQqBqrb 5v3EwJtMLcFtUHhu49/jnO6OgmIO+xVDdlUDfAnHwgQyGTbN9sGQFdd30sqjx+IcR5Yx 1ir9HtvVcf4W3IikTdr2Eme717cARK2Li2dmuFxDELgaWuk8VyAUOepesmh/TWvxuTqj 3hvXUuQzcdpPPu8oFNn2bQA75u5sQsvijvwa13VW0JEAfMoQJre0cf4H5IctiShnkW9s IKag== X-Gm-Message-State: AOAM530jRhTYl+t0WIRtaG+2+4Z0deNnAePciUfVBZN6m3AiOZF0Moi4 AFdL29HpSbzNEg0+PA9KIf4= X-Google-Smtp-Source: ABdhPJy6nmhaP70hLjuP7UUkSI2vRg+R6cLVH31ME6nZFnyfjO/2L99AjBImPmBY4M11zbaB7ha1TA== X-Received: by 2002:a17:90a:fd0b:: with SMTP id cv11mr10319170pjb.183.1615489754458; Thu, 11 Mar 2021 11:09:14 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:13 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 07/13] mm: vmscan: add shrinker_info_protected() helper Date: Thu, 11 Mar 2021 11:08:39 -0800 Message-Id: <20210311190845.9708-8-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The shrinker_info is dereferenced in a couple of places via rcu_dereference_protected with different calling conventions, for example, using mem_cgroup_nodeinfo helper or dereferencing memcg->nodeinfo[nid]->shrinker_info. And the later patch will add more dereference places. So extract the dereference into a helper to make the code more readable. No functional change. Acked-by: Roman Gushchin Acked-by: Kirill Tkhai Acked-by: Vlastimil Babka Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 7fdfdacf9a1f..ef9f1531a6ee 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -192,6 +192,13 @@ static inline int shrinker_map_size(int nr_items) return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); } +static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, + int nid) +{ + return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + lockdep_is_held(&shrinker_rwsem)); +} + static int expand_one_shrinker_info(struct mem_cgroup *memcg, int size, int old_size) { @@ -201,7 +208,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, for_each_node(nid) { pn = memcg->nodeinfo[nid]; - old = rcu_dereference_protected(pn->shrinker_info, true); + old = shrinker_info_protected(memcg, nid); /* Not yet online memcg */ if (!old) return 0; @@ -232,7 +239,7 @@ void free_shrinker_info(struct mem_cgroup *memcg) for_each_node(nid) { pn = memcg->nodeinfo[nid]; - info = rcu_dereference_protected(pn->shrinker_info, true); + info = shrinker_info_protected(memcg, nid); kvfree(info); rcu_assign_pointer(pn->shrinker_info, NULL); } @@ -675,8 +682,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (!down_read_trylock(&shrinker_rwsem)) return 0; - info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - true); + info = shrinker_info_protected(memcg, nid); if (unlikely(!info)) goto unlock; From patchwork Thu Mar 11 19:08:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132499 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E835EC43332 for ; Thu, 11 Mar 2021 19:10:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BE25764F0C for ; Thu, 11 Mar 2021 19:10:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230397AbhCKTJs (ORCPT ); Thu, 11 Mar 2021 14:09:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229884AbhCKTJR (ORCPT ); Thu, 11 Mar 2021 14:09:17 -0500 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 798F3C061574; Thu, 11 Mar 2021 11:09:17 -0800 (PST) Received: by mail-pj1-x1029.google.com with SMTP id kr3-20020a17090b4903b02900c096fc01deso9822664pjb.4; Thu, 11 Mar 2021 11:09:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YP+QHK4if14P2cPRJw16+qngvdLDRS9w75Y8NgWmP/4=; b=UMjzDnKIdce/WKJadmRH1OB/wiqQKKr+f1T+PfUDbCDkayv4q0SdCmFnxCvebi1rmK N0X+YvJKnGtGn/u5SmEzUuZuWZmPf0Xzb5cMXwth9/bWEi+Km1Ro1LMWQIXQlLNX7aMt FJ9LDWfDcvXOLwBsKG0KkLK2O2YsOgAJHSzPaKzCXI0TXt/KR9uBTseGJj0o6y3u5RmK ezEVLLxogRYTP8XAjxzK0ifp7zhKzVu1/9ebNHs6DsCuhHe9muaqwY4VKeFF/N4xl+gP eDWje8rFc72MqW0JxBm6Htkeu4oAwzwSgf5GCerc0BydB0mrdHuunXPOwuMSa2wqjCr5 O/nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YP+QHK4if14P2cPRJw16+qngvdLDRS9w75Y8NgWmP/4=; b=LQAbyp2bSBIwz6mN/u/7BOIqxw5D46MYQ+pi9sqToolTM6zMuFkk3zbzJvdHbGUAQt N4Qlf2JkCLuaWLwRYIux/5E5Iws+JcJ92NseC+Hl0K23HlOl5jT03nhIHWRdoVImTDAh fL+8TnZ8/FXJCekF5lhmH+UKP31xQcZQix/AfMTqUayU5xbGQDTmdMCvaKUPZhzSGlV3 KXwQzA6amDqKh1P40UJLf7UdjzM7A9Fw3c7hxizeQ0m0bPa7rzUnZwxrSx4ousb9YCDy seHbp3JSqZ7gMrceKsFop6TAPZCg46ofmVrLgwYcXCTeD3ECzkfACtt1b+Gd2+yMElA5 pwSw== X-Gm-Message-State: AOAM530RqKJ58RZNroRret+C7qU9YnfXjXopPFKnTxCRPTkaSkoogRXq GYZaIH2g8tb73v/9406yPDg= X-Google-Smtp-Source: ABdhPJwPuyvEh2XO3JP8w7da5IZt5jzq/Fiw92FtaiLp0S9sNPUyursRbVNVjwbuTZ8/7zTGL7GBjA== X-Received: by 2002:a17:902:cec8:b029:e4:a497:da8d with SMTP id d8-20020a170902cec8b02900e4a497da8dmr9643886plg.16.1615489756447; Thu, 11 Mar 2021 11:09:16 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:15 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 08/13] mm: vmscan: use a new flag to indicate shrinker is registered Date: Thu, 11 Mar 2021 11:08:40 -0800 Message-Id: <20210311190845.9708-9-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Currently registered shrinker is indicated by non-NULL shrinker->nr_deferred. This approach is fine with nr_deferred at the shrinker level, but the following patches will move MEMCG_AWARE shrinkers' nr_deferred to memcg level, so their shrinker->nr_deferred would always be NULL. This would prevent the shrinkers from unregistering correctly. Remove SHRINKER_REGISTERING since we could check if shrinker is registered successfully by the new flag. Acked-by: Kirill Tkhai Acked-by: Vlastimil Babka Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/shrinker.h | 7 ++++--- mm/vmscan.c | 40 +++++++++++++++------------------------- 2 files changed, 19 insertions(+), 28 deletions(-) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 0f80123650e2..1eac79ce57d4 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -79,13 +79,14 @@ struct shrinker { #define DEFAULT_SEEKS 2 /* A good number if you don't know better. */ /* Flags */ -#define SHRINKER_NUMA_AWARE (1 << 0) -#define SHRINKER_MEMCG_AWARE (1 << 1) +#define SHRINKER_REGISTERED (1 << 0) +#define SHRINKER_NUMA_AWARE (1 << 1) +#define SHRINKER_MEMCG_AWARE (1 << 2) /* * It just makes sense when the shrinker is also MEMCG_AWARE for now, * non-MEMCG_AWARE shrinker should not have this flag set. */ -#define SHRINKER_NONSLAB (1 << 2) +#define SHRINKER_NONSLAB (1 << 3) extern int prealloc_shrinker(struct shrinker *shrinker); extern void register_shrinker_prepared(struct shrinker *shrinker); diff --git a/mm/vmscan.c b/mm/vmscan.c index ef9f1531a6ee..34cf3d84309c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -316,19 +316,6 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) } } -/* - * We allow subsystems to populate their shrinker-related - * LRU lists before register_shrinker_prepared() is called - * for the shrinker, since we don't want to impose - * restrictions on their internal registration order. - * In this case shrink_slab_memcg() may find corresponding - * bit is set in the shrinkers map. - * - * This value is used by the function to detect registering - * shrinkers and to skip do_shrink_slab() calls for them. - */ -#define SHRINKER_REGISTERING ((struct shrinker *)~0UL) - static DEFINE_IDR(shrinker_idr); static int prealloc_memcg_shrinker(struct shrinker *shrinker) @@ -337,7 +324,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) down_write(&shrinker_rwsem); /* This may call shrinker, so it must use down_read_trylock() */ - id = idr_alloc(&shrinker_idr, SHRINKER_REGISTERING, 0, 0, GFP_KERNEL); + id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -360,9 +347,9 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) BUG_ON(id < 0); - down_write(&shrinker_rwsem); + lockdep_assert_held(&shrinker_rwsem); + idr_remove(&shrinker_idr, id); - up_write(&shrinker_rwsem); } static bool cgroup_reclaim(struct scan_control *sc) @@ -490,8 +477,11 @@ void free_prealloced_shrinker(struct shrinker *shrinker) if (!shrinker->nr_deferred) return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) + if (shrinker->flags & SHRINKER_MEMCG_AWARE) { + down_write(&shrinker_rwsem); unregister_memcg_shrinker(shrinker); + up_write(&shrinker_rwsem); + } kfree(shrinker->nr_deferred); shrinker->nr_deferred = NULL; @@ -501,10 +491,7 @@ void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); list_add_tail(&shrinker->list, &shrinker_list); -#ifdef CONFIG_MEMCG - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - idr_replace(&shrinker_idr, shrinker, shrinker->id); -#endif + shrinker->flags |= SHRINKER_REGISTERED; up_write(&shrinker_rwsem); } @@ -524,13 +511,16 @@ EXPORT_SYMBOL(register_shrinker); */ void unregister_shrinker(struct shrinker *shrinker) { - if (!shrinker->nr_deferred) + if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - unregister_memcg_shrinker(shrinker); + down_write(&shrinker_rwsem); list_del(&shrinker->list); + shrinker->flags &= ~SHRINKER_REGISTERED; + if (shrinker->flags & SHRINKER_MEMCG_AWARE) + unregister_memcg_shrinker(shrinker); up_write(&shrinker_rwsem); + kfree(shrinker->nr_deferred); shrinker->nr_deferred = NULL; } @@ -695,7 +685,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct shrinker *shrinker; shrinker = idr_find(&shrinker_idr, i); - if (unlikely(!shrinker || shrinker == SHRINKER_REGISTERING)) { + if (unlikely(!shrinker || !(shrinker->flags & SHRINKER_REGISTERED))) { if (!shrinker) clear_bit(i, info->map); continue; From patchwork Thu Mar 11 19:08:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132509 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2E95C43333 for ; Thu, 11 Mar 2021 19:10:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9696564E60 for ; Thu, 11 Mar 2021 19:10:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230404AbhCKTJs (ORCPT ); Thu, 11 Mar 2021 14:09:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35384 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230150AbhCKTJU (ORCPT ); Thu, 11 Mar 2021 14:09:20 -0500 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2EF9C061574; Thu, 11 Mar 2021 11:09:19 -0800 (PST) Received: by mail-pl1-x632.google.com with SMTP id a24so10679609plm.11; Thu, 11 Mar 2021 11:09:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=O2soj3VRJNBvxi9PWCn4DecTfTo8NOmelRWHo0xmQP0=; b=uRBsx+Kax6NkZ9uc+8SG6XZVcl8rkcflBM04WQkKe1hczZa1EixUiP1LbAu/TXH5hg aubdIpn0WNho5jsbSkooNSxqL+0DoAV7/yLncRMbbAPHNrWjm0Gx0FQ7c/1xiSsF0T7w NB1Cnzo8DSLhgxKQt7ME2aPCPXHczsjcj6JDCf4SK1FyAU2ZoLaSanGJLs89l2972BFU ns5j3EUPu5KWJXRwJ83uO2U93T1inYVPo7DB7nLqglpWDOIEky7EVHBnbI73wmB8P8Cp WvhGvR4WOhdI6nKogLkB1VXMSkTfFg/FEdhqY8kvdoWm5K6DD4F0QHchHO42vyknUHq0 2Chw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=O2soj3VRJNBvxi9PWCn4DecTfTo8NOmelRWHo0xmQP0=; b=JxiIKglifiH09/N52sE9koody0hFHz4mkOSpxYwo82O4F81KRxCcTxW5tb/3/I+XOW 7lX1Ae/an7dHslxhUmDU7N9wm5wONar6D4LRjzXDleyhX/yKWvq7D9MtoMFAJjPA7bRT pqp6ad3miyeGx6scp05OApjlPARzVTRGikyPClSQIg67se8Y9XgzMj1h0WNlOcQW+nX8 KJOk2mLDWASmgnUNki7+RmsorMacRbo291JYzJbfbIDuuXFPbaamsrZgOuw6H9RaH+/5 q12S80GmLIl0sKO8tTl0A+aCADg57Dq6w6p45wfwLq70idvfMcodOyAkF4Lofczo6wpj qXuQ== X-Gm-Message-State: AOAM533ncRW2cFq2sUxdLdxe8M2bAzQ0ufprTozFnLC/8SS3vdnQxryP 1akzK0o3A6qtGLSmnQ2valg= X-Google-Smtp-Source: ABdhPJxCLYddJof6Hlc1njrzRqI8meh0rFXR/jtVGiDahS5BvuBn7cw7/b5cJIri8oPuKD+CJDP5KQ== X-Received: by 2002:a17:902:aa0a:b029:e4:c090:ad76 with SMTP id be10-20020a170902aa0ab02900e4c090ad76mr9687150plb.2.1615489759072; Thu, 11 Mar 2021 11:09:19 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:18 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 09/13] mm: vmscan: add per memcg shrinker nr_deferred Date: Thu, 11 Mar 2021 11:08:41 -0800 Message-Id: <20210311190845.9708-10-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Currently the number of deferred objects are per shrinker, but some slabs, for example, vfs inode/dentry cache are per memcg, this would result in poor isolation among memcgs. The deferred objects typically are generated by __GFP_NOFS allocations, one memcg with excessive __GFP_NOFS allocations may blow up deferred objects, then other innocent memcgs may suffer from over shrink, excessive reclaim latency, etc. For example, two workloads run in memcgA and memcgB respectively, workload in B is vfs heavy workload. Workload in A generates excessive deferred objects, then B's vfs cache might be hit heavily (drop half of caches) by B's limit reclaim or global reclaim. We observed this hit in our production environment which was running vfs heavy workload shown as the below tracing log: <...>-409454 [016] .... 28286961.747146: mm_shrink_slab_start: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 1 objects to shrink 3641681686040 gfp_flags GFP_HIGHUSER_MOVABLE|__GFP_ZERO pgs_scanned 1 lru_pgs 15721 cache items 246404277 delta 31345 total_scan 123202138 <...>-409454 [022] .... 28287105.928018: mm_shrink_slab_end: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 1 unused scan count 3641681686040 new scan count 3641798379189 total_scan 602 last shrinker return val 123186855 The vfs cache and page cache ratio was 10:1 on this machine, and half of caches were dropped. This also resulted in significant amount of page caches were dropped due to inodes eviction. Make nr_deferred per memcg for memcg aware shrinkers would solve the unfairness and bring better isolation. The following patch will add nr_deferred to parent memcg when memcg offline. To preserve nr_deferred when reparenting memcgs to root, root memcg needs shrinker_info allocated too. When memcg is not enabled (!CONFIG_MEMCG or memcg disabled), the shrinker's nr_deferred would be used. And non memcg aware shrinkers use shrinker's nr_deferred all the time. Acked-by: Roman Gushchin Acked-by: Kirill Tkhai Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 7 +++-- mm/vmscan.c | 60 ++++++++++++++++++++++++++------------ 2 files changed, 46 insertions(+), 21 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index dc7d0e2cb3ad..24e735434a46 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -114,12 +114,13 @@ struct batched_lruvec_stat { }; /* - * Bitmap of shrinker::id corresponding to memcg-aware shrinkers, - * which have elements charged to this memcg. + * Bitmap and deferred work of shrinker::id corresponding to memcg-aware + * shrinkers, which have elements charged to this memcg. */ struct shrinker_info { struct rcu_head rcu; - unsigned long map[]; + atomic_long_t *nr_deferred; + unsigned long *map; }; /* diff --git a/mm/vmscan.c b/mm/vmscan.c index 34cf3d84309c..397f3b67bad8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -187,11 +187,17 @@ static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG static int shrinker_nr_max; +/* The shrinker_info is expanded in a batch of BITS_PER_LONG */ static inline int shrinker_map_size(int nr_items) { return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); } +static inline int shrinker_defer_size(int nr_items) +{ + return (round_up(nr_items, BITS_PER_LONG) * sizeof(atomic_long_t)); +} + static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { @@ -200,11 +206,13 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, } static int expand_one_shrinker_info(struct mem_cgroup *memcg, - int size, int old_size) + int map_size, int defer_size, + int old_map_size, int old_defer_size) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; int nid; + int size = map_size + defer_size; for_each_node(nid) { pn = memcg->nodeinfo[nid]; @@ -217,9 +225,16 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!new) return -ENOMEM; - /* Set all old bits, clear all new bits */ - memset(new->map, (int)0xff, old_size); - memset((void *)new->map + old_size, 0, size - old_size); + new->nr_deferred = (atomic_long_t *)(new + 1); + new->map = (void *)new->nr_deferred + defer_size; + + /* map: set all old bits, clear all new bits */ + memset(new->map, (int)0xff, old_map_size); + memset((void *)new->map + old_map_size, 0, map_size - old_map_size); + /* nr_deferred: copy old values, clear all new values */ + memcpy(new->nr_deferred, old->nr_deferred, old_defer_size); + memset((void *)new->nr_deferred + old_defer_size, 0, + defer_size - old_defer_size); rcu_assign_pointer(pn->shrinker_info, new); kvfree_rcu(old, rcu); @@ -234,9 +249,6 @@ void free_shrinker_info(struct mem_cgroup *memcg) struct shrinker_info *info; int nid; - if (mem_cgroup_is_root(memcg)) - return; - for_each_node(nid) { pn = memcg->nodeinfo[nid]; info = shrinker_info_protected(memcg, nid); @@ -249,12 +261,12 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) { struct shrinker_info *info; int nid, size, ret = 0; - - if (mem_cgroup_is_root(memcg)) - return 0; + int map_size, defer_size = 0; down_write(&shrinker_rwsem); - size = shrinker_map_size(shrinker_nr_max); + map_size = shrinker_map_size(shrinker_nr_max); + defer_size = shrinker_defer_size(shrinker_nr_max); + size = map_size + defer_size; for_each_node(nid) { info = kvzalloc_node(sizeof(*info) + size, GFP_KERNEL, nid); if (!info) { @@ -262,6 +274,8 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) ret = -ENOMEM; break; } + info->nr_deferred = (atomic_long_t *)(info + 1); + info->map = (void *)info->nr_deferred + defer_size; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -269,15 +283,21 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } +static inline bool need_expand(int nr_max) +{ + return round_up(nr_max, BITS_PER_LONG) > + round_up(shrinker_nr_max, BITS_PER_LONG); +} + static int expand_shrinker_info(int new_id) { - int size, old_size, ret = 0; + int ret = 0; int new_nr_max = new_id + 1; + int map_size, defer_size = 0; + int old_map_size, old_defer_size = 0; struct mem_cgroup *memcg; - size = shrinker_map_size(new_nr_max); - old_size = shrinker_map_size(shrinker_nr_max); - if (size <= old_size) + if (!need_expand(new_nr_max)) goto out; if (!root_mem_cgroup) @@ -285,11 +305,15 @@ static int expand_shrinker_info(int new_id) lockdep_assert_held(&shrinker_rwsem); + map_size = shrinker_map_size(new_nr_max); + defer_size = shrinker_defer_size(new_nr_max); + old_map_size = shrinker_map_size(shrinker_nr_max); + old_defer_size = shrinker_defer_size(shrinker_nr_max); + memcg = mem_cgroup_iter(NULL, NULL, NULL); do { - if (mem_cgroup_is_root(memcg)) - continue; - ret = expand_one_shrinker_info(memcg, size, old_size); + ret = expand_one_shrinker_info(memcg, map_size, defer_size, + old_map_size, old_defer_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; From patchwork Thu Mar 11 19:08:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A82ECC4360C for ; Thu, 11 Mar 2021 19:10:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5D3B564EC0 for ; Thu, 11 Mar 2021 19:10:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229821AbhCKTJu (ORCPT ); Thu, 11 Mar 2021 14:09:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230152AbhCKTJW (ORCPT ); Thu, 11 Mar 2021 14:09:22 -0500 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 99C94C061760; Thu, 11 Mar 2021 11:09:21 -0800 (PST) Received: by mail-pj1-x102e.google.com with SMTP id q6-20020a17090a4306b02900c42a012202so9820439pjg.5; Thu, 11 Mar 2021 11:09:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tkw5+G90n94OUjiIXFC/vgHLYhl7hCOYiRf2Qw5jx3o=; b=AZcAO2+Zu7/HIsDOkfadeet7K1FjCBUqPJHcDw10r6Ip/XSXQgclt7/on3IYyaVxzb gVfFraVv9Ztw5I3oKVe3BZ+0bbxXZc9dxi9YjiQKKhZ54f4Mm7t538LK+afS4g9WAmlk ZUI3rjA2+8XWNmrElOpXt/U5+ruaHwy9YwDgDjAvMK8uy0435JphBeu107Q5p1bmAE38 EBMmd2n59UEAmYKPBEd1hi/iFeUyPjyP5NgQd9ECU+/ZD81eV3zRtWvSdYjad3VbncTI drbXs6PpDSyksEWLlBYEYfq+beef6niS2B+C13hy1CNOZ7jK1J/pQ424C5TVHyf6yocq o2Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tkw5+G90n94OUjiIXFC/vgHLYhl7hCOYiRf2Qw5jx3o=; b=qPusX28c9SRE7bAKbcvp49QwfFZco15+0eCeRgxvh9Qea+tjlohkCKiYXtnpoHjNeJ vBiBCRIQ9MfYiPnq1YQDtbUmh7764Kh0hBJDenabwRFnHZ1l2Sfrc3pPL2cpyOIQWZro gW18zq4R6GXUoabMoXm3Fo0GpNQv2bw7VzCqr9RfpUnIbXeq7TyKdTgM8aKbQnoNhfm1 lb/4//WG+I5jEFCp29fs68arfJrKiH9cpO9JatIISc9keH77SYiYSeserPXWLfXQdXDB A0BlVEs92JElkyHy4euGnGXFS9sXbt/JmT5k6s2c2ZUQuiH9xxRhD/hhxmxmLfsQmQMi U7tQ== X-Gm-Message-State: AOAM531va8A9hJ5pelr8KmGCOnTYS4vpEaHnmg66Jrw0r5HwAXfj5oaS smJXoRZGdVsmc9cNaLpWxd8= X-Google-Smtp-Source: ABdhPJxnH70sPhLOC3ftHwNORmFlXsCr7F56porR1CAZav7poDXAWwFtgfvr0AYbNHuZhf+apfLeCg== X-Received: by 2002:a17:902:7c8e:b029:e5:f70f:b7ab with SMTP id y14-20020a1709027c8eb02900e5f70fb7abmr9749952pll.32.1615489761244; Thu, 11 Mar 2021 11:09:21 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:20 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 10/13] mm: vmscan: use per memcg nr_deferred of shrinker Date: Thu, 11 Mar 2021 11:08:42 -0800 Message-Id: <20210311190845.9708-11-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Use per memcg's nr_deferred for memcg aware shrinkers. The shrinker's nr_deferred will be used in the following cases: 1. Non memcg aware shrinkers 2. !CONFIG_MEMCG 3. memcg is disabled by boot parameter Acked-by: Roman Gushchin Acked-by: Kirill Tkhai Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 78 ++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 66 insertions(+), 12 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 397f3b67bad8..5bc6975cb635 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -376,6 +376,24 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) idr_remove(&shrinker_idr, id); } +static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + + info = shrinker_info_protected(memcg, nid); + return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); +} + +static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + + info = shrinker_info_protected(memcg, nid); + return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); +} + static bool cgroup_reclaim(struct scan_control *sc) { return sc->target_mem_cgroup; @@ -414,6 +432,18 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) { } +static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + return 0; +} + +static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + return 0; +} + static bool cgroup_reclaim(struct scan_control *sc) { return false; @@ -425,6 +455,39 @@ static bool writeback_throttling_sane(struct scan_control *sc) } #endif +static long xchg_nr_deferred(struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (sc->memcg && + (shrinker->flags & SHRINKER_MEMCG_AWARE)) + return xchg_nr_deferred_memcg(nid, shrinker, + sc->memcg); + + return atomic_long_xchg(&shrinker->nr_deferred[nid], 0); +} + + +static long add_nr_deferred(long nr, struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (sc->memcg && + (shrinker->flags & SHRINKER_MEMCG_AWARE)) + return add_nr_deferred_memcg(nr, nid, shrinker, + sc->memcg); + + return atomic_long_add_return(nr, &shrinker->nr_deferred[nid]); +} + /* * This misses isolated pages which are not accounted for to save counters. * As the data only determines if reclaim or compaction continues, it is @@ -561,14 +624,10 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, long freeable; long nr; long new_nr; - int nid = shrinkctl->nid; long batch_size = shrinker->batch ? shrinker->batch : SHRINK_BATCH; long scanned = 0, next_deferred; - if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) - nid = 0; - freeable = shrinker->count_objects(shrinker, shrinkctl); if (freeable == 0 || freeable == SHRINK_EMPTY) return freeable; @@ -578,7 +637,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, * and zero it so that other concurrent shrinker invocations * don't also do this scanning work. */ - nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0); + nr = xchg_nr_deferred(shrinker, shrinkctl); total_scan = nr; if (shrinker->seeks) { @@ -669,14 +728,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, next_deferred = 0; /* * move the unused scan count back into the shrinker in a - * manner that handles concurrent updates. If we exhausted the - * scan, there is no need to do an update. + * manner that handles concurrent updates. */ - if (next_deferred > 0) - new_nr = atomic_long_add_return(next_deferred, - &shrinker->nr_deferred[nid]); - else - new_nr = atomic_long_read(&shrinker->nr_deferred[nid]); + new_nr = add_nr_deferred(next_deferred, shrinker, shrinkctl); trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); return freed; From patchwork Thu Mar 11 19:08:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132505 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DA88C432C3 for ; Thu, 11 Mar 2021 19:10:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DF16D64EC0 for ; Thu, 11 Mar 2021 19:10:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230410AbhCKTJt (ORCPT ); Thu, 11 Mar 2021 14:09:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35402 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230162AbhCKTJY (ORCPT ); Thu, 11 Mar 2021 14:09:24 -0500 Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9EC39C061574; Thu, 11 Mar 2021 11:09:23 -0800 (PST) Received: by mail-pl1-x630.google.com with SMTP id d8so10682420plg.10; Thu, 11 Mar 2021 11:09:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KRjLPceQFZWfN56pzAL4S7TxF614EkVjGd+2AFI1iu8=; b=Ba3D+YLmVCJ+UrsBxNrusvOurFMF0GizvxXy+rbFsNb20EJIN3RNgoNWWh29ly8tw1 nwxuuvd0heENI0w8dr4FLMw7jhR+Okm1bLT2nwfqyGR5ZPmmK4Tf1gybOnD7BHGX+aIr M2As+IkvjxMoc6yjkBTEsv+stBsNyg6GyW1SKRZbakQyo6hqc3SD5r2HAMlia1fTWcKh 6e7PCIuSZPQ8/9NIVLE1ynbs7pFmidWychavvHD9AYqsEvOyXSZWenF+CUQABb240Y1W LMFfTJDUrzJsEsWcTv3baolU9PdD5mTuEmqQiE5MNyqF/zJmb5g6vUzuq8JGitbwnptL W90Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KRjLPceQFZWfN56pzAL4S7TxF614EkVjGd+2AFI1iu8=; b=A0Q3qCZFeghV46a9ZOrSlosN91kM25k/VTGKurO+LbhiwFxuwMY38iObHgD2Ylsing zz3ZxSdEIKY+t3bPXcd2XTK74QHgsoi/3+W4K04hAKt3cHZKiLY1f74ryL/8r4Du4upW rt68fGm9U+cSQRYUL6x1ImKZ8I61owljKEluZCvHWSqgzRHEmxsYtCF9zoDGG1QgMdRv mSFVlAUs4vtO9vbq4b3XNiHuaZiXz8cZVHg3xTReNFJ/jvxsvjujRNipl+m8agP84Az4 Bai0VKmUnSkb12n3cLW4aFbsxqQWpHDHh/pTk6o/xuVa1eWvb6YRbp/nrii4IN8Gcq6x CvCg== X-Gm-Message-State: AOAM532QsXoZ+qjjpiwolXjsrasiAlmzcHAXRoqzwsFLWXTikrACus7n OSPYtv4Km9Cd7pI1Ej31zzg= X-Google-Smtp-Source: ABdhPJyi+9KnFTLz7v5ac0t6Oomsbt4Jf4POupwR6yJqXXAM05xu/dsaKrVr9lkiWkjJtFGJ48k1rw== X-Received: by 2002:a17:902:f547:b029:e4:6dbc:6593 with SMTP id h7-20020a170902f547b02900e46dbc6593mr9511144plf.4.1615489763282; Thu, 11 Mar 2021 11:09:23 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:22 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 11/13] mm: vmscan: don't need allocate shrinker->nr_deferred for memcg aware shrinkers Date: Thu, 11 Mar 2021 11:08:43 -0800 Message-Id: <20210311190845.9708-12-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Now nr_deferred is available on per memcg level for memcg aware shrinkers, so don't need allocate shrinker->nr_deferred for such shrinkers anymore. The prealloc_memcg_shrinker() would return -ENOSYS if !CONFIG_MEMCG or memcg is disabled by kernel command line, then shrinker's SHRINKER_MEMCG_AWARE flag would be cleared. This makes the implementation of this patch simpler. Acked-by: Vlastimil Babka Reviewed-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 5bc6975cb635..324c34c6e5cf 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -346,6 +346,9 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) { int id, ret = -ENOMEM; + if (mem_cgroup_disabled()) + return -ENOSYS; + down_write(&shrinker_rwsem); /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); @@ -425,7 +428,7 @@ static bool writeback_throttling_sane(struct scan_control *sc) #else static int prealloc_memcg_shrinker(struct shrinker *shrinker) { - return 0; + return -ENOSYS; } static void unregister_memcg_shrinker(struct shrinker *shrinker) @@ -537,8 +540,18 @@ static unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru, */ int prealloc_shrinker(struct shrinker *shrinker) { - unsigned int size = sizeof(*shrinker->nr_deferred); + unsigned int size; + int err; + + if (shrinker->flags & SHRINKER_MEMCG_AWARE) { + err = prealloc_memcg_shrinker(shrinker); + if (err != -ENOSYS) + return err; + shrinker->flags &= ~SHRINKER_MEMCG_AWARE; + } + + size = sizeof(*shrinker->nr_deferred); if (shrinker->flags & SHRINKER_NUMA_AWARE) size *= nr_node_ids; @@ -546,28 +559,16 @@ int prealloc_shrinker(struct shrinker *shrinker) if (!shrinker->nr_deferred) return -ENOMEM; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - if (prealloc_memcg_shrinker(shrinker)) - goto free_deferred; - } - return 0; - -free_deferred: - kfree(shrinker->nr_deferred); - shrinker->nr_deferred = NULL; - return -ENOMEM; } void free_prealloced_shrinker(struct shrinker *shrinker) { - if (!shrinker->nr_deferred) - return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) { down_write(&shrinker_rwsem); unregister_memcg_shrinker(shrinker); up_write(&shrinker_rwsem); + return; } kfree(shrinker->nr_deferred); From patchwork Thu Mar 11 19:08:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50ADDC43603 for ; Thu, 11 Mar 2021 19:10:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 12E0864E60 for ; Thu, 11 Mar 2021 19:10:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230426AbhCKTJv (ORCPT ); Thu, 11 Mar 2021 14:09:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230182AbhCKTJ0 (ORCPT ); Thu, 11 Mar 2021 14:09:26 -0500 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF62DC061574; Thu, 11 Mar 2021 11:09:25 -0800 (PST) Received: by mail-pj1-x1032.google.com with SMTP id kr3-20020a17090b4903b02900c096fc01deso9822832pjb.4; Thu, 11 Mar 2021 11:09:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=387U6EUXBh7mLus/UDVh51X6DyrD4kEMfbx59TkIzlA=; b=ckPBzV+wjhR0jD6bD3xGl+wIqtfYTy22s8zEq9ESXN0mhXp9+zcVvvHlw3gxtFQPKT HRMXK7vIOLygJ54Uifyobx+ZVSjYyKNy3EUYp7N5A10/cn2ZlqM0x+OH6NI47iSMV4Gk 4oOY5NFCM5rH6st4OXi5JWySRl05hCS+SFhxmvxx5l8B4JLWzDBzbIIbNXgy3SHcEtEE cq7hJO5htnHv3DWlJ4wTfFnmUCijyxR70rjXUGMmOLGtWrctfVpkI9sxv25xJGWaF7O8 gpY+b5g+vtqaSyUqWYe3TJ8T90RfZZDRKBWkwBB3zzBgYF9r0WMGR5MI5AP4mEGB6cxk 03jQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=387U6EUXBh7mLus/UDVh51X6DyrD4kEMfbx59TkIzlA=; b=UQZljw/7loRn+IBIruo3/HU9KtVt5F3Ms/g5P0fIRY/KD9AwaS2VGbkH+wUo02vZuo 2dhiwaIi/duVh2+ALR8rv54qAoxlvQm+cKBAzGSFoVHe6TbFyN+TsQLG4SsrLIEboaQY Lw1VvbsIUUQ8TgHLSEk5GygZTNLrhr/q9qUKEvXEg0oQ4djxu8WolVQQQBOQNNl9/D09 mkZQuDVbUtYpjZEDwWU2CM7y6zj58bz0hiVLCgW/7M7sACIVYxKpYfuglxNxepq4pmVp l4D7oh/BclpjqAKfg+JODA4tBsWJ5OF8m6suJ8neGTi6uRiWkh4cVkycyq4IqPQljhEG E6aw== X-Gm-Message-State: AOAM531rKrFQ1wWuBY5Iq1HRcd+Izk/O1oMlDGs9J9CdF8B9ENocF+wI kTw6+lrwD135GPnE947tl3E= X-Google-Smtp-Source: ABdhPJyQw3LropgZkq0gCBA33nFDKXoIlKKNSZisQrGtPL1ucO8aoBJm/2M9nRLHLxAcIIw7eGjC5g== X-Received: by 2002:a17:902:d64a:b029:e6:30a6:64e3 with SMTP id y10-20020a170902d64ab02900e630a664e3mr9933203plh.28.1615489765521; Thu, 11 Mar 2021 11:09:25 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:24 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 12/13] mm: memcontrol: reparent nr_deferred when memcg offline Date: Thu, 11 Mar 2021 11:08:44 -0800 Message-Id: <20210311190845.9708-13-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Now shrinker's nr_deferred is per memcg for memcg aware shrinkers, add to parent's corresponding nr_deferred when memcg offline. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 1 + mm/vmscan.c | 24 ++++++++++++++++++++++++ 3 files changed, 26 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 24e735434a46..4064c9dda534 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1537,6 +1537,7 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) int alloc_shrinker_info(struct mem_cgroup *memcg); void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); +void reparent_shrinker_deferred(struct mem_cgroup *memcg); #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 35d44afdd9fc..a945dfc85156 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5167,6 +5167,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) page_counter_set_low(&memcg->memory, 0); memcg_offline_kmem(memcg); + reparent_shrinker_deferred(memcg); wb_memcg_offline(memcg); drain_all_stock(memcg); diff --git a/mm/vmscan.c b/mm/vmscan.c index 324c34c6e5cf..d0791ebd6761 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -397,6 +397,30 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } +void reparent_shrinker_deferred(struct mem_cgroup *memcg) +{ + int i, nid; + long nr; + struct mem_cgroup *parent; + struct shrinker_info *child_info, *parent_info; + + parent = parent_mem_cgroup(memcg); + if (!parent) + parent = root_mem_cgroup; + + /* Prevent from concurrent shrinker_info expand */ + down_read(&shrinker_rwsem); + for_each_node(nid) { + child_info = shrinker_info_protected(memcg, nid); + parent_info = shrinker_info_protected(parent, nid); + for (i = 0; i < shrinker_nr_max; i++) { + nr = atomic_long_read(&child_info->nr_deferred[i]); + atomic_long_add(nr, &parent_info->nr_deferred[i]); + } + } + up_read(&shrinker_rwsem); +} + static bool cgroup_reclaim(struct scan_control *sc) { return sc->target_mem_cgroup; From patchwork Thu Mar 11 19:08:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12132511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BD45C4321A for ; Thu, 11 Mar 2021 19:10:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7C0A764F13 for ; Thu, 11 Mar 2021 19:10:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230431AbhCKTJv (ORCPT ); Thu, 11 Mar 2021 14:09:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230198AbhCKTJ2 (ORCPT ); Thu, 11 Mar 2021 14:09:28 -0500 Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F179CC061574; Thu, 11 Mar 2021 11:09:27 -0800 (PST) Received: by mail-pg1-x531.google.com with SMTP id p21so14283309pgl.12; Thu, 11 Mar 2021 11:09:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EGK/9wMv8jRkB25hW3W7zHOqmtj2V0f0Nz5E9gUfq7E=; b=ruApEZXuvYHuRSS42zVIXrt9gJn10TsmfS0rhjKeh7yTcyMqOtl+EUi621sWLY3dre JeLTadgUNwsgqpPn2aJrvqDzLnXySmAWzha19/qTRCmT0xHyh1WuhXdQpSIysjiUKTKV F4ShMw4xHCgAc1jmekBhITwEAR07rska7p6IgcsYESGaTX6NpLcQTE708grIJflhLBQk urtp6/waSHfkEb2Pz/w6zSGviQxZc+zw5JPdO9L4+q5QeSUIScgmLd0g65+QkyHuvM/F 8yZpU8GH3N4MLvLnAtlV8qvp7VWvd5rbMz9MtLhgCwr2+TD9hjppVEbgOxLBShEunfr4 F1xQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EGK/9wMv8jRkB25hW3W7zHOqmtj2V0f0Nz5E9gUfq7E=; b=YEvF7oVuHv8Sk7hE93MYiHzy9BHmQifJFcnKjoYeIwzl01sW9YGReZgEZO4f3KAB+n s6kzJb0Ou/MXnv9T8mWedlBk33g/alBKvNY8Z4k174sbWLeTxYynkAxkESU70bFNm+i5 bvD2+K8f697btghznhrvbBVCIrigsFuuAvoL+ZkddE2Fm795EGJW8z1WBhGi5BsXFQsh BBBQJecwp8J66T6GGrmNoWdaIgoUQm3z8B87X9455/LWM5qOq06txxSIbzE6PdZZKQIj QsKPde6Bj141L/OOOTBwuKHLOWEOj+yzrIcZRCpVV0ABhxxUnauZ4usr/Af4squMlgfT zfrw== X-Gm-Message-State: AOAM532Arwd3O+w8pZHeOsfqiaBiG9dft4kPq9j5FcBAAJT4fUiQY8Bu 9+mZQL8nhL2JKVIPvWpHiDk= X-Google-Smtp-Source: ABdhPJzYGwUvbjrr9D3b2YYS9YyC7I1ThLnks5DFJYnKtTpSfEdsyC9fjHwBMOZQXfqjjGMp8cSnLw== X-Received: by 2002:a63:1343:: with SMTP id 3mr8397519pgt.166.1615489767493; Thu, 11 Mar 2021 11:09:27 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.09.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:09:26 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 13/13] mm: vmscan: shrink deferred objects proportional to priority Date: Thu, 11 Mar 2021 11:08:45 -0800 Message-Id: <20210311190845.9708-14-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210311190845.9708-1-shy828301@gmail.com> References: <20210311190845.9708-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The number of deferred objects might get windup to an absurd number, and it results in clamp of slab objects. It is undesirable for sustaining workingset. So shrink deferred objects proportional to priority and cap nr_deferred to twice of cache items. The idea is borrowed from Dave Chinner's patch: https://lore.kernel.org/linux-xfs/20191031234618.15403-13-david@fromorbit.com/ Tested with kernel build and vfs metadata heavy workload in our production environment, no regression is spotted so far. Signed-off-by: Yang Shi --- mm/vmscan.c | 46 +++++++++++----------------------------------- 1 file changed, 11 insertions(+), 35 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d0791ebd6761..163616e78a4e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -664,7 +664,6 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, */ nr = xchg_nr_deferred(shrinker, shrinkctl); - total_scan = nr; if (shrinker->seeks) { delta = freeable >> priority; delta *= 4; @@ -678,37 +677,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, delta = freeable / 2; } + total_scan = nr >> priority; total_scan += delta; - if (total_scan < 0) { - pr_err("shrink_slab: %pS negative objects to delete nr=%ld\n", - shrinker->scan_objects, total_scan); - total_scan = freeable; - next_deferred = nr; - } else - next_deferred = total_scan; - - /* - * We need to avoid excessive windup on filesystem shrinkers - * due to large numbers of GFP_NOFS allocations causing the - * shrinkers to return -1 all the time. This results in a large - * nr being built up so when a shrink that can do some work - * comes along it empties the entire cache due to nr >>> - * freeable. This is bad for sustaining a working set in - * memory. - * - * Hence only allow the shrinker to scan the entire cache when - * a large delta change is calculated directly. - */ - if (delta < freeable / 4) - total_scan = min(total_scan, freeable / 2); - - /* - * Avoid risking looping forever due to too large nr value: - * never try to free more than twice the estimate number of - * freeable entries. - */ - if (total_scan > freeable * 2) - total_scan = freeable * 2; + total_scan = min(total_scan, (2 * freeable)); trace_mm_shrink_slab_start(shrinker, shrinkctl, nr, freeable, delta, total_scan, priority); @@ -747,10 +718,15 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, cond_resched(); } - if (next_deferred >= scanned) - next_deferred -= scanned; - else - next_deferred = 0; + /* + * The deferred work is increased by any new work (delta) that wasn't + * done, decreased by old deferred work that was done now. + * + * And it is capped to two times of the freeable items. + */ + next_deferred = max_t(long, (nr + delta - scanned), 0); + next_deferred = min(next_deferred, (2 * freeable)); + /* * move the unused scan count back into the shrinker in a * manner that handles concurrent updates.