From patchwork Wed Jan 27 23:33:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12051209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27FBFC433DB for ; Wed, 27 Jan 2021 23:33:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B413161601 for ; Wed, 27 Jan 2021 23:33:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B413161601 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 53BD86B006E; Wed, 27 Jan 2021 18:33:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C7D06B0070; Wed, 27 Jan 2021 18:33:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 319516B0071; Wed, 27 Jan 2021 18:33:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0028.hostedemail.com [216.40.44.28]) by kanga.kvack.org (Postfix) with ESMTP id 1DBE96B006E for ; Wed, 27 Jan 2021 18:33:55 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id DEE2F362E for ; Wed, 27 Jan 2021 23:33:54 +0000 (UTC) X-FDA: 77753159988.03.swing46_230f5b52759b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id C19C328A4E8 for ; Wed, 27 Jan 2021 23:33:54 +0000 (UTC) X-HE-Tag: swing46_230f5b52759b X-Filterd-Recvd-Size: 4123 Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 23:33:54 +0000 (UTC) Received: by mail-pg1-f175.google.com with SMTP id b21so2810297pgk.7 for ; Wed, 27 Jan 2021 15:33:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IYLQLgfEaFjCvTsN4FkoUtw26E88jjRQPD7HteqNuNs=; b=fVhQu50/1FFP1iQUMRi/V3eB1Md5hlpC59mDgdwc9B945ZzyL9Z4ZZZkf2RbkBzriH 1mLmt6p+kp5wXoHXyAlA4agAYM2SuSZkBu0q6LOtkmwh/GEKP7+r1l+E3TFwvSjtfLQf MyUSx9MdvFruk85ZiEFPx9QlW2ZPAsN9sl1N4NZjFsn+hiwQtXEN6Jp5v9knQHzuYuwA GiJ3hpyCPKIDmCyyBsVdx6PgCFwFUiViqmBlKNCQ9ManwCS3v9/43PpzNEebADTHta67 xdsDyPHcryNmlwXrYLQF73QMBh2X5umHlGhDE4Qg9mlH6tNnfi4oVK7eoWSm+Y5jWX1G VN0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IYLQLgfEaFjCvTsN4FkoUtw26E88jjRQPD7HteqNuNs=; b=qa7KKKMjuP36oUGiXuapGDtU3AcRlsdwDQ4Bj9o29ZV5kB0IOTBZQo77xluOP5p39G CDhpvXrWO820dDEcHFuxH/yiwULNfZLMDCAgHM8T0ZJ+loQFbZysfJIqC+zCKcN5Avh5 3GCKwQduaZ7AcZxLYuQasWWxNzc9pj2zHi/rtkgMeSqOfF+2lO8EUCPAUyPAqSprVKIO rX+pObwURflRB1tY+8A4w1S664Z3F60iu7ato+B5nBQFZI7UXqy9Aev/iNBsnAz3VZD1 SEKHfC+kEDBquNwbPahxSeJp87m+eXpNQIkrE4MSjh0vKmxow6Bpm899nfaBT3b+cKcv CAjw== X-Gm-Message-State: AOAM531E8bf1ghf1d9HAq9vrO9ENIewHegHIyabsGnzNtpc+/UzyNoSc zi2HrAYIug7Hvk30/UoOCB8= X-Google-Smtp-Source: ABdhPJz5IsrFDmE2tZbDrvhPA00PE+Jp+IUlNVHV9LIeCdCOqtE8kXCvzP0WOvGajeFJqqCnB7g3cw== X-Received: by 2002:a63:1f54:: with SMTP id q20mr13701298pgm.135.1611790433585; Wed, 27 Jan 2021 15:33:53 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id 124sm3498648pfd.59.2021.01.27.15.33.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 15:33:52 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 01/11] mm: vmscan: use nid from shrink_control for tracepoint Date: Wed, 27 Jan 2021 15:33:35 -0800 Message-Id: <20210127233345.339910-2-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210127233345.339910-1-shy828301@gmail.com> References: <20210127233345.339910-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The tracepoint's nid should show what node the shrink happens on, the start tracepoint uses nid from shrinkctl, but the nid might be set to 0 before end tracepoint if the shrinker is not NUMA aware, so the traceing log may show the shrink happens on one node but end up on the other node. It seems confusing. And the following patch will remove using nid directly in do_shrink_slab(), this patch also helps cleanup the code. Signed-off-by: Yang Shi Acked-by: Vlastimil Babka --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b1b574ad199d..b512dd5e3a1c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -535,7 +535,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, else new_nr = atomic_long_read(&shrinker->nr_deferred[nid]); - trace_mm_shrink_slab_end(shrinker, nid, freed, nr, new_nr, total_scan); + trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); return freed; } From patchwork Wed Jan 27 23:33:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12051211 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA26FC433DB for ; Wed, 27 Jan 2021 23:33:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 800E061601 for ; Wed, 27 Jan 2021 23:33:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 800E061601 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 22FF66B0070; Wed, 27 Jan 2021 18:33:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 203516B0071; Wed, 27 Jan 2021 18:33:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0AABC6B0072; Wed, 27 Jan 2021 18:33:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0164.hostedemail.com [216.40.44.164]) by kanga.kvack.org (Postfix) with ESMTP id E3E4F6B0070 for ; Wed, 27 Jan 2021 18:33:57 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id AD016180AD838 for ; Wed, 27 Jan 2021 23:33:57 +0000 (UTC) X-FDA: 77753160114.22.head66_2a044732759b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id 548C11802B157 for ; Wed, 27 Jan 2021 23:33:57 +0000 (UTC) X-HE-Tag: head66_2a044732759b X-Filterd-Recvd-Size: 14881 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf29.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 23:33:56 +0000 (UTC) Received: by mail-pf1-f174.google.com with SMTP id m6so2490572pfk.1 for ; Wed, 27 Jan 2021 15:33:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EuCgItErx3mwuUfHlTHM7xjza45qOuWeaGuXNrAI598=; b=fL1vXxuh469ps/ZMe7z3gDHMlaLENwKYPEUvymAjnDXjdgSBwB2DMDEHdntpScegMn GbSAATL3R88s9axg9mBEqicycWnjus4JZ8F83bZ7GesdTbnTAtJK/3uOGET/RHrmr+uI 66MnjK7wY5fxeddtqCmYFZ/vlxvicCD34T0vhBb+DhQpau4C3a5oGAYj0RA4ZvI980t3 iO+WKxBFC1m2LeuuMAtzHi83Zkpr4Wm7XF7YkHUW0qXFFzFPQWhg0rou1I3bpG92Z4Mc Qbf0c4l4PZfCAr+eaVa7QDDktxbppIg9EyXxf3sxabiMWrvy5JG0QtzFYK85h3G7X6Hm JAtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EuCgItErx3mwuUfHlTHM7xjza45qOuWeaGuXNrAI598=; b=d0yhr3+bXQ+eMzGhmVeyO9VIfUVKqgXYS5uf0o/mU60hrDwvSO0KR/QbU8vPRKrYyD okqQHgzku2C6a4mDgbjApU0roCa2vbtu0LEeYwai0xwPl89gLx8hEmw4lkHAO4HGJWb1 t3IEyukH3kkATjZT8FB2x+ESZ39qOFOAs9/bvL/J01inhAqNxbfJseYFFINUJvG0cyIh elVIlFvv/jGHDbaT9VfiwWs58nEWfmyy3GOJqQJ8vI9n7nFxF8/DFxICwZhgrM4xfWoq CQHpGDIPR4JBmwlB2his2gA2FgdklAzlDFcARJ00BCyzRzy0F3Wsdf8OkYFA2MytzFto qxrw== X-Gm-Message-State: AOAM530WijAWA1WphEPCoGyUQax4w0nbU6Tro0RboLRdC/ehkaw5IX78 ZFigUbPTC4YCeFpLFA0MgFk= X-Google-Smtp-Source: ABdhPJynCu+Hpo9rcWyzbCKoIAjclQBtHH+5BgAdOkIw/tZhe8K92z1tTByxnkim2Qzn6ljdiETYxA== X-Received: by 2002:a63:5f93:: with SMTP id t141mr13638281pgb.299.1611790435797; Wed, 27 Jan 2021 15:33:55 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id 124sm3498648pfd.59.2021.01.27.15.33.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 15:33:54 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 02/11] mm: vmscan: consolidate shrinker_maps handling code Date: Wed, 27 Jan 2021 15:33:36 -0800 Message-Id: <20210127233345.339910-3-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210127233345.339910-1-shy828301@gmail.com> References: <20210127233345.339910-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shrinker map management is not purely memcg specific, it is at the intersection between memory cgroup and shrinkers. It's allocation and assignment of a structure, and the only memcg bit is the map is being stored in a memcg structure. So move the shrinker_maps handling code into vmscan.c for tighter integration with shrinker code, and remove the "memcg_" prefix. There is no functional change. Signed-off-by: Yang Shi Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 12 ++-- mm/huge_memory.c | 4 +- mm/list_lru.c | 6 +- mm/memcontrol.c | 130 +------------------------------------ mm/vmscan.c | 130 ++++++++++++++++++++++++++++++++++++- 5 files changed, 142 insertions(+), 140 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index eeb0b52203e9..0ee2924991fb 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1581,10 +1581,10 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -extern int memcg_expand_shrinker_maps(int new_id); - -extern void memcg_set_shrinker_bit(struct mem_cgroup *memcg, - int nid, int shrinker_id); +extern int alloc_shrinker_maps(struct mem_cgroup *memcg); +extern void free_shrinker_maps(struct mem_cgroup *memcg); +extern void set_shrinker_bit(struct mem_cgroup *memcg, + int nid, int shrinker_id); #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; @@ -1594,8 +1594,8 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -static inline void memcg_set_shrinker_bit(struct mem_cgroup *memcg, - int nid, int shrinker_id) +static inline void set_shrinker_bit(struct mem_cgroup *memcg, + int nid, int shrinker_id) { } #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9237976abe72..05190d7f32ae 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2823,8 +2823,8 @@ void deferred_split_huge_page(struct page *page) ds_queue->split_queue_len++; #ifdef CONFIG_MEMCG if (memcg) - memcg_set_shrinker_bit(memcg, page_to_nid(page), - deferred_split_shrinker.id); + set_shrinker_bit(memcg, page_to_nid(page), + deferred_split_shrinker.id); #endif } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); diff --git a/mm/list_lru.c b/mm/list_lru.c index fe230081690b..628030fa5f69 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -125,8 +125,8 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item) list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) - memcg_set_shrinker_bit(memcg, nid, - lru_shrinker_id(lru)); + set_shrinker_bit(memcg, nid, + lru_shrinker_id(lru)); nlru->nr_items++; spin_unlock(&nlru->lock); return true; @@ -548,7 +548,7 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, if (src->nr_items) { dst->nr_items += src->nr_items; - memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); + set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e2de77b5bcc2..f5c9a0d2160b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -397,130 +397,6 @@ DEFINE_STATIC_KEY_FALSE(memcg_kmem_enabled_key); EXPORT_SYMBOL(memcg_kmem_enabled_key); #endif -static int memcg_shrinker_map_size; -static DEFINE_MUTEX(memcg_shrinker_map_mutex); - -static void memcg_free_shrinker_map_rcu(struct rcu_head *head) -{ - kvfree(container_of(head, struct memcg_shrinker_map, rcu)); -} - -static int memcg_expand_one_shrinker_map(struct mem_cgroup *memcg, - int size, int old_size) -{ - struct memcg_shrinker_map *new, *old; - int nid; - - lockdep_assert_held(&memcg_shrinker_map_mutex); - - for_each_node(nid) { - old = rcu_dereference_protected( - mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); - /* Not yet online memcg */ - if (!old) - return 0; - - new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); - if (!new) - return -ENOMEM; - - /* Set all old bits, clear all new bits */ - memset(new->map, (int)0xff, old_size); - memset((void *)new->map + old_size, 0, size - old_size); - - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, new); - call_rcu(&old->rcu, memcg_free_shrinker_map_rcu); - } - - return 0; -} - -static void memcg_free_shrinker_maps(struct mem_cgroup *memcg) -{ - struct mem_cgroup_per_node *pn; - struct memcg_shrinker_map *map; - int nid; - - if (mem_cgroup_is_root(memcg)) - return; - - for_each_node(nid) { - pn = mem_cgroup_nodeinfo(memcg, nid); - map = rcu_dereference_protected(pn->shrinker_map, true); - if (map) - kvfree(map); - rcu_assign_pointer(pn->shrinker_map, NULL); - } -} - -static int memcg_alloc_shrinker_maps(struct mem_cgroup *memcg) -{ - struct memcg_shrinker_map *map; - int nid, size, ret = 0; - - if (mem_cgroup_is_root(memcg)) - return 0; - - mutex_lock(&memcg_shrinker_map_mutex); - size = memcg_shrinker_map_size; - for_each_node(nid) { - map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); - if (!map) { - memcg_free_shrinker_maps(memcg); - ret = -ENOMEM; - break; - } - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); - } - mutex_unlock(&memcg_shrinker_map_mutex); - - return ret; -} - -int memcg_expand_shrinker_maps(int new_id) -{ - int size, old_size, ret = 0; - struct mem_cgroup *memcg; - - size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); - old_size = memcg_shrinker_map_size; - if (size <= old_size) - return 0; - - mutex_lock(&memcg_shrinker_map_mutex); - if (!root_mem_cgroup) - goto unlock; - - for_each_mem_cgroup(memcg) { - if (mem_cgroup_is_root(memcg)) - continue; - ret = memcg_expand_one_shrinker_map(memcg, size, old_size); - if (ret) { - mem_cgroup_iter_break(NULL, memcg); - goto unlock; - } - } -unlock: - if (!ret) - memcg_shrinker_map_size = size; - mutex_unlock(&memcg_shrinker_map_mutex); - return ret; -} - -void memcg_set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) -{ - if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { - struct memcg_shrinker_map *map; - - rcu_read_lock(); - map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, map->map); - rcu_read_unlock(); - } -} - /** * mem_cgroup_css_from_page - css of the memcg associated with a page * @page: page of interest @@ -5370,11 +5246,11 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) struct mem_cgroup *memcg = mem_cgroup_from_css(css); /* - * A memcg must be visible for memcg_expand_shrinker_maps() + * A memcg must be visible for expand_shrinker_maps() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (memcg_alloc_shrinker_maps(memcg)) { + if (alloc_shrinker_maps(memcg)) { mem_cgroup_id_remove(memcg); return -ENOMEM; } @@ -5438,7 +5314,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) vmpressure_cleanup(&memcg->vmpressure); cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); - memcg_free_shrinker_maps(memcg); + free_shrinker_maps(memcg); memcg_free_kmem(memcg); mem_cgroup_free(memcg); } diff --git a/mm/vmscan.c b/mm/vmscan.c index b512dd5e3a1c..d950cead66ca 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -185,6 +185,132 @@ static LIST_HEAD(shrinker_list); static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG + +static int memcg_shrinker_map_size; +static DEFINE_MUTEX(memcg_shrinker_map_mutex); + +static void free_shrinker_map_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct memcg_shrinker_map, rcu)); +} + +static int expand_one_shrinker_map(struct mem_cgroup *memcg, + int size, int old_size) +{ + struct memcg_shrinker_map *new, *old; + int nid; + + lockdep_assert_held(&memcg_shrinker_map_mutex); + + for_each_node(nid) { + old = rcu_dereference_protected( + mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); + /* Not yet online memcg */ + if (!old) + return 0; + + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); + if (!new) + return -ENOMEM; + + /* Set all old bits, clear all new bits */ + memset(new->map, (int)0xff, old_size); + memset((void *)new->map + old_size, 0, size - old_size); + + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, new); + call_rcu(&old->rcu, free_shrinker_map_rcu); + } + + return 0; +} + +void free_shrinker_maps(struct mem_cgroup *memcg) +{ + struct mem_cgroup_per_node *pn; + struct memcg_shrinker_map *map; + int nid; + + if (mem_cgroup_is_root(memcg)) + return; + + for_each_node(nid) { + pn = mem_cgroup_nodeinfo(memcg, nid); + map = rcu_dereference_protected(pn->shrinker_map, true); + if (map) + kvfree(map); + rcu_assign_pointer(pn->shrinker_map, NULL); + } +} + +int alloc_shrinker_maps(struct mem_cgroup *memcg) +{ + struct memcg_shrinker_map *map; + int nid, size, ret = 0; + + if (mem_cgroup_is_root(memcg)) + return 0; + + mutex_lock(&memcg_shrinker_map_mutex); + size = memcg_shrinker_map_size; + for_each_node(nid) { + map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); + if (!map) { + free_shrinker_maps(memcg); + ret = -ENOMEM; + break; + } + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); + } + mutex_unlock(&memcg_shrinker_map_mutex); + + return ret; +} + +static int expand_shrinker_maps(int new_id) +{ + int size, old_size, ret = 0; + struct mem_cgroup *memcg; + + size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); + old_size = memcg_shrinker_map_size; + if (size <= old_size) + return 0; + + mutex_lock(&memcg_shrinker_map_mutex); + if (!root_mem_cgroup) + goto unlock; + + memcg = mem_cgroup_iter(NULL, NULL, NULL); + do { + if (mem_cgroup_is_root(memcg)) + continue; + ret = expand_one_shrinker_map(memcg, size, old_size); + if (ret) { + mem_cgroup_iter_break(NULL, memcg); + goto unlock; + } + } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); +unlock: + if (!ret) + memcg_shrinker_map_size = size; + mutex_unlock(&memcg_shrinker_map_mutex); + return ret; +} + +void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) +{ + if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { + struct memcg_shrinker_map *map; + + rcu_read_lock(); + map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, map->map); + rcu_read_unlock(); + } +} + /* * We allow subsystems to populate their shrinker-related * LRU lists before register_shrinker_prepared() is called @@ -212,7 +338,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) goto unlock; if (id >= shrinker_nr_max) { - if (memcg_expand_shrinker_maps(id)) { + if (expand_shrinker_maps(id)) { idr_remove(&shrinker_idr, id); goto unlock; } @@ -601,7 +727,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; else - memcg_set_shrinker_bit(memcg, nid, i); + set_shrinker_bit(memcg, nid, i); } freed += ret; From patchwork Wed Jan 27 23:33:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12051213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFA4CC433E0 for ; Wed, 27 Jan 2021 23:34:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 97A2061601 for ; Wed, 27 Jan 2021 23:34:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 97A2061601 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C8D806B0071; Wed, 27 Jan 2021 18:33:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C40956B0072; Wed, 27 Jan 2021 18:33:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B07616B0073; Wed, 27 Jan 2021 18:33:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0127.hostedemail.com [216.40.44.127]) by kanga.kvack.org (Postfix) with ESMTP id 9CB566B0071 for ; Wed, 27 Jan 2021 18:33:59 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 666B1362E for ; Wed, 27 Jan 2021 23:33:59 +0000 (UTC) X-FDA: 77753160198.24.humor40_49152662759b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id 4E9081A4A5 for ; Wed, 27 Jan 2021 23:33:59 +0000 (UTC) X-HE-Tag: humor40_49152662759b X-Filterd-Recvd-Size: 6175 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 23:33:58 +0000 (UTC) Received: by mail-pg1-f180.google.com with SMTP id c132so2823682pga.3 for ; Wed, 27 Jan 2021 15:33:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9KysAzfeahGm6mPyUWMF7gAQ+XHowxRld+CYS+eHKLY=; b=tA1KOlZp5xzhh+BzLlPHRYZMj0u9g7hiVvgYM969FGcVOE5V3HXcV2Mz8TbSbEi1Bw xhDchcWpoAY76GfDuwo9A0kj/Km4hWBpVmHW0LKImZ7rqH6p+oAwjKBSmDEue9vcaMIo UcNGYy1VikvDBtNDM8j+H8nj6zJa/LaYljFXew3+vEaAVcUrvpsTE+XRVnfxJ6wgAowy vuiaJE/KrFo38K+DanTTwLsWG922IvmyoCk4/2d/pPcvTO5wdt+4gOShqo2x7vaeAcSg 67QS4XCaw6zrPx9OJ6yFWGfsmoRDXAK5fjPT6cePC5J3TY/aGjXF7SY/X8H8Ox0VFpEs VCAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9KysAzfeahGm6mPyUWMF7gAQ+XHowxRld+CYS+eHKLY=; b=sxK/Qk/TAvuhDlcZ62Zni9NIe318FwZ/tWHvxbg+FJkgKN9HwYu8IoAx4cZAZbCuUA ruaTRYOqL3pqhW2z7Ulie66fOuAeI9O8QJ3J1iPuSwjOo3Z61oUYxaA7rMaicWB+wXIr MkVJ4v2qXjEYwC8wg1sld8sd2LjhVwhjpJ32QOfA18NRIc21E1KKuHlHSu406ps4LyvA mwlhbrI44s69OBgXRb+svuygWLpioZRCdV+06NBZZyQwmmcDh3pQZGSuN29pGgYYEMvA SBR3zYnpSGEMxmqaHlYs7fRWlq1qTunSHKGtm3CJhvegs/fCBbpwgmpVFQLeN22IdZiB XdrA== X-Gm-Message-State: AOAM5335cJjXusftrQvefl/rZh8WcGZM/wq+zwHvJXRbLEJ9RjH2yc1j p01rqPmm3VzaM2MIh9YNKx7f7c5qFG1Dcw== X-Google-Smtp-Source: ABdhPJw+ON6VKgYslWXLUXhFUVv8hlxDi+3h51p1DMxFCwmfU/ejOOO4FMb5xGih7lE2VLaiQPUQ8w== X-Received: by 2002:a63:1159:: with SMTP id 25mr13765977pgr.321.1611790438071; Wed, 27 Jan 2021 15:33:58 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id 124sm3498648pfd.59.2021.01.27.15.33.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 15:33:56 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 03/11] mm: vmscan: use shrinker_rwsem to protect shrinker_maps allocation Date: Wed, 27 Jan 2021 15:33:37 -0800 Message-Id: <20210127233345.339910-4-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210127233345.339910-1-shy828301@gmail.com> References: <20210127233345.339910-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since memcg_shrinker_map_size just can be changed under holding shrinker_rwsem exclusively, the read side can be protected by holding read lock, so it sounds superfluous to have a dedicated mutex. Kirill Tkhai suggested use write lock since: * We want the assignment to shrinker_maps is visible for shrink_slab_memcg(). * The rcu_dereference_protected() dereferrencing in shrink_slab_memcg(), but in case of we use READ lock in alloc_shrinker_maps(), the dereferrencing is not actually protected. * READ lock makes alloc_shrinker_info() racy against memory allocation fail. alloc_shrinker_info()->free_shrinker_info() may free memory right after shrink_slab_memcg() dereferenced it. You may say shrink_slab_memcg()->mem_cgroup_online() protects us from it? Yes, sure, but this is not the thing we want to remember in the future, since this spreads modularity. And a test with heavy paging workload didn't show write lock makes things worse. Signed-off-by: Yang Shi Acked-by: Vlastimil Babka --- mm/vmscan.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d950cead66ca..d3f3701dfcd2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -187,7 +187,6 @@ static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG static int memcg_shrinker_map_size; -static DEFINE_MUTEX(memcg_shrinker_map_mutex); static void free_shrinker_map_rcu(struct rcu_head *head) { @@ -200,8 +199,6 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, struct memcg_shrinker_map *new, *old; int nid; - lockdep_assert_held(&memcg_shrinker_map_mutex); - for_each_node(nid) { old = rcu_dereference_protected( mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); @@ -250,7 +247,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) if (mem_cgroup_is_root(memcg)) return 0; - mutex_lock(&memcg_shrinker_map_mutex); + down_write(&shrinker_rwsem); size = memcg_shrinker_map_size; for_each_node(nid) { map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); @@ -261,7 +258,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) } rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); } - mutex_unlock(&memcg_shrinker_map_mutex); + up_write(&shrinker_rwsem); return ret; } @@ -276,9 +273,8 @@ static int expand_shrinker_maps(int new_id) if (size <= old_size) return 0; - mutex_lock(&memcg_shrinker_map_mutex); if (!root_mem_cgroup) - goto unlock; + goto out; memcg = mem_cgroup_iter(NULL, NULL, NULL); do { @@ -287,13 +283,13 @@ static int expand_shrinker_maps(int new_id) ret = expand_one_shrinker_map(memcg, size, old_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); - goto unlock; + goto out; } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); -unlock: +out: if (!ret) memcg_shrinker_map_size = size; - mutex_unlock(&memcg_shrinker_map_mutex); + return ret; } From patchwork Wed Jan 27 23:33:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12051215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AB9DC433DB for ; Wed, 27 Jan 2021 23:34:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B6E3E64DCE for ; Wed, 27 Jan 2021 23:34:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B6E3E64DCE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CB4EF6B0072; Wed, 27 Jan 2021 18:34:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C3DE76B0073; Wed, 27 Jan 2021 18:34:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE1C46B0074; Wed, 27 Jan 2021 18:34:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0096.hostedemail.com [216.40.44.96]) by kanga.kvack.org (Postfix) with ESMTP id 95AD26B0072 for ; Wed, 27 Jan 2021 18:34:01 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 58FF3181AEF39 for ; Wed, 27 Jan 2021 23:34:01 +0000 (UTC) X-FDA: 77753160282.24.lead90_5605a122759b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id 3B52A1A4A0 for ; Wed, 27 Jan 2021 23:34:01 +0000 (UTC) X-HE-Tag: lead90_5605a122759b X-Filterd-Recvd-Size: 5550 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 23:34:00 +0000 (UTC) Received: by mail-pl1-f172.google.com with SMTP id 31so2043604plb.10 for ; Wed, 27 Jan 2021 15:34:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=65bxltQ5JyyivvvnTuxpyri2gVYu+eURJIDc8089YFI=; b=fF/4lvKdCeIWSpDsS59C71/1pkx5orm4veM66Uq59nJjBqNxFLFJiO1FFck57I6rg+ kFTyXYD/RvF9uSoP+TJpeDbb++3BEcWLvxpQbyseTcvDccweUbooPirPVUfeUtXOUToL oGkxd99AYI4VyVsgsIpbJTzwfIixg2ABX0z+8ddBIgWfIjnFVpGDQ9Z2fAD6/v01hAjA 1e/27ekiV+R9rDAj6lVzyWU5lWCB5gBcIOJi/DVEDZHSo/qm2GxS8F610GHfAMC/b/ZQ BqQm4/uEdh7tRXGiwL/8cOOea3R7vj1RIyo4BxRwn5EjX2fkJlhlCMnH0krkQ+eyLgHs /w+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=65bxltQ5JyyivvvnTuxpyri2gVYu+eURJIDc8089YFI=; b=ucTWHXXePign1xedHEBvDcw+nxHSHnxZOK0KF3VdQMvmyjpbCAkeOoFfRNMFa+kJMX c0FwsrCXfJBMEQZ4EQkBsSXT45KapIKozjakYRVTXINarI8AyX3PXC31+pEHQ/Q9WBVu 1JuWzXlVRsZ9HUUXCePSCuYsKBbEXtd7rWMJtPndi+A0fRAMleKuo5fX5oUqNYJMty+q j1J1QPNfuLeCDaeBSvCgUaem7qc/pne7g6AHM904f/eez6FanDM2gK4nMS1VtXR7BOpj O/48V1BUVjCvoPuQ7qG4SnnoKV8q0HzD+w62PyhFfv+Ti7r+MfnQJyppd2zY/6tgrTo3 UfzQ== X-Gm-Message-State: AOAM533ZF63q7RZvz5u6zjHalG4kpnm26CyieHaUzWznpB/fd1cOzyZO qKLim7UxhQ9Efasf0lodWO0= X-Google-Smtp-Source: ABdhPJyEROsqkycAYaLMggreyUDXfEmKv4d5fu9Tu67KGSVlcjm+y4TKGJ/ZQPQKP4WS1C3zBGyMrg== X-Received: by 2002:a17:90a:12c4:: with SMTP id b4mr7896770pjg.98.1611790440132; Wed, 27 Jan 2021 15:34:00 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id 124sm3498648pfd.59.2021.01.27.15.33.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 15:33:59 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 04/11] mm: vmscan: remove memcg_shrinker_map_size Date: Wed, 27 Jan 2021 15:33:38 -0800 Message-Id: <20210127233345.339910-5-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210127233345.339910-1-shy828301@gmail.com> References: <20210127233345.339910-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Both memcg_shrinker_map_size and shrinker_nr_max is maintained, but actually the map size can be calculated via shrinker_nr_max, so it seems unnecessary to keep both. Remove memcg_shrinker_map_size since shrinker_nr_max is also used by iterating the bit map. Signed-off-by: Yang Shi --- mm/vmscan.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d3f3701dfcd2..847369c19775 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -185,8 +185,7 @@ static LIST_HEAD(shrinker_list); static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG - -static int memcg_shrinker_map_size; +static int shrinker_nr_max; static void free_shrinker_map_rcu(struct rcu_head *head) { @@ -248,7 +247,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) return 0; down_write(&shrinker_rwsem); - size = memcg_shrinker_map_size; + size = (shrinker_nr_max / BITS_PER_LONG + 1) * sizeof(unsigned long); for_each_node(nid) { map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); if (!map) { @@ -266,12 +265,13 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) static int expand_shrinker_maps(int new_id) { int size, old_size, ret = 0; + int new_nr_max = new_id + 1; struct mem_cgroup *memcg; - size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); - old_size = memcg_shrinker_map_size; + size = (new_nr_max / BITS_PER_LONG + 1) * sizeof(unsigned long); + old_size = (shrinker_nr_max / BITS_PER_LONG + 1) * sizeof(unsigned long); if (size <= old_size) - return 0; + goto out; if (!root_mem_cgroup) goto out; @@ -286,9 +286,10 @@ static int expand_shrinker_maps(int new_id) goto out; } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); + out: if (!ret) - memcg_shrinker_map_size = size; + shrinker_nr_max = new_nr_max; return ret; } @@ -321,7 +322,6 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) #define SHRINKER_REGISTERING ((struct shrinker *)~0UL) static DEFINE_IDR(shrinker_idr); -static int shrinker_nr_max; static int prealloc_memcg_shrinker(struct shrinker *shrinker) { @@ -338,8 +338,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) idr_remove(&shrinker_idr, id); goto unlock; } - - shrinker_nr_max = id + 1; } shrinker->id = id; ret = 0; From patchwork Wed Jan 27 23:33:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12051217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12D0FC433DB for ; Wed, 27 Jan 2021 23:34:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B63F464DCE for ; Wed, 27 Jan 2021 23:34:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B63F464DCE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1426B6B0073; Wed, 27 Jan 2021 18:34:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F3C96B0074; Wed, 27 Jan 2021 18:34:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F270F6B0075; Wed, 27 Jan 2021 18:34:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D9BCE6B0073 for ; Wed, 27 Jan 2021 18:34:03 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A6BDB180AD83A for ; Wed, 27 Jan 2021 23:34:03 +0000 (UTC) X-FDA: 77753160366.14.look08_23021062759b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 8899F18229818 for ; Wed, 27 Jan 2021 23:34:03 +0000 (UTC) X-HE-Tag: look08_23021062759b X-Filterd-Recvd-Size: 11866 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 23:34:02 +0000 (UTC) Received: by mail-pj1-f42.google.com with SMTP id lw17so3784533pjb.0 for ; Wed, 27 Jan 2021 15:34:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eldz5buhjLQX8ZhgXpsDFlEXdXLwzL3iAl6Sndbwg64=; b=V8bQMse538cRFrcGy+0faJYyKmmxUfwF3XKDVy/LJSgAzwOM5xDQ+E+K5lnxYzphAG 0tcoJvEUkuwpn9C18o2mDmaN5uhgUQZ2jKKIQSJlYk7PeDLNzGx402SdJh01boPNOVym 8NjbbiPPTKadWIbcAc29d3K/YmKZaODJScZi/dhBj6ZcXYbjUEa5IcBkb+B0A2YeHrjz /Vg4+i/t2BldOnJyFQA1mJrm+D48UXdy6xo9yllTfKDzjLBdSP018uQkaVNSR52PN9p+ 8p2QhCQCyVmEQsYR8FfSp5luWyuLAYgQ2f3gIh2mnyvNy9I5tllxGwyV39RgjYYpOTPs pJJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eldz5buhjLQX8ZhgXpsDFlEXdXLwzL3iAl6Sndbwg64=; b=BeGd451X4rRXQjUx9HeMnGkw27cMnRtj0Bn/l+lRttgewXtOKybVzooLiagZBoAk5c fh43Fd2S+uxsWtztQjT3TrFU6mlKAAmIcqbLElYEedRZZSIcZHiDxVCBFlTQWc/RHwWQ cu9YYdMirKBgd52aW8Xx/tICsG3KaZUgxWrZmHLGikfPKZAiqVSZNqqzjRQ5UIx/qGTq JJWZ6gGuACtXvP+tY57l8ESgKYIj2fX+j73Z+rFgsP+KRfYudvByqWolpnHoktcPfRlh QXW+OE2A5TxWoDgjot8fDceVeD7O/sOxArp2XRwgnAzdjeh5lEgPffQAZy5wOwyliDC0 ht7A== X-Gm-Message-State: AOAM532Lb2+OOM9SHZkPpBn8ItF62KossySRNv+mkVLAGsLI/L+xpoXj dC/FWXaIRwn3baOirBehp7I= X-Google-Smtp-Source: ABdhPJwiJy4YuOICE6saPZTJ9q4PJifTh0ZiGir8F9Dgp9dDrs9RiMASvgfymltyxR9D2szPTSOd9A== X-Received: by 2002:a17:903:18a:b029:de:5aaa:e246 with SMTP id z10-20020a170903018ab02900de5aaae246mr13654988plg.70.1611790442302; Wed, 27 Jan 2021 15:34:02 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id 124sm3498648pfd.59.2021.01.27.15.34.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 15:34:01 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 05/11] mm: memcontrol: rename shrinker_map to shrinker_info Date: Wed, 27 Jan 2021 15:33:39 -0800 Message-Id: <20210127233345.339910-6-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210127233345.339910-1-shy828301@gmail.com> References: <20210127233345.339910-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The following patch is going to add nr_deferred into shrinker_map, the change will make shrinker_map not only include map anymore, so rename it to a more general name. And this should make the patch adding nr_deferred cleaner and readable and make review easier. Rename "memcg_shrinker_info" to "shrinker_info" as well. Signed-off-by: Yang Shi Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 8 ++--- mm/memcontrol.c | 6 ++-- mm/vmscan.c | 64 +++++++++++++++++++------------------- 3 files changed, 39 insertions(+), 39 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 0ee2924991fb..62b888b88a5f 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -96,7 +96,7 @@ struct lruvec_stat { * Bitmap of shrinker::id corresponding to memcg-aware shrinkers, * which have elements charged to this memcg. */ -struct memcg_shrinker_map { +struct shrinker_info { struct rcu_head rcu; unsigned long map[]; }; @@ -118,7 +118,7 @@ struct mem_cgroup_per_node { struct mem_cgroup_reclaim_iter iter; - struct memcg_shrinker_map __rcu *shrinker_map; + struct shrinker_info __rcu *shrinker_info; struct rb_node tree_node; /* RB tree node */ unsigned long usage_in_excess;/* Set to the value by which */ @@ -1581,8 +1581,8 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -extern int alloc_shrinker_maps(struct mem_cgroup *memcg); -extern void free_shrinker_maps(struct mem_cgroup *memcg); +extern int alloc_shrinker_info(struct mem_cgroup *memcg); +extern void free_shrinker_info(struct mem_cgroup *memcg); extern void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); #else diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f5c9a0d2160b..f64ad0d044d9 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5246,11 +5246,11 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) struct mem_cgroup *memcg = mem_cgroup_from_css(css); /* - * A memcg must be visible for expand_shrinker_maps() + * A memcg must be visible for expand_shrinker_info() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (alloc_shrinker_maps(memcg)) { + if (alloc_shrinker_info(memcg)) { mem_cgroup_id_remove(memcg); return -ENOMEM; } @@ -5314,7 +5314,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) vmpressure_cleanup(&memcg->vmpressure); cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); - free_shrinker_maps(memcg); + free_shrinker_info(memcg); memcg_free_kmem(memcg); mem_cgroup_free(memcg); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 847369c19775..92e917033797 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -187,20 +187,20 @@ static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG static int shrinker_nr_max; -static void free_shrinker_map_rcu(struct rcu_head *head) +static void free_shrinker_info_rcu(struct rcu_head *head) { - kvfree(container_of(head, struct memcg_shrinker_map, rcu)); + kvfree(container_of(head, struct shrinker_info, rcu)); } -static int expand_one_shrinker_map(struct mem_cgroup *memcg, +static int expand_one_shrinker_info(struct mem_cgroup *memcg, int size, int old_size) { - struct memcg_shrinker_map *new, *old; + struct shrinker_info *new, *old; int nid; for_each_node(nid) { old = rcu_dereference_protected( - mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); + mem_cgroup_nodeinfo(memcg, nid)->shrinker_info, true); /* Not yet online memcg */ if (!old) return 0; @@ -213,17 +213,17 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, memset(new->map, (int)0xff, old_size); memset((void *)new->map + old_size, 0, size - old_size); - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, new); - call_rcu(&old->rcu, free_shrinker_map_rcu); + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, new); + call_rcu(&old->rcu, free_shrinker_info_rcu); } return 0; } -void free_shrinker_maps(struct mem_cgroup *memcg) +void free_shrinker_info(struct mem_cgroup *memcg) { struct mem_cgroup_per_node *pn; - struct memcg_shrinker_map *map; + struct shrinker_info *info; int nid; if (mem_cgroup_is_root(memcg)) @@ -231,16 +231,16 @@ void free_shrinker_maps(struct mem_cgroup *memcg) for_each_node(nid) { pn = mem_cgroup_nodeinfo(memcg, nid); - map = rcu_dereference_protected(pn->shrinker_map, true); - if (map) - kvfree(map); - rcu_assign_pointer(pn->shrinker_map, NULL); + info = rcu_dereference_protected(pn->shrinker_info, true); + if (info) + kvfree(info); + rcu_assign_pointer(pn->shrinker_info, NULL); } } -int alloc_shrinker_maps(struct mem_cgroup *memcg) +int alloc_shrinker_info(struct mem_cgroup *memcg) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; int nid, size, ret = 0; if (mem_cgroup_is_root(memcg)) @@ -249,20 +249,20 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) down_write(&shrinker_rwsem); size = (shrinker_nr_max / BITS_PER_LONG + 1) * sizeof(unsigned long); for_each_node(nid) { - map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); - if (!map) { - free_shrinker_maps(memcg); + info = kvzalloc_node(sizeof(*info) + size, GFP_KERNEL, nid); + if (!info) { + free_shrinker_info(memcg); ret = -ENOMEM; break; } - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); return ret; } -static int expand_shrinker_maps(int new_id) +static int expand_shrinker_info(int new_id) { int size, old_size, ret = 0; int new_nr_max = new_id + 1; @@ -280,7 +280,7 @@ static int expand_shrinker_maps(int new_id) do { if (mem_cgroup_is_root(memcg)) continue; - ret = expand_one_shrinker_map(memcg, size, old_size); + ret = expand_one_shrinker_info(memcg, size, old_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -297,13 +297,13 @@ static int expand_shrinker_maps(int new_id) void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; rcu_read_lock(); - map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); + info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); - set_bit(shrinker_id, map->map); + set_bit(shrinker_id, info->map); rcu_read_unlock(); } } @@ -334,7 +334,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) goto unlock; if (id >= shrinker_nr_max) { - if (expand_shrinker_maps(id)) { + if (expand_shrinker_info(id)) { idr_remove(&shrinker_idr, id); goto unlock; } @@ -663,7 +663,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, int priority) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; unsigned long ret, freed = 0; int i; @@ -673,12 +673,12 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (!down_read_trylock(&shrinker_rwsem)) return 0; - map = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_map, - true); - if (unlikely(!map)) + info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + true); + if (unlikely(!info)) goto unlock; - for_each_set_bit(i, map->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, shrinker_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -689,7 +689,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, shrinker = idr_find(&shrinker_idr, i); if (unlikely(!shrinker || shrinker == SHRINKER_REGISTERING)) { if (!shrinker) - clear_bit(i, map->map); + clear_bit(i, info->map); continue; } @@ -700,7 +700,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, ret = do_shrink_slab(&sc, shrinker, priority); if (ret == SHRINK_EMPTY) { - clear_bit(i, map->map); + clear_bit(i, info->map); /* * After the shrinker reported that it had no objects to * free, but before we cleared the corresponding bit in From patchwork Wed Jan 27 23:33:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12051219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41417C433E0 for ; Wed, 27 Jan 2021 23:34:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EFA2264D7F for ; Wed, 27 Jan 2021 23:34:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EFA2264D7F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 600606B0074; Wed, 27 Jan 2021 18:34:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 562526B0075; Wed, 27 Jan 2021 18:34:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 477DC6B0078; Wed, 27 Jan 2021 18:34:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0129.hostedemail.com [216.40.44.129]) by kanga.kvack.org (Postfix) with ESMTP id 2FFF16B0074 for ; Wed, 27 Jan 2021 18:34:06 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id EFA391EF2 for ; Wed, 27 Jan 2021 23:34:05 +0000 (UTC) X-FDA: 77753160450.29.fowl08_2a0dc242759b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id BAC0D180868EE for ; Wed, 27 Jan 2021 23:34:05 +0000 (UTC) X-HE-Tag: fowl08_2a0dc242759b X-Filterd-Recvd-Size: 7068 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 23:34:05 +0000 (UTC) Received: by mail-pf1-f170.google.com with SMTP id i63so2480297pfg.7 for ; Wed, 27 Jan 2021 15:34:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Z0qOZRcKtz9kja21tsQM1jjJMKJ1BMp88S9zi7WuYMo=; b=PwacIXXYOOrNNuWAIrBAFfvozxj3s/deo2VW6K4hOuVhtA5CdaNNq8PVUQdBzYT+mB lIyQUvi+l2SK/6A9Rv+2gkay1eD1LfWFs6NYt0eBiNC5bj1NL0zMbrHPc3FnfOgePrkw qD78ylPN7I/ryYcTMiW8IbdqyifrYbU8OUetpefcEWH6O1h4IMZ37Iqr70bCjeUt/Oip MyPtcDkCtStbW4G8fDLtyQBFUQCn2ocFSpIMHxsqTwc2aGYsm4mepS2PO7zC0ctCpBLB E7usm6cMpfMPnrVKoedBGoZECuvRFncT9Tr6RwQu3QmEPUV6lOtNaeKP9zAyhM13RsUy N2Xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Z0qOZRcKtz9kja21tsQM1jjJMKJ1BMp88S9zi7WuYMo=; b=qIpyYp2zEAxencKwprO7KFMBLhmWIaNVcvW0+zXi0x0FUR9K6CYxD0ldQ4gG4yzImx 0da/nRgNFfsE1ZCYU6hcdnfxTNGQo0j2mWduxLg11ARI12aU++N5UGqMSlS9+5gvAv3B 7RDEak7A1C0n/S7CsrOU+QZoDU/W/FfeWYbyQ45Vas5pvGsKQTgPe6cfKM3/Fn4Q/RFB 3Nq8uyRnuRjgqESo/VztkvFtuf+dC0yQEL9aQW36Wbz8jfbSJbLIg7w9DUQXBHjPKyXL mr6vbQ4JF0mE0RRiaZLQPuubvutawwbpHqJWSiNPWTzwPS+xZ0N2bP7cjnr+9haqxT9p sDgg== X-Gm-Message-State: AOAM532VoriJvXwRfCKXRWArk3UOgCR2V5eUofItw05dLpmKDYubB5Zw t9YyJt41iekgz3Q6kF0s/9w= X-Google-Smtp-Source: ABdhPJwmNRsc3/SzIbQ591nJDEumm52L/rIIR4+QoGy+oHboi6njK9SDr2+KcbtNnfp6wdACwZwwZA== X-Received: by 2002:a63:ee4a:: with SMTP id n10mr13603776pgk.375.1611790444512; Wed, 27 Jan 2021 15:34:04 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id 124sm3498648pfd.59.2021.01.27.15.34.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 15:34:03 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 06/11] mm: vmscan: use a new flag to indicate shrinker is registered Date: Wed, 27 Jan 2021 15:33:40 -0800 Message-Id: <20210127233345.339910-7-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210127233345.339910-1-shy828301@gmail.com> References: <20210127233345.339910-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently registered shrinker is indicated by non-NULL shrinker->nr_deferred. This approach is fine with nr_deferred at the shrinker level, but the following patches will move MEMCG_AWARE shrinkers' nr_deferred to memcg level, so their shrinker->nr_deferred would always be NULL. This would prevent the shrinkers from unregistering correctly. Remove SHRINKER_REGISTERING since we could check if shrinker is registered successfully by the new flag. Signed-off-by: Yang Shi --- include/linux/shrinker.h | 7 ++++--- mm/vmscan.c | 27 +++++++++------------------ 2 files changed, 13 insertions(+), 21 deletions(-) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 0f80123650e2..1eac79ce57d4 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -79,13 +79,14 @@ struct shrinker { #define DEFAULT_SEEKS 2 /* A good number if you don't know better. */ /* Flags */ -#define SHRINKER_NUMA_AWARE (1 << 0) -#define SHRINKER_MEMCG_AWARE (1 << 1) +#define SHRINKER_REGISTERED (1 << 0) +#define SHRINKER_NUMA_AWARE (1 << 1) +#define SHRINKER_MEMCG_AWARE (1 << 2) /* * It just makes sense when the shrinker is also MEMCG_AWARE for now, * non-MEMCG_AWARE shrinker should not have this flag set. */ -#define SHRINKER_NONSLAB (1 << 2) +#define SHRINKER_NONSLAB (1 << 3) extern int prealloc_shrinker(struct shrinker *shrinker); extern void register_shrinker_prepared(struct shrinker *shrinker); diff --git a/mm/vmscan.c b/mm/vmscan.c index 92e917033797..256896d157d4 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -308,19 +308,6 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) } } -/* - * We allow subsystems to populate their shrinker-related - * LRU lists before register_shrinker_prepared() is called - * for the shrinker, since we don't want to impose - * restrictions on their internal registration order. - * In this case shrink_slab_memcg() may find corresponding - * bit is set in the shrinkers map. - * - * This value is used by the function to detect registering - * shrinkers and to skip do_shrink_slab() calls for them. - */ -#define SHRINKER_REGISTERING ((struct shrinker *)~0UL) - static DEFINE_IDR(shrinker_idr); static int prealloc_memcg_shrinker(struct shrinker *shrinker) @@ -329,7 +316,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) down_write(&shrinker_rwsem); /* This may call shrinker, so it must use down_read_trylock() */ - id = idr_alloc(&shrinker_idr, SHRINKER_REGISTERING, 0, 0, GFP_KERNEL); + id = idr_alloc(&shrinker_idr, NULL, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -496,6 +483,7 @@ void register_shrinker_prepared(struct shrinker *shrinker) if (shrinker->flags & SHRINKER_MEMCG_AWARE) idr_replace(&shrinker_idr, shrinker, shrinker->id); #endif + shrinker->flags |= SHRINKER_REGISTERED; up_write(&shrinker_rwsem); } @@ -515,13 +503,16 @@ EXPORT_SYMBOL(register_shrinker); */ void unregister_shrinker(struct shrinker *shrinker) { - if (!shrinker->nr_deferred) + if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - unregister_memcg_shrinker(shrinker); + down_write(&shrinker_rwsem); list_del(&shrinker->list); + shrinker->flags &= ~SHRINKER_REGISTERED; up_write(&shrinker_rwsem); + + if (shrinker->flags & SHRINKER_MEMCG_AWARE) + unregister_memcg_shrinker(shrinker); kfree(shrinker->nr_deferred); shrinker->nr_deferred = NULL; } @@ -687,7 +678,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct shrinker *shrinker; shrinker = idr_find(&shrinker_idr, i); - if (unlikely(!shrinker || shrinker == SHRINKER_REGISTERING)) { + if (unlikely(!shrinker || !(shrinker->flags & SHRINKER_REGISTERED))) { if (!shrinker) clear_bit(i, info->map); continue; From patchwork Wed Jan 27 23:33:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12051221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7912DC433DB for ; Wed, 27 Jan 2021 23:34:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2B59E64DCE for ; Wed, 27 Jan 2021 23:34:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2B59E64DCE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A6A256B0075; Wed, 27 Jan 2021 18:34:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A1CD76B0078; Wed, 27 Jan 2021 18:34:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 909F26B007B; Wed, 27 Jan 2021 18:34:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0153.hostedemail.com [216.40.44.153]) by kanga.kvack.org (Postfix) with ESMTP id 78B4E6B0075 for ; Wed, 27 Jan 2021 18:34:08 -0500 (EST) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 46BB11EF2 for ; Wed, 27 Jan 2021 23:34:08 +0000 (UTC) X-FDA: 77753160576.08.stew30_60067102759b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin08.hostedemail.com (Postfix) with ESMTP id 2AE021819E785 for ; Wed, 27 Jan 2021 23:34:08 +0000 (UTC) X-HE-Tag: stew30_60067102759b X-Filterd-Recvd-Size: 9631 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 23:34:07 +0000 (UTC) Received: by mail-pf1-f172.google.com with SMTP id e19so2475874pfh.6 for ; Wed, 27 Jan 2021 15:34:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WHXdiiNMkNIAWPj5jAlj2sLIoqe+YK5UhAhpnAub6bQ=; b=XUz/mnHzPZ0b2SdudDflQsr3JoDRs7VbyKkLSCfhQe6Atrj870akiuiHzicOsNC1U9 UrBgAr7ctO78waaNbPTTjYtO/Y8Z0G6602/tc57gp6rNAcjr/g0xOysF3OT++wD+FLVK GT8HXiGXyXbaHEeJ3YiAV7PDVhTGZJw6YEMNHGrBBJBEVygW+JAR+LjIw43nSH9/HgqX O7ND8PemQx7W+GKSITDGA/Mz1LdUQhNIM0Ir/zhGPBC+lNp7b3UPNGU6y4HA8iYg7431 TNxgTBVypR7vYY2z52W6ndgjsCDK7ojsUdeqkxNzltJcikR4QI1ruUq5VUl6Ks1KrX0c YFzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WHXdiiNMkNIAWPj5jAlj2sLIoqe+YK5UhAhpnAub6bQ=; b=h61Mx2y9RNgOZzHkB+4WCQ1sWB8ElLM2m7SvX2u0FXy/LIMBo78qH36IkyCkuBz99g 9/D1i4sYF+IDEV0Pu+mkLWOlVo6losCLebn1UI5zWGPIYqgkwiAmbk0zeaMltLPHG/ar EIVRVH40fsXRUovF1ffGJyHhJrdxvN7iYogMGeO+0/aFMVXMMo8CLZWoY030FVpNVjC2 aD2tsEAWGOdTu2S1NHn0r+PGINmIoMz3R4J+BDs2NJsbRMQFbAnR50ywIMSNuj4ABTRV 8wS5tGQ3EhKu/x/jBe7QR3Bmod0ra1le/xMxZRkeNQoCL+BVGQQnb577Tz/o85ZTGxkq a5VQ== X-Gm-Message-State: AOAM533dOc7Ck4FFhAs3j4d/otDa9n+OU1Fpx6vtSBJEfFN2/QUdIHB1 BEY8rDT1ezMyuPO8bo+8G3M= X-Google-Smtp-Source: ABdhPJxQDAc8sTA2KFNgfn9BXLtBZmKiY+DnypennfBAvw8iw98VGDS460FyGUPM2ksRCNKOBypLWQ== X-Received: by 2002:a63:1519:: with SMTP id v25mr13596877pgl.217.1611790446884; Wed, 27 Jan 2021 15:34:06 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id 124sm3498648pfd.59.2021.01.27.15.34.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 15:34:05 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 07/11] mm: vmscan: add per memcg shrinker nr_deferred Date: Wed, 27 Jan 2021 15:33:41 -0800 Message-Id: <20210127233345.339910-8-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210127233345.339910-1-shy828301@gmail.com> References: <20210127233345.339910-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently the number of deferred objects are per shrinker, but some slabs, for example, vfs inode/dentry cache are per memcg, this would result in poor isolation among memcgs. The deferred objects typically are generated by __GFP_NOFS allocations, one memcg with excessive __GFP_NOFS allocations may blow up deferred objects, then other innocent memcgs may suffer from over shrink, excessive reclaim latency, etc. For example, two workloads run in memcgA and memcgB respectively, workload in B is vfs heavy workload. Workload in A generates excessive deferred objects, then B's vfs cache might be hit heavily (drop half of caches) by B's limit reclaim or global reclaim. We observed this hit in our production environment which was running vfs heavy workload shown as the below tracing log: <...>-409454 [016] .... 28286961.747146: mm_shrink_slab_start: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 1 objects to shrink 3641681686040 gfp_flags GFP_HIGHUSER_MOVABLE|__GFP_ZERO pgs_scanned 1 lru_pgs 15721 cache items 246404277 delta 31345 total_scan 123202138 <...>-409454 [022] .... 28287105.928018: mm_shrink_slab_end: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 1 unused scan count 3641681686040 new scan count 3641798379189 total_scan 602 last shrinker return val 123186855 The vfs cache and page cache ration was 10:1 on this machine, and half of caches were dropped. This also resulted in significant amount of page caches were dropped due to inodes eviction. Make nr_deferred per memcg for memcg aware shrinkers would solve the unfairness and bring better isolation. When memcg is not enabled (!CONFIG_MEMCG or memcg disabled), the shrinker's nr_deferred would be used. And non memcg aware shrinkers use shrinker's nr_deferred all the time. Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 7 +++--- mm/vmscan.c | 48 +++++++++++++++++++++++++------------- 2 files changed, 36 insertions(+), 19 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 62b888b88a5f..e0384367e07d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -93,12 +93,13 @@ struct lruvec_stat { }; /* - * Bitmap of shrinker::id corresponding to memcg-aware shrinkers, - * which have elements charged to this memcg. + * Bitmap and deferred work of shrinker::id corresponding to memcg-aware + * shrinkers, which have elements charged to this memcg. */ struct shrinker_info { struct rcu_head rcu; - unsigned long map[]; + unsigned long *map; + atomic_long_t *nr_deferred; }; /* diff --git a/mm/vmscan.c b/mm/vmscan.c index 256896d157d4..20be0db291fe 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -187,16 +187,21 @@ static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG static int shrinker_nr_max; +#define NR_MAX_TO_SHR_MAP_SIZE(nr_max) \ + ((nr_max / BITS_PER_LONG + 1) * sizeof(unsigned long)) + static void free_shrinker_info_rcu(struct rcu_head *head) { kvfree(container_of(head, struct shrinker_info, rcu)); } static int expand_one_shrinker_info(struct mem_cgroup *memcg, - int size, int old_size) + int m_size, int d_size, + int old_m_size, int old_d_size) { struct shrinker_info *new, *old; int nid; + int size = m_size + d_size; for_each_node(nid) { old = rcu_dereference_protected( @@ -209,9 +214,15 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!new) return -ENOMEM; - /* Set all old bits, clear all new bits */ - memset(new->map, (int)0xff, old_size); - memset((void *)new->map + old_size, 0, size - old_size); + new->map = (unsigned long *)(new + 1); + new->nr_deferred = (void *)new->map + m_size; + + /* map: set all old bits, clear all new bits */ + memset(new->map, (int)0xff, old_m_size); + memset((void *)new->map + old_m_size, 0, m_size - old_m_size); + /* nr_deferred: copy old values, clear all new values */ + memcpy(new->nr_deferred, old->nr_deferred, old_d_size); + memset((void *)new->nr_deferred + old_d_size, 0, d_size - old_d_size); rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, new); call_rcu(&old->rcu, free_shrinker_info_rcu); @@ -226,9 +237,6 @@ void free_shrinker_info(struct mem_cgroup *memcg) struct shrinker_info *info; int nid; - if (mem_cgroup_is_root(memcg)) - return; - for_each_node(nid) { pn = mem_cgroup_nodeinfo(memcg, nid); info = rcu_dereference_protected(pn->shrinker_info, true); @@ -242,12 +250,13 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) { struct shrinker_info *info; int nid, size, ret = 0; - - if (mem_cgroup_is_root(memcg)) - return 0; + int m_size, d_size = 0; down_write(&shrinker_rwsem); - size = (shrinker_nr_max / BITS_PER_LONG + 1) * sizeof(unsigned long); + m_size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); + d_size = shrinker_nr_max * sizeof(atomic_long_t); + size = m_size + d_size; + for_each_node(nid) { info = kvzalloc_node(sizeof(*info) + size, GFP_KERNEL, nid); if (!info) { @@ -255,6 +264,8 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) ret = -ENOMEM; break; } + info->map = (unsigned long *)(info + 1); + info->nr_deferred = (void *)info->map + m_size; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -266,10 +277,16 @@ static int expand_shrinker_info(int new_id) { int size, old_size, ret = 0; int new_nr_max = new_id + 1; + int m_size, d_size = 0; + int old_m_size, old_d_size = 0; struct mem_cgroup *memcg; - size = (new_nr_max / BITS_PER_LONG + 1) * sizeof(unsigned long); - old_size = (shrinker_nr_max / BITS_PER_LONG + 1) * sizeof(unsigned long); + m_size = NR_MAX_TO_SHR_MAP_SIZE(new_nr_max); + d_size = new_nr_max * sizeof(atomic_long_t); + size = m_size + d_size; + old_m_size = NR_MAX_TO_SHR_MAP_SIZE(shrinker_nr_max); + old_d_size = shrinker_nr_max * sizeof(atomic_long_t); + old_size = old_m_size + old_d_size; if (size <= old_size) goto out; @@ -278,9 +295,8 @@ static int expand_shrinker_info(int new_id) memcg = mem_cgroup_iter(NULL, NULL, NULL); do { - if (mem_cgroup_is_root(memcg)) - continue; - ret = expand_one_shrinker_info(memcg, size, old_size); + ret = expand_one_shrinker_info(memcg, m_size, d_size, + old_m_size, old_d_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; From patchwork Wed Jan 27 23:33:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12051223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEF9BC433E0 for ; Wed, 27 Jan 2021 23:34:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5C13464D7F for ; Wed, 27 Jan 2021 23:34:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5C13464D7F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 009416B0080; Wed, 27 Jan 2021 18:34:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EFA7A6B007D; Wed, 27 Jan 2021 18:34:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DFAFC6B007E; Wed, 27 Jan 2021 18:34:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0171.hostedemail.com [216.40.44.171]) by kanga.kvack.org (Postfix) with ESMTP id C9AA36B0078 for ; Wed, 27 Jan 2021 18:34:12 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 92EF1181AEF32 for ; Wed, 27 Jan 2021 23:34:12 +0000 (UTC) X-FDA: 77753160744.22.brick92_441216e2759b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id 3CB861803B8DC for ; Wed, 27 Jan 2021 23:34:10 +0000 (UTC) X-HE-Tag: brick92_441216e2759b X-Filterd-Recvd-Size: 8333 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 23:34:09 +0000 (UTC) Received: by mail-pj1-f47.google.com with SMTP id u4so2681514pjn.4 for ; Wed, 27 Jan 2021 15:34:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KjfIUTm7EyLVGh5kUmhdOqVLNJ4R1Pj2IbptP7JZXIY=; b=CJGu6zGoPLDDnlpgOZuwWPp5CWHGWPlM7mMwoQ0ox5SqOSYYVS5mq9g8fqQEiW9V2H XgRxKyb1+rTefGeCguck/SZyFuSAXs9DnQjGQNpl7we4T/YKjYZ6uBdDkb7EEc4X4PtM YbK8jzIXjJKyq5w27JRArLsiyxFfepRLTdq45qlJjRtNOZlzqqJNcdVlhKFb0gwGOhBg F2eMiUlzjiGI0mhL3gz2cEG1CL50nND1/42YUNuOgIRDVnxffBzoI4DueKCvpPn+pU3S CdqkmaTB+n0BgXDkuJ5F1ufWkQOxEnzai+vb7GmdMYUCRcvdAnxn3GeG+MJZK4K1leYE 4Plw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KjfIUTm7EyLVGh5kUmhdOqVLNJ4R1Pj2IbptP7JZXIY=; b=a85iuaoKLj06UAMd0JZxS5x25K8Rv+ywJGWEy6kdLXJjaDbITp8Y1bji7WPs5cVfyG KwkgzOmHtmgUePLiN60KKhf4Wc2pmVPRW/1A1zo5X/BRaiPFIVw3wU8qEYlorpEQiAS/ FBYd5tQKVmn384h5u7rH1ek/3psfGRIFrrkMoR7Ui8L0zuZAdUxqDdnh3rausdx+cb5n HZDZGO8cJeONBA5a06IKl7xrzoIqlOLQvG+HhSb10Z02FHmlClCd6HrorGFP4Qy64KUX CTcNBjzQW55qvATTb03GDog+mIc7ERHVBSDXBkH5BdzKyMdYDQMQAMN5aKHFLYR1e2/m HwCg== X-Gm-Message-State: AOAM530rz9ZmZlDSNXF0b9vw8yWWNc66/597lBsaAaXOAwk+Caa5BZBT IVcE6K1nEz3L8d3MBhInBfw= X-Google-Smtp-Source: ABdhPJx+sxQ8J5zuiqS4QK0aWVfSSOBe71N3VVys+iKEeQQxQLYUqN96B/84mTeDUBCUzzatPEJ2fQ== X-Received: by 2002:a17:90a:da02:: with SMTP id e2mr8105516pjv.173.1611790448927; Wed, 27 Jan 2021 15:34:08 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id 124sm3498648pfd.59.2021.01.27.15.34.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 15:34:08 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 08/11] mm: vmscan: use per memcg nr_deferred of shrinker Date: Wed, 27 Jan 2021 15:33:42 -0800 Message-Id: <20210127233345.339910-9-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210127233345.339910-1-shy828301@gmail.com> References: <20210127233345.339910-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use per memcg's nr_deferred for memcg aware shrinkers. The shrinker's nr_deferred will be used in the following cases: 1. Non memcg aware shrinkers 2. !CONFIG_MEMCG 3. memcg is disabled by boot parameter Signed-off-by: Yang Shi --- mm/vmscan.c | 87 ++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 73 insertions(+), 14 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 20be0db291fe..e1f8960f5cf6 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -205,7 +205,8 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, for_each_node(nid) { old = rcu_dereference_protected( - mem_cgroup_nodeinfo(memcg, nid)->shrinker_info, true); + mem_cgroup_nodeinfo(memcg, nid)->shrinker_info, + lockdep_is_held(&shrinker_rwsem)); /* Not yet online memcg */ if (!old) return 0; @@ -239,7 +240,8 @@ void free_shrinker_info(struct mem_cgroup *memcg) for_each_node(nid) { pn = mem_cgroup_nodeinfo(memcg, nid); - info = rcu_dereference_protected(pn->shrinker_info, true); + info = rcu_dereference_protected(pn->shrinker_info, + lockdep_is_held(&shrinker_rwsem)); if (info) kvfree(info); rcu_assign_pointer(pn->shrinker_info, NULL); @@ -360,6 +362,27 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) up_write(&shrinker_rwsem); } +static long count_nr_deferred_memcg(int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + + info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + lockdep_is_held(&shrinker_rwsem)); + return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); +} + +static long set_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + + info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + lockdep_is_held(&shrinker_rwsem)); + + return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); +} + static bool cgroup_reclaim(struct scan_control *sc) { return sc->target_mem_cgroup; @@ -398,6 +421,18 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) { } +static long count_nr_deferred_memcg(int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + return 0; +} + +static long set_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + return 0; +} + static bool cgroup_reclaim(struct scan_control *sc) { return false; @@ -409,6 +444,39 @@ static bool writeback_throttling_sane(struct scan_control *sc) } #endif +static long count_nr_deferred(struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (sc->memcg && + (shrinker->flags & SHRINKER_MEMCG_AWARE)) + return count_nr_deferred_memcg(nid, shrinker, + sc->memcg); + + return atomic_long_xchg(&shrinker->nr_deferred[nid], 0); +} + + +static long set_nr_deferred(long nr, struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (sc->memcg && + (shrinker->flags & SHRINKER_MEMCG_AWARE)) + return set_nr_deferred_memcg(nr, nid, shrinker, + sc->memcg); + + return atomic_long_add_return(nr, &shrinker->nr_deferred[nid]); +} + /* * This misses isolated pages which are not accounted for to save counters. * As the data only determines if reclaim or compaction continues, it is @@ -545,14 +613,10 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, long freeable; long nr; long new_nr; - int nid = shrinkctl->nid; long batch_size = shrinker->batch ? shrinker->batch : SHRINK_BATCH; long scanned = 0, next_deferred; - if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) - nid = 0; - freeable = shrinker->count_objects(shrinker, shrinkctl); if (freeable == 0 || freeable == SHRINK_EMPTY) return freeable; @@ -562,7 +626,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, * and zero it so that other concurrent shrinker invocations * don't also do this scanning work. */ - nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0); + nr = count_nr_deferred(shrinker, shrinkctl); total_scan = nr; if (shrinker->seeks) { @@ -653,14 +717,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, next_deferred = 0; /* * move the unused scan count back into the shrinker in a - * manner that handles concurrent updates. If we exhausted the - * scan, there is no need to do an update. + * manner that handles concurrent updates. */ - if (next_deferred > 0) - new_nr = atomic_long_add_return(next_deferred, - &shrinker->nr_deferred[nid]); - else - new_nr = atomic_long_read(&shrinker->nr_deferred[nid]); + new_nr = set_nr_deferred(next_deferred, shrinker, shrinkctl); trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); return freed; From patchwork Wed Jan 27 23:33:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12051225 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BC30C433E6 for ; Wed, 27 Jan 2021 23:34:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EFFB364D7F for ; Wed, 27 Jan 2021 23:34:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EFFB364D7F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 25DE86B0078; Wed, 27 Jan 2021 18:34:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 144006B007E; Wed, 27 Jan 2021 18:34:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0073A6B0078; Wed, 27 Jan 2021 18:34:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0188.hostedemail.com [216.40.44.188]) by kanga.kvack.org (Postfix) with ESMTP id DAEED6B007B for ; Wed, 27 Jan 2021 18:34:12 -0500 (EST) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A48068249980 for ; Wed, 27 Jan 2021 23:34:12 +0000 (UTC) X-FDA: 77753160744.10.birth19_38011b22759b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id 83E7516A4AA for ; Wed, 27 Jan 2021 23:34:12 +0000 (UTC) X-HE-Tag: birth19_38011b22759b X-Filterd-Recvd-Size: 5665 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 23:34:11 +0000 (UTC) Received: by mail-pl1-f180.google.com with SMTP id 31so2044002plb.10 for ; Wed, 27 Jan 2021 15:34:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=A25C2yzorA2J3gVn3nsaaXJeMnBXVSTs3vvI0YY4hDA=; b=WiNuvn47Mp5t41ts2kRKGgt9BbuxXM5u4SJJ63aPMMDxGo8sV5xwtpPfaFHY67SRgm HCycw0iUMUzn2SEgIiJA0Vc9VR8Gkq4lIujR53Wch3vwGYsYA8xixp2q1bRMmxREGOjM pedM0KFhy8EXOBdZVz/9meVGcVEGvtOjOHETUPg6HV2ZucFrpv11jKWNKFYMtnqcvUB8 +yckZbVdTj6oK4S7hSnrLLfyycArgFKh3BCtD15MzK7QMmlZbza0OnD6suquKbs+C4N9 QmbhTijxgXEzmpH/DMB2hLGDgJ9cDMX6EvIVZq6vZTvxicbzc/Dkq+PbZUFfJsYDPVwh Bd5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=A25C2yzorA2J3gVn3nsaaXJeMnBXVSTs3vvI0YY4hDA=; b=kyPy/Ng09iDGyfRief0hPxXEX9f7MyWQ/OlQJs50A9aa7OJKGoZw2vP+I8/NRHoYM+ zrYnfw4DGnlKsxM3b8PEQwjNgFVo3NE4wYFc6XdYPMqO6vhc1nUe5rnkRdjMjx/ExO0u ahSqAvsSZSbXCjlHMKC63qAnp74MZ32Yk+58RENkA35Zvl6DlSeE3mJxf88OLXdXNwfo fH2b2l5irMfvOPuV7mRAJ97HVbst46MlU6oOUUyxfUAUXOmWofjdCnChGpTX5Ib4+sbq T9te3yNJNZnScVCaR16PrxeZ7MFLzlcZHdziovpv8YjI4wgTwpsQOIlnGJRVz8UmvhVm yguA== X-Gm-Message-State: AOAM530G5RctZVFMwlDXVtiSmtkK/FBQ5kJi9f45UpC+EiH7rL3j8DwR jB9ZIPEvrBKK2Gi2ssCYgvM= X-Google-Smtp-Source: ABdhPJxcg6SxlofTZRZskbI8gjSTaNdJroCzzy078FkO4MpMnwzQMsW/UX1X0UAWCI7/gCFeJql8iQ== X-Received: by 2002:a17:90b:60c:: with SMTP id gb12mr8128306pjb.125.1611790451282; Wed, 27 Jan 2021 15:34:11 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id 124sm3498648pfd.59.2021.01.27.15.34.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 15:34:10 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 09/11] mm: vmscan: don't need allocate shrinker->nr_deferred for memcg aware shrinkers Date: Wed, 27 Jan 2021 15:33:43 -0800 Message-Id: <20210127233345.339910-10-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210127233345.339910-1-shy828301@gmail.com> References: <20210127233345.339910-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now nr_deferred is available on per memcg level for memcg aware shrinkers, so don't need allocate shrinker->nr_deferred for such shrinkers anymore. The prealloc_memcg_shrinker() would return -ENOSYS if !CONFIG_MEMCG or memcg is disabled by kernel command line, then shrinker's SHRINKER_MEMCG_AWARE flag would be cleared. This makes the implementation of this patch simpler. Signed-off-by: Yang Shi Acked-by: Vlastimil Babka --- mm/vmscan.c | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index e1f8960f5cf6..0373d7619d7b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -332,6 +332,9 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) { int id, ret = -ENOMEM; + if (mem_cgroup_disabled()) + return -ENOSYS; + down_write(&shrinker_rwsem); /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, NULL, 0, 0, GFP_KERNEL); @@ -414,7 +417,7 @@ static bool writeback_throttling_sane(struct scan_control *sc) #else static int prealloc_memcg_shrinker(struct shrinker *shrinker) { - return 0; + return -ENOSYS; } static void unregister_memcg_shrinker(struct shrinker *shrinker) @@ -525,8 +528,20 @@ unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru, int zone */ int prealloc_shrinker(struct shrinker *shrinker) { - unsigned int size = sizeof(*shrinker->nr_deferred); + unsigned int size; + int err; + + if (shrinker->flags & SHRINKER_MEMCG_AWARE) { + err = prealloc_memcg_shrinker(shrinker); + if (!err) + return 0; + if (err != -ENOSYS) + return err; + + shrinker->flags &= ~SHRINKER_MEMCG_AWARE; + } + size = sizeof(*shrinker->nr_deferred); if (shrinker->flags & SHRINKER_NUMA_AWARE) size *= nr_node_ids; @@ -534,26 +549,14 @@ int prealloc_shrinker(struct shrinker *shrinker) if (!shrinker->nr_deferred) return -ENOMEM; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - if (prealloc_memcg_shrinker(shrinker)) - goto free_deferred; - } return 0; - -free_deferred: - kfree(shrinker->nr_deferred); - shrinker->nr_deferred = NULL; - return -ENOMEM; } void free_prealloced_shrinker(struct shrinker *shrinker) { - if (!shrinker->nr_deferred) - return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - unregister_memcg_shrinker(shrinker); + return unregister_memcg_shrinker(shrinker); kfree(shrinker->nr_deferred); shrinker->nr_deferred = NULL; From patchwork Wed Jan 27 23:33:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12051227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E5DAC433DB for ; Wed, 27 Jan 2021 23:34:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 04E1361601 for ; Wed, 27 Jan 2021 23:34:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 04E1361601 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A99386B007B; Wed, 27 Jan 2021 18:34:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D3A16B007D; Wed, 27 Jan 2021 18:34:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 848286B007E; Wed, 27 Jan 2021 18:34:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0244.hostedemail.com [216.40.44.244]) by kanga.kvack.org (Postfix) with ESMTP id 67A7B6B007B for ; Wed, 27 Jan 2021 18:34:15 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 302328249980 for ; Wed, 27 Jan 2021 23:34:15 +0000 (UTC) X-FDA: 77753160870.01.spy56_3c066ea2759b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 121C71004D5BA for ; Wed, 27 Jan 2021 23:34:15 +0000 (UTC) X-HE-Tag: spy56_3c066ea2759b X-Filterd-Recvd-Size: 5756 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 23:34:14 +0000 (UTC) Received: by mail-pj1-f49.google.com with SMTP id a20so2434138pjs.1 for ; Wed, 27 Jan 2021 15:34:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rtMUSqzuW0mmPVrpVmtzg3xeLjYzKxf0vIpZFRWhJNw=; b=IK7YmjHgj3pG2Gx8I4pi5CXEEK/MPNLJUX4VO9Jdoy/OhHpqD4N9Rjybc3tE1VprDL faSkbMaZzSd+bXQJfD4wq3KQ+rS+HR6XRRzQatXUhi3fcnuH786tYm+8LpHKV8oINHnC x3ih9cNljKRSf1cnl7ytq9eMTTqSA14ESeIVH7QMzIXStQxo6pfiI47aUCdph1Trlhc4 udhsPCMwEfjg6Swin7MBa6JurkEsyX3wvV5tlL5oibECFNES/9DjFgfZabm5IWGD8cgp WYU8ECzM6bi9wEzHsAJHbDQ4i9wx/++eUPrBv+6UWIFqhabLqRtMe3seEOYCHAAXBRfr j+Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rtMUSqzuW0mmPVrpVmtzg3xeLjYzKxf0vIpZFRWhJNw=; b=DOZ3kDsT1U4EpaWx9NChZMIOhL+6eg3V5wRnbTkiD6gXcwszrT6i4AUsjCDHJQxM71 kPVSVx3vMHQdnx7K476bRa2GXErnNcYct1bJZJ7dvRQcEnvUfSxUL8lwVM4zb00nEbep e5SVF8PZc3+Rj7N/p2T228/O9t3g1/8CE4VsdNp/totuFX42OYpznAZ4GCCN0vMuNlwN 72LftrzghvEA7IflS6FrduG4lBVfwDxyKBvuGYeoQACr773rwl0FF9IK2Q/IeRj9BI4R fAQfV9zn2naCIWe9d1EddkDUcXKpS8a5C4ml7eb3wwaHo6DGVEjOTxmD3pjiV2rV4Hl+ QoNA== X-Gm-Message-State: AOAM5310HllvDIeLxOIlPAx801XiyT48jXTNnzxhSXmGm1fCCeIRkqwk tDEbS8p3TPmS9XulRWQxWzY= X-Google-Smtp-Source: ABdhPJwJbUr7AEV5afH0AkrswES2yI+fQ1dNzepkpXY5WMSGLXOLIFnWa24NcIjMjMn7Ku8CPLOEgA== X-Received: by 2002:a17:90a:7e84:: with SMTP id j4mr2214547pjl.167.1611790453815; Wed, 27 Jan 2021 15:34:13 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id 124sm3498648pfd.59.2021.01.27.15.34.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 15:34:12 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 10/11] mm: memcontrol: reparent nr_deferred when memcg offline Date: Wed, 27 Jan 2021 15:33:44 -0800 Message-Id: <20210127233345.339910-11-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210127233345.339910-1-shy828301@gmail.com> References: <20210127233345.339910-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now shrinker's nr_deferred is per memcg for memcg aware shrinkers, add to parent's corresponding nr_deferred when memcg offline. Signed-off-by: Yang Shi Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 1 + mm/vmscan.c | 31 +++++++++++++++++++++++++++++++ 3 files changed, 33 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e0384367e07d..fe1375f08881 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1586,6 +1586,7 @@ extern int alloc_shrinker_info(struct mem_cgroup *memcg); extern void free_shrinker_info(struct mem_cgroup *memcg); extern void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); +extern void reparent_shrinker_deferred(struct mem_cgroup *memcg); #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f64ad0d044d9..21f36b73f36a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5282,6 +5282,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) page_counter_set_low(&memcg->memory, 0); memcg_offline_kmem(memcg); + reparent_shrinker_deferred(memcg); wb_memcg_offline(memcg); drain_all_stock(memcg); diff --git a/mm/vmscan.c b/mm/vmscan.c index 0373d7619d7b..55ad91a26ba3 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -386,6 +386,37 @@ static long set_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } +static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, + int nid) +{ + return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + lockdep_is_held(&shrinker_rwsem)); +} + +void reparent_shrinker_deferred(struct mem_cgroup *memcg) +{ + int i, nid; + long nr; + struct mem_cgroup *parent; + struct shrinker_info *child_info, *parent_info; + + parent = parent_mem_cgroup(memcg); + if (!parent) + parent = root_mem_cgroup; + + /* Prevent from concurrent shrinker_info expand */ + down_read(&shrinker_rwsem); + for_each_node(nid) { + child_info = shrinker_info_protected(memcg, nid); + parent_info = shrinker_info_protected(parent, nid); + for (i = 0; i < shrinker_nr_max; i++) { + nr = atomic_long_read(&child_info->nr_deferred[i]); + atomic_long_add(nr, &parent_info->nr_deferred[i]); + } + } + up_read(&shrinker_rwsem); +} + static bool cgroup_reclaim(struct scan_control *sc) { return sc->target_mem_cgroup; From patchwork Wed Jan 27 23:33:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12051229 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37963C433E6 for ; Wed, 27 Jan 2021 23:34:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E91FE64D7F for ; Wed, 27 Jan 2021 23:34:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E91FE64D7F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D57286B007D; Wed, 27 Jan 2021 18:34:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C8C7B6B007E; Wed, 27 Jan 2021 18:34:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2CFB6B0081; Wed, 27 Jan 2021 18:34:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0246.hostedemail.com [216.40.44.246]) by kanga.kvack.org (Postfix) with ESMTP id 94D266B007D for ; Wed, 27 Jan 2021 18:34:17 -0500 (EST) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 515EB8249980 for ; Wed, 27 Jan 2021 23:34:17 +0000 (UTC) X-FDA: 77753160954.27.ocean16_5a087032759b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id 368B63D663 for ; Wed, 27 Jan 2021 23:34:17 +0000 (UTC) X-HE-Tag: ocean16_5a087032759b X-Filterd-Recvd-Size: 6036 Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 Jan 2021 23:34:16 +0000 (UTC) Received: by mail-pg1-f178.google.com with SMTP id g15so2809647pgu.9 for ; Wed, 27 Jan 2021 15:34:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6zTr7oQng8A44UhAdRsCMp9TEBoUovtHbqgUeR1gt90=; b=ufz7fAq+D1Ldzv3KQIQSXZJiFZzJBw07ab2lzgQc/xuHOlmS+AJXgqzLu0ip18ohT8 JaE7UH03ZVgMWDLZUo4zt+NJLC0MyzS9x/iCNVun2KLczlFHn6KUEs1ISJyWcpLqsO+5 Uubhk+CWE8XQ5f94DHoDndqULxC30pSdqwurFm4eUs6pyEdBULbSA8b4iJ+xtBQ7g9Vz +eEortvkZ9BZVCes8RtcDa9eM3QutksUtdWBOxxts+d19fAF/iQ70rykYqSSRub/FI1k z9mTe2t8ETz37O5vFOsRcf9RGnwUhXRFFTjzn/3VQ7XMlisXynda4Kpf9UL8+5jfSViK KLQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6zTr7oQng8A44UhAdRsCMp9TEBoUovtHbqgUeR1gt90=; b=Lgn5nruxUwkFUw8lymXUO/czn4OYuaoLny1/zS/gx/MTiHxlI7vE62HQWKjs8RIqKX xulYSKclULx7Zw616JMaErncWqc0P0kh8M5rg1zrW2VQykp29xMSzTCSMqQgTKVzsKg0 BK7NzWwqbr31P/8FN3ouZs/8MkM+qweWqEu6qEIIZcephi0oSyN5qtRp2J8MyTD00qD9 u2GZNlxw3w6SwJplESywAUayvbGJze1UqPjCjRii3rRT8j97N0p+ge4YLu2Tcul5t13Q P19CgkfXgGkFIfPNU5svIRFJx2MsgpoPVWroaGjA2nzNpgPcuJFU8qUKs/PRX9mHJ42V SDYA== X-Gm-Message-State: AOAM530qCxxWqT5cqDHIggxxAade9q6XpnTtawyS2KaTVVG+hwSjEzxk 5mJWwOnAMnGC9QYoRAn2WSk= X-Google-Smtp-Source: ABdhPJxRoYG80XjL2szA2Xx9L3qJm5uTvO6Za+n3fNgIluivnAlS4ESMxzMdazerK1m7Lg4dmWhFyg== X-Received: by 2002:a63:da46:: with SMTP id l6mr14006921pgj.134.1611790455956; Wed, 27 Jan 2021 15:34:15 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id 124sm3498648pfd.59.2021.01.27.15.34.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 15:34:15 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 11/11] mm: vmscan: shrink deferred objects proportional to priority Date: Wed, 27 Jan 2021 15:33:45 -0800 Message-Id: <20210127233345.339910-12-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210127233345.339910-1-shy828301@gmail.com> References: <20210127233345.339910-1-shy828301@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The number of deferred objects might get windup to an absurd number, and it results in clamp of slab objects. It is undesirable for sustaining workingset. So shrink deferred objects proportional to priority and cap nr_deferred to twice of cache items. The idea is borrowed fron Dave Chinner's patch: https://lore.kernel.org/linux-xfs/20191031234618.15403-13-david@fromorbit.com/ Tested with kernel build and vfs metadata heavy workload, no regression is spotted so far. But it still may have regression for some corner cases. Signed-off-by: Yang Shi --- mm/vmscan.c | 40 +++++----------------------------------- 1 file changed, 5 insertions(+), 35 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 55ad91a26ba3..471d037d735e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -662,7 +662,6 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, */ nr = count_nr_deferred(shrinker, shrinkctl); - total_scan = nr; if (shrinker->seeks) { delta = freeable >> priority; delta *= 4; @@ -676,37 +675,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, delta = freeable / 2; } + total_scan = nr >> priority; total_scan += delta; - if (total_scan < 0) { - pr_err("shrink_slab: %pS negative objects to delete nr=%ld\n", - shrinker->scan_objects, total_scan); - total_scan = freeable; - next_deferred = nr; - } else - next_deferred = total_scan; - - /* - * We need to avoid excessive windup on filesystem shrinkers - * due to large numbers of GFP_NOFS allocations causing the - * shrinkers to return -1 all the time. This results in a large - * nr being built up so when a shrink that can do some work - * comes along it empties the entire cache due to nr >>> - * freeable. This is bad for sustaining a working set in - * memory. - * - * Hence only allow the shrinker to scan the entire cache when - * a large delta change is calculated directly. - */ - if (delta < freeable / 4) - total_scan = min(total_scan, freeable / 2); - - /* - * Avoid risking looping forever due to too large nr value: - * never try to free more than twice the estimate number of - * freeable entries. - */ - if (total_scan > freeable * 2) - total_scan = freeable * 2; + total_scan = min(total_scan, (2 * freeable)); trace_mm_shrink_slab_start(shrinker, shrinkctl, nr, freeable, delta, total_scan, priority); @@ -745,10 +716,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, cond_resched(); } - if (next_deferred >= scanned) - next_deferred -= scanned; - else - next_deferred = 0; + next_deferred = max_t(long, (nr - scanned), 0) + total_scan; + next_deferred = min(next_deferred, (2 * freeable)); + /* * move the unused scan count back into the shrinker in a * manner that handles concurrent updates.