From patchwork Wed Mar 10 17:45:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12128863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AA04C433DB for ; Wed, 10 Mar 2021 17:47:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 41F2864F1D for ; Wed, 10 Mar 2021 17:47:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 41F2864F1D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D0CA08D01D5; Wed, 10 Mar 2021 12:47:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CE43F8D01ED; Wed, 10 Mar 2021 12:47:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB70B8D01D5; Wed, 10 Mar 2021 12:47:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0253.hostedemail.com [216.40.44.253]) by kanga.kvack.org (Postfix) with ESMTP id A236F8D01D5 for ; Wed, 10 Mar 2021 12:47:18 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 5884F82499A8 for ; Wed, 10 Mar 2021 17:47:18 +0000 (UTC) X-FDA: 77904696156.16.8C70211 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf03.hostedemail.com (Postfix) with ESMTP id 9AF1BC001C47 for ; Wed, 10 Mar 2021 17:46:15 +0000 (UTC) Received: by mail-pj1-f44.google.com with SMTP id kr3-20020a17090b4903b02900c096fc01deso7898541pjb.4 for ; Wed, 10 Mar 2021 09:46:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Xfk/5hVVyAFMJCQGuT9bj2CxEXYg++EdZvSN0UtrQ3I=; b=PuEAhbWT7AWcBZuIbjQlpcFqTlG4qBBXxKDlR3RNaDYY+O8Cuc1PJsqDEbN4ODv1qW WgkRvJEqkqUwjyt9mVwA4bjt6ilyzahwH3Rd8zrbkgvnRKJXyV/GZMJTR1/jfvMSEXAx XL+DZezGFRC3uyMWhkvVe0f5vledaguk3UoqU81EXYpTWVl4lzTH7Ab1Hc8tYXCuSQIT 7A0gVhxnpuNNfe1nn2lz0YRZvSAsn2Qy5yoQZUCUflax5lXpCNZ7Np6piNXmsstFi7ZJ 6+o9dHfCGvDbuoUFt3vb/VTRZy3AQiJSHFx6D4Hh0z+JmnKtHChQdb+Q7ocSD7SYtjKp 8PtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Xfk/5hVVyAFMJCQGuT9bj2CxEXYg++EdZvSN0UtrQ3I=; b=Tg2Tkc6hYakXfN4AgNZ1yeObYammIlRNUAJVaTG8WL9z31ym7TSyL0Qt/3P3QxiiDy Pgs5MRV4F2ly2G7fkyCvyceyoqHt2QfQ6nSr5vvxugs8Vh1GoYDmkZAWs6HaneKrF/R7 bZwxighgfvXLvYpf6bkuUKjDLIObRhDgU0WSqlfyl6Sb9obXTNEoEbhZf1+1KnH6dzSv ruZ4op9RhbHez3OFflMQdySp4PjnSOTm7haUpRf8Cv2C0yWGk3C7KuBOdNtRq8kLXM26 nVkQ+ha3qPtOSnuQklcse/racV/NxwKA1spBj8ziZG841aGvAnyW48Fu+m+iWCN9bKyi M9Bw== X-Gm-Message-State: AOAM533XT1CPY69fZJBWvISz3YSlh7UUej+qRsKGmbxVo88fGTN8Dkjd 4HJU5myK8ryAycgc2KbYyy8= X-Google-Smtp-Source: ABdhPJwo0iozQkYECa6RLbEYLMRLVPxCBqIC2dkepEvixp18RaJjtZQmoxcIUawP1U67Dep2wKAuoA== X-Received: by 2002:a17:902:e78e:b029:e4:84c1:51ce with SMTP id cp14-20020a170902e78eb02900e484c151cemr3827631plb.25.1615398376908; Wed, 10 Mar 2021 09:46:16 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d6sm145804pfq.109.2021.03.10.09.46.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Mar 2021 09:46:16 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v9 PATCH 01/13] mm: vmscan: use nid from shrink_control for tracepoint Date: Wed, 10 Mar 2021 09:45:51 -0800 Message-Id: <20210310174603.5093-2-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210310174603.5093-1-shy828301@gmail.com> References: <20210310174603.5093-1-shy828301@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 9AF1BC001C47 X-Stat-Signature: 1n1zedxtc7i7z6w51jb6sm6je139qttd Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf03; identity=mailfrom; envelope-from=""; helo=mail-pj1-f44.google.com; client-ip=209.85.216.44 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615398375-985105 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The tracepoint's nid should show what node the shrink happens on, the start tracepoint uses nid from shrinkctl, but the nid might be set to 0 before end tracepoint if the shrinker is not NUMA aware, so the tracing log may show the shrink happens on one node but end up on the other node. It seems confusing. And the following patch will remove using nid directly in do_shrink_slab(), this patch also helps cleanup the code. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Reviewed-by: Shakeel Butt Acked-by: Roman Gushchin Signed-off-by: Yang Shi --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 562e87cbd7a1..31d116ea59a9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -536,7 +536,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, else new_nr = atomic_long_read(&shrinker->nr_deferred[nid]); - trace_mm_shrink_slab_end(shrinker, nid, freed, nr, new_nr, total_scan); + trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); return freed; } From patchwork Wed Mar 10 17:45:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12128865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD3D7C433E6 for ; Wed, 10 Mar 2021 17:47:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6994864EDD for ; Wed, 10 Mar 2021 17:47:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6994864EDD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 014F78D01EE; Wed, 10 Mar 2021 12:47:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F2E9E8D01ED; Wed, 10 Mar 2021 12:47:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D357A8D01EE; Wed, 10 Mar 2021 12:47:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0033.hostedemail.com [216.40.44.33]) by kanga.kvack.org (Postfix) with ESMTP id B89058D01ED for ; Wed, 10 Mar 2021 12:47:20 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 7B7EF5DD6 for ; Wed, 10 Mar 2021 17:47:20 +0000 (UTC) X-FDA: 77904696240.14.DF26D93 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf15.hostedemail.com (Postfix) with ESMTP id 19FA0A0049CA for ; Wed, 10 Mar 2021 17:46:18 +0000 (UTC) Received: by mail-pj1-f50.google.com with SMTP id t18so2039676pjs.3 for ; Wed, 10 Mar 2021 09:46:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ObeMVyV5O8vY+PHncdJMACZNksVYw+fJr38C23xI8TE=; b=sGcjbrUlbr984MgGT9icAbhOGN/txpCwRbO//C2jZZsuwUtT5YKkOZrAPUuKH8cNwT cxPC1MBB47mQ7VZdWf9koqjQg5GHsOsUNWaZHki08RBWeWXBNaV3f5+TmFwi0u4rdSgx +o1eTUqZK7TsKquJ485ogz3/gM5wMgJEKdS5Cz2V8OzkhsVzwWRaIvEYPAF96wgO885w LN+V62WIvq71enmmz0NYCStQRxAzzPpmkZxM1DqBPs1E9B3Afe3DwS/jebn4X/o6dObz 1X5x0u/9JsaflQrde7B3G4nXFXvOlrneZbzqMR7r/RNOzBwndSFmqQXFKRQ4doBVlqlT lhqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ObeMVyV5O8vY+PHncdJMACZNksVYw+fJr38C23xI8TE=; b=VQNXHm7dYLRDmZfP1NzkEJETtAHsMUMvti+viAnsfkUgjomFKNvpEik4XZcVAjBvEV JsMLcRgkAyR4Dz8yNe4aJ9H6zhJ838N3Nn/f0iwtKdRp0NoDELawmw4f10ylY82wLka8 7CfEhISXycvJ1l/XY6W1QLBKTosBwUkxmXzomYmUA4FahVYvrR/GB8C9rwcYunmSDtSj YRjlSeBqMfBHQZflNghsVFPj+jzZwy1yRKVd44lGSKYQzpS5QiPPTaehY0ublvr968VH pFY5IGWG3C/9V4PQmnJH5MnSuTRSTbS9X0k+yjJSm8fF+h3l7rBqnVCYjUUAZvvpbeKX JVug== X-Gm-Message-State: AOAM5312lVWeUsfTdOM9UBEW42vX/XdI8PiO1doYO6B8KS4DjnN/pm96 wMt5TvwBheQKZg0r5yeiZJ0dw85r/ep+wg== X-Google-Smtp-Source: ABdhPJwKLm7eKFn37xSSBdu3st3mDITd5Yzav6KEaKhbDBsEWV7b8AbnhmXiSpIpl8EhjU3ZsmhTpA== X-Received: by 2002:a17:902:da8d:b029:e5:c7d9:81f2 with SMTP id j13-20020a170902da8db02900e5c7d981f2mr4086791plx.21.1615398378781; Wed, 10 Mar 2021 09:46:18 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d6sm145804pfq.109.2021.03.10.09.46.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Mar 2021 09:46:18 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v9 PATCH 02/13] mm: vmscan: consolidate shrinker_maps handling code Date: Wed, 10 Mar 2021 09:45:52 -0800 Message-Id: <20210310174603.5093-3-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210310174603.5093-1-shy828301@gmail.com> References: <20210310174603.5093-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 61yoi4gqbpoyxisu5rwj78tig8ofuw6a X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 19FA0A0049CA Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf15; identity=mailfrom; envelope-from=""; helo=mail-pj1-f50.google.com; client-ip=209.85.216.50 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615398378-476682 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shrinker map management is not purely memcg specific, it is at the intersection between memory cgroup and shrinkers. It's allocation and assignment of a structure, and the only memcg bit is the map is being stored in a memcg structure. So move the shrinker_maps handling code into vmscan.c for tighter integration with shrinker code, and remove the "memcg_" prefix. There is no functional change. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 11 ++-- mm/huge_memory.c | 4 +- mm/list_lru.c | 6 +- mm/memcontrol.c | 129 +----------------------------------- mm/vmscan.c | 131 ++++++++++++++++++++++++++++++++++++- 5 files changed, 141 insertions(+), 140 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e6dc793d587d..fb2b7ef298ec 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1565,10 +1565,9 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -extern int memcg_expand_shrinker_maps(int new_id); - -extern void memcg_set_shrinker_bit(struct mem_cgroup *memcg, - int nid, int shrinker_id); +int alloc_shrinker_maps(struct mem_cgroup *memcg); +void free_shrinker_maps(struct mem_cgroup *memcg); +void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; @@ -1578,8 +1577,8 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -static inline void memcg_set_shrinker_bit(struct mem_cgroup *memcg, - int nid, int shrinker_id) +static inline void set_shrinker_bit(struct mem_cgroup *memcg, + int nid, int shrinker_id) { } #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 395c75111d33..e8008d2f8497 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2842,8 +2842,8 @@ void deferred_split_huge_page(struct page *page) ds_queue->split_queue_len++; #ifdef CONFIG_MEMCG if (memcg) - memcg_set_shrinker_bit(memcg, page_to_nid(page), - deferred_split_shrinker.id); + set_shrinker_bit(memcg, page_to_nid(page), + deferred_split_shrinker.id); #endif } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); diff --git a/mm/list_lru.c b/mm/list_lru.c index 6f067b6b935f..cd58790d0fb3 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -125,8 +125,8 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item) list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) - memcg_set_shrinker_bit(memcg, nid, - lru_shrinker_id(lru)); + set_shrinker_bit(memcg, nid, + lru_shrinker_id(lru)); nlru->nr_items++; spin_unlock(&nlru->lock); return true; @@ -540,7 +540,7 @@ static void memcg_drain_list_lru_node(struct list_lru *lru, int nid, if (src->nr_items) { dst->nr_items += src->nr_items; - memcg_set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); + set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 845eec01ef9d..a89c960f768e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -402,129 +402,6 @@ DEFINE_STATIC_KEY_FALSE(memcg_kmem_enabled_key); EXPORT_SYMBOL(memcg_kmem_enabled_key); #endif -static int memcg_shrinker_map_size; -static DEFINE_MUTEX(memcg_shrinker_map_mutex); - -static void memcg_free_shrinker_map_rcu(struct rcu_head *head) -{ - kvfree(container_of(head, struct memcg_shrinker_map, rcu)); -} - -static int memcg_expand_one_shrinker_map(struct mem_cgroup *memcg, - int size, int old_size) -{ - struct memcg_shrinker_map *new, *old; - int nid; - - lockdep_assert_held(&memcg_shrinker_map_mutex); - - for_each_node(nid) { - old = rcu_dereference_protected( - mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); - /* Not yet online memcg */ - if (!old) - return 0; - - new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); - if (!new) - return -ENOMEM; - - /* Set all old bits, clear all new bits */ - memset(new->map, (int)0xff, old_size); - memset((void *)new->map + old_size, 0, size - old_size); - - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, new); - call_rcu(&old->rcu, memcg_free_shrinker_map_rcu); - } - - return 0; -} - -static void memcg_free_shrinker_maps(struct mem_cgroup *memcg) -{ - struct mem_cgroup_per_node *pn; - struct memcg_shrinker_map *map; - int nid; - - if (mem_cgroup_is_root(memcg)) - return; - - for_each_node(nid) { - pn = mem_cgroup_nodeinfo(memcg, nid); - map = rcu_dereference_protected(pn->shrinker_map, true); - kvfree(map); - rcu_assign_pointer(pn->shrinker_map, NULL); - } -} - -static int memcg_alloc_shrinker_maps(struct mem_cgroup *memcg) -{ - struct memcg_shrinker_map *map; - int nid, size, ret = 0; - - if (mem_cgroup_is_root(memcg)) - return 0; - - mutex_lock(&memcg_shrinker_map_mutex); - size = memcg_shrinker_map_size; - for_each_node(nid) { - map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); - if (!map) { - memcg_free_shrinker_maps(memcg); - ret = -ENOMEM; - break; - } - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); - } - mutex_unlock(&memcg_shrinker_map_mutex); - - return ret; -} - -int memcg_expand_shrinker_maps(int new_id) -{ - int size, old_size, ret = 0; - struct mem_cgroup *memcg; - - size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); - old_size = memcg_shrinker_map_size; - if (size <= old_size) - return 0; - - mutex_lock(&memcg_shrinker_map_mutex); - if (!root_mem_cgroup) - goto unlock; - - for_each_mem_cgroup(memcg) { - if (mem_cgroup_is_root(memcg)) - continue; - ret = memcg_expand_one_shrinker_map(memcg, size, old_size); - if (ret) { - mem_cgroup_iter_break(NULL, memcg); - goto unlock; - } - } -unlock: - if (!ret) - memcg_shrinker_map_size = size; - mutex_unlock(&memcg_shrinker_map_mutex); - return ret; -} - -void memcg_set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) -{ - if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { - struct memcg_shrinker_map *map; - - rcu_read_lock(); - map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, map->map); - rcu_read_unlock(); - } -} - /** * mem_cgroup_css_from_page - css of the memcg associated with a page * @page: page of interest @@ -5349,11 +5226,11 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) struct mem_cgroup *memcg = mem_cgroup_from_css(css); /* - * A memcg must be visible for memcg_expand_shrinker_maps() + * A memcg must be visible for expand_shrinker_maps() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (memcg_alloc_shrinker_maps(memcg)) { + if (alloc_shrinker_maps(memcg)) { mem_cgroup_id_remove(memcg); return -ENOMEM; } @@ -5417,7 +5294,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) vmpressure_cleanup(&memcg->vmpressure); cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); - memcg_free_shrinker_maps(memcg); + free_shrinker_maps(memcg); memcg_free_kmem(memcg); mem_cgroup_free(memcg); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 31d116ea59a9..ad164f3af9a0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -185,6 +185,131 @@ static LIST_HEAD(shrinker_list); static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG + +static int memcg_shrinker_map_size; +static DEFINE_MUTEX(memcg_shrinker_map_mutex); + +static void free_shrinker_map_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct memcg_shrinker_map, rcu)); +} + +static int expand_one_shrinker_map(struct mem_cgroup *memcg, + int size, int old_size) +{ + struct memcg_shrinker_map *new, *old; + int nid; + + lockdep_assert_held(&memcg_shrinker_map_mutex); + + for_each_node(nid) { + old = rcu_dereference_protected( + mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); + /* Not yet online memcg */ + if (!old) + return 0; + + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); + if (!new) + return -ENOMEM; + + /* Set all old bits, clear all new bits */ + memset(new->map, (int)0xff, old_size); + memset((void *)new->map + old_size, 0, size - old_size); + + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, new); + call_rcu(&old->rcu, free_shrinker_map_rcu); + } + + return 0; +} + +void free_shrinker_maps(struct mem_cgroup *memcg) +{ + struct mem_cgroup_per_node *pn; + struct memcg_shrinker_map *map; + int nid; + + if (mem_cgroup_is_root(memcg)) + return; + + for_each_node(nid) { + pn = mem_cgroup_nodeinfo(memcg, nid); + map = rcu_dereference_protected(pn->shrinker_map, true); + kvfree(map); + rcu_assign_pointer(pn->shrinker_map, NULL); + } +} + +int alloc_shrinker_maps(struct mem_cgroup *memcg) +{ + struct memcg_shrinker_map *map; + int nid, size, ret = 0; + + if (mem_cgroup_is_root(memcg)) + return 0; + + mutex_lock(&memcg_shrinker_map_mutex); + size = memcg_shrinker_map_size; + for_each_node(nid) { + map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); + if (!map) { + free_shrinker_maps(memcg); + ret = -ENOMEM; + break; + } + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); + } + mutex_unlock(&memcg_shrinker_map_mutex); + + return ret; +} + +static int expand_shrinker_maps(int new_id) +{ + int size, old_size, ret = 0; + struct mem_cgroup *memcg; + + size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); + old_size = memcg_shrinker_map_size; + if (size <= old_size) + return 0; + + mutex_lock(&memcg_shrinker_map_mutex); + if (!root_mem_cgroup) + goto unlock; + + memcg = mem_cgroup_iter(NULL, NULL, NULL); + do { + if (mem_cgroup_is_root(memcg)) + continue; + ret = expand_one_shrinker_map(memcg, size, old_size); + if (ret) { + mem_cgroup_iter_break(NULL, memcg); + goto unlock; + } + } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); +unlock: + if (!ret) + memcg_shrinker_map_size = size; + mutex_unlock(&memcg_shrinker_map_mutex); + return ret; +} + +void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) +{ + if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { + struct memcg_shrinker_map *map; + + rcu_read_lock(); + map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, map->map); + rcu_read_unlock(); + } +} + /* * We allow subsystems to populate their shrinker-related * LRU lists before register_shrinker_prepared() is called @@ -212,7 +337,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) goto unlock; if (id >= shrinker_nr_max) { - if (memcg_expand_shrinker_maps(id)) { + if (expand_shrinker_maps(id)) { idr_remove(&shrinker_idr, id); goto unlock; } @@ -590,7 +715,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, * case, we invoke the shrinker one more time and reset * the bit if it reports that it is not empty anymore. * The memory barrier here pairs with the barrier in - * memcg_set_shrinker_bit(): + * set_shrinker_bit(): * * list_lru_add() shrink_slab_memcg() * list_add_tail() clear_bit() @@ -602,7 +727,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; else - memcg_set_shrinker_bit(memcg, nid, i); + set_shrinker_bit(memcg, nid, i); } freed += ret; From patchwork Wed Mar 10 17:45:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12128815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1ACE9C433DB for ; Wed, 10 Mar 2021 17:46:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A7F6964F45 for ; Wed, 10 Mar 2021 17:46:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A7F6964F45 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 287F58D01E3; Wed, 10 Mar 2021 12:46:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 20FC58D01D5; Wed, 10 Mar 2021 12:46:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 08A7F8D01E3; Wed, 10 Mar 2021 12:46:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0091.hostedemail.com [216.40.44.91]) by kanga.kvack.org (Postfix) with ESMTP id E30998D01D5 for ; Wed, 10 Mar 2021 12:46:22 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 9F7F02484 for ; Wed, 10 Mar 2021 17:46:22 +0000 (UTC) X-FDA: 77904693804.14.C28E7B6 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf04.hostedemail.com (Postfix) with ESMTP id 4205E3EB for ; Wed, 10 Mar 2021 17:46:21 +0000 (UTC) Received: by mail-pg1-f177.google.com with SMTP id w34so10807671pga.8 for ; Wed, 10 Mar 2021 09:46:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aiM+lLYNubKNlhU5HOyGVLNEOVuVKT3O2pdKzgZrP7U=; b=L38uPzNxzeHdFm5yL8UeAwlmZ/6ul0Jc5NXK/TEsJNntKbqK6cuERyzxVNFntfJzpR MSzJpTbHGADjG5e0kPevhzKwvDipYOrp83GOb6zMtXAc21yCbXlVx3+JOat8amDsl9A3 PTMGkBWbFpO1O8w1MTpxwQqsKgxG5OIZKCfOOWjUzgU9BKA9d0QQUT7HX3agjv/wcoj5 lgrUlMfRhBiBUexwdNJayXurHC/gLCKLU2IpjXJzlbwNUW9qxioTi4laOh/qlTn5pyaw Oj9wV5XtmrJLnwyqWdj0wRAnfRG/19ZxvcBPqxOIF1QB4cPnGlMFJtNoKATCeAEGAeNN pfqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aiM+lLYNubKNlhU5HOyGVLNEOVuVKT3O2pdKzgZrP7U=; b=ewYHjOPv/gzokLkhcuwWWoPCN6w0zDCowB35Nhk1DO+Z68H8xKmSpCI8a8gez6Hdba +bC8C4bvIeQy+eJj7WJ7w3IGuh5DKtSyRGAAal/Uo1GOCSlZiUOcoq2vyrRBScas5o35 6xX5O0HD3Zh7IyaGf+/SMMRd62PNkbsZ1mDidSALtW1sZEWJyJ2rVan0CPjBoqdJtgPE I52zZGGqkgCJB0SKXFob2DCzE0QR0BAtLfKJy/PCLB4phDhD/rUdokOS5gSiE1s5Dmdc Q0xy/3JO8gDqTzVtRKiK61hxYnE0B7aPLLrC/2r4PBphzi+968WL2htTCOz59JlKPnfC U5Og== X-Gm-Message-State: AOAM5310Q7MHnLWTGcttSgthJ9gRPPnh0ZX7BqarVw3/cBMTNyf3lOeT nF+cQrIIvUOkheRM9V3R5fs= X-Google-Smtp-Source: ABdhPJyCE7pWMJCZS0OOk6anncpki25SoIBg7kTzYHtJ/2KgB4+490RPqKke3EkJ2ZeqQpgNIbP8rQ== X-Received: by 2002:aa7:888b:0:b029:1ec:df4a:4da2 with SMTP id z11-20020aa7888b0000b02901ecdf4a4da2mr3830929pfe.66.1615398381083; Wed, 10 Mar 2021 09:46:21 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d6sm145804pfq.109.2021.03.10.09.46.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Mar 2021 09:46:20 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v9 PATCH 03/13] mm: vmscan: use shrinker_rwsem to protect shrinker_maps allocation Date: Wed, 10 Mar 2021 09:45:53 -0800 Message-Id: <20210310174603.5093-4-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210310174603.5093-1-shy828301@gmail.com> References: <20210310174603.5093-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: jz975x8uceiqz9x44x5739bqtjzfrwb7 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 4205E3EB Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf04; identity=mailfrom; envelope-from=""; helo=mail-pg1-f177.google.com; client-ip=209.85.215.177 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615398381-869911 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since memcg_shrinker_map_size just can be changed under holding shrinker_rwsem exclusively, the read side can be protected by holding read lock, so it sounds superfluous to have a dedicated mutex. Kirill Tkhai suggested use write lock since: * We want the assignment to shrinker_maps is visible for shrink_slab_memcg(). * The rcu_dereference_protected() dereferrencing in shrink_slab_memcg(), but in case of we use READ lock in alloc_shrinker_maps(), the dereferrencing is not actually protected. * READ lock makes alloc_shrinker_info() racy against memory allocation fail. alloc_shrinker_info()->free_shrinker_info() may free memory right after shrink_slab_memcg() dereferenced it. You may say shrink_slab_memcg()->mem_cgroup_online() protects us from it? Yes, sure, but this is not the thing we want to remember in the future, since this spreads modularity. And a test with heavy paging workload didn't show write lock makes things worse. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index ad164f3af9a0..75fd8038a6c8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -187,7 +187,6 @@ static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG static int memcg_shrinker_map_size; -static DEFINE_MUTEX(memcg_shrinker_map_mutex); static void free_shrinker_map_rcu(struct rcu_head *head) { @@ -200,8 +199,6 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, struct memcg_shrinker_map *new, *old; int nid; - lockdep_assert_held(&memcg_shrinker_map_mutex); - for_each_node(nid) { old = rcu_dereference_protected( mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); @@ -249,7 +246,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) if (mem_cgroup_is_root(memcg)) return 0; - mutex_lock(&memcg_shrinker_map_mutex); + down_write(&shrinker_rwsem); size = memcg_shrinker_map_size; for_each_node(nid) { map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); @@ -260,7 +257,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) } rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); } - mutex_unlock(&memcg_shrinker_map_mutex); + up_write(&shrinker_rwsem); return ret; } @@ -275,9 +272,10 @@ static int expand_shrinker_maps(int new_id) if (size <= old_size) return 0; - mutex_lock(&memcg_shrinker_map_mutex); if (!root_mem_cgroup) - goto unlock; + goto out; + + lockdep_assert_held(&shrinker_rwsem); memcg = mem_cgroup_iter(NULL, NULL, NULL); do { @@ -286,13 +284,13 @@ static int expand_shrinker_maps(int new_id) ret = expand_one_shrinker_map(memcg, size, old_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); - goto unlock; + goto out; } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); -unlock: +out: if (!ret) memcg_shrinker_map_size = size; - mutex_unlock(&memcg_shrinker_map_mutex); + return ret; } From patchwork Wed Mar 10 17:45:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12128817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33475C433E0 for ; Wed, 10 Mar 2021 17:46:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CCF1264F1D for ; Wed, 10 Mar 2021 17:46:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CCF1264F1D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6CD918D01E4; Wed, 10 Mar 2021 12:46:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 658BA8D01D5; Wed, 10 Mar 2021 12:46:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F77A8D01E4; Wed, 10 Mar 2021 12:46:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0020.hostedemail.com [216.40.44.20]) by kanga.kvack.org (Postfix) with ESMTP id 35DFC8D01D5 for ; Wed, 10 Mar 2021 12:46:25 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E425E181AF5C2 for ; Wed, 10 Mar 2021 17:46:24 +0000 (UTC) X-FDA: 77904693888.29.4098911 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by imf21.hostedemail.com (Postfix) with ESMTP id 9A75FE005F1C for ; Wed, 10 Mar 2021 17:46:22 +0000 (UTC) Received: by mail-pf1-f173.google.com with SMTP id t85so7296398pfc.13 for ; Wed, 10 Mar 2021 09:46:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8qnm1r39F1+nLF2q+MJEWtJZ2OA03LbHXXfmJXBg/bU=; b=bQha59oGw/ATUzMVQxON9irQJ6V4Iq4VT34my0MPRxVqoD3pc3R7GOZ4Ihw9+D3w7J QbcWdPEOKc7kLp9evqhFteAsW6sH5UhtJNH9G8COXvScubfZr5yqOEPqb6FpdRZ1d7z7 IH68e2svWzJIzb+jG1kbN/SK5aHjP1dtN9op49wnjmKxIzjTUusOBvm5SD52yI2sOXN4 LCEZYzkLc1103vE/bmiPc59xEU2imubQ8rKnH7rx2ssNjYyNBPTQy2XSHdGOLD/e/qQV UbBghQvT4rK1Hea+MZEy2+QfEqswO7+A4Vj4uIq2h9AMIUk5AzoqbOzCpuTf62uCGU/D A9lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8qnm1r39F1+nLF2q+MJEWtJZ2OA03LbHXXfmJXBg/bU=; b=HxRunU0o33IpIxxK4+kSJVWcf8zwzB1H75/1tbOmnpjhacu1TxTheqEKlUKq3TVHVO g4PzKo9ixl3Zi2uRI884SKw/nL+E2XBTzea9gWryX9lH+Rq9hVDNwn4S0QNj+4F0VvRG NZ0Hy6LR/JcbgV58Xw7P1IBbHLqSXTLbpouQfeyRyMhjvssp3+hF9OVeXb5mUb+ZLLTq fM8FMyc0BZQYs85S90FYh1MV7FZdJedPSYspSx4b21k58dGTSY4CFM5FzTivs9w0EIAn CDPSDtDPZidflB5zr6UCbjY/+xWGY/xu3soOU5uQ7wN1NKv5E8Y8FbarGlmRtHSbHdrb D8Tw== X-Gm-Message-State: AOAM532wFOGGCDzoosj6eSOwd5BwOzUadAVLkll1C6na1pZdYGPQrWic SVzOBWsfriEUBDNV0yD1tfA= X-Google-Smtp-Source: ABdhPJzBn8tAF3hvcNEKpLzZbv6EQt380V9sSRJOIaoGi/hfNxA10rSwz5p+lofpE1kxGrEMUTQNhg== X-Received: by 2002:a65:6642:: with SMTP id z2mr3695109pgv.214.1615398383207; Wed, 10 Mar 2021 09:46:23 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d6sm145804pfq.109.2021.03.10.09.46.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Mar 2021 09:46:22 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v9 PATCH 04/13] mm: vmscan: remove memcg_shrinker_map_size Date: Wed, 10 Mar 2021 09:45:54 -0800 Message-Id: <20210310174603.5093-5-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210310174603.5093-1-shy828301@gmail.com> References: <20210310174603.5093-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 5aebkth3znwq5nyg88kogu6qbafbciqc X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 9A75FE005F1C Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf21; identity=mailfrom; envelope-from=""; helo=mail-pf1-f173.google.com; client-ip=209.85.210.173 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615398382-135163 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Both memcg_shrinker_map_size and shrinker_nr_max is maintained, but actually the map size can be calculated via shrinker_nr_max, so it seems unnecessary to keep both. Remove memcg_shrinker_map_size since shrinker_nr_max is also used by iterating the bit map. Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Acked-by: Vlastimil Babka Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 75fd8038a6c8..bda67e1ac84b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -185,8 +185,12 @@ static LIST_HEAD(shrinker_list); static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG +static int shrinker_nr_max; -static int memcg_shrinker_map_size; +static inline int shrinker_map_size(int nr_items) +{ + return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); +} static void free_shrinker_map_rcu(struct rcu_head *head) { @@ -247,7 +251,7 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) return 0; down_write(&shrinker_rwsem); - size = memcg_shrinker_map_size; + size = shrinker_map_size(shrinker_nr_max); for_each_node(nid) { map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); if (!map) { @@ -265,12 +269,13 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) static int expand_shrinker_maps(int new_id) { int size, old_size, ret = 0; + int new_nr_max = new_id + 1; struct mem_cgroup *memcg; - size = DIV_ROUND_UP(new_id + 1, BITS_PER_LONG) * sizeof(unsigned long); - old_size = memcg_shrinker_map_size; + size = shrinker_map_size(new_nr_max); + old_size = shrinker_map_size(shrinker_nr_max); if (size <= old_size) - return 0; + goto out; if (!root_mem_cgroup) goto out; @@ -289,7 +294,7 @@ static int expand_shrinker_maps(int new_id) } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); out: if (!ret) - memcg_shrinker_map_size = size; + shrinker_nr_max = new_nr_max; return ret; } @@ -322,7 +327,6 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) #define SHRINKER_REGISTERING ((struct shrinker *)~0UL) static DEFINE_IDR(shrinker_idr); -static int shrinker_nr_max; static int prealloc_memcg_shrinker(struct shrinker *shrinker) { @@ -339,8 +343,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) idr_remove(&shrinker_idr, id); goto unlock; } - - shrinker_nr_max = id + 1; } shrinker->id = id; ret = 0; From patchwork Wed Mar 10 17:45:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12128819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF344C433DB for ; Wed, 10 Mar 2021 17:46:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 96CA464F1D for ; Wed, 10 Mar 2021 17:46:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 96CA464F1D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D0FCC8D01E5; Wed, 10 Mar 2021 12:46:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CE7978D01D5; Wed, 10 Mar 2021 12:46:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B60CD8D01E5; Wed, 10 Mar 2021 12:46:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0083.hostedemail.com [216.40.44.83]) by kanga.kvack.org (Postfix) with ESMTP id 9526C8D01D5 for ; Wed, 10 Mar 2021 12:46:26 -0500 (EST) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4DFE41809CA52 for ; Wed, 10 Mar 2021 17:46:26 +0000 (UTC) X-FDA: 77904693972.04.F20E63F Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf12.hostedemail.com (Postfix) with ESMTP id 4CADD238F for ; Wed, 10 Mar 2021 17:46:22 +0000 (UTC) Received: by mail-pf1-f182.google.com with SMTP id l7so12607553pfd.3 for ; Wed, 10 Mar 2021 09:46:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wWUIbkbeLDVeoKhz04HlwT305yWIUKH2N08ijUlQ/LU=; b=RQpXtzHP6YLnQC3J09daZ1GQ7Qy/ODQBOv6H7Z+LvqIS0e7lDgCcOoaNdrznD3roDt uXG/8UkhVivBJ8R+uu0pyD7qtpuNK+1eCaa6UEA6Dzcjetx7RIE67JyTSZ5xtx4/XZAU FsggF8rrvj+qXONgVtNy+LbbkKjX4JqcNtC1AwMVW1WyLWzJ1tA/UMRQDyKyU6Wkv6od UAHFvYXXOHo5zV6UlRE6sFxrR4vBxmNfqGqIXhBGwvWO5CU2ru8uApAI5iX6vp0N6pAM PoV6D18R+RvVz7ohociHFzTV3UkHctNUy3+zaK2C1vtCMH1hSL3WSRx8n/2PEdMrs/Ve 6WwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wWUIbkbeLDVeoKhz04HlwT305yWIUKH2N08ijUlQ/LU=; b=TuTVBGb9fHTCd8s1VrJ6oOldinDDdsq7CTXA2Dxe/bPWHlmZ026OJO/N8a+YWBLnyD 7FQjaZkBE0ZW2doXH+Qcwx/Te+O6/bIfucFR2G+7nEt2aEBiPNtvoh+iTNfO5Yvy4Amy f1mB/A81XLIpMxrkgtGjGkK3Ro4LnHSme8ch7HvG/DgEwdYjcVGJw+zMcbKBmPezzUor QQgJ2vsls84tY0Z1mz4E0HboUgllYuQd+Mj5iIidXIk7QKLLIpTcP1o/DPO6xTtOfkvv 29Go1phX1AX6lrVQOnd+7R/XjpygXdAdOxJxuguGNfhhmXuIBH9u7Vtq1HRHvAIOURb+ P2tg== X-Gm-Message-State: AOAM532Sr+b/0JlHRHHfiZhjjwk5MwUDhR5nypGCLJFDR7u9CFNyD/C9 +J2vkF2RNuOISRISiPvVyIg= X-Google-Smtp-Source: ABdhPJwG6polu59DK+rBm2AeJddhoqFoysCxyy7Lndk7oMONee3tmfH81DaB99JjMA4u3gQUliU3Dg== X-Received: by 2002:a63:c84a:: with SMTP id l10mr3724168pgi.159.1615398385023; Wed, 10 Mar 2021 09:46:25 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d6sm145804pfq.109.2021.03.10.09.46.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Mar 2021 09:46:24 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v9 PATCH 05/13] mm: vmscan: use kvfree_rcu instead of call_rcu Date: Wed, 10 Mar 2021 09:45:55 -0800 Message-Id: <20210310174603.5093-6-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210310174603.5093-1-shy828301@gmail.com> References: <20210310174603.5093-1-shy828301@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4CADD238F X-Stat-Signature: tigyic7jebfir49a85jky975b6qpzdb7 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf12; identity=mailfrom; envelope-from=""; helo=mail-pf1-f182.google.com; client-ip=209.85.210.182 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615398382-647451 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Using kvfree_rcu() to free the old shrinker_maps instead of call_rcu(). We don't have to define a dedicated callback for call_rcu() anymore. Acked-by: Roman Gushchin Acked-by: Kirill Tkhai Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index bda67e1ac84b..c9898e66011e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -192,11 +192,6 @@ static inline int shrinker_map_size(int nr_items) return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); } -static void free_shrinker_map_rcu(struct rcu_head *head) -{ - kvfree(container_of(head, struct memcg_shrinker_map, rcu)); -} - static int expand_one_shrinker_map(struct mem_cgroup *memcg, int size, int old_size) { @@ -219,7 +214,7 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, memset((void *)new->map + old_size, 0, size - old_size); rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, new); - call_rcu(&old->rcu, free_shrinker_map_rcu); + kvfree_rcu(old, rcu); } return 0; From patchwork Wed Mar 10 17:45:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12128821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DA38C433E0 for ; Wed, 10 Mar 2021 17:46:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E622864EDD for ; Wed, 10 Mar 2021 17:46:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E622864EDD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 260698D01E6; Wed, 10 Mar 2021 12:46:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2121F8D01D5; Wed, 10 Mar 2021 12:46:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 063848D01E6; Wed, 10 Mar 2021 12:46:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DC3478D01D5 for ; Wed, 10 Mar 2021 12:46:28 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A0DF918140B74 for ; Wed, 10 Mar 2021 17:46:28 +0000 (UTC) X-FDA: 77904694056.19.6F95A2B Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf06.hostedemail.com (Postfix) with ESMTP id 57765C000C6B for ; Wed, 10 Mar 2021 17:46:28 +0000 (UTC) Received: by mail-pf1-f182.google.com with SMTP id s21so1645969pfm.1 for ; Wed, 10 Mar 2021 09:46:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2W3ntTrXqJM/mWjhYznbaelKG6bWKsePb0Ao/1PIRAY=; b=rIXhaC5Iki0FteZltEViFBuKC2WQJmHTczF1sDVbvHzQEsocI1b8KimJcJJ9h1PK3d hDqDujx/g9plz4rDssRAyG53/VMiisuKHoWDAqAs4UWVoGmdTCgYsZHtARMv96JBnf0L SUuUO+e5iMODTtGcm/+3pKAJubLup3ugfKQ3Lu1K/5HD8wsakBES/BOYEd1BFtmHNEUC MTv08xK90bQjeWiDt+vyAycuHbHwymrAxQ9i22yVf0IC21+Tl1vDO65igqPxf/3u/7GC B6Z7L04Hc5LtEBA5B2rni4MA4KZIb8MAEyjFjSu04TOOQIX3kjQJw48jR2WHhx6kRckF BgEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2W3ntTrXqJM/mWjhYznbaelKG6bWKsePb0Ao/1PIRAY=; b=WSTCpVvQPZ/58xIXYwSdq1+2VoF5jgchkZNlsJ5bI7o8jwLM25UqVi6y7b7R8rjPqu FCKeCk0DFuAvVWYp39y/vOpDt98U7QL7A4/lAhjbcosTTQEMwkM/ARkzpiVCOQE/0oKN Lbsv+3bKdrS1BgisL/65fvFhcmh6PRlicSotv4LzC083LZgfA6vqQWMUX6gxjhBJ1NhB 02qFSrZJqZsUsVTS3AKdb+gIzBoK/ZbGmo4h8d3MmFHa8SQ6VwuRLD+Vi+ZbU2uUZqTr GY73LykRu95DHV0r4wOE0MI+H9XXHiHK2gZxQ9wDNxK/HLDT7uj4X5f8CoW79/zlDtI+ 5vbg== X-Gm-Message-State: AOAM531kuyJ0IUXbJIrEzzdXTWzC2q9ks/sG9HYuPoauTY2PSNorCxI6 FcdWZ29ETU8XqR9P2mcqOSg= X-Google-Smtp-Source: ABdhPJy6hWQf8gzXhJQdLdSQs/jiVAM24ALfP1sKNe+juvOErAoTmQ8kndGD/Yxo1X0R4jgbO//iFQ== X-Received: by 2002:a62:62c1:0:b029:1ee:7ad:8cb3 with SMTP id w184-20020a6262c10000b02901ee07ad8cb3mr4067466pfb.21.1615398387208; Wed, 10 Mar 2021 09:46:27 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d6sm145804pfq.109.2021.03.10.09.46.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Mar 2021 09:46:26 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v9 PATCH 06/13] mm: memcontrol: rename shrinker_map to shrinker_info Date: Wed, 10 Mar 2021 09:45:56 -0800 Message-Id: <20210310174603.5093-7-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210310174603.5093-1-shy828301@gmail.com> References: <20210310174603.5093-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: dpf7ra3jo3kzu74abydi8jkh9bnpc8xm X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 57765C000C6B Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=mail-pf1-f182.google.com; client-ip=209.85.210.182 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615398388-295234 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The following patch is going to add nr_deferred into shrinker_map, the change will make shrinker_map not only include map anymore, so rename it to "memcg_shrinker_info". And this should make the patch adding nr_deferred cleaner and readable and make review easier. Also remove the "memcg_" prefix. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 8 +++--- mm/memcontrol.c | 6 ++-- mm/vmscan.c | 58 +++++++++++++++++++------------------- 3 files changed, 36 insertions(+), 36 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index fb2b7ef298ec..308a202f1de2 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -100,7 +100,7 @@ struct batched_lruvec_stat { * Bitmap of shrinker::id corresponding to memcg-aware shrinkers, * which have elements charged to this memcg. */ -struct memcg_shrinker_map { +struct shrinker_info { struct rcu_head rcu; unsigned long map[]; }; @@ -128,7 +128,7 @@ struct mem_cgroup_per_node { struct mem_cgroup_reclaim_iter iter; - struct memcg_shrinker_map __rcu *shrinker_map; + struct shrinker_info __rcu *shrinker_info; struct rb_node tree_node; /* RB tree node */ unsigned long usage_in_excess;/* Set to the value by which */ @@ -1565,8 +1565,8 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) return false; } -int alloc_shrinker_maps(struct mem_cgroup *memcg); -void free_shrinker_maps(struct mem_cgroup *memcg); +int alloc_shrinker_info(struct mem_cgroup *memcg); +void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); #else #define mem_cgroup_sockets_enabled 0 diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a89c960f768e..edd8a06c751f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5226,11 +5226,11 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) struct mem_cgroup *memcg = mem_cgroup_from_css(css); /* - * A memcg must be visible for expand_shrinker_maps() + * A memcg must be visible for expand_shrinker_info() * by the time the maps are allocated. So, we allocate maps * here, when for_each_mem_cgroup() can't skip it. */ - if (alloc_shrinker_maps(memcg)) { + if (alloc_shrinker_info(memcg)) { mem_cgroup_id_remove(memcg); return -ENOMEM; } @@ -5294,7 +5294,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) vmpressure_cleanup(&memcg->vmpressure); cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); - free_shrinker_maps(memcg); + free_shrinker_info(memcg); memcg_free_kmem(memcg); mem_cgroup_free(memcg); } diff --git a/mm/vmscan.c b/mm/vmscan.c index c9898e66011e..7f3c00e76fd1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -192,15 +192,15 @@ static inline int shrinker_map_size(int nr_items) return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); } -static int expand_one_shrinker_map(struct mem_cgroup *memcg, - int size, int old_size) +static int expand_one_shrinker_info(struct mem_cgroup *memcg, + int size, int old_size) { - struct memcg_shrinker_map *new, *old; + struct shrinker_info *new, *old; int nid; for_each_node(nid) { old = rcu_dereference_protected( - mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); + mem_cgroup_nodeinfo(memcg, nid)->shrinker_info, true); /* Not yet online memcg */ if (!old) return 0; @@ -213,17 +213,17 @@ static int expand_one_shrinker_map(struct mem_cgroup *memcg, memset(new->map, (int)0xff, old_size); memset((void *)new->map + old_size, 0, size - old_size); - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, new); + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, new); kvfree_rcu(old, rcu); } return 0; } -void free_shrinker_maps(struct mem_cgroup *memcg) +void free_shrinker_info(struct mem_cgroup *memcg) { struct mem_cgroup_per_node *pn; - struct memcg_shrinker_map *map; + struct shrinker_info *info; int nid; if (mem_cgroup_is_root(memcg)) @@ -231,15 +231,15 @@ void free_shrinker_maps(struct mem_cgroup *memcg) for_each_node(nid) { pn = mem_cgroup_nodeinfo(memcg, nid); - map = rcu_dereference_protected(pn->shrinker_map, true); - kvfree(map); - rcu_assign_pointer(pn->shrinker_map, NULL); + info = rcu_dereference_protected(pn->shrinker_info, true); + kvfree(info); + rcu_assign_pointer(pn->shrinker_info, NULL); } } -int alloc_shrinker_maps(struct mem_cgroup *memcg) +int alloc_shrinker_info(struct mem_cgroup *memcg) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; int nid, size, ret = 0; if (mem_cgroup_is_root(memcg)) @@ -248,20 +248,20 @@ int alloc_shrinker_maps(struct mem_cgroup *memcg) down_write(&shrinker_rwsem); size = shrinker_map_size(shrinker_nr_max); for_each_node(nid) { - map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); - if (!map) { - free_shrinker_maps(memcg); + info = kvzalloc_node(sizeof(*info) + size, GFP_KERNEL, nid); + if (!info) { + free_shrinker_info(memcg); ret = -ENOMEM; break; } - rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); return ret; } -static int expand_shrinker_maps(int new_id) +static int expand_shrinker_info(int new_id) { int size, old_size, ret = 0; int new_nr_max = new_id + 1; @@ -281,7 +281,7 @@ static int expand_shrinker_maps(int new_id) do { if (mem_cgroup_is_root(memcg)) continue; - ret = expand_one_shrinker_map(memcg, size, old_size); + ret = expand_one_shrinker_info(memcg, size, old_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -297,13 +297,13 @@ static int expand_shrinker_maps(int new_id) void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; rcu_read_lock(); - map = rcu_dereference(memcg->nodeinfo[nid]->shrinker_map); + info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); - set_bit(shrinker_id, map->map); + set_bit(shrinker_id, info->map); rcu_read_unlock(); } } @@ -334,7 +334,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) goto unlock; if (id >= shrinker_nr_max) { - if (expand_shrinker_maps(id)) { + if (expand_shrinker_info(id)) { idr_remove(&shrinker_idr, id); goto unlock; } @@ -664,7 +664,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, int priority) { - struct memcg_shrinker_map *map; + struct shrinker_info *info; unsigned long ret, freed = 0; int i; @@ -674,12 +674,12 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (!down_read_trylock(&shrinker_rwsem)) return 0; - map = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_map, - true); - if (unlikely(!map)) + info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + true); + if (unlikely(!info)) goto unlock; - for_each_set_bit(i, map->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, shrinker_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -690,7 +690,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, shrinker = idr_find(&shrinker_idr, i); if (unlikely(!shrinker || shrinker == SHRINKER_REGISTERING)) { if (!shrinker) - clear_bit(i, map->map); + clear_bit(i, info->map); continue; } @@ -701,7 +701,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, ret = do_shrink_slab(&sc, shrinker, priority); if (ret == SHRINK_EMPTY) { - clear_bit(i, map->map); + clear_bit(i, info->map); /* * After the shrinker reported that it had no objects to * free, but before we cleared the corresponding bit in From patchwork Wed Mar 10 17:45:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12128867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9075CC433DB for ; Wed, 10 Mar 2021 17:47:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3D11E64EDD for ; Wed, 10 Mar 2021 17:47:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3D11E64EDD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D348F8D01EF; Wed, 10 Mar 2021 12:47:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D0B1B8D01ED; Wed, 10 Mar 2021 12:47:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAC7F8D01EF; Wed, 10 Mar 2021 12:47:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0205.hostedemail.com [216.40.44.205]) by kanga.kvack.org (Postfix) with ESMTP id 9A3008D01ED for ; Wed, 10 Mar 2021 12:47:30 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 5EFA4181AF5C2 for ; Wed, 10 Mar 2021 17:47:30 +0000 (UTC) X-FDA: 77904696660.21.614C62E Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by imf01.hostedemail.com (Postfix) with ESMTP id 52E342000380 for ; Wed, 10 Mar 2021 17:46:30 +0000 (UTC) Received: by mail-pg1-f178.google.com with SMTP id w34so10807893pga.8 for ; Wed, 10 Mar 2021 09:46:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Py/EOd4SfsVAE1wGGKDRwg9/efRB8yrzYfbXqq9sd00=; b=Xpwz5NuPcHO0UONgccK0lxYzdukCMeF0Re0h4f0uNRyBewcR9stOMZVAtRxz61w3Z5 BATO2TMLuqZ8FLGXpMgv9hV89IVkjrs5pMV2bbdKKA4sMJGez7zOY8rg+Usr+I0NDL7a WNfeuaIsyxwsfVOlfaEWD+fI+QZql3ZG6F4QZ6miI/TXrDnK2bFokzxTdh9jXJ0apslx w6WFNjigZFUzJS8ruS63XTqLSeg2C+6pPQUo7MRNxYa9YCLuyRUHGq7oa/5w3/jqnldZ LOr7MlAofEyyLVfsWbZHhlhY2cx6mT2acQeHanKOn215EMFA2SbUfZMjWJv5ZdJUIY9w ChhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Py/EOd4SfsVAE1wGGKDRwg9/efRB8yrzYfbXqq9sd00=; b=esEVcVXbOSRVqzi+ZkjweSbSsNCabn+Wp00ER8r4kU5QXSld3C2wn+G4L6tffa8L+0 yVx68Zgb5ty1gz5/UU21+M3IHLAErvpfacRpissaq2+r8esqep+cvBfCeRLlAGckf0gp tnomvXyw3e0GIohhzIV5turYFAmBwXUn05g3cPjIDSRlkpJezESol60U+/aeYkXgJE42 gqEhSs5xydjIZPwYlmIc+vZkY/HdL6pa1QzDi41dCk448iAq+6nE1MfY4eVVDbC0u8QN 6oYmoTdFO5Cu4BmayeNvmQYww9C5zUPM/Y/43HHw91U/B6It0j1LpMOBSYRfagJanUmz jsuQ== X-Gm-Message-State: AOAM532ynh+WlpUo402DehyY9dwGV52HxBIQhMHGrlwQw5uVJRiLh1V+ mw/K1kAWKbZYgCCbI8+mKow= X-Google-Smtp-Source: ABdhPJy2ZsOHvx2Hmu5HfY7PtlhmBiqC9ADbXQxKs8wOp8se+DZgPiNAvxMx83CNwYNWVOw6WhzXtA== X-Received: by 2002:aa7:83cf:0:b029:1ee:f550:7d48 with SMTP id j15-20020aa783cf0000b02901eef5507d48mr4054966pfn.12.1615398389031; Wed, 10 Mar 2021 09:46:29 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d6sm145804pfq.109.2021.03.10.09.46.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Mar 2021 09:46:28 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v9 PATCH 07/13] mm: vmscan: add shrinker_info_protected() helper Date: Wed, 10 Mar 2021 09:45:57 -0800 Message-Id: <20210310174603.5093-8-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210310174603.5093-1-shy828301@gmail.com> References: <20210310174603.5093-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: tugpsumr6djud97qs6p5nzp1i7c8ri79 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 52E342000380 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf01; identity=mailfrom; envelope-from=""; helo=mail-pg1-f178.google.com; client-ip=209.85.215.178 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615398390-327856 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shrinker_info is dereferenced in a couple of places via rcu_dereference_protected with different calling conventions, for example, using mem_cgroup_nodeinfo helper or dereferencing memcg->nodeinfo[nid]->shrinker_info. And the later patch will add more dereference places. So extract the dereference into a helper to make the code more readable. No functional change. Acked-by: Roman Gushchin Acked-by: Kirill Tkhai Acked-by: Vlastimil Babka Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 7f3c00e76fd1..c0d04f242917 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -192,6 +192,13 @@ static inline int shrinker_map_size(int nr_items) return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); } +static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, + int nid) +{ + return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + lockdep_is_held(&shrinker_rwsem)); +} + static int expand_one_shrinker_info(struct mem_cgroup *memcg, int size, int old_size) { @@ -199,8 +206,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, int nid; for_each_node(nid) { - old = rcu_dereference_protected( - mem_cgroup_nodeinfo(memcg, nid)->shrinker_info, true); + old = shrinker_info_protected(memcg, nid); /* Not yet online memcg */ if (!old) return 0; @@ -231,7 +237,7 @@ void free_shrinker_info(struct mem_cgroup *memcg) for_each_node(nid) { pn = mem_cgroup_nodeinfo(memcg, nid); - info = rcu_dereference_protected(pn->shrinker_info, true); + info = shrinker_info_protected(memcg, nid); kvfree(info); rcu_assign_pointer(pn->shrinker_info, NULL); } @@ -674,8 +680,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (!down_read_trylock(&shrinker_rwsem)) return 0; - info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - true); + info = shrinker_info_protected(memcg, nid); if (unlikely(!info)) goto unlock; From patchwork Wed Mar 10 17:45:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12128823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA308C433DB for ; Wed, 10 Mar 2021 17:46:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 808D764F1D for ; Wed, 10 Mar 2021 17:46:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 808D764F1D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 239AC8D01E7; Wed, 10 Mar 2021 12:46:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 20FF08D01D5; Wed, 10 Mar 2021 12:46:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FFEB8D01E7; Wed, 10 Mar 2021 12:46:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0034.hostedemail.com [216.40.44.34]) by kanga.kvack.org (Postfix) with ESMTP id EB26E8D01D5 for ; Wed, 10 Mar 2021 12:46:32 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 9B8728249980 for ; Wed, 10 Mar 2021 17:46:32 +0000 (UTC) X-FDA: 77904694224.24.1C46E83 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf03.hostedemail.com (Postfix) with ESMTP id D53B5C001C5F for ; Wed, 10 Mar 2021 17:46:29 +0000 (UTC) Received: by mail-pl1-f177.google.com with SMTP id a24so8826512plm.11 for ; Wed, 10 Mar 2021 09:46:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9fzXcQw4lBeUc4+bt/8wnQKsoR0U89o0QoJXEyYgW5c=; b=FB2Yz9RmCu+ccD2Dwwe/AYp1avMlu8LfTcD9I/7Gkv1HQtjz9zXwn8q9snSDKkBaOR Zmt5nDGnCdt+LLNUHbIjOmgX2dSrfPgvj0KqfBCk/9GvlYT980VQrjV8xpu7fgpMyK6f nXvGEhzI6pwyU1kepjOtJ1wIYlzLxogcub7t4cOLohSg4FQfhhnf1WSnMAhk05Q0IqAl p4SekSkSINOsiM7LvObXt/kMivy57B5UvPhushop8U+a7PlgNjOX5FrrZfmHAdl0SHYr o91epyagM874nS7cOdKfFA9NGDRNS6pTzCNG0mbvAvPOBMgW+L0nuFK97z9Z41i6MKK1 PacA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9fzXcQw4lBeUc4+bt/8wnQKsoR0U89o0QoJXEyYgW5c=; b=ftQNt9ceJncbLAVPa2cuBu4h/LrluR+9yaKV66Ba48afG+5FFs84D68DqKJCZz1WJZ cnZCeKaQbWV+AY6DIUuFYSnl+X5ZDLN+wsqSWhmbV+YGYURM3kDWIXEUPkOKGGMl4WTU CxoA63XjHJEmagiruTBxGTKQlzLmtWB4pBSFc44fkODTG9GpRsFrhQ155YniK35Bts5X 5kKAkk/5DFD0gk32YDjkPq5AyaDIyUAkj21ClT2MI3L9NFTFGHIT6TBYHpEA7scx6tf1 PxdB+KYP0Fyxh0fHqVis84xrjcirW0ip5xw+J9Hf2WE3g8QhbUSkOBkPPDMemhLde+ty 4ihA== X-Gm-Message-State: AOAM530/30cVQfzv8tHlBkgGwZiRcxvxbsUQu25SV9jWpNalyUJrk7+z hYeyK00qZEiNdDeHyaBGyq0= X-Google-Smtp-Source: ABdhPJz8789kCVyakl8XlWuYd//XMEoDUSopV1h4rWINLev2b+o9UbsIsldLfTnNXgAxWJQtsoMRow== X-Received: by 2002:a17:90a:bf04:: with SMTP id c4mr4610626pjs.170.1615398391251; Wed, 10 Mar 2021 09:46:31 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d6sm145804pfq.109.2021.03.10.09.46.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Mar 2021 09:46:30 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v9 PATCH 08/13] mm: vmscan: use a new flag to indicate shrinker is registered Date: Wed, 10 Mar 2021 09:45:58 -0800 Message-Id: <20210310174603.5093-9-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210310174603.5093-1-shy828301@gmail.com> References: <20210310174603.5093-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 7s3s6w5cxfpcmbottg8ngqxianyczg75 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: D53B5C001C5F Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf03; identity=mailfrom; envelope-from=""; helo=mail-pl1-f177.google.com; client-ip=209.85.214.177 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615398389-15619 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently registered shrinker is indicated by non-NULL shrinker->nr_deferred. This approach is fine with nr_deferred at the shrinker level, but the following patches will move MEMCG_AWARE shrinkers' nr_deferred to memcg level, so their shrinker->nr_deferred would always be NULL. This would prevent the shrinkers from unregistering correctly. Remove SHRINKER_REGISTERING since we could check if shrinker is registered successfully by the new flag. Acked-by: Kirill Tkhai Acked-by: Vlastimil Babka Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/shrinker.h | 7 ++++--- mm/vmscan.c | 40 +++++++++++++++------------------------- 2 files changed, 19 insertions(+), 28 deletions(-) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 0f80123650e2..1eac79ce57d4 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -79,13 +79,14 @@ struct shrinker { #define DEFAULT_SEEKS 2 /* A good number if you don't know better. */ /* Flags */ -#define SHRINKER_NUMA_AWARE (1 << 0) -#define SHRINKER_MEMCG_AWARE (1 << 1) +#define SHRINKER_REGISTERED (1 << 0) +#define SHRINKER_NUMA_AWARE (1 << 1) +#define SHRINKER_MEMCG_AWARE (1 << 2) /* * It just makes sense when the shrinker is also MEMCG_AWARE for now, * non-MEMCG_AWARE shrinker should not have this flag set. */ -#define SHRINKER_NONSLAB (1 << 2) +#define SHRINKER_NONSLAB (1 << 3) extern int prealloc_shrinker(struct shrinker *shrinker); extern void register_shrinker_prepared(struct shrinker *shrinker); diff --git a/mm/vmscan.c b/mm/vmscan.c index c0d04f242917..d0876970601e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -314,19 +314,6 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) } } -/* - * We allow subsystems to populate their shrinker-related - * LRU lists before register_shrinker_prepared() is called - * for the shrinker, since we don't want to impose - * restrictions on their internal registration order. - * In this case shrink_slab_memcg() may find corresponding - * bit is set in the shrinkers map. - * - * This value is used by the function to detect registering - * shrinkers and to skip do_shrink_slab() calls for them. - */ -#define SHRINKER_REGISTERING ((struct shrinker *)~0UL) - static DEFINE_IDR(shrinker_idr); static int prealloc_memcg_shrinker(struct shrinker *shrinker) @@ -335,7 +322,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) down_write(&shrinker_rwsem); /* This may call shrinker, so it must use down_read_trylock() */ - id = idr_alloc(&shrinker_idr, SHRINKER_REGISTERING, 0, 0, GFP_KERNEL); + id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -358,9 +345,9 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) BUG_ON(id < 0); - down_write(&shrinker_rwsem); + lockdep_assert_held(&shrinker_rwsem); + idr_remove(&shrinker_idr, id); - up_write(&shrinker_rwsem); } static bool cgroup_reclaim(struct scan_control *sc) @@ -488,8 +475,11 @@ void free_prealloced_shrinker(struct shrinker *shrinker) if (!shrinker->nr_deferred) return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) + if (shrinker->flags & SHRINKER_MEMCG_AWARE) { + down_write(&shrinker_rwsem); unregister_memcg_shrinker(shrinker); + up_write(&shrinker_rwsem); + } kfree(shrinker->nr_deferred); shrinker->nr_deferred = NULL; @@ -499,10 +489,7 @@ void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); list_add_tail(&shrinker->list, &shrinker_list); -#ifdef CONFIG_MEMCG - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - idr_replace(&shrinker_idr, shrinker, shrinker->id); -#endif + shrinker->flags |= SHRINKER_REGISTERED; up_write(&shrinker_rwsem); } @@ -522,13 +509,16 @@ EXPORT_SYMBOL(register_shrinker); */ void unregister_shrinker(struct shrinker *shrinker) { - if (!shrinker->nr_deferred) + if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - unregister_memcg_shrinker(shrinker); + down_write(&shrinker_rwsem); list_del(&shrinker->list); + shrinker->flags &= ~SHRINKER_REGISTERED; + if (shrinker->flags & SHRINKER_MEMCG_AWARE) + unregister_memcg_shrinker(shrinker); up_write(&shrinker_rwsem); + kfree(shrinker->nr_deferred); shrinker->nr_deferred = NULL; } @@ -693,7 +683,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct shrinker *shrinker; shrinker = idr_find(&shrinker_idr, i); - if (unlikely(!shrinker || shrinker == SHRINKER_REGISTERING)) { + if (unlikely(!shrinker || !(shrinker->flags & SHRINKER_REGISTERED))) { if (!shrinker) clear_bit(i, info->map); continue; From patchwork Wed Mar 10 17:45:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12128825 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31BD2C433E6 for ; Wed, 10 Mar 2021 17:46:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C5C5E64F1D for ; Wed, 10 Mar 2021 17:46:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C5C5E64F1D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 56DAD8D01E8; Wed, 10 Mar 2021 12:46:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D1EB8D01D5; Wed, 10 Mar 2021 12:46:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D4398D01E8; Wed, 10 Mar 2021 12:46:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0114.hostedemail.com [216.40.44.114]) by kanga.kvack.org (Postfix) with ESMTP id 148AE8D01D5 for ; Wed, 10 Mar 2021 12:46:35 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id C3639181AF5C2 for ; Wed, 10 Mar 2021 17:46:34 +0000 (UTC) X-FDA: 77904694308.12.1CC6DFE Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf11.hostedemail.com (Postfix) with ESMTP id 18EC220053C8 for ; Wed, 10 Mar 2021 17:46:28 +0000 (UTC) Received: by mail-pl1-f178.google.com with SMTP id d8so8828100plg.10 for ; Wed, 10 Mar 2021 09:46:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ehi3hOTq936nu/4xzqEoUNb+p2/SBTsgXG0kobshYf4=; b=DkxAKwp8pjrUF4XXxpugAYfT0pOQlbBLmQm0LwEcFScxBF6c7XqO15AWjDXAYwyvtl vYoUg3g1HveTrNzIEu5oTxjQnrwLUVXcGty1DImaOYqBAUOl2VHR3ngCC+N+SRqkXI67 QhzbKkHO3MM8zSRb3QbbCHIKRJk3xiZszssy6to7ZIlr/Cn08RIJxCQZLfvqP8nrMnxd OUzYxdTkjwOj0RYkJJIX86rqwru8o7UsHAHiOdDd8d3PVlYX1smD6yiIsgZ5hnQK8lsk 4706G3Jua+Gzi4AskzElyX/HTY2onxk0MK0F4YqPFGMwypO5Rg7DBrCrINFTSLeLthDV McIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ehi3hOTq936nu/4xzqEoUNb+p2/SBTsgXG0kobshYf4=; b=f9cWtnfGmeDPTG7mrreb2x4PuK9KQFoZn4HhtMLY5JOZiuPLRJjQfW+6ewXiKc9gS5 g5NuTbGpLtXgoHFlYNywiZRpmBo+pWWXmY8PnmYC61ISJKgbrAur9kRhGFXsToin2LJ1 2J7rI3mwNO8SUOzrguJN8nmfFo+9Fqucnoc81X5EnGREwM/1Ki9dROeo7AbWVTs0TpAt NdBOtmL1AGbYtV3QWejKdmHXoEbHcX2240jwNm8el2E8tI3DQKPWM8NXsOvy2pnpgYVt y8a8TRNXeW8Z2C/ZUWox1OV+g6GnLCIZ6hLT7O0YaSal7qBdd6RC/XAUa5WB6/Gz5QmJ dJQA== X-Gm-Message-State: AOAM533bTYQhROFrsRvGlpc62Rgitdyq5yt0Z+4CweT2a76FjJS1cD2a tZl6sYSQ2EkrkAs0KqXuwDA= X-Google-Smtp-Source: ABdhPJywFd3ZIqnBOJV9QyFTNUlMg2/rEy0nzP+FtWd7b4R8iOG2b7nOhV0tN9AgCJ8Y5h6q2QPDdA== X-Received: by 2002:a17:90a:d90a:: with SMTP id c10mr4702892pjv.13.1615398393441; Wed, 10 Mar 2021 09:46:33 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d6sm145804pfq.109.2021.03.10.09.46.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Mar 2021 09:46:32 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v9 PATCH 09/13] mm: vmscan: add per memcg shrinker nr_deferred Date: Wed, 10 Mar 2021 09:45:59 -0800 Message-Id: <20210310174603.5093-10-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210310174603.5093-1-shy828301@gmail.com> References: <20210310174603.5093-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: o5tr78zunxy94bk8pw1whdkgyrna65i7 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 18EC220053C8 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf11; identity=mailfrom; envelope-from=""; helo=mail-pl1-f178.google.com; client-ip=209.85.214.178 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615398388-269276 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently the number of deferred objects are per shrinker, but some slabs, for example, vfs inode/dentry cache are per memcg, this would result in poor isolation among memcgs. The deferred objects typically are generated by __GFP_NOFS allocations, one memcg with excessive __GFP_NOFS allocations may blow up deferred objects, then other innocent memcgs may suffer from over shrink, excessive reclaim latency, etc. For example, two workloads run in memcgA and memcgB respectively, workload in B is vfs heavy workload. Workload in A generates excessive deferred objects, then B's vfs cache might be hit heavily (drop half of caches) by B's limit reclaim or global reclaim. We observed this hit in our production environment which was running vfs heavy workload shown as the below tracing log: <...>-409454 [016] .... 28286961.747146: mm_shrink_slab_start: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 1 objects to shrink 3641681686040 gfp_flags GFP_HIGHUSER_MOVABLE|__GFP_ZERO pgs_scanned 1 lru_pgs 15721 cache items 246404277 delta 31345 total_scan 123202138 <...>-409454 [022] .... 28287105.928018: mm_shrink_slab_end: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 1 unused scan count 3641681686040 new scan count 3641798379189 total_scan 602 last shrinker return val 123186855 The vfs cache and page cache ratio was 10:1 on this machine, and half of caches were dropped. This also resulted in significant amount of page caches were dropped due to inodes eviction. Make nr_deferred per memcg for memcg aware shrinkers would solve the unfairness and bring better isolation. The following patch will add nr_deferred to parent memcg when memcg offline. To preserve nr_deferred when reparenting memcgs to root, root memcg needs shrinker_info allocated too. When memcg is not enabled (!CONFIG_MEMCG or memcg disabled), the shrinker's nr_deferred would be used. And non memcg aware shrinkers use shrinker's nr_deferred all the time. Acked-by: Roman Gushchin Acked-by: Kirill Tkhai Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 7 +++-- mm/vmscan.c | 60 ++++++++++++++++++++++++++------------ 2 files changed, 46 insertions(+), 21 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 308a202f1de2..42a4facb5b7c 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -97,12 +97,13 @@ struct batched_lruvec_stat { }; /* - * Bitmap of shrinker::id corresponding to memcg-aware shrinkers, - * which have elements charged to this memcg. + * Bitmap and deferred work of shrinker::id corresponding to memcg-aware + * shrinkers, which have elements charged to this memcg. */ struct shrinker_info { struct rcu_head rcu; - unsigned long map[]; + atomic_long_t *nr_deferred; + unsigned long *map; }; /* diff --git a/mm/vmscan.c b/mm/vmscan.c index d0876970601e..ae82afe6cec6 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -187,11 +187,17 @@ static DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG static int shrinker_nr_max; +/* The shrinker_info is expanded in a batch of BITS_PER_LONG */ static inline int shrinker_map_size(int nr_items) { return (DIV_ROUND_UP(nr_items, BITS_PER_LONG) * sizeof(unsigned long)); } +static inline int shrinker_defer_size(int nr_items) +{ + return (round_up(nr_items, BITS_PER_LONG) * sizeof(atomic_long_t)); +} + static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { @@ -200,10 +206,12 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, } static int expand_one_shrinker_info(struct mem_cgroup *memcg, - int size, int old_size) + int map_size, int defer_size, + int old_map_size, int old_defer_size) { struct shrinker_info *new, *old; int nid; + int size = map_size + defer_size; for_each_node(nid) { old = shrinker_info_protected(memcg, nid); @@ -215,9 +223,16 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!new) return -ENOMEM; - /* Set all old bits, clear all new bits */ - memset(new->map, (int)0xff, old_size); - memset((void *)new->map + old_size, 0, size - old_size); + new->nr_deferred = (atomic_long_t *)(new + 1); + new->map = (void *)new->nr_deferred + defer_size; + + /* map: set all old bits, clear all new bits */ + memset(new->map, (int)0xff, old_map_size); + memset((void *)new->map + old_map_size, 0, map_size - old_map_size); + /* nr_deferred: copy old values, clear all new values */ + memcpy(new->nr_deferred, old->nr_deferred, old_defer_size); + memset((void *)new->nr_deferred + old_defer_size, 0, + defer_size - old_defer_size); rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, new); kvfree_rcu(old, rcu); @@ -232,9 +247,6 @@ void free_shrinker_info(struct mem_cgroup *memcg) struct shrinker_info *info; int nid; - if (mem_cgroup_is_root(memcg)) - return; - for_each_node(nid) { pn = mem_cgroup_nodeinfo(memcg, nid); info = shrinker_info_protected(memcg, nid); @@ -247,12 +259,12 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) { struct shrinker_info *info; int nid, size, ret = 0; - - if (mem_cgroup_is_root(memcg)) - return 0; + int map_size, defer_size = 0; down_write(&shrinker_rwsem); - size = shrinker_map_size(shrinker_nr_max); + map_size = shrinker_map_size(shrinker_nr_max); + defer_size = shrinker_defer_size(shrinker_nr_max); + size = map_size + defer_size; for_each_node(nid) { info = kvzalloc_node(sizeof(*info) + size, GFP_KERNEL, nid); if (!info) { @@ -260,6 +272,8 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) ret = -ENOMEM; break; } + info->nr_deferred = (atomic_long_t *)(info + 1); + info->map = (void *)info->nr_deferred + defer_size; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -267,15 +281,21 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } +static inline bool need_expand(int nr_max) +{ + return round_up(nr_max, BITS_PER_LONG) > + round_up(shrinker_nr_max, BITS_PER_LONG); +} + static int expand_shrinker_info(int new_id) { - int size, old_size, ret = 0; + int ret = 0; int new_nr_max = new_id + 1; + int map_size, defer_size = 0; + int old_map_size, old_defer_size = 0; struct mem_cgroup *memcg; - size = shrinker_map_size(new_nr_max); - old_size = shrinker_map_size(shrinker_nr_max); - if (size <= old_size) + if (!need_expand(new_nr_max)) goto out; if (!root_mem_cgroup) @@ -283,11 +303,15 @@ static int expand_shrinker_info(int new_id) lockdep_assert_held(&shrinker_rwsem); + map_size = shrinker_map_size(new_nr_max); + defer_size = shrinker_defer_size(new_nr_max); + old_map_size = shrinker_map_size(shrinker_nr_max); + old_defer_size = shrinker_defer_size(shrinker_nr_max); + memcg = mem_cgroup_iter(NULL, NULL, NULL); do { - if (mem_cgroup_is_root(memcg)) - continue; - ret = expand_one_shrinker_info(memcg, size, old_size); + ret = expand_one_shrinker_info(memcg, map_size, defer_size, + old_map_size, old_defer_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; From patchwork Wed Mar 10 17:46:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12128827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10236C433E0 for ; Wed, 10 Mar 2021 17:46:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ACED964F45 for ; Wed, 10 Mar 2021 17:46:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ACED964F45 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2118F8D01E9; Wed, 10 Mar 2021 12:46:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1EBC98D01D5; Wed, 10 Mar 2021 12:46:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 016BC8D01E9; Wed, 10 Mar 2021 12:46:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0228.hostedemail.com [216.40.44.228]) by kanga.kvack.org (Postfix) with ESMTP id DADFB8D01D5 for ; Wed, 10 Mar 2021 12:46:36 -0500 (EST) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 957728249980 for ; Wed, 10 Mar 2021 17:46:36 +0000 (UTC) X-FDA: 77904694392.10.82964E9 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by imf28.hostedemail.com (Postfix) with ESMTP id 2D74A20053DC for ; Wed, 10 Mar 2021 17:46:37 +0000 (UTC) Received: by mail-pj1-f45.google.com with SMTP id lr10-20020a17090b4b8ab02900dd61b95c5eso5384249pjb.4 for ; Wed, 10 Mar 2021 09:46:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AJz0Wb5sPGubosZGWYITbWJb9prsfgWGS78iJTHia7k=; b=hwmGVy7R7pl3/e+v47GWlbKqHBOLmJ94Qrk62MCC030HH4KX6PRx0iAmUE1p8pDzNi F/LSOLjE0cet/nr83buTzNrZpM0dpyEyr/hVUGQIvBIsIxsyHtWURh0uCjYjU9ENb4Db dPOL3E+vRSg5NRdseOw6yYlVj952heRYvZEcYigrIWTi8WNFatQt9NKijhRkVKb1AUtC 97HCsLisryN80Ldfxr/nv+TBd4VNWwtltn8jtn+gxcTl5OgJxM2DJsgKC+AMxOPy2VQ5 Oequx25raFoyykWstClKOop08hSLpFylGK/uB6UEO7khh5mvsikyMfrVyAqtDIqTaMcI LBqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AJz0Wb5sPGubosZGWYITbWJb9prsfgWGS78iJTHia7k=; b=B3c6gVgiW9boWjOydwmJZvSPu+hyiPVTYAB83wJGBG7xLxNWH06Xmhl3A8qppUcJ8+ C51KPNy49siB4HdFPAAnjzQcr1kCw1VNDstu1l/01OGeAWRprqFLYWarb2QP4JZNiYwm GE3MrZlTYbB07kLQ5e06MgbdnI4omAwsePtQHXAY3lIB/e2haQUeQewd88cjVOgniDmn fIr+hcIIT1w0glObY9PFDyC3R41QDwrVnoNA4DPCmCdFvXQd6YZbI5IpuCVb2jCICa3S HgX7ifU988f0t5M2vh7AuftdS/YpOuc0ho7ETbQOaVAbbRioUd/7pr1qvqi8r83E1Zue o0sw== X-Gm-Message-State: AOAM531aYbcAeMGcFECPA1sqLMHxbjDIRR4VKa8DwS8xqxh92Z8PHdPW 9nr5QbV61zNT5F0pcqK9tas= X-Google-Smtp-Source: ABdhPJxd01rtfsohsp6B/Mcet9kZtJ3KN9JVwchG8OO5e7MCaQ8wJTNHPWJpN2zoiIerX1hfoH8USQ== X-Received: by 2002:a17:90b:609:: with SMTP id gb9mr4568728pjb.209.1615398395320; Wed, 10 Mar 2021 09:46:35 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d6sm145804pfq.109.2021.03.10.09.46.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Mar 2021 09:46:34 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v9 PATCH 10/13] mm: vmscan: use per memcg nr_deferred of shrinker Date: Wed, 10 Mar 2021 09:46:00 -0800 Message-Id: <20210310174603.5093-11-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210310174603.5093-1-shy828301@gmail.com> References: <20210310174603.5093-1-shy828301@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2D74A20053DC X-Stat-Signature: nxb619b7utpmk4zgpkcu99gjygonpxy8 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf28; identity=mailfrom; envelope-from=""; helo=mail-pj1-f45.google.com; client-ip=209.85.216.45 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615398397-193209 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use per memcg's nr_deferred for memcg aware shrinkers. The shrinker's nr_deferred will be used in the following cases: 1. Non memcg aware shrinkers 2. !CONFIG_MEMCG 3. memcg is disabled by boot parameter Acked-by: Roman Gushchin Acked-by: Kirill Tkhai Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 78 ++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 66 insertions(+), 12 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index ae82afe6cec6..326f0e0c4356 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -374,6 +374,24 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) idr_remove(&shrinker_idr, id); } +static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + + info = shrinker_info_protected(memcg, nid); + return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); +} + +static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + struct shrinker_info *info; + + info = shrinker_info_protected(memcg, nid); + return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); +} + static bool cgroup_reclaim(struct scan_control *sc) { return sc->target_mem_cgroup; @@ -412,6 +430,18 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) { } +static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + return 0; +} + +static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, + struct mem_cgroup *memcg) +{ + return 0; +} + static bool cgroup_reclaim(struct scan_control *sc) { return false; @@ -423,6 +453,39 @@ static bool writeback_throttling_sane(struct scan_control *sc) } #endif +static long xchg_nr_deferred(struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (sc->memcg && + (shrinker->flags & SHRINKER_MEMCG_AWARE)) + return xchg_nr_deferred_memcg(nid, shrinker, + sc->memcg); + + return atomic_long_xchg(&shrinker->nr_deferred[nid], 0); +} + + +static long add_nr_deferred(long nr, struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (sc->memcg && + (shrinker->flags & SHRINKER_MEMCG_AWARE)) + return add_nr_deferred_memcg(nr, nid, shrinker, + sc->memcg); + + return atomic_long_add_return(nr, &shrinker->nr_deferred[nid]); +} + /* * This misses isolated pages which are not accounted for to save counters. * As the data only determines if reclaim or compaction continues, it is @@ -559,14 +622,10 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, long freeable; long nr; long new_nr; - int nid = shrinkctl->nid; long batch_size = shrinker->batch ? shrinker->batch : SHRINK_BATCH; long scanned = 0, next_deferred; - if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) - nid = 0; - freeable = shrinker->count_objects(shrinker, shrinkctl); if (freeable == 0 || freeable == SHRINK_EMPTY) return freeable; @@ -576,7 +635,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, * and zero it so that other concurrent shrinker invocations * don't also do this scanning work. */ - nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0); + nr = xchg_nr_deferred(shrinker, shrinkctl); total_scan = nr; if (shrinker->seeks) { @@ -667,14 +726,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, next_deferred = 0; /* * move the unused scan count back into the shrinker in a - * manner that handles concurrent updates. If we exhausted the - * scan, there is no need to do an update. + * manner that handles concurrent updates. */ - if (next_deferred > 0) - new_nr = atomic_long_add_return(next_deferred, - &shrinker->nr_deferred[nid]); - else - new_nr = atomic_long_read(&shrinker->nr_deferred[nid]); + new_nr = add_nr_deferred(next_deferred, shrinker, shrinkctl); trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); return freed; From patchwork Wed Mar 10 17:46:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12128829 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB504C433DB for ; Wed, 10 Mar 2021 17:46:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6171C64F1D for ; Wed, 10 Mar 2021 17:46:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6171C64F1D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BFD758D01EA; Wed, 10 Mar 2021 12:46:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BD57A8D01D5; Wed, 10 Mar 2021 12:46:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93E4C8D01EA; Wed, 10 Mar 2021 12:46:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0188.hostedemail.com [216.40.44.188]) by kanga.kvack.org (Postfix) with ESMTP id 6FE858D01D5 for ; Wed, 10 Mar 2021 12:46:39 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 2B34218140B80 for ; Wed, 10 Mar 2021 17:46:39 +0000 (UTC) X-FDA: 77904694518.02.4ED7DA4 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf16.hostedemail.com (Postfix) with ESMTP id 7BF5B8019154 for ; Wed, 10 Mar 2021 17:46:38 +0000 (UTC) Received: by mail-pg1-f177.google.com with SMTP id e6so11877129pgk.5 for ; Wed, 10 Mar 2021 09:46:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FaxJJOVv7LgdCo9yE7MuyRBn9qNHRZNOLyYzzoINruU=; b=c/DvCf9Nn8eVdeoELtlskBSsceIxBGXO+pGtMzYQWwDbGAtl4IBbN02T5i3OP/VsdG 9ICBeRVNqvOzLaUf9kj/zkRR1MwWIGq1b2vnhEwMwqjm9iikDM8eH5nCHNHOpnpDbiHC KDqyAvntuiCVKY10oIjRqz0QJH56nZd+oIZ+xJk93ySIBWQ+TiP4icuGCGHpEmbTS2U0 R1WZVYs5c3cbmzAm2fDvz53zAxv/Kx/hMt62h3VtrTsfFONIOg+eg73mRcFALzWY9BKf Axti/UDZq8bIreJS7rNwhmq80BgMaS1aYmhDYmPUNyk1/wxY3BdD9N5s8mr03Sh9kDNm 245Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FaxJJOVv7LgdCo9yE7MuyRBn9qNHRZNOLyYzzoINruU=; b=L0aCDrGnruUyHgylIkFF1ZEtixj+1l7YwEjJ1HAQs8Kw/8uXG290yOxvtq1KpNgUMX Kkv4no8qGcupk/oUle0BWGpQ/LS8vN2oDRKb6y+SwM3I0JcKTLn590AEUOQCmwR8B9wq u5MkQ6L6F+03i2oJHw808qPTcxGu6t4OcNDYJw9AHnQAp7TV9dHdeAknJFC0EvRoansp sKEbL3iPIU+snOVw01h08jXNTePx8Eb/CFvCvbNbVPsBo/TRlwLbl9ORZMpmh9ONThYw ni+XCS3AC71PNwdDSD+oU5zuXtRjE9iMU6WT8tcSvqF7L3ELeMS5IsOrbSkeYPzQInAb 2njg== X-Gm-Message-State: AOAM533z5MtZI46ul7hAKeFxD6kdJcMNGHaMfuko20jGIv4KxPbsgsIs 77/YRbfhJFIdBsJLSsb0kos= X-Google-Smtp-Source: ABdhPJzcqpIQDE7VuURNT6V1tgUthG66dbUo1skbXfxFFImty1y9kA3R8cyFlnyHF2RAg89YmpUSOw== X-Received: by 2002:a05:6a00:8d2:b029:1f1:5b57:85ae with SMTP id s18-20020a056a0008d2b02901f15b5785aemr3840622pfu.60.1615398397512; Wed, 10 Mar 2021 09:46:37 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d6sm145804pfq.109.2021.03.10.09.46.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Mar 2021 09:46:36 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v9 PATCH 11/13] mm: vmscan: don't need allocate shrinker->nr_deferred for memcg aware shrinkers Date: Wed, 10 Mar 2021 09:46:01 -0800 Message-Id: <20210310174603.5093-12-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210310174603.5093-1-shy828301@gmail.com> References: <20210310174603.5093-1-shy828301@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7BF5B8019154 X-Stat-Signature: j5zbrewkxuhf5yx1xw7uarj1xuapq77w Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=mail-pg1-f177.google.com; client-ip=209.85.215.177 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615398398-105793 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now nr_deferred is available on per memcg level for memcg aware shrinkers, so don't need allocate shrinker->nr_deferred for such shrinkers anymore. The prealloc_memcg_shrinker() would return -ENOSYS if !CONFIG_MEMCG or memcg is disabled by kernel command line, then shrinker's SHRINKER_MEMCG_AWARE flag would be cleared. This makes the implementation of this patch simpler. Acked-by: Vlastimil Babka Reviewed-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- mm/vmscan.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 326f0e0c4356..cf25c78661d1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -344,6 +344,9 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) { int id, ret = -ENOMEM; + if (mem_cgroup_disabled()) + return -ENOSYS; + down_write(&shrinker_rwsem); /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); @@ -423,7 +426,7 @@ static bool writeback_throttling_sane(struct scan_control *sc) #else static int prealloc_memcg_shrinker(struct shrinker *shrinker) { - return 0; + return -ENOSYS; } static void unregister_memcg_shrinker(struct shrinker *shrinker) @@ -535,8 +538,18 @@ static unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru, */ int prealloc_shrinker(struct shrinker *shrinker) { - unsigned int size = sizeof(*shrinker->nr_deferred); + unsigned int size; + int err; + + if (shrinker->flags & SHRINKER_MEMCG_AWARE) { + err = prealloc_memcg_shrinker(shrinker); + if (err != -ENOSYS) + return err; + shrinker->flags &= ~SHRINKER_MEMCG_AWARE; + } + + size = sizeof(*shrinker->nr_deferred); if (shrinker->flags & SHRINKER_NUMA_AWARE) size *= nr_node_ids; @@ -544,28 +557,16 @@ int prealloc_shrinker(struct shrinker *shrinker) if (!shrinker->nr_deferred) return -ENOMEM; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - if (prealloc_memcg_shrinker(shrinker)) - goto free_deferred; - } - return 0; - -free_deferred: - kfree(shrinker->nr_deferred); - shrinker->nr_deferred = NULL; - return -ENOMEM; } void free_prealloced_shrinker(struct shrinker *shrinker) { - if (!shrinker->nr_deferred) - return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) { down_write(&shrinker_rwsem); unregister_memcg_shrinker(shrinker); up_write(&shrinker_rwsem); + return; } kfree(shrinker->nr_deferred); From patchwork Wed Mar 10 17:46:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12128831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4900CC433E0 for ; Wed, 10 Mar 2021 17:46:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E934564F48 for ; Wed, 10 Mar 2021 17:46:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E934564F48 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9171A8D01EB; Wed, 10 Mar 2021 12:46:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A50B8D01D5; Wed, 10 Mar 2021 12:46:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F5F48D01EB; Wed, 10 Mar 2021 12:46:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0246.hostedemail.com [216.40.44.246]) by kanga.kvack.org (Postfix) with ESMTP id 5677D8D01D5 for ; Wed, 10 Mar 2021 12:46:41 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 10A57181AF5C2 for ; Wed, 10 Mar 2021 17:46:41 +0000 (UTC) X-FDA: 77904694602.01.44E77D0 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by imf12.hostedemail.com (Postfix) with ESMTP id 0D5EA3D6 for ; Wed, 10 Mar 2021 17:46:36 +0000 (UTC) Received: by mail-pf1-f175.google.com with SMTP id x7so9137193pfi.7 for ; Wed, 10 Mar 2021 09:46:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=G7y5azUFXSsyavdHIsaT9AMO5ymd4GoyVzQs6Ax3Mec=; b=s23IbToCEQiW2LfwWgwPLKu/K9irz9puJBxy4pj77zsIN1F4mhp6kFZ0fvAnr8bRW/ nix8GUrtk1m2rNPjffKA5w++RSNotRLG7AMG0xYRpHpPfBFWIUw4cF40Z2oBVIdBfUGA 868+iPgOzhmkwBGjSPjYxsi0LRNDzHWeu72Dqiv+ZVOhlsmhAMYW2MgSvsoUx0am3akz BBQWSKepzU2L5MzrdVv24FZ/ZWf60yQVGJVNFsVXol0aW3SU69apMVL+roJ8a/GFu7A6 rlkKCAnvXOe/sHkc1noaSpLzmzsINLfKBvsYZEASTt4GBr+owzhw8Aj4afFDFZALePnS Qatg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=G7y5azUFXSsyavdHIsaT9AMO5ymd4GoyVzQs6Ax3Mec=; b=I6qY0b5ZSq88mfA6sqwitPCTvSFmbInW7cILkyIm3BoiJhE2FndWSHSN5Otq7yixpd NarE1j/s21Losjbr/1FXXNm8biaU71jaIM/ATN6pxF2j5a744EWVdQCn/BwG9QjQkb92 WKf0jCplDJdtPxYp2eIgS14k2+DRPFs+EvLyDOM6wx2WPu4o0cJgnSj7aQhJJHEy756J MlK0uHztcfo0zmcQtHLbSG3C0Gbnc/aBXfRIuc5UkVwL9pZz/jlhCf6qOKgaspYJHMYH 61GdRSCC8WAe+oro+K2yHfEYBpeBIr9FXwIIvq9B/tjz1WQF6T1ZHaa1506bSyfRADCn PM+g== X-Gm-Message-State: AOAM533NLeasja5P4g2YiNnIflHBumMHSimnhbqgzx6DZnM24ihY0Cq/ +mwR+IliDRoe90Gxe33Fc4I= X-Google-Smtp-Source: ABdhPJzvRhb2rWON3l0wObFgGYmePKCSygAJAPGbZZV40nSzwpJc9kOk6TMgIw6aZvx6zbZDlOoUew== X-Received: by 2002:a63:2a16:: with SMTP id q22mr3743667pgq.211.1615398399711; Wed, 10 Mar 2021 09:46:39 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d6sm145804pfq.109.2021.03.10.09.46.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Mar 2021 09:46:38 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v9 PATCH 12/13] mm: memcontrol: reparent nr_deferred when memcg offline Date: Wed, 10 Mar 2021 09:46:02 -0800 Message-Id: <20210310174603.5093-13-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210310174603.5093-1-shy828301@gmail.com> References: <20210310174603.5093-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: n4q879u79q6w9icaz9hccwgexip9p4zw X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 0D5EA3D6 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf12; identity=mailfrom; envelope-from=""; helo=mail-pf1-f175.google.com; client-ip=209.85.210.175 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615398396-135994 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now shrinker's nr_deferred is per memcg for memcg aware shrinkers, add to parent's corresponding nr_deferred when memcg offline. Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 1 + mm/vmscan.c | 24 ++++++++++++++++++++++++ 3 files changed, 26 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 42a4facb5b7c..2c76fe53fb6d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1569,6 +1569,7 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) int alloc_shrinker_info(struct mem_cgroup *memcg); void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); +void reparent_shrinker_deferred(struct mem_cgroup *memcg); #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index edd8a06c751f..dacb1c6087ea 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5262,6 +5262,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) page_counter_set_low(&memcg->memory, 0); memcg_offline_kmem(memcg); + reparent_shrinker_deferred(memcg); wb_memcg_offline(memcg); drain_all_stock(memcg); diff --git a/mm/vmscan.c b/mm/vmscan.c index cf25c78661d1..9a2dfeaa79f4 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -395,6 +395,30 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } +void reparent_shrinker_deferred(struct mem_cgroup *memcg) +{ + int i, nid; + long nr; + struct mem_cgroup *parent; + struct shrinker_info *child_info, *parent_info; + + parent = parent_mem_cgroup(memcg); + if (!parent) + parent = root_mem_cgroup; + + /* Prevent from concurrent shrinker_info expand */ + down_read(&shrinker_rwsem); + for_each_node(nid) { + child_info = shrinker_info_protected(memcg, nid); + parent_info = shrinker_info_protected(parent, nid); + for (i = 0; i < shrinker_nr_max; i++) { + nr = atomic_long_read(&child_info->nr_deferred[i]); + atomic_long_add(nr, &parent_info->nr_deferred[i]); + } + } + up_read(&shrinker_rwsem); +} + static bool cgroup_reclaim(struct scan_control *sc) { return sc->target_mem_cgroup; From patchwork Wed Mar 10 17:46:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12128833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F16DCC433DB for ; Wed, 10 Mar 2021 17:46:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 96D9464F1D for ; Wed, 10 Mar 2021 17:46:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 96D9464F1D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 87B278D01EC; Wed, 10 Mar 2021 12:46:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 852C78D01D5; Wed, 10 Mar 2021 12:46:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F1D88D01EC; Wed, 10 Mar 2021 12:46:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0120.hostedemail.com [216.40.44.120]) by kanga.kvack.org (Postfix) with ESMTP id 525558D01D5 for ; Wed, 10 Mar 2021 12:46:43 -0500 (EST) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id F23F88151 for ; Wed, 10 Mar 2021 17:46:42 +0000 (UTC) X-FDA: 77904694644.13.048A446 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf20.hostedemail.com (Postfix) with ESMTP id C578F2385 for ; Wed, 10 Mar 2021 17:46:38 +0000 (UTC) Received: by mail-pl1-f179.google.com with SMTP id a24so8826754plm.11 for ; Wed, 10 Mar 2021 09:46:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sHD52Sa+hkPXZ82uTwxwMMAdUbtnhYs3jM+eAa3hQC4=; b=PMvgCUnE80nILtQ2zlCJoYAgUvQmTVJucA6+y4K4wxwO6YJrKobwbUSLVLeis7S1KP d6UAmZakJ6HVzaoCYaOUOiQ87cpXlRynpv8Bdx94Osz6Ftkzr6wCePVsT1sWHtlg9t+u iXaL0vCXp4wW9yJa4ABJ8J0FK+LgqGPjx3Z/Wa4iu0fa97r1AiOiMX+vt9tT3ltyErqS tORkHF48vI39D1syGD3K2TzbOOs6S52RzC2oIl8HKh5dum7GPMi623Bz2mD2aUUIGQ3Z 5Tiqo/hn89JdQ0CGpkcxgi9hm/yMlCRAPcoNLPrMQszIe6HMuu6cHctnB4murDsq3NOk MqEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sHD52Sa+hkPXZ82uTwxwMMAdUbtnhYs3jM+eAa3hQC4=; b=LJe6P4t/im0CHU2ZVMZkYy6eWKU2PLrGPfk/LwNp2ejjKQwpdlQ10AQB+EZzRZiOFO AukTiCs4/KII4kWNJoL4LpEsj9k5yNdJbEO4I4ZC+vaFQ60K8az4W4jK7JbwY7lgMbOt gdW6UhCwBqDEewIEBnrEXNOoRMwBcsqYNYPZC+GinY9Hu/d2wD2F6npVvMNLZIskX4NR TVbcDTki5fLk/yjTScNJmXyXJAz4nXPVied/3nsneh+PfQmnt5Y9jmZLb7Cr8yVZ1R59 9r6KVpoNV+tCqPSAaP8x30VG9H1atd5OEw7r41fFIi0AVMhrGASzcMg2R2JS86WT17cQ zlRQ== X-Gm-Message-State: AOAM530xzoVG/MHHzxroygqR5ICIih8uVMGuEnNoRglJLkyX/LUfWCck dpB0KiFdYO+JDCFnRVmyKsM= X-Google-Smtp-Source: ABdhPJy0/FEbV2UYtiznaebUra6fBNIUYPi6n9765r1hA2Y/cHKhUMldo7IUnb9SWEgqSkqvqxTrdQ== X-Received: by 2002:a17:90a:8908:: with SMTP id u8mr4549214pjn.135.1615398401563; Wed, 10 Mar 2021 09:46:41 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d6sm145804pfq.109.2021.03.10.09.46.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Mar 2021 09:46:41 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v9 PATCH 13/13] mm: vmscan: shrink deferred objects proportional to priority Date: Wed, 10 Mar 2021 09:46:03 -0800 Message-Id: <20210310174603.5093-14-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210310174603.5093-1-shy828301@gmail.com> References: <20210310174603.5093-1-shy828301@gmail.com> MIME-Version: 1.0 X-Stat-Signature: f65g7c45961uctzyus5bt3rg4fe851xn X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: C578F2385 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf20; identity=mailfrom; envelope-from=""; helo=mail-pl1-f179.google.com; client-ip=209.85.214.179 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615398398-427262 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The number of deferred objects might get windup to an absurd number, and it results in clamp of slab objects. It is undesirable for sustaining workingset. So shrink deferred objects proportional to priority and cap nr_deferred to twice of cache items. The idea is borrowed from Dave Chinner's patch: https://lore.kernel.org/linux-xfs/20191031234618.15403-13-david@fromorbit.com/ Tested with kernel build and vfs metadata heavy workload in our production environment, no regression is spotted so far. Signed-off-by: Yang Shi --- mm/vmscan.c | 46 +++++++++++----------------------------------- 1 file changed, 11 insertions(+), 35 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9a2dfeaa79f4..6a0a91b23597 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -662,7 +662,6 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, */ nr = xchg_nr_deferred(shrinker, shrinkctl); - total_scan = nr; if (shrinker->seeks) { delta = freeable >> priority; delta *= 4; @@ -676,37 +675,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, delta = freeable / 2; } + total_scan = nr >> priority; total_scan += delta; - if (total_scan < 0) { - pr_err("shrink_slab: %pS negative objects to delete nr=%ld\n", - shrinker->scan_objects, total_scan); - total_scan = freeable; - next_deferred = nr; - } else - next_deferred = total_scan; - - /* - * We need to avoid excessive windup on filesystem shrinkers - * due to large numbers of GFP_NOFS allocations causing the - * shrinkers to return -1 all the time. This results in a large - * nr being built up so when a shrink that can do some work - * comes along it empties the entire cache due to nr >>> - * freeable. This is bad for sustaining a working set in - * memory. - * - * Hence only allow the shrinker to scan the entire cache when - * a large delta change is calculated directly. - */ - if (delta < freeable / 4) - total_scan = min(total_scan, freeable / 2); - - /* - * Avoid risking looping forever due to too large nr value: - * never try to free more than twice the estimate number of - * freeable entries. - */ - if (total_scan > freeable * 2) - total_scan = freeable * 2; + total_scan = min(total_scan, (2 * freeable)); trace_mm_shrink_slab_start(shrinker, shrinkctl, nr, freeable, delta, total_scan, priority); @@ -745,10 +716,15 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, cond_resched(); } - if (next_deferred >= scanned) - next_deferred -= scanned; - else - next_deferred = 0; + /* + * The deferred work is increased by any new work (delta) that wasn't + * done, decreased by old deferred work that was done now. + * + * And it is capped to two times of the freeable items. + */ + next_deferred = max_t(long, (nr + delta - scanned), 0); + next_deferred = min(next_deferred, (2 * freeable)); + /* * move the unused scan count back into the shrinker in a * manner that handles concurrent updates.