From patchwork Mon Dec 14 22:37:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 11973271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DD69C2BB48 for ; Mon, 14 Dec 2020 22:38:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3183722573 for ; Mon, 14 Dec 2020 22:38:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2502207AbgLNWi2 (ORCPT ); Mon, 14 Dec 2020 17:38:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36930 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2441091AbgLNWiX (ORCPT ); Mon, 14 Dec 2020 17:38:23 -0500 Received: from mail-pj1-x1042.google.com (mail-pj1-x1042.google.com [IPv6:2607:f8b0:4864:20::1042]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C7833C061793; Mon, 14 Dec 2020 14:37:42 -0800 (PST) Received: by mail-pj1-x1042.google.com with SMTP id v1so7254570pjr.2; Mon, 14 Dec 2020 14:37:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JK0RA9az0H36swxW1b9xoMtxd77STDT0bYjsifkqcRo=; b=F/5xiYvM5I0EZ1qGc6HKxC5DbVx8mFwcf3JfwHFHN3kcB1Qk1grQjgn8+kQBdPQSM1 alxCOT2x5/FoazJu6JQkdG4GitO3MZsQEYSYtsjaQ1/F2TyZ2E92jXroDuFdytjhvKmO dUMJHFDGxiZ2cM0lQ8CMcNHXBU+sBDQMm1b8ZrivVoAmViSUsIHutgMvOtj9kgkl7msA r6T7n+glgZi4S5CqtQQj/F+wqQ5y/4lc6pyB3xHEAzlcLA6PXgtMgM8YYxEhmnF6oWq1 NSaUa2RVk8XMOmsEyNxFja/5Ndj8gKg6XpSZHHj7mV5zYjfZg4HN/c8FLUq3S1DFx9Fl I8Fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JK0RA9az0H36swxW1b9xoMtxd77STDT0bYjsifkqcRo=; b=CPhq/pbxDSREFJw6hIZ81c0LSZliDYEvECvjfuy7WyLgCLr30yNMTFjRXxHrnQ2h0U IwQ8qwgyYQWl1ZGk4XWtMncRiKd5MKGUgE+VL0ud1jMxzeh3C2uZOnKLXvQl7Z7cosEb qHzOoLEPd7/ydJCBW5FbYkxrJ1vho/sQr+Y89Hp+AHp5i74pDhljbAGVVNX6tU9b1X9H PS1ExzKGPOaSC4vMsCcR8Y20nW9x8cD8eox84JJXMGVDn5uMRCgQ7D+YwAO0MpMODQp8 mj2JPALPqIZWKfJaxaPgfkg8i8KtoEL8bHAhQgcepWyNA0pS57yjUKxlFZ1PWEGbhZiR hdMw== X-Gm-Message-State: AOAM530mwtIOoZBM068cX9OL3nzRbFXlYj5zG16wvwdlhccQ89Ql6Eqe xocKR1f+xWSn8/YgjR4ZgTl1ehPZUpWXDg== X-Google-Smtp-Source: ABdhPJyD+/0faNVKRhmAJZsCjeKks1k242JFR+RcoNYxKcHgo8xWCG5eLgy8LarRv/qmPAu9krxmMw== X-Received: by 2002:a17:90a:5581:: with SMTP id c1mr6843745pji.86.1607985462413; Mon, 14 Dec 2020 14:37:42 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d4sm20610758pfo.127.2020.12.14.14.37.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 14:37:41 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v2 PATCH 1/9] mm: vmscan: use nid from shrink_control for tracepoint Date: Mon, 14 Dec 2020 14:37:14 -0800 Message-Id: <20201214223722.232537-2-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201214223722.232537-1-shy828301@gmail.com> References: <20201214223722.232537-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The tracepoint's nid should show what node the shrink happens on, the start tracepoint uses nid from shrinkctl, but the nid might be set to 0 before end tracepoint if the shrinker is not NUMA aware, so the traceing log may show the shrink happens on one node but end up on the other node. It seems confusing. And the following patch will remove using nid directly in do_shrink_slab(), this patch also helps cleanup the code. Signed-off-by: Yang Shi --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 7b4e31eac2cf..48c06c48b97e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -537,7 +537,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, else new_nr = atomic_long_read(&shrinker->nr_deferred[nid]); - trace_mm_shrink_slab_end(shrinker, nid, freed, nr, new_nr, total_scan); + trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); return freed; } From patchwork Mon Dec 14 22:37:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 11973309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D7ADC2BB40 for ; Mon, 14 Dec 2020 22:44:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C68DE224F9 for ; Mon, 14 Dec 2020 22:44:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2502315AbgLNWia (ORCPT ); Mon, 14 Dec 2020 17:38:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36940 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2441179AbgLNWiZ (ORCPT ); Mon, 14 Dec 2020 17:38:25 -0500 Received: from mail-pl1-x642.google.com (mail-pl1-x642.google.com [IPv6:2607:f8b0:4864:20::642]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D14CC061794; Mon, 14 Dec 2020 14:37:45 -0800 (PST) Received: by mail-pl1-x642.google.com with SMTP id 4so9747674plk.5; Mon, 14 Dec 2020 14:37:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ozVKgrz/cuTdhl0CMZyCRzGO4Sb1PcBOLEI+PfNUSHM=; b=dXTQDcP72TdGCCJV+w/cgvaAaD/MLr2B5g5F2KSNONWtAZWOrf4mUN2HiA0vEeHEcK mIdVk7cX8wqk32sJglGFovFRe9g17sNd9lz4PWd6gnOEUagZ7k0cfxk9LsNabtlnHoIH K5V13cL6igyAu0466QaJrJHXMIXQaEoAlJ2rCYucdM8mhULTLteNYDGA14m2Sb1m15Ur cDzckkG3rRz9pDmouO3zJm9mPEaPtswg3Ql+oC+VsTkihgfEfbOVXn6tSZkZZHZpHwCJ yDgYlgKsPsUFqLDbti8/VKsZZ3l139LvxDf/arZ3LGUfghanuwCfHYoAKv10FumTEAmX 5ajg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ozVKgrz/cuTdhl0CMZyCRzGO4Sb1PcBOLEI+PfNUSHM=; b=T4mUC570oOAy15o//umuG5ThwaZfyCHDQqHu716gpW4PH0BBD+emh7EQiZFuBVwrI7 Xmml4GKNGTVwjg4X0KtTwmu/0XIwxTYDknnuo/lVqUbjnV1YOExEyU7Fbh2OyI/U1yNi nIAbjhrrSfsAKU3+5l+Hy0m3KA3jMU6agr6MePYnDHYth3UWIVQe3kf53/7ZUPqmCCe0 qE4n4YUS4uEfd6Hulyy2H65eU2K7B6o7yQBKD9UiyYZ58ipVHk3bS+WvDGIO5i5Bt96D 5lh8yKIPJ8wcHx2DgGFCfsTIZCTN36oCuvB3H+gv1FElemQjkUpUIgekT7qb/5HJC9Uk 3Eyg== X-Gm-Message-State: AOAM532cH7YSOwDPSdbmVv1s84YIyrZNEGOYqKQdoa5J6N/LlatF/w7t vxAHUpU14oy5BnMUHK49//c= X-Google-Smtp-Source: ABdhPJwNKKhYDEpsImErYCx1i9yXj/MdY2yg4ExDO6Zf1q/mFSBm62tiTbO3AGPAWepsMKNUSLxTfw== X-Received: by 2002:a17:90a:f0c5:: with SMTP id fa5mr27964263pjb.144.1607985464867; Mon, 14 Dec 2020 14:37:44 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d4sm20610758pfo.127.2020.12.14.14.37.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 14:37:43 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v2 PATCH 2/9] mm: memcontrol: use shrinker_rwsem to protect shrinker_maps allocation Date: Mon, 14 Dec 2020 14:37:15 -0800 Message-Id: <20201214223722.232537-3-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201214223722.232537-1-shy828301@gmail.com> References: <20201214223722.232537-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Since memcg_shrinker_map_size just can be changd under holding shrinker_rwsem exclusively, the read side can be protected by holding read lock, so it sounds superfluous to have a dedicated mutex. This should not exacerbate the contention to shrinker_rwsem since just one read side critical section is added. Signed-off-by: Yang Shi Acked-by: Johannes Weiner --- mm/internal.h | 1 + mm/memcontrol.c | 17 +++++++---------- mm/vmscan.c | 2 +- 3 files changed, 9 insertions(+), 11 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index c43ccdddb0f6..10c79d199aaa 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -108,6 +108,7 @@ extern unsigned long highest_memmap_pfn; /* * in mm/vmscan.c: */ +extern struct rw_semaphore shrinker_rwsem; extern int isolate_lru_page(struct page *page); extern void putback_lru_page(struct page *page); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 29459a6ce1c7..ed942734235f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -394,8 +394,8 @@ DEFINE_STATIC_KEY_FALSE(memcg_kmem_enabled_key); EXPORT_SYMBOL(memcg_kmem_enabled_key); #endif +/* It is only can be changed with holding shrinker_rwsem exclusively */ static int memcg_shrinker_map_size; -static DEFINE_MUTEX(memcg_shrinker_map_mutex); static void memcg_free_shrinker_map_rcu(struct rcu_head *head) { @@ -408,8 +408,6 @@ static int memcg_expand_one_shrinker_map(struct mem_cgroup *memcg, struct memcg_shrinker_map *new, *old; int nid; - lockdep_assert_held(&memcg_shrinker_map_mutex); - for_each_node(nid) { old = rcu_dereference_protected( mem_cgroup_nodeinfo(memcg, nid)->shrinker_map, true); @@ -458,7 +456,7 @@ static int memcg_alloc_shrinker_maps(struct mem_cgroup *memcg) if (mem_cgroup_is_root(memcg)) return 0; - mutex_lock(&memcg_shrinker_map_mutex); + down_read(&shrinker_rwsem); size = memcg_shrinker_map_size; for_each_node(nid) { map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid); @@ -469,7 +467,7 @@ static int memcg_alloc_shrinker_maps(struct mem_cgroup *memcg) } rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_map, map); } - mutex_unlock(&memcg_shrinker_map_mutex); + up_read(&shrinker_rwsem); return ret; } @@ -484,9 +482,8 @@ int memcg_expand_shrinker_maps(int new_id) if (size <= old_size) return 0; - mutex_lock(&memcg_shrinker_map_mutex); if (!root_mem_cgroup) - goto unlock; + goto out; for_each_mem_cgroup(memcg) { if (mem_cgroup_is_root(memcg)) @@ -494,13 +491,13 @@ int memcg_expand_shrinker_maps(int new_id) ret = memcg_expand_one_shrinker_map(memcg, size, old_size); if (ret) { mem_cgroup_iter_break(NULL, memcg); - goto unlock; + goto out; } } -unlock: +out: if (!ret) memcg_shrinker_map_size = size; - mutex_unlock(&memcg_shrinker_map_mutex); + return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 48c06c48b97e..912c044301dd 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -184,7 +184,7 @@ static void set_task_reclaim_state(struct task_struct *task, } static LIST_HEAD(shrinker_list); -static DECLARE_RWSEM(shrinker_rwsem); +DECLARE_RWSEM(shrinker_rwsem); #ifdef CONFIG_MEMCG /* From patchwork Mon Dec 14 22:37:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 11973275 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D130C4361B for ; Mon, 14 Dec 2020 22:39:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5694E20732 for ; Mon, 14 Dec 2020 22:39:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2502369AbgLNWib (ORCPT ); Mon, 14 Dec 2020 17:38:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2441180AbgLNWi2 (ORCPT ); Mon, 14 Dec 2020 17:38:28 -0500 Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C31CDC06179C; Mon, 14 Dec 2020 14:37:47 -0800 (PST) Received: by mail-pl1-x62a.google.com with SMTP id x12so9275997plr.10; Mon, 14 Dec 2020 14:37:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RnXK+gZ1C2/dQ0iAczaSS+4uBnqZQVQH1YKLl+NZ61M=; b=GGQmRYRtTlBV1mtp1fkqgUKL0n34FOR26CqFeArLaZtWKB97t753yJch9PooXHpLPB FFL31FyLa6SpRP+3/Z0kWL3bXc0aEaQP7CmkOOgzgIxDiSSIJTe9E0UDD+D7+IOMXNdq G32XeAoRxQ4Gj9cZLsbnwzlpUj1jOUS1JaEU/TlA8GRMDiBhfEAq6+Z8gPbS7KQNUQO+ 047aDRXN/m70MRnprmi6cLoOwFaRYNUAcDtCC4tLBVY/sYP3ED7zOjg9FjRrwMyYi+K1 YIh/vfq2/gW4y2Tdc7GdFKtnUOKWimotpGDHPEzQGyU0c2quOU94BM3rJ0TF8VKN2gK9 6y3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RnXK+gZ1C2/dQ0iAczaSS+4uBnqZQVQH1YKLl+NZ61M=; b=eU4ZSDzxaaeSvAOlJ6Z4cYJshDq9p2tlOPtdcQ0KjZRso59m90vgtvFrx/u8uya+uu Iabxi6NzGqypJWqCBMGJdaZk052zpFs/93jPqfBTz/KTrbl0MobwJQD91fZg7YSU6nPY 7DoWMV+FiT8yZrOK+jGNYDQWYfkMc9SuaMWt+vSJQoptqVYvRCdyEedOOF/nrDOReLbA TQ/HVSklzCQvjjG18p4oRBYxCnuiIS5zeuhYxvXaBRmV0FuvD68qkmDFW69pUVTyU2OK Vf0dONaRG2yTTXcYMxV2AY3Mw7zx5lmWz5OyrHpitflVpm9eoUNG6WZEAAKeMcMECURC HQag== X-Gm-Message-State: AOAM530cuRr9+8ETGCZL1I36u/hT+Zy1FjMsf0ou9AY41s6Nhop//aAF r5qLnsJpKg5DB/3qCGWWJ/g= X-Google-Smtp-Source: ABdhPJx2Cv9p2Tq9vADJWiKsvIEIe5Ww975Xm0kVR2bMbuwE9HC7vTBHlQ9cBXkEeK2Pe9I1qyYDNg== X-Received: by 2002:a17:90a:e7cc:: with SMTP id kb12mr27236213pjb.234.1607985467358; Mon, 14 Dec 2020 14:37:47 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d4sm20610758pfo.127.2020.12.14.14.37.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 14:37:46 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v2 PATCH 3/9] mm: vmscan: guarantee shrinker_slab_memcg() sees valid shrinker_maps for online memcg Date: Mon, 14 Dec 2020 14:37:16 -0800 Message-Id: <20201214223722.232537-4-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201214223722.232537-1-shy828301@gmail.com> References: <20201214223722.232537-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The shrink_slab_memcg() races with mem_cgroup_css_online(). A visibility of CSS_ONLINE flag in shrink_slab_memcg()->mem_cgroup_online() does not guarantee that we will see memcg->nodeinfo[nid]->shrinker_maps != NULL. This may occur because of processor reordering on !x86. This seems like the below case: CPU A CPU B store shrinker_map load CSS_ONLINE store CSS_ONLINE load shrinker_map So the memory ordering could be guaranteed by smp_wmb()/smp_rmb() pair. The memory barriers pair will guarantee the ordering between shrinker_deferred and CSS_ONLINE for the following patches as well. Signed-off-by: Yang Shi --- mm/memcontrol.c | 7 +++++++ mm/vmscan.c | 8 +++++--- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ed942734235f..3d4ddbb84a01 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5406,6 +5406,13 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) return -ENOMEM; } + /* + * Barrier for CSS_ONLINE, so that shrink_slab_memcg() sees shirnker_maps + * and shrinker_deferred before CSS_ONLINE. It pairs with the read barrier + * in shrink_slab_memcg(). + */ + smp_wmb(); + /* Online state pins memcg ID, memcg ID pins CSS */ refcount_set(&memcg->id.ref, 1); css_get(css); diff --git a/mm/vmscan.c b/mm/vmscan.c index 912c044301dd..9b31b9c419ec 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -552,13 +552,15 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (!mem_cgroup_online(memcg)) return 0; + /* Pairs with write barrier in mem_cgroup_css_online() */ + smp_rmb(); + if (!down_read_trylock(&shrinker_rwsem)) return 0; + /* Once memcg is online it can't be NULL */ map = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_map, true); - if (unlikely(!map)) - goto unlock; for_each_set_bit(i, map->map, shrinker_nr_max) { struct shrink_control sc = { @@ -612,7 +614,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, break; } } -unlock: + up_read(&shrinker_rwsem); return freed; } From patchwork Mon Dec 14 22:37:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 11973277 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DAABC4361B for ; Mon, 14 Dec 2020 22:39:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1F99A22509 for ; Mon, 14 Dec 2020 22:39:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2502500AbgLNWjC (ORCPT ); Mon, 14 Dec 2020 17:39:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2502413AbgLNWiy (ORCPT ); Mon, 14 Dec 2020 17:38:54 -0500 Received: from mail-pg1-x542.google.com (mail-pg1-x542.google.com [IPv6:2607:f8b0:4864:20::542]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B974C0617A6; Mon, 14 Dec 2020 14:37:50 -0800 (PST) Received: by mail-pg1-x542.google.com with SMTP id t37so13749464pga.7; Mon, 14 Dec 2020 14:37:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=buF0Bc96au1TIy8Azp9h1XTsIalRZi79CS3NEeVVksI=; b=OaUcYrYVf/eeMpxa1NLV/j1JCTA3035IB9EUkfK3Pj1tDDE9eF3GHvc1LmiNegkWnH vZKB9fyYio5aTMn3q269OXiEB6rOIRy9cFsX3Ve5v8g9/cHgo8OSh9K+KhokHrHtNS4T HuR8/EwA3E9eyzcqZAOrmJ9iT0iZvl2SgPhk/fK3FuvkiJjvH7kamIhCNcF5cyrJD1Os GQN68PXwvsxq5aQR44ulyWqZ6s84fCyZmdJ0k/wxHnooGLkH69L6I/se5zmYCtglPmbm HxGN/G/Oq7BpsavM8km/yznT8JI7JISm7VYSEpRfFRfKj4peKfG6mKuY/gDBvNMcZ0/u g2/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=buF0Bc96au1TIy8Azp9h1XTsIalRZi79CS3NEeVVksI=; b=tJYQ4kOTu79yJcGcLrRbqH4IWyTRBFo3PZc/kR8r2r73jaIv40UQ2THO6HCb6h18ch e8bTliPZ+5rCSYIl6fJ9BKMTmHsijyMK8e10kKBOREiCQX0Sq1aAFRFZWqO1pneEvN8Y I5OoCB94hMrqz3PrhmgWXwCMhC0wkvpgfjyDvNQkXhfDAecSfLjJYwoF9KaPEv4H+V0F pdaQccIPKwhSpmuzl6urPTeFWLlwl5O9B+AD6TSk/sEeR03Nl3GZEy68Jjva/B+xHprc 2KN06LYEg4d3yymtGLvPvo49moxOL+VuW5zhQKDdjvquNgNeZ/95CjI+IwBaDpvynDWO 8pDw== X-Gm-Message-State: AOAM5311nzS4YI+MLGmrx7lfa5iZIiGvhaJ9OPe4vjuOMMxIr0m36lC2 Pmg/1kFcAAgYq3SuEW5PTWY= X-Google-Smtp-Source: ABdhPJwzng310WgdKPBA4Pp2tm1EIsAo8wEKWhFrdn5P24aXLStOtFuZeSLbkUX67IKRJTf9pkPdCQ== X-Received: by 2002:a63:5d59:: with SMTP id o25mr25614645pgm.218.1607985469792; Mon, 14 Dec 2020 14:37:49 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d4sm20610758pfo.127.2020.12.14.14.37.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 14:37:48 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v2 PATCH 4/9] mm: vmscan: use a new flag to indicate shrinker is registered Date: Mon, 14 Dec 2020 14:37:17 -0800 Message-Id: <20201214223722.232537-5-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201214223722.232537-1-shy828301@gmail.com> References: <20201214223722.232537-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Currently registered shrinker is indicated by non-NULL shrinker->nr_deferred. This approach is fine with nr_deferred at the shrinker level, but the following patches will move MEMCG_AWARE shrinkers' nr_deferred to memcg level, so their shrinker->nr_deferred would always be NULL. This would prevent the shrinkers from unregistering correctly. Introduce a new flag to indicate if shrinker is registered or not. Signed-off-by: Yang Shi --- include/linux/shrinker.h | 7 ++++--- mm/vmscan.c | 13 +++++++++---- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 0f80123650e2..1eac79ce57d4 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -79,13 +79,14 @@ struct shrinker { #define DEFAULT_SEEKS 2 /* A good number if you don't know better. */ /* Flags */ -#define SHRINKER_NUMA_AWARE (1 << 0) -#define SHRINKER_MEMCG_AWARE (1 << 1) +#define SHRINKER_REGISTERED (1 << 0) +#define SHRINKER_NUMA_AWARE (1 << 1) +#define SHRINKER_MEMCG_AWARE (1 << 2) /* * It just makes sense when the shrinker is also MEMCG_AWARE for now, * non-MEMCG_AWARE shrinker should not have this flag set. */ -#define SHRINKER_NONSLAB (1 << 2) +#define SHRINKER_NONSLAB (1 << 3) extern int prealloc_shrinker(struct shrinker *shrinker); extern void register_shrinker_prepared(struct shrinker *shrinker); diff --git a/mm/vmscan.c b/mm/vmscan.c index 9b31b9c419ec..16c9d2aeeb26 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -378,6 +378,7 @@ void register_shrinker_prepared(struct shrinker *shrinker) if (shrinker->flags & SHRINKER_MEMCG_AWARE) idr_replace(&shrinker_idr, shrinker, shrinker->id); #endif + shrinker->flags |= SHRINKER_REGISTERED; up_write(&shrinker_rwsem); } @@ -397,13 +398,17 @@ EXPORT_SYMBOL(register_shrinker); */ void unregister_shrinker(struct shrinker *shrinker) { - if (!shrinker->nr_deferred) - return; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - unregister_memcg_shrinker(shrinker); down_write(&shrinker_rwsem); + if (!(shrinker->flags & SHRINKER_REGISTERED)) { + up_write(&shrinker_rwsem); + return; + } list_del(&shrinker->list); + shrinker->flags &= ~SHRINKER_REGISTERED; up_write(&shrinker_rwsem); + + if (shrinker->flags & SHRINKER_MEMCG_AWARE) + unregister_memcg_shrinker(shrinker); kfree(shrinker->nr_deferred); shrinker->nr_deferred = NULL; } From patchwork Mon Dec 14 22:37:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 11973281 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FDECC4361B for ; Mon, 14 Dec 2020 22:39:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2F0F420732 for ; Mon, 14 Dec 2020 22:39:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2502662AbgLNWjR (ORCPT ); Mon, 14 Dec 2020 17:39:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2502521AbgLNWjG (ORCPT ); Mon, 14 Dec 2020 17:39:06 -0500 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76446C06138C; Mon, 14 Dec 2020 14:37:52 -0800 (PST) Received: by mail-pl1-x644.google.com with SMTP id y8so9744628plp.8; Mon, 14 Dec 2020 14:37:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ssXnZiLmnDnhCBaKkF0ViwDaE3pTbiD3bgrO3REdu1o=; b=Kdyag+afxDemndXxDu2neVdeMrMnJnpsM79GpbcCiOrzOzOecSQNwuUwNkXBaAfsUm amDvyRcr96/3GBLplUfOrS53VyimB/bD7uc8cL7ubwF/h2bPq8ljNIV1s7se7hquiP9c 3RvhvrRw/x7f2Khxqt8p3gz0hMoWA+vH1wdzGMUQd4f/e2P0RPJwesMT33kdRgRR7ZQa tTYXem2DTnlGvmP2C9FZi3W0VC37WChq1xtvTnSlMHjlpmKdvYYrKxL/nXqDKvyRHxDf CTV27ONp3jyp+fxK52/nnbpluX0KBiHPx8zrahhBuZE/0VU2+hJ6L+reyM7zj7X9D8qb /LOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ssXnZiLmnDnhCBaKkF0ViwDaE3pTbiD3bgrO3REdu1o=; b=P/xCT1SlHLe+DQaGq7dkaWkzxmQmwubK0fBe4X1VGdZ6TQOyEP57QRRUepTCvZNbjv e3+l68i/iBHT2Edd4HKKmhXJHbXebWCuwYLgCeFA7ZJyCKLfQwowIa3EKdReO+0v/Uhc CTaxNKv6LEU5yflxZp59nBSqYzXZoR8ZAjFdXwfP3CUTLwt0Aw/zZGvVaPmL36r/Ptmv MS88lEAmsrvkC+JBNtDxRyiXkbWlytOefTk4x80jicH7eJ2VK/OPgWkYKhMc5c0NrTYf U1geNlkg7aSZ26iksWYiN6AO+T9+vnJL0wOFvA/6Q3Yp2VYjpdkBK6seZHC4sVmN2LBB sdPA== X-Gm-Message-State: AOAM5315nhbIhQUQzZC2IBW4HtyGmRUD17zDqgtAu7eLl7spZt3UQoYU 8GCagvFxABKlMYQ9A4DtGsM= X-Google-Smtp-Source: ABdhPJyWkKVRTRNzs5DbHGN2QcNv2wKI3ximz4j2UUbDkjomVlfb5vfSXO6RIa099VMG93AU7IgDsw== X-Received: by 2002:a17:90b:4b07:: with SMTP id lx7mr28007598pjb.230.1607985471993; Mon, 14 Dec 2020 14:37:51 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d4sm20610758pfo.127.2020.12.14.14.37.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 14:37:51 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v2 PATCH 5/9] mm: memcontrol: add per memcg shrinker nr_deferred Date: Mon, 14 Dec 2020 14:37:18 -0800 Message-Id: <20201214223722.232537-6-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201214223722.232537-1-shy828301@gmail.com> References: <20201214223722.232537-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Currently the number of deferred objects are per shrinker, but some slabs, for example, vfs inode/dentry cache are per memcg, this would result in poor isolation among memcgs. The deferred objects typically are generated by __GFP_NOFS allocations, one memcg with excessive __GFP_NOFS allocations may blow up deferred objects, then other innocent memcgs may suffer from over shrink, excessive reclaim latency, etc. For example, two workloads run in memcgA and memcgB respectively, workload in B is vfs heavy workload. Workload in A generates excessive deferred objects, then B's vfs cache might be hit heavily (drop half of caches) by B's limit reclaim or global reclaim. We observed this hit in our production environment which was running vfs heavy workload shown as the below tracing log: <...>-409454 [016] .... 28286961.747146: mm_shrink_slab_start: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 1 objects to shrink 3641681686040 gfp_flags GFP_HIGHUSER_MOVABLE|__GFP_ZERO pgs_scanned 1 lru_pgs 15721 cache items 246404277 delta 31345 total_scan 123202138 <...>-409454 [022] .... 28287105.928018: mm_shrink_slab_end: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 1 unused scan count 3641681686040 new scan count 3641798379189 total_scan 602 last shrinker return val 123186855 The vfs cache and page cache ration was 10:1 on this machine, and half of caches were dropped. This also resulted in significant amount of page caches were dropped due to inodes eviction. Make nr_deferred per memcg for memcg aware shrinkers would solve the unfairness and bring better isolation. When memcg is not enabled (!CONFIG_MEMCG or memcg disabled), the shrinker's nr_deferred would be used. And non memcg aware shrinkers use shrinker's nr_deferred all the time. Signed-off-by: Yang Shi --- include/linux/memcontrol.h | 9 +++ mm/memcontrol.c | 110 ++++++++++++++++++++++++++++++++++++- mm/vmscan.c | 4 ++ 3 files changed, 120 insertions(+), 3 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 922a7f600465..1b343b268359 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -92,6 +92,13 @@ struct lruvec_stat { long count[NR_VM_NODE_STAT_ITEMS]; }; + +/* Shrinker::id indexed nr_deferred of memcg-aware shrinkers. */ +struct memcg_shrinker_deferred { + struct rcu_head rcu; + atomic_long_t nr_deferred[]; +}; + /* * Bitmap of shrinker::id corresponding to memcg-aware shrinkers, * which have elements charged to this memcg. @@ -119,6 +126,7 @@ struct mem_cgroup_per_node { struct mem_cgroup_reclaim_iter iter; struct memcg_shrinker_map __rcu *shrinker_map; + struct memcg_shrinker_deferred __rcu *shrinker_deferred; struct rb_node tree_node; /* RB tree node */ unsigned long usage_in_excess;/* Set to the value by which */ @@ -1489,6 +1497,7 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) } extern int memcg_expand_shrinker_maps(int new_id); +extern int memcg_expand_shrinker_deferred(int new_id); extern void memcg_set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 3d4ddbb84a01..321d1818ce3d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -394,14 +394,20 @@ DEFINE_STATIC_KEY_FALSE(memcg_kmem_enabled_key); EXPORT_SYMBOL(memcg_kmem_enabled_key); #endif -/* It is only can be changed with holding shrinker_rwsem exclusively */ +/* They are only can be changed with holding shrinker_rwsem exclusively */ static int memcg_shrinker_map_size; +static int memcg_shrinker_deferred_size; static void memcg_free_shrinker_map_rcu(struct rcu_head *head) { kvfree(container_of(head, struct memcg_shrinker_map, rcu)); } +static void memcg_free_shrinker_deferred_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct memcg_shrinker_deferred, rcu)); +} + static int memcg_expand_one_shrinker_map(struct mem_cgroup *memcg, int size, int old_size) { @@ -430,6 +436,34 @@ static int memcg_expand_one_shrinker_map(struct mem_cgroup *memcg, return 0; } +static int memcg_expand_one_shrinker_deferred(struct mem_cgroup *memcg, + int size, int old_size) +{ + struct memcg_shrinker_deferred *new, *old; + int nid; + + for_each_node(nid) { + old = rcu_dereference_protected( + mem_cgroup_nodeinfo(memcg, nid)->shrinker_deferred, true); + /* Not yet online memcg */ + if (!old) + return 0; + + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); + if (!new) + return -ENOMEM; + + /* Copy all old values, and clear all new ones */ + memcpy((void *)new->nr_deferred, (void *)old->nr_deferred, old_size); + memset((void *)new->nr_deferred + old_size, 0, size - old_size); + + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_deferred, new); + call_rcu(&old->rcu, memcg_free_shrinker_deferred_rcu); + } + + return 0; +} + static void memcg_free_shrinker_maps(struct mem_cgroup *memcg) { struct mem_cgroup_per_node *pn; @@ -448,6 +482,21 @@ static void memcg_free_shrinker_maps(struct mem_cgroup *memcg) } } +static void memcg_free_shrinker_deferred(struct mem_cgroup *memcg) +{ + struct mem_cgroup_per_node *pn; + struct memcg_shrinker_deferred *deferred; + int nid; + + for_each_node(nid) { + pn = mem_cgroup_nodeinfo(memcg, nid); + deferred = rcu_dereference_protected(pn->shrinker_deferred, true); + if (deferred) + kvfree(deferred); + rcu_assign_pointer(pn->shrinker_deferred, NULL); + } +} + static int memcg_alloc_shrinker_maps(struct mem_cgroup *memcg) { struct memcg_shrinker_map *map; @@ -472,6 +521,27 @@ static int memcg_alloc_shrinker_maps(struct mem_cgroup *memcg) return ret; } +static int memcg_alloc_shrinker_deferred(struct mem_cgroup *memcg) +{ + struct memcg_shrinker_deferred *deferred; + int nid, size, ret = 0; + + down_read(&shrinker_rwsem); + size = memcg_shrinker_deferred_size; + for_each_node(nid) { + deferred = kvzalloc_node(sizeof(*deferred) + size, GFP_KERNEL, nid); + if (!deferred) { + memcg_free_shrinker_deferred(memcg); + ret = -ENOMEM; + break; + } + rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_deferred, deferred); + } + up_read(&shrinker_rwsem); + + return ret; +} + int memcg_expand_shrinker_maps(int new_id) { int size, old_size, ret = 0; @@ -501,6 +571,33 @@ int memcg_expand_shrinker_maps(int new_id) return ret; } +int memcg_expand_shrinker_deferred(int new_id) +{ + int size, old_size, ret = 0; + struct mem_cgroup *memcg; + + size = (new_id + 1) * sizeof(atomic_long_t); + old_size = memcg_shrinker_deferred_size; + if (size <= old_size) + return 0; + + if (!root_mem_cgroup) + goto out; + + for_each_mem_cgroup(memcg) { + ret = memcg_expand_one_shrinker_deferred(memcg, size, old_size); + if (ret) { + mem_cgroup_iter_break(NULL, memcg); + goto out; + } + } +out: + if (!ret) + memcg_shrinker_deferred_size = size; + + return ret; +} + void memcg_set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { @@ -5397,8 +5494,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) struct mem_cgroup *memcg = mem_cgroup_from_css(css); /* - * A memcg must be visible for memcg_expand_shrinker_maps() - * by the time the maps are allocated. So, we allocate maps + * A memcg must be visible for memcg_expand_shrinker_{maps|deferred}() + * by the time the maps are allocated. So, we allocate maps and deferred * here, when for_each_mem_cgroup() can't skip it. */ if (memcg_alloc_shrinker_maps(memcg)) { @@ -5406,6 +5503,12 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) return -ENOMEM; } + if (memcg_alloc_shrinker_deferred(memcg)) { + memcg_free_shrinker_maps(memcg); + mem_cgroup_id_remove(memcg); + return -ENOMEM; + } + /* * Barrier for CSS_ONLINE, so that shrink_slab_memcg() sees shirnker_maps * and shrinker_deferred before CSS_ONLINE. It pairs with the read barrier @@ -5473,6 +5576,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); memcg_free_shrinker_maps(memcg); + memcg_free_shrinker_deferred(memcg); memcg_free_kmem(memcg); mem_cgroup_free(memcg); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 16c9d2aeeb26..bf34167dd67e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -219,6 +219,10 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) goto unlock; } + if (memcg_expand_shrinker_deferred(id)) { + idr_remove(&shrinker_idr, id); + goto unlock; + } shrinker_nr_max = id + 1; } shrinker->id = id; From patchwork Mon Dec 14 22:37:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 11973305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56498C4361B for ; Mon, 14 Dec 2020 22:42:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1FE2622509 for ; Mon, 14 Dec 2020 22:42:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2441157AbgLNWlK (ORCPT ); Mon, 14 Dec 2020 17:41:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2502537AbgLNWjG (ORCPT ); Mon, 14 Dec 2020 17:39:06 -0500 Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6E7AC061285; Mon, 14 Dec 2020 14:37:54 -0800 (PST) Received: by mail-pg1-x544.google.com with SMTP id g18so13777395pgk.1; Mon, 14 Dec 2020 14:37:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+kmdzUyomriPoEmgNJZkhkyxllzW++vSvJDg1wBbRWU=; b=iI8SQ4uIBy6rSF6WJgwZJtcCMKt06dXZSxVquYF/3xfi7j7bZXBscDEUmCgzi5GewB IkE8ILNOew3ChA1nW75iqnez79iWw1ndLmdEo8i3ua18h3SXGZmqO0ZMh1+wRiabjzqR u457fTskXBgUezEgKLBY80N5NDt6zlCNjwuLuHFKLHmuCBZW8+xvjRE2ntee027Cahhr NM9BA3PYx/1thRe828BQdlVujz2/6yVbURUpZFHYoG9XQbBcF0YdiW1At1cCihtAuj12 e4tXSPFTIhhJ5e/n3mwY124qc/D8hTOntHETVdST3zZcxe1yU7NptmaHaERLmo9+Wwta EO1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+kmdzUyomriPoEmgNJZkhkyxllzW++vSvJDg1wBbRWU=; b=V0/tNCieSBE5g/I0MkVuTL6cx/gzttn7Nuo3s5Ll3bvT/l3zyQcpt8/WowTnDBgYlj PYZ/lBH80iNrh7PTonHrC6mECX9q2+Z0i5eqFb1B9H2lqeK/dBXKmrnnBgqzDvuQAqEZ AOG3GVPStcKW3VETcWNBjvzFSXgzSzyuzHNn3uNIeuW2X8Y/heys7EQ3e2yivDaVOP4c CsHEZQOn8W9rveIuTSTPTGz+s1oqou2HoyTUfYwv4LWL/j49u9TnUYzPxgFMuWD2oSJX gZ6RVUc3KpSAldL0ffEmVvDGAfJZGy8xTuw0/HmaV6uXiSbDWF7N643tDD2KaQVR28Rm Fy2A== X-Gm-Message-State: AOAM530jfsGmolA6O0jVJ16o2+iAh+YCjb104cDeXkyGnjFWp95hNQoP 2fI2bXcqbcWtSpE3jSnfqYo= X-Google-Smtp-Source: ABdhPJz/LPX+WFR6Oci5zsfqoxLPRAVL4rXMXlUq7w+UJS5jvacni+aMkFdG5kal9SInUPXavYNuhQ== X-Received: by 2002:aa7:963c:0:b029:19d:dcd3:b2ae with SMTP id r28-20020aa7963c0000b029019ddcd3b2aemr25803975pfg.76.1607985474398; Mon, 14 Dec 2020 14:37:54 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d4sm20610758pfo.127.2020.12.14.14.37.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 14:37:53 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v2 PATCH 6/9] mm: vmscan: use per memcg nr_deferred of shrinker Date: Mon, 14 Dec 2020 14:37:19 -0800 Message-Id: <20201214223722.232537-7-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201214223722.232537-1-shy828301@gmail.com> References: <20201214223722.232537-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Use per memcg's nr_deferred for memcg aware shrinkers. The shrinker's nr_deferred will be used in the following cases: 1. Non memcg aware shrinkers 2. !CONFIG_MEMCG 3. memcg is disabled by boot parameter Signed-off-by: Yang Shi --- mm/vmscan.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 83 insertions(+), 11 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index bf34167dd67e..bce8cf44eca2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -203,6 +203,12 @@ DECLARE_RWSEM(shrinker_rwsem); static DEFINE_IDR(shrinker_idr); static int shrinker_nr_max; +static inline bool is_deferred_memcg_aware(struct shrinker *shrinker) +{ + return (shrinker->flags & SHRINKER_MEMCG_AWARE) && + !mem_cgroup_disabled(); +} + static int prealloc_memcg_shrinker(struct shrinker *shrinker) { int id, ret = -ENOMEM; @@ -271,7 +277,58 @@ static bool writeback_throttling_sane(struct scan_control *sc) #endif return false; } + +static inline long count_nr_deferred(struct shrinker *shrinker, + struct shrink_control *sc) +{ + bool per_memcg_deferred = is_deferred_memcg_aware(shrinker) && sc->memcg; + struct memcg_shrinker_deferred *deferred; + struct mem_cgroup *memcg = sc->memcg; + int nid = sc->nid; + int id = shrinker->id; + long nr; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (per_memcg_deferred) { + deferred = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_deferred, + true); + nr = atomic_long_xchg(&deferred->nr_deferred[id], 0); + } else + nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0); + + return nr; +} + +static inline long set_nr_deferred(long nr, struct shrinker *shrinker, + struct shrink_control *sc) +{ + bool per_memcg_deferred = is_deferred_memcg_aware(shrinker) && sc->memcg; + struct memcg_shrinker_deferred *deferred; + struct mem_cgroup *memcg = sc->memcg; + int nid = sc->nid; + int id = shrinker->id; + long new_nr; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + if (per_memcg_deferred) { + deferred = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_deferred, + true); + new_nr = atomic_long_add_return(nr, &deferred->nr_deferred[id]); + } else + new_nr = atomic_long_add_return(nr, &shrinker->nr_deferred[nid]); + + return new_nr; +} #else +static inline bool is_deferred_memcg_aware(struct shrinker *shrinker) +{ + return false; +} + static int prealloc_memcg_shrinker(struct shrinker *shrinker) { return 0; @@ -290,6 +347,29 @@ static bool writeback_throttling_sane(struct scan_control *sc) { return true; } + +static inline long count_nr_deferred(struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + return atomic_long_xchg(&shrinker->nr_deferred[nid], 0); +} + +static inline long set_nr_deferred(long nr, struct shrinker *shrinker, + struct shrink_control *sc) +{ + int nid = sc->nid; + + if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) + nid = 0; + + return atomic_long_add_return(nr, + &shrinker->nr_deferred[nid]); +} #endif /* @@ -429,13 +509,10 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, long freeable; long nr; long new_nr; - int nid = shrinkctl->nid; long batch_size = shrinker->batch ? shrinker->batch : SHRINK_BATCH; long scanned = 0, next_deferred; - if (!(shrinker->flags & SHRINKER_NUMA_AWARE)) - nid = 0; freeable = shrinker->count_objects(shrinker, shrinkctl); if (freeable == 0 || freeable == SHRINK_EMPTY) @@ -446,7 +523,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, * and zero it so that other concurrent shrinker invocations * don't also do this scanning work. */ - nr = atomic_long_xchg(&shrinker->nr_deferred[nid], 0); + nr = count_nr_deferred(shrinker, shrinkctl); total_scan = nr; if (shrinker->seeks) { @@ -537,14 +614,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, next_deferred = 0; /* * move the unused scan count back into the shrinker in a - * manner that handles concurrent updates. If we exhausted the - * scan, there is no need to do an update. + * manner that handles concurrent updates. */ - if (next_deferred > 0) - new_nr = atomic_long_add_return(next_deferred, - &shrinker->nr_deferred[nid]); - else - new_nr = atomic_long_read(&shrinker->nr_deferred[nid]); + new_nr = set_nr_deferred(next_deferred, shrinker, shrinkctl); trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan); return freed; From patchwork Mon Dec 14 22:37:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 11973279 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80258C2BB48 for ; Mon, 14 Dec 2020 22:39:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4F52022509 for ; Mon, 14 Dec 2020 22:39:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2502692AbgLNWjZ (ORCPT ); Mon, 14 Dec 2020 17:39:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37064 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2502576AbgLNWjM (ORCPT ); Mon, 14 Dec 2020 17:39:12 -0500 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 20437C061248; Mon, 14 Dec 2020 14:37:57 -0800 (PST) Received: by mail-pf1-x441.google.com with SMTP id m6so3426661pfm.6; Mon, 14 Dec 2020 14:37:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SadIRfB1ozL+BvgNvi56acGK40sGczeZRbLfcOjsgUI=; b=Mc+/YW20XwmOUSCsmjbxlbhWLn0udfDF1hb/tFJNMCo26ixOoN4By9QolVjs97PCec OsGBYh8BNUOvZrxJdfXaWJP7hwH9ofY7oommJ7G+EXI5WXBBZv9A1gWNQb6wYY/Gu8Pr y7qK59t7ZdEHMj/A29Kbg2Stu7UteCZvXBPx5QN5VjvG55neVSTwoUcRrRZATYVQq1+j sLl+9sM+yMHaC339qQHpau96kBWG90pfG+0EVfBmEmFcjatC2mX+U2j2Hx7L9vbXdaGX SiBpXoYwWKsqJoqzhFhgjkjALrzhvgF9hunAQTYfmtDdbvm42WW6Ll9PO/QZK02jtOSv frGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SadIRfB1ozL+BvgNvi56acGK40sGczeZRbLfcOjsgUI=; b=KhEAgHocDIdZ/3x6S+6oaTahwLoa6DGGMad4i0OgvEfghPMi/yBAp3wq9+NsLogK1F vFE6pwnwzmH3cga0bPbhL57uGivxb83A4UiHjyIKW5HuZX2+zK42zWUDPhvoFX8eSlYs +nrFGuQm26PZTk6MJ0cNvnQZZETOiSNwalxL6mEhWgFoQBeoAMIWE9O3sXfjUOLaZ8vR 6YZtmnynXaTyLAf33cOg5y5ASUyRYXT6PJ7/jXTYHVvZre3DA2N3V1o4LHWX6u3qVbfF QbYq0pKExY9EZewUstwYg5jpwtDhL3ZCQbhz9/fiAjn2Xuf54Zt8FE97s2S6g748k2zB DXtw== X-Gm-Message-State: AOAM533icloDdMI3w7DQ7LAIRqTo8nhN/TBd9oAcjuD48qKUedvsOHuC EwgLmjN2U/9q/9X6WTbbEWk= X-Google-Smtp-Source: ABdhPJxLNoStEmuqK/WqKcnP4+Gjaztl1jLEFeEywadhkRJ+a/8qnPy/51MZKB/GCDpJoTDGEkDQVQ== X-Received: by 2002:a05:6a00:1596:b029:19d:96b8:6eab with SMTP id u22-20020a056a001596b029019d96b86eabmr25860773pfk.38.1607985476760; Mon, 14 Dec 2020 14:37:56 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d4sm20610758pfo.127.2020.12.14.14.37.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 14:37:55 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v2 PATCH 7/9] mm: vmscan: don't need allocate shrinker->nr_deferred for memcg aware shrinkers Date: Mon, 14 Dec 2020 14:37:20 -0800 Message-Id: <20201214223722.232537-8-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201214223722.232537-1-shy828301@gmail.com> References: <20201214223722.232537-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Now nr_deferred is available on per memcg level for memcg aware shrinkers, so don't need allocate shrinker->nr_deferred for such shrinkers anymore. Signed-off-by: Yang Shi --- mm/vmscan.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index bce8cf44eca2..8d5bfd818acd 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -420,7 +420,15 @@ unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru, int zone */ int prealloc_shrinker(struct shrinker *shrinker) { - unsigned int size = sizeof(*shrinker->nr_deferred); + unsigned int size; + + if (is_deferred_memcg_aware(shrinker)) { + if (prealloc_memcg_shrinker(shrinker)) + return -ENOMEM; + return 0; + } + + size = sizeof(*shrinker->nr_deferred); if (shrinker->flags & SHRINKER_NUMA_AWARE) size *= nr_node_ids; @@ -429,26 +437,18 @@ int prealloc_shrinker(struct shrinker *shrinker) if (!shrinker->nr_deferred) return -ENOMEM; - if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - if (prealloc_memcg_shrinker(shrinker)) - goto free_deferred; - } - return 0; - -free_deferred: - kfree(shrinker->nr_deferred); - shrinker->nr_deferred = NULL; - return -ENOMEM; } void free_prealloced_shrinker(struct shrinker *shrinker) { - if (!shrinker->nr_deferred) + if (is_deferred_memcg_aware(shrinker)) { + unregister_memcg_shrinker(shrinker); return; + } - if (shrinker->flags & SHRINKER_MEMCG_AWARE) - unregister_memcg_shrinker(shrinker); + if (!shrinker->nr_deferred) + return; kfree(shrinker->nr_deferred); shrinker->nr_deferred = NULL; From patchwork Mon Dec 14 22:37:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 11973307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 582EFC4361B for ; Mon, 14 Dec 2020 22:43:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1BE8320732 for ; Mon, 14 Dec 2020 22:43:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2441142AbgLNWlK (ORCPT ); Mon, 14 Dec 2020 17:41:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2502578AbgLNWjM (ORCPT ); Mon, 14 Dec 2020 17:39:12 -0500 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71286C061257; Mon, 14 Dec 2020 14:37:59 -0800 (PST) Received: by mail-pf1-x441.google.com with SMTP id t8so13163831pfg.8; Mon, 14 Dec 2020 14:37:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=w3fRQtpX8VRWJVNuVlihrDSl4Ew2OVDi3uoQ+orHFSM=; b=TUWsLN2EIgkfe9UBwtBIZfE2DYuDAvqDdRMI2Mzue4kMvvvzUiNj6Hple87HPp2G/p m2vn8lIGJ3ccErxvg6+mSJA9OIBTGnlxjN6kAz2Ha+JqXXUyhU6ToD/1s9ggBTx8my9u GDBG3+FZfNKeJlTdz+jGhB9ials3Yak4nSI6MqItr76HUvhNN/v50JsTz7G9COEt0C8Z fRLPSSHRlyYvfdIhxC8cpseF5JQJl/PbjXNlLlCtz45Czf8pup73ukEbKZSEmX/yD+JD NHII5vcveR9S7an5u1MUIetWNpbfHppz2X41tSHAnIv/Y2ezkciJBMStXI6b4xSAf1R0 XaBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=w3fRQtpX8VRWJVNuVlihrDSl4Ew2OVDi3uoQ+orHFSM=; b=K5Vr8ZK7eRd99/AhtfUh9ZNSmaQe1ip1XgxQoFUYTwbusHjK5K7jhIKxkybZ+8AwF5 pL7eyZ56Yk1i0uu+IMJfXb86qYdTwdaJMrNa5cGs2JXgbH1lkbxf3JxAH2bdnBiBm5f6 Nl3H6RTVSilpnLk85s9ZMyjIL57Yckx59SriCbT9FTOrZds5bfq64oIH11zXuABxG2+o IHwC6UHYl9PM42iM6kesLBe97H2m8mwCwkPK2yP9MN+SEcYta7RpnGTnTbpl/gSTJvJW 8s1hmWiPMJtdkf0QjhPyjxn/5mMtMsza+mt2MLVRkpNDLNZ/utvlJgS6CEk4PoDEm155 NFKA== X-Gm-Message-State: AOAM531xqAZHr1yRsdNZf1tnT+wB/rH0UNsjk+Vpirwv+qDbOU9cxTEV Yv0Chq0CLLWe8EBG73xdaZ4= X-Google-Smtp-Source: ABdhPJy5whBOYsOd14m3qSnV22diJRbUamHDpaAsnC9yQbAgm+Kzx7vaAAYra+01E60AsBNY91361Q== X-Received: by 2002:aa7:85d2:0:b029:1a2:73fe:5c28 with SMTP id z18-20020aa785d20000b02901a273fe5c28mr14313012pfn.40.1607985479060; Mon, 14 Dec 2020 14:37:59 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d4sm20610758pfo.127.2020.12.14.14.37.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 14:37:58 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v2 PATCH 8/9] mm: memcontrol: reparent nr_deferred when memcg offline Date: Mon, 14 Dec 2020 14:37:21 -0800 Message-Id: <20201214223722.232537-9-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201214223722.232537-1-shy828301@gmail.com> References: <20201214223722.232537-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Now shrinker's nr_deferred is per memcg for memcg aware shrinkers, add to parent's corresponding nr_deferred when memcg offline. Signed-off-by: Yang Shi --- include/linux/shrinker.h | 4 ++++ mm/memcontrol.c | 24 ++++++++++++++++++++++++ mm/vmscan.c | 2 +- 3 files changed, 29 insertions(+), 1 deletion(-) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 1eac79ce57d4..85cfc910dde4 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -78,6 +78,10 @@ struct shrinker { }; #define DEFAULT_SEEKS 2 /* A good number if you don't know better. */ +#ifdef CONFIG_MEMCG +extern int shrinker_nr_max; +#endif + /* Flags */ #define SHRINKER_REGISTERED (1 << 0) #define SHRINKER_NUMA_AWARE (1 << 1) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 321d1818ce3d..1f191a15bee1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -59,6 +59,7 @@ #include #include #include +#include #include "internal.h" #include #include @@ -612,6 +613,28 @@ void memcg_set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) } } +static void memcg_reparent_shrinker_deferred(struct mem_cgroup *memcg) +{ + int i, nid; + long nr; + struct mem_cgroup *parent; + struct memcg_shrinker_deferred *child_deferred, *parent_deferred; + + parent = parent_mem_cgroup(memcg); + if (!parent) + parent = root_mem_cgroup; + + for_each_node(nid) { + child_deferred = memcg->nodeinfo[nid]->shrinker_deferred; + parent_deferred = parent->nodeinfo[nid]->shrinker_deferred; + for (i = 0; i < shrinker_nr_max; i ++) { + nr = atomic_long_read(&child_deferred->nr_deferred[i]); + atomic_long_add(nr, + &parent_deferred->nr_deferred[i]); + } + } +} + /** * mem_cgroup_css_from_page - css of the memcg associated with a page * @page: page of interest @@ -5543,6 +5566,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) page_counter_set_low(&memcg->memory, 0); memcg_offline_kmem(memcg); + memcg_reparent_shrinker_deferred(memcg); wb_memcg_offline(memcg); drain_all_stock(memcg); diff --git a/mm/vmscan.c b/mm/vmscan.c index 8d5bfd818acd..693a41e89969 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -201,7 +201,7 @@ DECLARE_RWSEM(shrinker_rwsem); #define SHRINKER_REGISTERING ((struct shrinker *)~0UL) static DEFINE_IDR(shrinker_idr); -static int shrinker_nr_max; +int shrinker_nr_max; static inline bool is_deferred_memcg_aware(struct shrinker *shrinker) { From patchwork Mon Dec 14 22:37:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 11973283 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D05CDC2BB9A for ; Mon, 14 Dec 2020 22:41:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9B0C622509 for ; Mon, 14 Dec 2020 22:41:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2441180AbgLNWlK (ORCPT ); Mon, 14 Dec 2020 17:41:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37068 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2502582AbgLNWjM (ORCPT ); Mon, 14 Dec 2020 17:39:12 -0500 Received: from mail-pg1-x541.google.com (mail-pg1-x541.google.com [IPv6:2607:f8b0:4864:20::541]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD524C0611C5; Mon, 14 Dec 2020 14:38:01 -0800 (PST) Received: by mail-pg1-x541.google.com with SMTP id w5so12982470pgj.3; Mon, 14 Dec 2020 14:38:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4UvK5zU8YmbSA4ZaOjSKDv3yX4mccXYxyaNC7iOnySk=; b=GeoybizuMfpDpfAV+/8gXc4rn4T8bM6oj5/eYEvqje82syR/3rKUqljo8Tm4WFWUg1 1w3F/pPUjwfEfRfpHW9pcmwhju62t7p3ZHTesDosxN1rma5RaKBYCUeBZL8cVQW7R79n OUaSC1f5iQFNeQHbcisZmna4WNL6WB3YRH+v5d/mMEW1oMAJ6BQBkqmf//LLdSlBMxP6 zsC0eX1wJB0kp1pCC5P6dLcZDzlLMAjPM3zUDse2W7bS6YBN5J/x6RhkXV1hKcLIa8zO D45Jw/7xDAM8F8o2M9p+L767QPqRyEGI77zEZbivRA/B5qGv7qFXpKsR3TvKy0dKlxBS Qmnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4UvK5zU8YmbSA4ZaOjSKDv3yX4mccXYxyaNC7iOnySk=; b=ip9r+PKY3/ePbc9+E2BIj2lHKwOb7cGBDO/8iL/wwESLBo7J++DoRCISL+CVq/J334 hnOQyQt1rIvzJ8rTTHhCIFPRkGsuaSsHLgrbC2TTp7TcX3oAtcizpLAF/w33o/qoZlxx HDpPKKMgxDkDIH5+pHCqISBrLYkYB7Zh2gRwyJ1u65Wknp0DlOjhit3vpBULYj3qC3WB WrBKR4voC3cxlmsOiWOZNwUQJ/cZ76kc2KHFdSupW3CHcDDNYEJK2EZ+FQndKpTpk+oq x7/QgEgxITskCCzQqLBsbkCZ4zkt7yheA73C5OF9vo8qHg5XLnz/mlbR4uB75EyUhhnM f5QQ== X-Gm-Message-State: AOAM530EGV/N4U6nhuWveCGx/+V6BOqdtJ6STFR7ofVVBibjG1Y5E6ME vvuRaJY3hBOY1ProZhjJlrY= X-Google-Smtp-Source: ABdhPJzYQzO1DELUNIEnqXV01z7jV6nNrNROlE526aXtT8QmQi4/cQdvSHCGGkYYtvsWa+qjj0okbA== X-Received: by 2002:a63:344b:: with SMTP id b72mr16684261pga.406.1607985481301; Mon, 14 Dec 2020 14:38:01 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id d4sm20610758pfo.127.2020.12.14.14.37.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 14:38:00 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v2 PATCH 9/9] mm: vmscan: shrink deferred objects proportional to priority Date: Mon, 14 Dec 2020 14:37:22 -0800 Message-Id: <20201214223722.232537-10-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201214223722.232537-1-shy828301@gmail.com> References: <20201214223722.232537-1-shy828301@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The number of deferred objects might get windup to an absurd number, and it results in clamp of slab objects. It is undesirable for sustaining workingset. So shrink deferred objects proportional to priority and cap nr_deferred to twice of cache items. Signed-off-by: Yang Shi --- mm/vmscan.c | 40 +++++----------------------------------- 1 file changed, 5 insertions(+), 35 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 693a41e89969..58f4a383f0df 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -525,7 +525,6 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, */ nr = count_nr_deferred(shrinker, shrinkctl); - total_scan = nr; if (shrinker->seeks) { delta = freeable >> priority; delta *= 4; @@ -539,37 +538,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, delta = freeable / 2; } + total_scan = nr >> priority; total_scan += delta; - if (total_scan < 0) { - pr_err("shrink_slab: %pS negative objects to delete nr=%ld\n", - shrinker->scan_objects, total_scan); - total_scan = freeable; - next_deferred = nr; - } else - next_deferred = total_scan; - - /* - * We need to avoid excessive windup on filesystem shrinkers - * due to large numbers of GFP_NOFS allocations causing the - * shrinkers to return -1 all the time. This results in a large - * nr being built up so when a shrink that can do some work - * comes along it empties the entire cache due to nr >>> - * freeable. This is bad for sustaining a working set in - * memory. - * - * Hence only allow the shrinker to scan the entire cache when - * a large delta change is calculated directly. - */ - if (delta < freeable / 4) - total_scan = min(total_scan, freeable / 2); - - /* - * Avoid risking looping forever due to too large nr value: - * never try to free more than twice the estimate number of - * freeable entries. - */ - if (total_scan > freeable * 2) - total_scan = freeable * 2; + total_scan = min(total_scan, (2 * freeable)); trace_mm_shrink_slab_start(shrinker, shrinkctl, nr, freeable, delta, total_scan, priority); @@ -608,10 +579,9 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, cond_resched(); } - if (next_deferred >= scanned) - next_deferred -= scanned; - else - next_deferred = 0; + next_deferred = max_t(long, (nr - scanned), 0) + total_scan; + next_deferred = min(next_deferred, (2 * freeable)); + /* * move the unused scan count back into the shrinker in a * manner that handles concurrent updates.