From patchwork Fri Oct 1 19:00:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 12531621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 789F0C433EF for ; Fri, 1 Oct 2021 19:00:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2D18B61A57 for ; Fri, 1 Oct 2021 19:00:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2D18B61A57 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 8B3FC94013B; Fri, 1 Oct 2021 15:00:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 863FC940121; Fri, 1 Oct 2021 15:00:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 77A6194013B; Fri, 1 Oct 2021 15:00:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0102.hostedemail.com [216.40.44.102]) by kanga.kvack.org (Postfix) with ESMTP id 683D6940121 for ; Fri, 1 Oct 2021 15:00:49 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 2FD993C650 for ; Fri, 1 Oct 2021 19:00:49 +0000 (UTC) X-FDA: 78648785418.14.2E4EB99 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf04.hostedemail.com (Postfix) with ESMTP id CD9C550056DA for ; Fri, 1 Oct 2021 19:00:48 +0000 (UTC) Received: by mail-pj1-f74.google.com with SMTP id x61-20020a17090a6c4300b0019f789f61bdso1911953pjj.0 for ; Fri, 01 Oct 2021 12:00:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc :content-transfer-encoding; bh=6eaOrx6B9WTfA79dzFjS+JPlNyBUQULetUEsCbO4l+A=; b=gBoqBjumM+ARyiIKWQvUhK9hG+lBFscSv/woVNTb0aHmNmPO90ZO9aLhckcOu1nx6Z kVncrTjjIVtsO6zP4Puly78rWvi5zI4syAoJg4/QIv4c4j8JhOwD5de25nNbf3s31hW9 vi3+1L86KkqQjIo144G+PdtXoCTd/6Q7Di62QOjw/PxHnJx3Hk0PMSZhnEh7NcaPxB0D wO52jucST0sPnaoXOGm89E8yqaj94mu0DhdcF8yZE0pkec7Q8IUwsEBsxqbyJN41GLgo NzechnLrbZvDq37r4XMfh7ag1p11dGoXLZ3K9KRG33RIg0CCOZVbukr4KuYbpRKGdLKl ISrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc :content-transfer-encoding; bh=6eaOrx6B9WTfA79dzFjS+JPlNyBUQULetUEsCbO4l+A=; b=eCioM04ParYhuIqoBLUhFFLSK53kkf9hHdjuJNPJYxBClpdkyQCOZpbkVFCA8UShrU 0zgD445XPjPYfi5RsdbQKdOgIP25qNwl7RS1/Rkno7dGdFiLQ/LXEgnTDdxBuYYwObD4 CVcjRZi7j9e+xwl20l573VylDdmy2PUKyuQqArvhOPsODofFqVvSYRSWRAVsBJb2L1fw 7uFnBtMWZk0XmnwquB50O3SPpNDgBop4gLgt5ilJoFtCH2zjCK1JcG1CCeRNt0qLWQkr jC0bxAkhMS0cpad+j18TPTVe+GIU9cgWHryDunDX/eegm8EXjz8ntQYmsuR30NnV6NnH kmZg== X-Gm-Message-State: AOAM531eH3NLXSZGXMoC8QH3XGh2Z7n6T8fk5jUNvP0i7HI2mH2z3e7i GQmgfNbVwUoga+UEVnjmy7iXhiEHB5n7Eg== X-Google-Smtp-Source: ABdhPJx6IgvBeYEk+UPLwvs9Zhx5drq1IwkilKeIvGG35/M5gj+3M4JrFlVfvTWWs6+j9lVZmuYqGmq0Gg2hVg== X-Received: from shakeelb.svl.corp.google.com ([2620:15c:2cd:202:af1a:f1cd:5283:aa67]) (user=shakeelb job=sendgmr) by 2002:a17:90b:1258:: with SMTP id gx24mr5127236pjb.205.1633114847633; Fri, 01 Oct 2021 12:00:47 -0700 (PDT) Date: Fri, 1 Oct 2021 12:00:39 -0700 Message-Id: <20211001190040.48086-1-shakeelb@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog Subject: [PATCH v2 1/2] memcg: flush stats only if updated From: Shakeel Butt To: Johannes Weiner , Michal Hocko Cc: " =?utf-8?q?Michal_Koutn=C3=BD?= " , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Shakeel Butt Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=gBoqBjum; spf=pass (imf04.hostedemail.com: domain of 331pXYQgKCAQwleoiipfksskpi.gsqpmry1-qqozego.svk@flex--shakeelb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=331pXYQgKCAQwleoiipfksskpi.gsqpmry1-qqozego.svk@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: CD9C550056DA X-Stat-Signature: zo5z3awkmm73der1imxwyih5uq5zyyt8 X-HE-Tag: 1633114848-780650 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: At the moment, the kernel flushes the memcg stats on every refault and also on every reclaim iteration. Although rstat maintains per-cpu update tree but on the flush the kernel still has to go through all the cpu rstat update tree to check if there is anything to flush. This patch adds the tracking on the stats update side to make flush side more clever by skipping the flush if there is no update. The stats update codepath is very sensitive performance wise for many workloads and benchmarks. So, we can not follow what the commit aa48e47e3906 ("memcg: infrastructure to flush memcg stats") did which was triggering async flush through queue_work() and caused a lot performance regression reports. That got reverted by the commit 1f828223b799 ("memcg: flush lruvec stats in the refault"). In this patch we kept the stats update codepath very minimal and let the stats reader side to flush the stats only when the updates are over a specific threshold. For now the threshold is (nr_cpus * CHARGE_BATCH). To evaluate the impact of this patch, an 8 GiB tmpfs file is created on a system with swap-on-zram and the file was pushed to swap through memory.force_empty interface. On reading the whole file, the memcg stat flush in the refault code path is triggered. With this patch, we observed 63% reduction in the read time of 8 GiB file. Signed-off-by: Shakeel Butt Acked-by: Johannes Weiner Cc: Michal Hocko Cc: "Michal Koutný" Cc: Andrew Morton Reviewed-by: Michal Koutný --- Changelog since v1: - Moved code and comment added as suggested by Johannes Weiner mm/memcontrol.c | 78 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 55 insertions(+), 23 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7c9d5703700e..25f55636ca37 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -103,11 +103,6 @@ static bool do_memsw_account(void) return !cgroup_subsys_on_dfl(memory_cgrp_subsys) && !cgroup_memory_noswap; } -/* memcg and lruvec stats flushing */ -static void flush_memcg_stats_dwork(struct work_struct *w); -static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); -static DEFINE_SPINLOCK(stats_flush_lock); - #define THRESHOLDS_EVENTS_TARGET 128 #define SOFTLIMIT_EVENTS_TARGET 1024 @@ -613,6 +608,56 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) return mz; } +/* + * memcg and lruvec stats flushing + * + * Many codepaths leading to stats update or read are performance sensitive and + * adding stats flushing in such codepaths is not desirable. So, to optimize the + * flushing the kernel does: + * + * 1) Periodically and asynchronously flush the stats every 2 seconds to not let + * rstat update tree grow unbounded. + * + * 2) Flush the stats synchronously on reader side only when there are more than + * (MEMCG_CHARGE_BATCH * nr_cpus) update events. Though this optimization + * will let stats be out of sync by atmost (MEMCG_CHARGE_BATCH * nr_cpus) but + * only for 2 seconds due to (1). + */ +static void flush_memcg_stats_dwork(struct work_struct *w); +static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); +static DEFINE_SPINLOCK(stats_flush_lock); +static DEFINE_PER_CPU(unsigned int, stats_updates); +static atomic_t stats_flush_threshold = ATOMIC_INIT(0); + +static inline void memcg_rstat_updated(struct mem_cgroup *memcg) +{ + cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); + if (!(__this_cpu_inc_return(stats_updates) % MEMCG_CHARGE_BATCH)) + atomic_inc(&stats_flush_threshold); +} + +static void __mem_cgroup_flush_stats(void) +{ + if (!spin_trylock(&stats_flush_lock)) + return; + + cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup); + atomic_set(&stats_flush_threshold, 0); + spin_unlock(&stats_flush_lock); +} + +void mem_cgroup_flush_stats(void) +{ + if (atomic_read(&stats_flush_threshold) > num_online_cpus()) + __mem_cgroup_flush_stats(); +} + +static void flush_memcg_stats_dwork(struct work_struct *w) +{ + mem_cgroup_flush_stats(); + queue_delayed_work(system_unbound_wq, &stats_flush_dwork, 2UL*HZ); +} + /** * __mod_memcg_state - update cgroup memory statistics * @memcg: the memory cgroup @@ -625,7 +670,7 @@ void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val) return; __this_cpu_add(memcg->vmstats_percpu->state[idx], val); - cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); + memcg_rstat_updated(memcg); } /* idx can be of type enum memcg_stat_item or node_stat_item. */ @@ -653,10 +698,12 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, memcg = pn->memcg; /* Update memcg */ - __mod_memcg_state(memcg, idx, val); + __this_cpu_add(memcg->vmstats_percpu->state[idx], val); /* Update lruvec */ __this_cpu_add(pn->lruvec_stats_percpu->state[idx], val); + + memcg_rstat_updated(memcg); } /** @@ -758,7 +805,7 @@ void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, return; __this_cpu_add(memcg->vmstats_percpu->events[idx], count); - cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); + memcg_rstat_updated(memcg); } static unsigned long memcg_events(struct mem_cgroup *memcg, int event) @@ -5342,21 +5389,6 @@ static void mem_cgroup_css_reset(struct cgroup_subsys_state *css) memcg_wb_domain_size_changed(memcg); } -void mem_cgroup_flush_stats(void) -{ - if (!spin_trylock(&stats_flush_lock)) - return; - - cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup); - spin_unlock(&stats_flush_lock); -} - -static void flush_memcg_stats_dwork(struct work_struct *w) -{ - mem_cgroup_flush_stats(); - queue_delayed_work(system_unbound_wq, &stats_flush_dwork, 2UL*HZ); -} - static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) { struct mem_cgroup *memcg = mem_cgroup_from_css(css); From patchwork Fri Oct 1 19:00:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 12531623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06C78C433EF for ; Fri, 1 Oct 2021 19:01:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 995AA61A57 for ; Fri, 1 Oct 2021 19:01:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 995AA61A57 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 420D694013C; Fri, 1 Oct 2021 15:01:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3D05F940121; Fri, 1 Oct 2021 15:01:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2BF7094013C; Fri, 1 Oct 2021 15:01:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0167.hostedemail.com [216.40.44.167]) by kanga.kvack.org (Postfix) with ESMTP id 1CF94940121 for ; Fri, 1 Oct 2021 15:01:00 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id B617D184558F1 for ; Fri, 1 Oct 2021 19:00:59 +0000 (UTC) X-FDA: 78648785838.22.39BB1D3 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf16.hostedemail.com (Postfix) with ESMTP id 6FD04F000369 for ; Fri, 1 Oct 2021 19:00:59 +0000 (UTC) Received: by mail-pg1-f202.google.com with SMTP id n22-20020a6563d6000000b0029261ffde9bso5574223pgv.22 for ; Fri, 01 Oct 2021 12:00:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=oA4D6jdogE7b/uxQa2O7tT0f2ZE2oMjI4z42jgRtcc8=; b=CwC2586zm2KkYzc/zO4TibKRvuGJhARCs0kdLQas3WOjM5X9DtAUcCSoLuF/k0hEsf Qud32VB81TbVE69nvAZnu/lgmWXRqUHZu/l5m1DD8yVCvRg2BkOk8Lh03i7d4azOgwpb k24FlBtLBbtPAiKELRevmpnJJOM0fSEVyyEu714vSz7JBVobN4ZXo0ibB2fuHo0v+1zE qkOw61zTfTidMcwKd8zyQSdHVPDEe23/Ls5YK2dVGWsVoV5I6WEIKNQ5M6ko8dRb/Qy8 kzC1BhoUajYaJ6MPwXM67juENfPQvzXTiRCBlfmuSt9qn+6M4EVnyaPf5aYX2hBpKhaX aqcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=oA4D6jdogE7b/uxQa2O7tT0f2ZE2oMjI4z42jgRtcc8=; b=sMQPtU7aOAhDdJHZ4G9+Jkbu7CoIyQ3XHt3lASYIBt+pFI58/hXBZWVGE6sUsb5wNK ix/Xl+4D5nJQyV+UL01T9R/C1xB9dKJa4h2QdyIKLZmol6Nh9z0vGI14C0k1zz5TIb2V vHDtM0SrCsyRueF0EGA4A3hujUSOGnNJTFn2FlluAjvShAvUZnyg7/qw7imhwlPH18Q7 9WGy8yhjzTGgd2doyQ6qEYOvLStzYHe39e09CfYGpq+Ba4SsIlUfCuCA7BVLgado/0Ln p8D42aKDSEHBL03hgo34uFXQCK9Jx4mVEbTwQNV+3K6VeclEnG133kQo5YWyr7qUn0Lq CDhA== X-Gm-Message-State: AOAM530rYlpF52HjNjDLeLkKraznGqgaUM7etX8E1ZSftG5ktjDWzcfJ +e14qQTyVmOVIxeMMbkVwil/t+wbhbJSWQ== X-Google-Smtp-Source: ABdhPJwPCkuJXkhCP4l5CIoXIN1d+5YKvrNGYw0bdo919HksBNmuVFGnfbbFfUCgWYWvrM3bDJKTfBFFUUIeiw== X-Received: from shakeelb.svl.corp.google.com ([2620:15c:2cd:202:af1a:f1cd:5283:aa67]) (user=shakeelb job=sendgmr) by 2002:a17:90b:4ac1:: with SMTP id mh1mr21669147pjb.238.1633114858394; Fri, 01 Oct 2021 12:00:58 -0700 (PDT) Date: Fri, 1 Oct 2021 12:00:40 -0700 In-Reply-To: <20211001190040.48086-1-shakeelb@google.com> Message-Id: <20211001190040.48086-2-shakeelb@google.com> Mime-Version: 1.0 References: <20211001190040.48086-1-shakeelb@google.com> X-Mailer: git-send-email 2.33.0.800.g4c38ced690-goog Subject: [PATCH v2 2/2] memcg: unify memcg stat flushing From: Shakeel Butt To: Johannes Weiner , Michal Hocko Cc: " =?utf-8?q?Michal_Koutn=C3=BD?= " , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Shakeel Butt X-Rspamd-Queue-Id: 6FD04F000369 X-Stat-Signature: cbbesrbwewq4rrgoxxnhsf4f8sg8n6tg Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=CwC2586z; spf=pass (imf16.hostedemail.com: domain of 36lpXYQgKCA87wpztt0qv33v0t.r310x29C-11zAprz.36v@flex--shakeelb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=36lpXYQgKCA87wpztt0qv33v0t.r310x29C-11zAprz.36v@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam06 X-HE-Tag: 1633114859-480736 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The memcg stats can be flushed in multiple context and potentially in parallel too. For example multiple parallel user space readers for memcg stats will contend on the rstat locks with each other. There is no need for that. We just need one flusher and everyone else can benefit. In addition after aa48e47e3906 ("memcg: infrastructure to flush memcg stats") the kernel periodically flush the memcg stats from the root, so, the other flushers will potentially have much less work to do. Signed-off-by: Shakeel Butt Acked-by: Johannes Weiner Cc: Michal Hocko Cc: "Michal Koutný" Cc: Andrew Morton --- Changelog since v1: - N/A mm/memcontrol.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 25f55636ca37..22d905f30a30 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -638,12 +638,14 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg) static void __mem_cgroup_flush_stats(void) { - if (!spin_trylock(&stats_flush_lock)) + unsigned long flag; + + if (!spin_trylock_irqsave(&stats_flush_lock, flag)) return; cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); - spin_unlock(&stats_flush_lock); + spin_unlock_irqrestore(&stats_flush_lock, flag); } void mem_cgroup_flush_stats(void) @@ -1462,7 +1464,7 @@ static char *memory_stat_format(struct mem_cgroup *memcg) * * Current memory state: */ - cgroup_rstat_flush(memcg->css.cgroup); + mem_cgroup_flush_stats(); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { u64 size; @@ -3566,8 +3568,7 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) unsigned long val; if (mem_cgroup_is_root(memcg)) { - /* mem_cgroup_threshold() calls here from irqsafe context */ - cgroup_rstat_flush_irqsafe(memcg->css.cgroup); + mem_cgroup_flush_stats(); val = memcg_page_state(memcg, NR_FILE_PAGES) + memcg_page_state(memcg, NR_ANON_MAPPED); if (swap) @@ -3948,7 +3949,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) int nid; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - cgroup_rstat_flush(memcg->css.cgroup); + mem_cgroup_flush_stats(); for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { seq_printf(m, "%s=%lu", stat->name, @@ -4020,7 +4021,7 @@ static int memcg_stat_show(struct seq_file *m, void *v) BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats)); - cgroup_rstat_flush(memcg->css.cgroup); + mem_cgroup_flush_stats(); for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { unsigned long nr; @@ -4523,7 +4524,7 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - cgroup_rstat_flush_irqsafe(memcg->css.cgroup); + mem_cgroup_flush_stats(); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); @@ -6408,7 +6409,7 @@ static int memory_numa_stat_show(struct seq_file *m, void *v) int i; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - cgroup_rstat_flush(memcg->css.cgroup); + mem_cgroup_flush_stats(); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { int nid;