From patchwork Tue Mar 12 22:33:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10850087 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6482A139A for ; Tue, 12 Mar 2019 22:34:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4DE4D29243 for ; Tue, 12 Mar 2019 22:34:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4195D296FE; Tue, 12 Mar 2019 22:34:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4D4A429243 for ; Tue, 12 Mar 2019 22:34:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A98F58E0005; Tue, 12 Mar 2019 18:34:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A22508E0002; Tue, 12 Mar 2019 18:34:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89F5A8E0005; Tue, 12 Mar 2019 18:34:11 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 3EBD58E0002 for ; Tue, 12 Mar 2019 18:34:11 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id 17so4242043pgw.12 for ; Tue, 12 Mar 2019 15:34:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=CNCy2SGf01agfPIUoI0mNB3tC3JUsDAW7lCdNygAAfg=; b=NriGDOs3o5gCOji6PgLa8A5qSp81tKy1NdRakdaiw02Y94ZlvoUnciC20r96bABKfY drr5v7LgrKmBfkRqREQwTuCSeCws6ytmzkcKDTKxYydZUHxUdKdR8C2FRy3VwWKIhRT5 Dlyr0OeWHU7PLP7+5PgZH2jOIpif3V5/ISvkZ7A96EWuH66CNBpaxmR/zJV3z3rx1aZr q6Ws2cvqGcfLAgmP+ng+vVsLRE0dcGIcOWnaqSu9T3r3QvosAi6mPH+3Yul/LBkMzyaI e3reD0UGb0Gf4LJZ+4csFKDBD1q8Wbp7nYBdGHq5/z3hE1EzZWRMV6VsB+xkItys/XTH ggKQ== X-Gm-Message-State: APjAAAVvm2jAqBeEluBUPmsdFDhSm20HWy+EDkX2iKhII9MrQC8aRAOA IrAMUXysO3+waFS7yHj+eA9iJLn60c/ZETEr58i++0NvPEf65ugotY/2b5CIUpp495PBxUO0PJB 3LTcSzo8a6LjTVaOq540Qvl2h3YsKNnBAyFBSDFbsbNCwRO+SjfYwCZrESfBSzvp5uU5rzog/oZ lCJ43hRtZfF6GLV+nmtaFPQ/h7jZoMpP7O8SBz7hMlLflUfWFZgrFSUOpC2LajnTzaIGIpWKpd6 D1mNI+X0HdAkMzvnAQG3MVfK8meu5IFB6Mo1LCQ/QIP3B7ponJeXz44m3TEOmVw4kW5psFfI400 BWcIeW+cgR0ikWHUcN7hj14hfBbW6l9TCCHB/AA8JArt5OzYbuAWePQxWvMbADvfN84ITSXVC0s W X-Received: by 2002:a65:6105:: with SMTP id z5mr2242389pgu.434.1552430050901; Tue, 12 Mar 2019 15:34:10 -0700 (PDT) X-Received: by 2002:a65:6105:: with SMTP id z5mr2242318pgu.434.1552430049690; Tue, 12 Mar 2019 15:34:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552430049; cv=none; d=google.com; s=arc-20160816; b=Pp3YK1f6HZYLXD1VOWpZIRPSdAs+bZlc2YXLY0ef+MI3YS/G1gMBnlbjLrS2i0DcIp z0yrLEMdOoRfjaC2UTjNP+hU21S0/kxguYxiXuMaSz3qSEK0I7p+WCay6VdaSJkc9CI+ RgSoBU0AlicCMKoy0Fpuyr57SsuPLEr+pQyxNcxeDkvLmYRN1+5Llh0lDHZpBDYdGOZp ZTW7R+qr40xJzlD8jURmJsnOG9NJmafwTBLCnn2wu8sCJuHAoDwYjvIRvjkxRHGw5ozl LlKi46x/yQbatLZNGy047+W761JH6mcN/TUrfjT680qewJvm7t6uH24+vvXLNjHLIttu +/rA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=CNCy2SGf01agfPIUoI0mNB3tC3JUsDAW7lCdNygAAfg=; b=U0H2fDUhX2v84l2YCKsQfzpNOGF21L2K7HL/Bh7J3aR6ZAJxU82Br1MXd7wPlPcrh7 WrkRW+sI8tJJ1ybzM4UkdNlNw4M1/KA9XiKmwFPhzkEIrMlEwig/EFv/0R5b06DZ4Xu1 G29OhRBgWJX1reArTJ3TZOcw87cNqFYR79m3fAlsUyJWXTTNckDX8P48ySe3tLpayDj/ b14x9M/B6ArZIWTq01g5OS9JajBQ8qU0UXQzmvCoAGu85JrCoTim9M6D4ZErLZ8dDDx8 p1IjA14jEUrbhf+8RUKonRQSOiyQURjRCQulhdVZJU3P7wyICuCWQvEXcz/7A0s/Lnbk WJWw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=lLhnV2bS; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id e63sor15418007pgc.0.2019.03.12.15.34.09 for (Google Transport Security); Tue, 12 Mar 2019 15:34:09 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=lLhnV2bS; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CNCy2SGf01agfPIUoI0mNB3tC3JUsDAW7lCdNygAAfg=; b=lLhnV2bSVxytqXr7GTraMYKDSJWcLPf1AJ4jhxWyzW7riU8COKYEbht7n1aK3jzLua hZFskTIFmO4ddHn0VBmA7bYJ+lFl+SNXELhkeYW3fDi7ozLe48oYQd5x2azT3GpXXV73 snsgq+EMuVdW43hSpU61qtsuGnlBNF5dIQo3wV/qA7depVjbEvySF6D3Y9URSlzsJRuS rgyqqo6yqBpRRcJIfcmjeY7+PbPOrCKCYDFZxdiMS7Vy6dY7/+R/r+L/x6Uc4M5/Ha8t 72sE8kqCVR8ZTKGrKod1FJCx7fmZL0MPdyJ3c7ggY/ltDF658f3+SEPS+sbTl9bL6P84 xIaA== X-Google-Smtp-Source: APXvYqyd55NWzBFVlPqDjX0SOkA4t3ShH69woO4U/IzykAEYfA+93dC952jtmZXrq8qIWhYCNo/o3w== X-Received: by 2002:a63:dc4a:: with SMTP id f10mr6801954pgj.231.1552430049131; Tue, 12 Mar 2019 15:34:09 -0700 (PDT) Received: from tower.thefacebook.com ([2620:10d:c090:200::1:3203]) by smtp.gmail.com with ESMTPSA id i13sm14680592pfo.106.2019.03.12.15.34.07 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 12 Mar 2019 15:34:08 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH v2 1/6] mm: prepare to premature release of memcg->vmstats_percpu Date: Tue, 12 Mar 2019 15:33:58 -0700 Message-Id: <20190312223404.28665-2-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190312223404.28665-1-guro@fb.com> References: <20190312223404.28665-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Prepare to handle premature release of memcg->vmstats_percpu data. Currently it's a generic pointer which is expected to be non-NULL during the whole life time of a memcg. Switch over to the rcu-protected pointer, and carefully check it for being non-NULL. This change is a required step towards dynamic premature release of percpu memcg data. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 40 +++++++++++++++++------- mm/memcontrol.c | 62 +++++++++++++++++++++++++++++--------- 2 files changed, 77 insertions(+), 25 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 534267947664..05ca77767c6a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -274,7 +274,7 @@ struct mem_cgroup { struct task_struct *move_lock_task; /* memory.stat */ - struct memcg_vmstats_percpu __percpu *vmstats_percpu; + struct memcg_vmstats_percpu __rcu /* __percpu */ *vmstats_percpu; MEMCG_PADDING(_pad2_); @@ -597,17 +597,26 @@ static inline unsigned long memcg_page_state(struct mem_cgroup *memcg, static inline void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; long x; if (mem_cgroup_disabled()) return; - x = val + __this_cpu_read(memcg->vmstats_percpu->stat[idx]); - if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { - atomic_long_add(x, &memcg->vmstats[idx]); - x = 0; + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + if (likely(vmstats_percpu)) { + x = val + __this_cpu_read(vmstats_percpu->stat[idx]); + if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { + atomic_long_add(x, &memcg->vmstats[idx]); + x = 0; + } + __this_cpu_write(vmstats_percpu->stat[idx], x); + } else { + atomic_long_add(val, &memcg->vmstats[idx]); } - __this_cpu_write(memcg->vmstats_percpu->stat[idx], x); + rcu_read_unlock(); } /* idx can be of type enum memcg_stat_item or node_stat_item */ @@ -740,17 +749,26 @@ static inline void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, unsigned long count) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; unsigned long x; if (mem_cgroup_disabled()) return; - x = count + __this_cpu_read(memcg->vmstats_percpu->events[idx]); - if (unlikely(x > MEMCG_CHARGE_BATCH)) { - atomic_long_add(x, &memcg->vmevents[idx]); - x = 0; + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + if (likely(vmstats_percpu)) { + x = count + __this_cpu_read(vmstats_percpu->events[idx]); + if (unlikely(x > MEMCG_CHARGE_BATCH)) { + atomic_long_add(x, &memcg->vmevents[idx]); + x = 0; + } + __this_cpu_write(vmstats_percpu->events[idx], x); + } else { + atomic_long_add(count, &memcg->vmevents[idx]); } - __this_cpu_write(memcg->vmstats_percpu->events[idx], x); + rcu_read_unlock(); } static inline void count_memcg_events(struct mem_cgroup *memcg, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c532f8685aa3..803c772f354b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -697,6 +697,8 @@ static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, struct page *page, bool compound, int nr_pages) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; + /* * Here, RSS means 'mapped anon' and anon's SwapCache. Shmem/tmpfs is * counted as CACHE even if it's on ANON LRU. @@ -722,7 +724,12 @@ static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, nr_pages = -nr_pages; /* for event */ } - __this_cpu_add(memcg->vmstats_percpu->nr_page_events, nr_pages); + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + if (likely(vmstats_percpu)) + __this_cpu_add(vmstats_percpu->nr_page_events, nr_pages); + rcu_read_unlock(); } unsigned long mem_cgroup_node_nr_lru_pages(struct mem_cgroup *memcg, @@ -756,10 +763,18 @@ static unsigned long mem_cgroup_nr_lru_pages(struct mem_cgroup *memcg, static bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg, enum mem_cgroup_events_target target) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; unsigned long val, next; + bool ret = false; - val = __this_cpu_read(memcg->vmstats_percpu->nr_page_events); - next = __this_cpu_read(memcg->vmstats_percpu->targets[target]); + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + if (!vmstats_percpu) + goto out; + + val = __this_cpu_read(vmstats_percpu->nr_page_events); + next = __this_cpu_read(vmstats_percpu->targets[target]); /* from time_after() in jiffies.h */ if ((long)(next - val) < 0) { switch (target) { @@ -775,10 +790,12 @@ static bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg, default: break; } - __this_cpu_write(memcg->vmstats_percpu->targets[target], next); - return true; + __this_cpu_write(vmstats_percpu->targets[target], next); + ret = true; } - return false; +out: + rcu_read_unlock(); + return ret; } /* @@ -2104,22 +2121,29 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) static int memcg_hotplug_cpu_dead(unsigned int cpu) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; struct memcg_stock_pcp *stock; struct mem_cgroup *memcg; stock = &per_cpu(memcg_stock, cpu); drain_stock(stock); + rcu_read_lock(); for_each_mem_cgroup(memcg) { int i; + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + for (i = 0; i < MEMCG_NR_STAT; i++) { int nid; long x; - x = this_cpu_xchg(memcg->vmstats_percpu->stat[i], 0); - if (x) - atomic_long_add(x, &memcg->vmstats[i]); + if (vmstats_percpu) { + x = this_cpu_xchg(vmstats_percpu->stat[i], 0); + if (x) + atomic_long_add(x, &memcg->vmstats[i]); + } if (i >= NR_VM_NODE_STAT_ITEMS) continue; @@ -2137,11 +2161,14 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) for (i = 0; i < NR_VM_EVENT_ITEMS; i++) { long x; - x = this_cpu_xchg(memcg->vmstats_percpu->events[i], 0); - if (x) - atomic_long_add(x, &memcg->vmevents[i]); + if (vmstats_percpu) { + x = this_cpu_xchg(vmstats_percpu->events[i], 0); + if (x) + atomic_long_add(x, &memcg->vmevents[i]); + } } } + rcu_read_unlock(); return 0; } @@ -4464,7 +4491,8 @@ static struct mem_cgroup *mem_cgroup_alloc(void) if (memcg->id.id < 0) goto fail; - memcg->vmstats_percpu = alloc_percpu(struct memcg_vmstats_percpu); + rcu_assign_pointer(memcg->vmstats_percpu, + alloc_percpu(struct memcg_vmstats_percpu)); if (!memcg->vmstats_percpu) goto fail; @@ -6054,6 +6082,7 @@ static void uncharge_batch(const struct uncharge_gather *ug) { unsigned long nr_pages = ug->nr_anon + ug->nr_file + ug->nr_kmem; unsigned long flags; + struct memcg_vmstats_percpu __percpu *vmstats_percpu; if (!mem_cgroup_is_root(ug->memcg)) { page_counter_uncharge(&ug->memcg->memory, nr_pages); @@ -6070,7 +6099,12 @@ static void uncharge_batch(const struct uncharge_gather *ug) __mod_memcg_state(ug->memcg, MEMCG_RSS_HUGE, -ug->nr_huge); __mod_memcg_state(ug->memcg, NR_SHMEM, -ug->nr_shmem); __count_memcg_events(ug->memcg, PGPGOUT, ug->pgpgout); - __this_cpu_add(ug->memcg->vmstats_percpu->nr_page_events, nr_pages); + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(ug->memcg->vmstats_percpu); + if (likely(vmstats_percpu)) + __this_cpu_add(vmstats_percpu->nr_page_events, nr_pages); + rcu_read_unlock(); memcg_check_events(ug->memcg, ug->dummy_page); local_irq_restore(flags); From patchwork Tue Mar 12 22:33:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10850089 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 787881515 for ; Tue, 12 Mar 2019 22:34:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 62CEF29243 for ; Tue, 12 Mar 2019 22:34:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 57197296FE; Tue, 12 Mar 2019 22:34:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C64E529243 for ; Tue, 12 Mar 2019 22:34:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 545A68E0007; Tue, 12 Mar 2019 18:34:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4CC4D8E0002; Tue, 12 Mar 2019 18:34:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3491D8E0007; Tue, 12 Mar 2019 18:34:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id DD7FB8E0002 for ; Tue, 12 Mar 2019 18:34:12 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id d5so4712331pfo.5 for ; Tue, 12 Mar 2019 15:34:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=oz/TqQdvSWg5Jr50nmxxA8VLsKG86DG+TYlsVRNN9G4=; b=MSP8bjw9NJS60OA/OWQjFoIzCPeQR6xJad9pTRpmmlB76at90onUSayfJb9xJNRM92 H1BeLcEYfwqtsP4UfXDMorK28V8CnwxCClgH3pnkwSmZ9zhYWWq6A27oOFAEFtxm20pv KzqjT2QalS6Aa5c2ef0N+vagjc3mVrhPsPDDa+WZBIYPKb5HunYJL4+U9rj39JNvyAby uR0hweunGMJ4yhJZia4pG7CRuVw7caD9u8avO1s5zM5pZUyxepC7/kSUGQliRiDgXOAY ennP2/L2grBFxRLc8n/5j2lAGWWP7flGWUZLGpQDq3YxA33+JMBLqStrM0t42O/v+JFx elmA== X-Gm-Message-State: APjAAAWF9vXCm20/yRhIJv40lS+pHX/BZzSIsw3Q26CHjStRhaUdMf/D eAPe7zIwKwSrfkE139ImXlzcY416bHj2c/H+XtLBgaqmxZSswi0Nd2hgQQCDnqvpQKC34bgGVRE OBYFTprmwYI6E6RqthpBSoyYhmpAmHSHNsu/IBAdOBao6yvu9fpidg7M8mldGeI9PqLYTJDFr9e t5FevklZTrm49yakSnCf8dB30nGJVty8TjivBj6xNtAHQoMC+Z1n4TEWWwuwwBSOJ0IxaY74OuE +j2jolGzlKJZVaeZD1Z4D+Um/Dx2S0qWS8QB7YoLMa8oBXmAf8Q+idwnfmkcSrPX0hNP1i+Wm+j 5D51Z7QQ4VgXTfb59Ic9pgbesieoWpbys8xd2vL7wWm6bXOoMTQe0t5ewuhtagJyc6rI/KFvQBd Y X-Received: by 2002:a65:4542:: with SMTP id x2mr22231750pgr.65.1552430052574; Tue, 12 Mar 2019 15:34:12 -0700 (PDT) X-Received: by 2002:a65:4542:: with SMTP id x2mr22231663pgr.65.1552430051044; Tue, 12 Mar 2019 15:34:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552430051; cv=none; d=google.com; s=arc-20160816; b=DcU9bbcgT8LAY5JLtvkZ1oBqzwYbkYMn9Yw3CLXaKai2qzdYbLaisJhWXDNOB51xbN 3FQo28GczOIH5jLhwh2dJzA1YqQpHzuys44kNfkOreMMJVMa00Ti9toPplrAD6Nt4YEF ZbgzRu3WCYf6sEYmSnZWc8eNnroz4KGvItAqeOnc6kMUeJhF0kEpxhi4Qoer643LlTDF 8O9dvR1jghLlphnrvKRhcSt5pbpbaL1Y/IyE75NyjlwBbhcHgdFvsRmWgjw9MjeyKMp8 2/pil9QgWXFZnXridbpb0R5Jm0i/aAnS81owDohk5Pq0F1GP+W2vNAkSPsr/NzhjLkMu yz2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=oz/TqQdvSWg5Jr50nmxxA8VLsKG86DG+TYlsVRNN9G4=; b=KViXYDbj3qDRIKvMn0/HPI1z0kV8nBnwP5AiWXwriPDl2bM+2DE9Q/8gx4Q1m8x1CT r+wUVtocLSQzZS/LsP+vvPfsKK3HcaG4tnbBk4PEV7y3rFyCvg/FuVJ61TvEAy7zbc7q EXx6JvzoStZ6flLQxSKRjW954ayPDeTilanAJgWUuhzesrVCcBp6z/6YK8KEtg7BIdJx n2PoHewpK3pNbnD+BpKiMSjtNdwpZ4UDLX4lD4geO7fok1B2CRliFI08i/QqK4I+/QwJ kqTWtfT1rt6CK+Ob2VxMSwfHRLFCUdQWO+/Y2Rl+E8CbRQwImnQsfpCBib9GsncoWY9l k30w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=LXstkKlU; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id q20sor15900074pgh.60.2019.03.12.15.34.10 for (Google Transport Security); Tue, 12 Mar 2019 15:34:11 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=LXstkKlU; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oz/TqQdvSWg5Jr50nmxxA8VLsKG86DG+TYlsVRNN9G4=; b=LXstkKlUo0CnIa3Kc9zF+D4gAlOlVCACVZQL55UAPdMLJPpES1HylsZEk9D3A62uSi fdBXQCsgnT34wO+Y3YeEVDZDz8SkhJqKZzNp1SLlupnP8AMhk6eQJq0gJEcndGpfsCyy gICdN5uWruLEXqIic9Da9FeFoaleFCceVwVaTlcBB5XxD0QQ9PtvlAWeqrQSIQVAEdW8 NwRAy+W1N0D9imr2hWnXWox7pYqy0heUoeImh6pY8pQi5y3XmlcbCifj9BumcodY0pl9 dhp6sT026DZ5kN93hNlwp4Tt1mOJtPMBEgrfBmcNZ3FLzMxDcbFGfjwXQtA/lQ6FqUiF 1WCw== X-Google-Smtp-Source: APXvYqxaTl30AAKSU0hP96o65+VdoqQtZLA+IvuzkQRRVNKj8cm5sDUVP3whhZO805D+hSJdmfCc0A== X-Received: by 2002:a63:3541:: with SMTP id c62mr36759041pga.157.1552430050528; Tue, 12 Mar 2019 15:34:10 -0700 (PDT) Received: from tower.thefacebook.com ([2620:10d:c090:200::1:3203]) by smtp.gmail.com with ESMTPSA id i13sm14680592pfo.106.2019.03.12.15.34.09 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 12 Mar 2019 15:34:09 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH v2 2/6] mm: prepare to premature release of per-node lruvec_stat_cpu Date: Tue, 12 Mar 2019 15:33:59 -0700 Message-Id: <20190312223404.28665-3-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190312223404.28665-1-guro@fb.com> References: <20190312223404.28665-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Similar to the memcg's vmstats_percpu, per-memcg per-node stats consists of percpu- and atomic counterparts, and we do expect that both coexist during the whole life-cycle of the memcg. To prepare for a premature release of percpu per-node data, let's pretend that lruvec_stat_cpu is a rcu-protected pointer, which can be NULL. This patch adds corresponding checks whenever required. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 21 +++++++++++++++------ mm/memcontrol.c | 14 +++++++++++--- 2 files changed, 26 insertions(+), 9 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 05ca77767c6a..8ac04632002a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -126,7 +126,7 @@ struct memcg_shrinker_map { struct mem_cgroup_per_node { struct lruvec lruvec; - struct lruvec_stat __percpu *lruvec_stat_cpu; + struct lruvec_stat __rcu /* __percpu */ *lruvec_stat_cpu; atomic_long_t lruvec_stat[NR_VM_NODE_STAT_ITEMS]; unsigned long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; @@ -682,6 +682,7 @@ static inline unsigned long lruvec_page_state(struct lruvec *lruvec, static inline void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val) { + struct lruvec_stat __percpu *lruvec_stat_cpu; struct mem_cgroup_per_node *pn; long x; @@ -697,12 +698,20 @@ static inline void __mod_lruvec_state(struct lruvec *lruvec, __mod_memcg_state(pn->memcg, idx, val); /* Update lruvec */ - x = val + __this_cpu_read(pn->lruvec_stat_cpu->count[idx]); - if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { - atomic_long_add(x, &pn->lruvec_stat[idx]); - x = 0; + rcu_read_lock(); + lruvec_stat_cpu = (struct lruvec_stat __percpu *) + rcu_dereference(pn->lruvec_stat_cpu); + if (likely(lruvec_stat_cpu)) { + x = val + __this_cpu_read(lruvec_stat_cpu->count[idx]); + if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { + atomic_long_add(x, &pn->lruvec_stat[idx]); + x = 0; + } + __this_cpu_write(lruvec_stat_cpu->count[idx], x); + } else { + atomic_long_add(val, &pn->lruvec_stat[idx]); } - __this_cpu_write(pn->lruvec_stat_cpu->count[idx], x); + rcu_read_unlock(); } static inline void mod_lruvec_state(struct lruvec *lruvec, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 803c772f354b..5ef4098f3f8d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2122,6 +2122,7 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) static int memcg_hotplug_cpu_dead(unsigned int cpu) { struct memcg_vmstats_percpu __percpu *vmstats_percpu; + struct lruvec_stat __percpu *lruvec_stat_cpu; struct memcg_stock_pcp *stock; struct mem_cgroup *memcg; @@ -2152,7 +2153,12 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) struct mem_cgroup_per_node *pn; pn = mem_cgroup_nodeinfo(memcg, nid); - x = this_cpu_xchg(pn->lruvec_stat_cpu->count[i], 0); + + lruvec_stat_cpu = (struct lruvec_stat __percpu*) + rcu_dereference(pn->lruvec_stat_cpu); + if (!lruvec_stat_cpu) + continue; + x = this_cpu_xchg(lruvec_stat_cpu->count[i], 0); if (x) atomic_long_add(x, &pn->lruvec_stat[i]); } @@ -4414,6 +4420,7 @@ struct mem_cgroup *mem_cgroup_from_id(unsigned short id) static int alloc_mem_cgroup_per_node_info(struct mem_cgroup *memcg, int node) { + struct lruvec_stat __percpu *lruvec_stat_cpu; struct mem_cgroup_per_node *pn; int tmp = node; /* @@ -4430,11 +4437,12 @@ static int alloc_mem_cgroup_per_node_info(struct mem_cgroup *memcg, int node) if (!pn) return 1; - pn->lruvec_stat_cpu = alloc_percpu(struct lruvec_stat); - if (!pn->lruvec_stat_cpu) { + lruvec_stat_cpu = alloc_percpu(struct lruvec_stat); + if (!lruvec_stat_cpu) { kfree(pn); return 1; } + rcu_assign_pointer(pn->lruvec_stat_cpu, lruvec_stat_cpu); lruvec_init(&pn->lruvec); pn->usage_in_excess = 0; From patchwork Tue Mar 12 22:34:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10850091 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0B72713B5 for ; Tue, 12 Mar 2019 22:34:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EC20C29243 for ; Tue, 12 Mar 2019 22:34:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DD962296FE; Tue, 12 Mar 2019 22:34:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5F74F29243 for ; Tue, 12 Mar 2019 22:34:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 372788E0009; Tue, 12 Mar 2019 18:34:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0FD188E0002; Tue, 12 Mar 2019 18:34:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E47788E0009; Tue, 12 Mar 2019 18:34:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 9B30E8E0002 for ; Tue, 12 Mar 2019 18:34:14 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id d128so4253953pgc.8 for ; Tue, 12 Mar 2019 15:34:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=WITBUxeGSIQmxb1SdGnY+GHNprTyqfSQ+9QDULNZwIA=; b=Zll7H4ujJS2lnuK/1bE2UrkAhJta2ZxoMQAdEg5YwI7PANLfKvAenI3PPJ13yknRT4 6qsThl38qB1wwmuwghfx6EfavzJ4eR0KUDNItw0sR0a+dfaOq+0DT9U7hH7NI9CicSYb 8he6l2Cx7RNgZOajS+qkbqylc5IvAlBiThVGQFc6uaWrtT2fF6lCSfcPlokaC6rXxQ1m FTqnN+TYmeZB7wVGK4lz0TEigduZCorTEtzTl7mTghG+0ijNgSXN9Y/cWFiINSQ23ma4 muG4QzkGT6YfuI1nzyA3G+jeFH27/0JnoOjPCyjYJLQ837tqBr2iu2885fYdz8P/8ToX EypA== X-Gm-Message-State: APjAAAXJrGRTm+QV2lPF5nM5w+BqHz7nJl/29joe3xcey+WNMKYebjLm L4LQgrE0QYiY30Rv5znWn/j138GmFbEPImdGzeXlY1sWha0DAf/oDyP3f8WPfHNjKKezezN3GO7 ksXnA3E0mZxkgJi3jzC8+6kumRmWTvfvev00cLt9xnGbmQRL+Mj6v5Ftw6/DILQxbWurYCPNgQa 529+svN4+JZWHlksUTKq/ZqrHiabIlKvwGWGxMnwQWeFt9QvpP/rElOz7EdYl9Ln44ggtxB4uf0 OYiGWkHsTCY+lf2BYWuEP3xWco1zn11WvS4awYPnJb45EhH1LGclG4nx0ZlW+yCVocqiw0AKx/6 7ssALXF0/k7as3V3daSifJ7jRiKDJI/zuaK2xJeKAzGrUL5/pl4jmiWpVEKJ/2VlzhpBVTDc2e/ N X-Received: by 2002:a17:902:e90b:: with SMTP id cs11mr41117478plb.197.1552430054298; Tue, 12 Mar 2019 15:34:14 -0700 (PDT) X-Received: by 2002:a17:902:e90b:: with SMTP id cs11mr41117391plb.197.1552430052779; Tue, 12 Mar 2019 15:34:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552430052; cv=none; d=google.com; s=arc-20160816; b=CSOCvV0vhYZvp+f7XGRIUbcRcwc/mWP2eOPEBVDrCu1SDW6jeZ7gGf6CekV1aJffzF n9OHdYyQ95kNdT1TghtK8uZDIrmHGwLcmF1xSRvt12/NTpK/KrU94+sNwcR+aFRcnhAk 5aEnrOdRKePc8gAEQnAARQSikAOWw9rjVn8JC2BYDGgeH44i96apKtPRxJVvjngn+Ucw HLzrjF06n8GClQBquG2TyuR+7/8HpOd6UhdPWLGEzCN7xuCiMxje03qDbBNb+fXeLpEn olD4oTwRW5lu+elbgHbQYc9B4bxU7Gv2xhL1XjwCurLTR1hwtzObXV+s+mglYiNR9+hy xIMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=WITBUxeGSIQmxb1SdGnY+GHNprTyqfSQ+9QDULNZwIA=; b=LDBN9q3wmvm+JNrH0GY+Np+QkHKbiZtB725lHKdSFzTGovcI74CcNuNduUyLNbQwY3 +KFwFmQirvuQu+DDRa+icGZT2DjXIViDyOF8dA6d7iBlUZvtojqMi9Ek6KUSTwoN0BHe Aggl9LPZZ25ioIILq4NymWQLeWpPV7VA7Kaxs74Rk/X/MBJx7ugu8S73ShbEAyg++vNH OdlYcqJQqwA4JXCY9Wf4r/ohcTs/lnuohvT0a8MiXZvoecNc/Xf9Ck4Xmv9nBR0sQ1NH 7lb4QNpk+V5ApBzrTXJaYSU2/Af++8NFaOoc1DaXpzJVTL15dMaugyl9b+dMjFknc6Pu aLTw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=gw9+V7Aw; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id r22sor3528196pgg.11.2019.03.12.15.34.12 for (Google Transport Security); Tue, 12 Mar 2019 15:34:12 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=gw9+V7Aw; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WITBUxeGSIQmxb1SdGnY+GHNprTyqfSQ+9QDULNZwIA=; b=gw9+V7AwT9BhELUS3izGcpK1i9cnVisDnS7huiwO0ueQQsRqDBRQcs9no5AD+mhOyZ exFVmMPw2TSbuSfU8RXhszBVIq2IRNjzA4mtCnLvggeTueonNHnhfmfvwJ+wWdMmuxif yfx6/r60CWxsOd4dvu0akIYSbexXfK9B4teuwUDLfkU5M1U/dIBlf6sk6I6RmOwOBDVZ ja8lbvLfjJk5bfajgTjeRrYumvVtsEtu3PLcP5IKHOiElvMZD5ehmsZ2yc5wfpWM81N6 2TH47iyF59ke+rxd5q+hAEHm1suK/m//n0NPrp9baJtPSctDOiPqvetpX4V/0zHsad1W E3sQ== X-Google-Smtp-Source: APXvYqwOCTEzdCd46KHhbLhv+IK8K9xe2Y6ShvmFeW2wxLpyyyNkLKjIUagfAk+pvoSIv5NI7DyUlg== X-Received: by 2002:a63:7e0e:: with SMTP id z14mr37330494pgc.436.1552430052235; Tue, 12 Mar 2019 15:34:12 -0700 (PDT) Received: from tower.thefacebook.com ([2620:10d:c090:200::1:3203]) by smtp.gmail.com with ESMTPSA id i13sm14680592pfo.106.2019.03.12.15.34.10 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 12 Mar 2019 15:34:11 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH v2 3/6] mm: release memcg percpu data prematurely Date: Tue, 12 Mar 2019 15:34:00 -0700 Message-Id: <20190312223404.28665-4-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190312223404.28665-1-guro@fb.com> References: <20190312223404.28665-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP To reduce the memory footprint of a dying memory cgroup, let's release massive percpu data (vmstats_percpu) as early as possible, and use atomic counterparts instead. A dying cgroup can remain in the dying state for quite a long time, being pinned in memory by any reference. For example, if a page mlocked by some other cgroup, is charged to the dying cgroup, it won't go away until the page will be released. A dying memory cgroup can have some memory activity (e.g. dirty pages can be flushed after cgroup removal), but in general it's not expected to be very active in comparison to living cgroups. So reducing the memory footprint by releasing percpu data and switching over to atomics seems to be a good trade off. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 4 ++++ mm/memcontrol.c | 24 +++++++++++++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 8ac04632002a..569337514230 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -275,6 +275,10 @@ struct mem_cgroup { /* memory.stat */ struct memcg_vmstats_percpu __rcu /* __percpu */ *vmstats_percpu; + struct memcg_vmstats_percpu __percpu *vmstats_percpu_offlined; + + /* used to release non-used percpu memory */ + struct rcu_head rcu; MEMCG_PADDING(_pad2_); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5ef4098f3f8d..efd5bc131a38 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4470,7 +4470,7 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg) for_each_node(node) free_mem_cgroup_per_node_info(memcg, node); - free_percpu(memcg->vmstats_percpu); + WARN_ON_ONCE(memcg->vmstats_percpu != NULL); kfree(memcg); } @@ -4613,6 +4613,26 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) return 0; } +static void percpu_rcu_free(struct rcu_head *rcu) +{ + struct mem_cgroup *memcg = container_of(rcu, struct mem_cgroup, rcu); + + free_percpu(memcg->vmstats_percpu_offlined); + WARN_ON_ONCE(memcg->vmstats_percpu); + + css_put(&memcg->css); +} + +static void mem_cgroup_offline_percpu(struct mem_cgroup *memcg) +{ + memcg->vmstats_percpu_offlined = (struct memcg_vmstats_percpu __percpu*) + rcu_dereference(memcg->vmstats_percpu); + rcu_assign_pointer(memcg->vmstats_percpu, NULL); + + css_get(&memcg->css); + call_rcu(&memcg->rcu, percpu_rcu_free); +} + static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) { struct mem_cgroup *memcg = mem_cgroup_from_css(css); @@ -4639,6 +4659,8 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) drain_all_stock(memcg); mem_cgroup_id_put(memcg); + + mem_cgroup_offline_percpu(memcg); } static void mem_cgroup_css_released(struct cgroup_subsys_state *css) From patchwork Tue Mar 12 22:34:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10850093 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7E000139A for ; Tue, 12 Mar 2019 22:34:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 69F8229243 for ; Tue, 12 Mar 2019 22:34:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5E0F6296FE; Tue, 12 Mar 2019 22:34:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB75029243 for ; Tue, 12 Mar 2019 22:34:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 34F5C8E0011; Tue, 12 Mar 2019 18:34:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2373D8E0002; Tue, 12 Mar 2019 18:34:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB78F8E0011; Tue, 12 Mar 2019 18:34:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id A83608E0002 for ; Tue, 12 Mar 2019 18:34:15 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id m10so4708840pfj.4 for ; Tue, 12 Mar 2019 15:34:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ZKgL4zeK+z0St01hNMYKSt6dKMvw92MkjAHduvDs3M4=; b=EpssZ/ABixEePqNf0mAttQvDUFYopscv4NGHaBAjHVP64nOcJN/v+aKRygKQtm2vHb QMDpvgcxTrDSHxPsPW1oLvDsRJfGk7L/R7xTNS8I4E6HrozIlQ6QIa2/65zFUMmzV5Ih ck3FaIh3FvVPUI9l6dOxJHfdzdtb73smNHGuvBbUbUTS8lmlBZ2FBva9QZsduNDfjeoa FWootkH5oTHt++SI5ug35rHT2U0yGhu13g0UvQV9GKnVkQWyoRwjZsGy95X5z1G5jtBv ZcK/nLMd+8JMLRSalEALM1OSqB2ewAexa2XQ4RjUuk5fT+Fko6Lm0M1ZLE+hr77llEiL Ou6w== X-Gm-Message-State: APjAAAWnw7+kHk5gZRixE3VEBdG1dMl1bpb9zB5QLHjn8Z00RUTE1+nZ tClR3N7GRdeup3VSF7EW0wMQM7QLMdglmtSUwCxcselJOPH1EOCbu0SfzjV14vPpUzhuXjZlyu5 uSk3Jk6aS60uTbKrnf2f+KS+KkO/QH9jLBQVt696vNhvnjVso2uQig6lzEllI/f9yts/1BMpAgW s/QX9L07venv/ptiDToSaHVkISzl9+nYoNXFFxowB9GegwbxaJmPRdqF/YPlM0K4Tn8b6DfSGVw rOp1WrXphUJiNHvwescbYk52s99oIKVpqM5xfcznO1h1Vo+dFetr5mG+ToTM90c5oRDF5RBTaAG rO59aEeFiDcZ7K+La0YBaZJV6k8JpOqqV6kkC1etabjzLobYf8neh947R+QbctRrjAmTTr8ftDX P X-Received: by 2002:a62:1d8c:: with SMTP id d134mr1634666pfd.185.1552430055342; Tue, 12 Mar 2019 15:34:15 -0700 (PDT) X-Received: by 2002:a62:1d8c:: with SMTP id d134mr1634579pfd.185.1552430053923; Tue, 12 Mar 2019 15:34:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552430053; cv=none; d=google.com; s=arc-20160816; b=x6e+GEbGo5UYqb/DsEsXxOBkJUjLuoESruHpnMJi2/MBJ1F/1tpPYjPhP2LHSK8z3G tPRRFn/KudGOtNxp4tnJ9aLFOtBhLEV2NxDFlaLzkh87JVtUeUfRo7tvwPm9A2gGDPqr auB4OfOSm1CvpTkcQtHw3qvqRYQlQTcZ9oI3M4seeRCyJ5CS38nbRyOetSDS97Vhx96V rxJrwBunENoXK6EyODvFqqb/hIqZiF1ev2wlIoBOvs0TAnFloXFLGDPjQkg+cpNkvkng W5GSi5sNKixa4m5l3M8K7zI6pROQOQEjvvJR8Gio6+AoxJwlwhzNXZE8ESig1YH5UhvU /vBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=ZKgL4zeK+z0St01hNMYKSt6dKMvw92MkjAHduvDs3M4=; b=pwgQMc1kn92N6/KH7eb6eVYPUEWV2rI7NuMWavGrF0uALqVF+BpOfGUnO37mEX7/ri k6K6sIqGNBV6WR5gH4HNZfC8p+r9X+9mrlY7IdyTgR5FL3eJnk3WTaLvDEjN59B3fO7V rB2oKLkkTn4MQf7H7iAevM3epTYb1bm88H6iixwWzHOv7tWh8QPitinoX/dBKvJcPr+J BCwXpOj2FybNa+9e2itWevpgdJvz1LIeyzWRy3t5BxXp31Vy0PQRTHpTpIs4s7wxBYLZ LkZ0V3cocez6q65GIUrJxfgS7qxV8up0Uwninln+oXuepnQJTWQYr03WvIhb7ROFb9oZ NqzA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=jrtS3ovn; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id g14sor15781213pgn.40.2019.03.12.15.34.13 for (Google Transport Security); Tue, 12 Mar 2019 15:34:13 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=jrtS3ovn; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZKgL4zeK+z0St01hNMYKSt6dKMvw92MkjAHduvDs3M4=; b=jrtS3ovnWaLclOmLoRw34CUAd+9ENpw1XWtfnLUP1GL2O+JypA4+PCOxzdusoMOk42 MwG2QHBlvBDAahr7Py5iS1cPYcTR4sq6AHI1i7JxSrfe4Z193iZBEtTWyzzO5M/qeVh8 y/5CRZ5bEQz7MH/ith0ogencVWwi/1sVtcswJ1X1i7uv6w689GJCGWNslFK0fan/knST Evo9ANJxmGdHkfNod5kaooDNe8L3Wnrvb215MwDQyPp9a6tvADnANRyz4q1sEuM5r9Gt Xmd1TzZfc/2bDNGBzFEqgP3VGvUWq+PFL50UCQgdo9wcjoPw7/LplLb/nOt9Xy/gnxY3 svCA== X-Google-Smtp-Source: APXvYqyS8FSfyNNo22nCX4/AyHeBT/RRTO7S/phayDoBXXlEplUx4YCEb0YjtOc8QhTdvGHx5uHWwg== X-Received: by 2002:a63:6193:: with SMTP id v141mr8446168pgb.392.1552430053392; Tue, 12 Mar 2019 15:34:13 -0700 (PDT) Received: from tower.thefacebook.com ([2620:10d:c090:200::1:3203]) by smtp.gmail.com with ESMTPSA id i13sm14680592pfo.106.2019.03.12.15.34.12 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 12 Mar 2019 15:34:12 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH v2 4/6] mm: release per-node memcg percpu data prematurely Date: Tue, 12 Mar 2019 15:34:01 -0700 Message-Id: <20190312223404.28665-5-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190312223404.28665-1-guro@fb.com> References: <20190312223404.28665-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Similar to memcg-level statistics, per-node data isn't expected to be hot after cgroup removal. Switching over to atomics and prematurely releasing percpu data helps to reduce the memory footprint of dying cgroups. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 24 +++++++++++++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 569337514230..f296693d102b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -127,6 +127,7 @@ struct mem_cgroup_per_node { struct lruvec lruvec; struct lruvec_stat __rcu /* __percpu */ *lruvec_stat_cpu; + struct lruvec_stat __percpu *lruvec_stat_cpu_offlined; atomic_long_t lruvec_stat[NR_VM_NODE_STAT_ITEMS]; unsigned long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index efd5bc131a38..1b5fe826d6d0 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4460,7 +4460,7 @@ static void free_mem_cgroup_per_node_info(struct mem_cgroup *memcg, int node) if (!pn) return; - free_percpu(pn->lruvec_stat_cpu); + WARN_ON_ONCE(pn->lruvec_stat_cpu != NULL); kfree(pn); } @@ -4616,7 +4616,17 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) static void percpu_rcu_free(struct rcu_head *rcu) { struct mem_cgroup *memcg = container_of(rcu, struct mem_cgroup, rcu); + int node; + + for_each_node(node) { + struct mem_cgroup_per_node *pn = memcg->nodeinfo[node]; + if (!pn) + continue; + + free_percpu(pn->lruvec_stat_cpu_offlined); + WARN_ON_ONCE(pn->lruvec_stat_cpu != NULL); + } free_percpu(memcg->vmstats_percpu_offlined); WARN_ON_ONCE(memcg->vmstats_percpu); @@ -4625,6 +4635,18 @@ static void percpu_rcu_free(struct rcu_head *rcu) static void mem_cgroup_offline_percpu(struct mem_cgroup *memcg) { + int node; + + for_each_node(node) { + struct mem_cgroup_per_node *pn = memcg->nodeinfo[node]; + + if (!pn) + continue; + + pn->lruvec_stat_cpu_offlined = (struct lruvec_stat __percpu *) + rcu_dereference(pn->lruvec_stat_cpu); + rcu_assign_pointer(pn->lruvec_stat_cpu, NULL); + } memcg->vmstats_percpu_offlined = (struct memcg_vmstats_percpu __percpu*) rcu_dereference(memcg->vmstats_percpu); rcu_assign_pointer(memcg->vmstats_percpu, NULL); From patchwork Tue Mar 12 22:34:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10850095 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EEC0B139A for ; Tue, 12 Mar 2019 22:34:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DA8C429243 for ; Tue, 12 Mar 2019 22:34:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CE505296FE; Tue, 12 Mar 2019 22:34:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 57D6529243 for ; Tue, 12 Mar 2019 22:34:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 19E108E0016; Tue, 12 Mar 2019 18:34:17 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 125D98E0002; Tue, 12 Mar 2019 18:34:17 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E8F9B8E0016; Tue, 12 Mar 2019 18:34:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 9B0108E0002 for ; Tue, 12 Mar 2019 18:34:16 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id d5so4712502pfo.5 for ; Tue, 12 Mar 2019 15:34:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=0AseUfXAAtgM7xE37ALVmimobdOHrae++QaZmnVS3p0=; b=LPQt1R3LJKLbWf0AmTsqwRqUnctLprPlneiQqWfX1/8bUVAoq9ICr5lQqJX8dEawMK uegyIzhIUYUYQEp2mqiKhJyPBrOlxt1oPbgJGS7MQ3XL6Bxm7BwjAIMEyFyRP60sdni0 ENn7ogIBbUkTEj5YHE78PHbEH13QTgrfi2CU9/UDUL1oty/uJyodQQQBk/BMvLXU/Ijx v/CLHePyiv3FdOnc3AYoHrD0eVeJPe80XJIADO8tWpIg9wMKgT4pyoi1lN16Sfd5qlu0 MyVrUCJalQvJnm50fhrc0s14va9tXBbxrDBP80SD4yb9NutABpVZteuy58+s9y6LfJTY kAnA== X-Gm-Message-State: APjAAAVKPgJ2RJVNbpHD9jbjR2/UCIHiRqpnJ29rKRH0eCPwa6cmOueh FOTIVGETp3TrUiFmKtIu2GzezkecPp/jVZWRV4n6a9TQcL0rv4X71kw+YxB3iDEEzHVKUiBztMI vhs9cNVyS3NyeJBKWLO8CdT5DOcqow1qUa3XqSX6o1idWfQR0RtSqXDtjzdB8uQLLV0H5lGPzrZ wyjf+dUMLoM88hquQF0TTT8fK3+nyGDZA5+18HFnUQOWUwNNqfW9xwNBZchwDY0yGQ9wWqlycIu In/qQJ7xV9C04wEJU9nKJkqMdVZTQFxPb+lgZgVs1FzM+4WiWU+DI/ZvRYUg5+DOi5mmbXIvArD gmWFI8tgatfCJmNNjJM48QQJ0xBK3TEK+Twh6qfnLIPWa3uq8KqzKdG/rJV6Sd6mbpmM4YB8K2r O X-Received: by 2002:a17:902:2ba7:: with SMTP id l36mr20272419plb.246.1552430056312; Tue, 12 Mar 2019 15:34:16 -0700 (PDT) X-Received: by 2002:a17:902:2ba7:: with SMTP id l36mr20272332plb.246.1552430054959; Tue, 12 Mar 2019 15:34:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552430054; cv=none; d=google.com; s=arc-20160816; b=luD9qDRSyqrot6kIdKlKoM1qTf1x1VVOurjPIZB/o9ZPwvX3iw6zc0JYYeHIagNAP+ EbdXBXIvQHI0Uy+OoVBzVEIlZbv5SmUfwNq+xA8FYZKlJG+fBohQHvHY9HUif9O6ZBcv eBMaT67WPaRG6mHQwrCw2Kyyg0ChelZ5zUahBOe83qmsh4K6N5Pc+vuHVbm7yQjB8LfY lQIppuc4PXlSqIX7er+fP8/apA6IQunhQkgebddEc8/8UarbXnaZfQuUf4LIeneLS1L1 aIf2qvBgoYXULJcLsmhrtw/59eJvI4Po5Bo9SYhLFhOVyfo7XBzB1DqnsZS5E27H4DsI /lJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=0AseUfXAAtgM7xE37ALVmimobdOHrae++QaZmnVS3p0=; b=zxpEKmOj9WveSsdcc/Q45mGwFfOnZXlHdHOBaRE7UPjMRbNIYtvTHvf5pAqOa5NHoK 6vzs75f5LBYnLOMuPt8q1YLmEfD522ly/qZkEgXox+JRAd3xuB2k/dLKtceSeJU4GCm9 kq+ljcJShjq1kFhDsaceTpDn8g92ClLfwlJPW0AjEQjiJFKQ+G7svfFYtyxnkDzC79C5 JA7zx/J/uYfWIQEpgnifuV3wOiHcfyVrLDDLlhlkKAkgBMBLley7VD17n9fMYPbsdA/L vWv89j3jQi3iK6/H73TkyprfG2oK7dCNx8mjbm4+aRvb0AWWsUuywrCBvBnmCmoZuVUr 49qQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=hlmEsoAp; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id x1sor15122845plb.45.2019.03.12.15.34.14 for (Google Transport Security); Tue, 12 Mar 2019 15:34:14 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=hlmEsoAp; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0AseUfXAAtgM7xE37ALVmimobdOHrae++QaZmnVS3p0=; b=hlmEsoApvA/mPdl0iBW06MNbW54pe9uXKLI0ahcw+KfOFDqPsNwozyuf/2FhiVycfD knJ0Vxm7F0KP7fmIjk3HSFKX7vxKdbqVR/ncWSecT8LiNZj8ed3QWLFwKx4cNWGcvC9l U+r7VBADW1oe+7Q32r16t4eTIfTU/p6TXMKLGiJbf9dHzgUYXD/79078ZFmDbKAEotUk GY0aTSay4wBRp5nV4d/00Yon9XA6n8xKekA/Di4Nn6DeVujfOekkgZdqIS1BykhG95E1 Wh/Sd0PNA2IlQP24StTYHp5SWr4M+KoUkYW35t9ukZTak76552zqkiyBUFTdIRHC23Lv 3QkQ== X-Google-Smtp-Source: APXvYqw+oQJ4ndnWt7iHQgCj/cQPi1Rt1P8zv210NZF5fOTuQOXgHm6K3xRS+YSiomUIUFri8uU6TA== X-Received: by 2002:a17:902:968b:: with SMTP id n11mr42163572plp.316.1552430054499; Tue, 12 Mar 2019 15:34:14 -0700 (PDT) Received: from tower.thefacebook.com ([2620:10d:c090:200::1:3203]) by smtp.gmail.com with ESMTPSA id i13sm14680592pfo.106.2019.03.12.15.34.13 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 12 Mar 2019 15:34:13 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH v2 5/6] mm: flush memcg percpu stats and events before releasing Date: Tue, 12 Mar 2019 15:34:02 -0700 Message-Id: <20190312223404.28665-6-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190312223404.28665-1-guro@fb.com> References: <20190312223404.28665-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Flush percpu stats and events data to corresponding before releasing percpu memory. Although per-cpu stats are never exactly precise, dropping them on floor regularly may lead to an accumulation of an error. So, it's safer to flush them before releasing. To minimize the number of atomic updates, let's sum all stats/events on all cpus locally, and then make a single update per entry. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- mm/memcontrol.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1b5fe826d6d0..0f18bf2afea8 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2119,6 +2119,56 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) mutex_unlock(&percpu_charge_mutex); } +/* + * Flush all per-cpu stats and events into atomics. + * Try to minimize the number of atomic writes by gathering data from + * all cpus locally, and then make one atomic update. + * No locking is required, because no one has an access to + * the offlined percpu data. + */ +static void memcg_flush_offline_percpu(struct mem_cgroup *memcg) +{ + struct memcg_vmstats_percpu __percpu *vmstats_percpu; + struct lruvec_stat __percpu *lruvec_stat_cpu; + struct mem_cgroup_per_node *pn; + int cpu, i; + long x; + + vmstats_percpu = memcg->vmstats_percpu_offlined; + + for (i = 0; i < MEMCG_NR_STAT; i++) { + int nid; + + x = 0; + for_each_possible_cpu(cpu) + x += per_cpu(vmstats_percpu->stat[i], cpu); + if (x) + atomic_long_add(x, &memcg->vmstats[i]); + + if (i >= NR_VM_NODE_STAT_ITEMS) + continue; + + for_each_node(nid) { + pn = mem_cgroup_nodeinfo(memcg, nid); + lruvec_stat_cpu = pn->lruvec_stat_cpu_offlined; + + x = 0; + for_each_possible_cpu(cpu) + x += per_cpu(lruvec_stat_cpu->count[i], cpu); + if (x) + atomic_long_add(x, &pn->lruvec_stat[i]); + } + } + + for (i = 0; i < NR_VM_EVENT_ITEMS; i++) { + x = 0; + for_each_possible_cpu(cpu) + x += per_cpu(vmstats_percpu->events[i], cpu); + if (x) + atomic_long_add(x, &memcg->vmevents[i]); + } +} + static int memcg_hotplug_cpu_dead(unsigned int cpu) { struct memcg_vmstats_percpu __percpu *vmstats_percpu; @@ -4618,6 +4668,8 @@ static void percpu_rcu_free(struct rcu_head *rcu) struct mem_cgroup *memcg = container_of(rcu, struct mem_cgroup, rcu); int node; + memcg_flush_offline_percpu(memcg); + for_each_node(node) { struct mem_cgroup_per_node *pn = memcg->nodeinfo[node]; From patchwork Tue Mar 12 22:34:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10850099 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6EFB413B5 for ; Tue, 12 Mar 2019 22:34:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5B96229243 for ; Tue, 12 Mar 2019 22:34:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4F9BB296FE; Tue, 12 Mar 2019 22:34:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BDB5929243 for ; Tue, 12 Mar 2019 22:34:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 65DC68E0018; Tue, 12 Mar 2019 18:34:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5E7FD8E0002; Tue, 12 Mar 2019 18:34:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 47F4B8E0018; Tue, 12 Mar 2019 18:34:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 7F9C28E0002 for ; Tue, 12 Mar 2019 18:34:18 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id p9so4687176pfn.9 for ; Tue, 12 Mar 2019 15:34:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=PLL/4LfAnZv9OSID2l2Kyrqdn4GFKp657vXZGXPSxO0=; b=D7JGwKTNj3fn9QInxKQqxGgYxjcpXw2MXba85R6sp6oQqwsmZ7wNaBVTALNM7h0HL0 y5WHPfNACdPNpqD7JnMfTorbSgHKCmB3v0hyU7pmEL64IBNGRFn0l1Gd2rCxS56izYm4 cVwaiVxUTCOdAjdwTHwcX5Q6XSAnRj1JCxjDZQkGuKOtydwqjjUBlxa2mR6exAuqZ1wJ 0TsTr9ir4m7tnfmIcvCouQm0dfSjJbN1zT+rVVjD2zvsDer5OXACg4yDilAGv27E3ntR lCJ/xkNwiH/VKToAg5QjSjTofUDP0qubW/UbWk0GF4jqda7cqfm3t+z9Q836rs6eBj4S IVlA== X-Gm-Message-State: APjAAAXRVyy6X1kT777fg2EmbtmeCDnBKSQQRI+JtO0POm2TlDVzfeOJ JwrdZaSWErOBZt0Sa6ZAaLjGKvX8g6BttwTiQCIa7i8ETEVFt/BhA0Pqu6+aEjpTqWSI+aTzvb0 TnFHBX0IoUzQ5SCXy+tcgouoTlPU7oKOiA6tCEN609kI6FaLCRNlcskgzcrNQAqAT7zzgeWjvq/ LKJe0xvMujqmfky1rilKF3Ke6+4zxDeuDpKR/0ppJcKwR2XRvlIHsTgCjse2gujvgcR8zOPVg50 MMwuKoTuDMW6gg0PU3tnzgBXbvQJM9LlS4xTSV0AP5+kpdhFC9KWNxrHPjVJWnP31sfyX1xmqq4 z7EFw0byyY9GNEZIkNu30QcpYWOeAtruCuvVXNsGdhf7wPgbAqDQ2np6wvS3DRvTaup4ODQKEvy L X-Received: by 2002:a63:d502:: with SMTP id c2mr15712735pgg.290.1552430058172; Tue, 12 Mar 2019 15:34:18 -0700 (PDT) X-Received: by 2002:a63:d502:: with SMTP id c2mr15712679pgg.290.1552430057163; Tue, 12 Mar 2019 15:34:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552430057; cv=none; d=google.com; s=arc-20160816; b=RcZzHLF15K6/au67z9UUX3orPYmI/bHddpwAs+dEaRL117rFgagGkZs9L4z7RTk5Ii f+pdg62oGuJ7N13jbGQpFpIsNhlkxcMXbBcyfrXdBkv4N8WENpgskmig4MM2QcKPmwIQ G8oZo/E1Zwr1uchr3uVgP7WcNm5EMmCxVNr+Ghjru5k+3QEpgH2QFt24fWTZwu7TmU5W 1lPlqvqcHYbK1G/mgShxY8f/iPE7gi9zUQJtBuOB0icJwOHDWYMHguXkqArq9rPoTyys GBfSKwI9F4FXoBSoeq/RqXZ+s4ic0BNjd7l0ScYX4diJ20/jlMApX2uSLCy6nHI3bTTi qlTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=PLL/4LfAnZv9OSID2l2Kyrqdn4GFKp657vXZGXPSxO0=; b=tPJ11AJ59SzZxKjqZFVLGfx+N2M6jtU/oThAw7naI0EICj6jC6d/VQm+QfqqHCe9RO 4wkSpVvydS4EeJr0gbdSF42u2FauotiBRxvehF4QjrldCa3KXyB8Y/KUVH+QO7KhJS++ 6GtyQJkAQUInTM/3z5pWkXvwUnyEc85cVVV6Y1Q+bLaefCW6wCGTzCFl/jri6ADvGNJr e1UregSXIuep8xNEQbSzsurJ6v69SiXGkFjJO7SLa7MzcRfn3jWx5L0DHSfqhC9NHaCK QfyXjZSwSp6TWJfRpyD7XSRWLeyonz7FsDlo1rNW+scEa+0Pm1r7wAPKOQfbTpRWT3Fw GPhw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=k7QPsFLm; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id be3sor15229860plb.25.2019.03.12.15.34.17 for (Google Transport Security); Tue, 12 Mar 2019 15:34:17 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=k7QPsFLm; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PLL/4LfAnZv9OSID2l2Kyrqdn4GFKp657vXZGXPSxO0=; b=k7QPsFLmc+eBwxBeIp3C8Q/G3nJeZYxu7XnCay7LwQzvYDlcex8ZmzvLEALm8Ne5rw SLlbWdGCJNNNylqAovxOpBlwHo49mZQmWXeGGDXwyl+jLrKRlypwlZLICtOsya6U8/88 ks2uXUfUmxVwbCUqjNelbhwLdaTnFwnlcnHLdIhCFHmh3+XU5EdrQpfzjqb28c/31aEh MWgHKZOniE/gg/g/aN3rT2rpFpy6or34LenQ6dTPqDklOxnQM6FsOdnsHRZ5GK6LSJxr VikGyXRKTXAJblZ/1TJmD13Dvb2uX0PrEt3TBBMrq/zR0FyJPp5nDwvQQHtrumlswmgj UotQ== X-Google-Smtp-Source: APXvYqzMJr1pK4nMS01UgtE1/G7AaJ3jZ+2slx1gvSttry8dUdcDTbmwyNqFCWr4j4KvJIjGUZ1mbQ== X-Received: by 2002:a17:902:2bc7:: with SMTP id l65mr42472606plb.79.1552430056682; Tue, 12 Mar 2019 15:34:16 -0700 (PDT) Received: from tower.thefacebook.com ([2620:10d:c090:200::1:3203]) by smtp.gmail.com with ESMTPSA id i13sm14680592pfo.106.2019.03.12.15.34.15 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 12 Mar 2019 15:34:16 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH v2 6/6] mm: refactor memcg_hotplug_cpu_dead() to use memcg_flush_offline_percpu() Date: Tue, 12 Mar 2019 15:34:04 -0700 Message-Id: <20190312223404.28665-8-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190312223404.28665-1-guro@fb.com> References: <20190312223404.28665-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It's possible to remove a big chunk of the redundant code by making memcg_flush_offline_percpu() to take cpumask as an argument and flush percpu data on all cpus belonging to the mask instead of all possible cpus. Then memcg_hotplug_cpu_dead() can call it with a single CPU bit set. This approach allows to remove all duplicated code, but safe the performance optimization made in memcg_flush_offline_percpu(): only one atomic operation per data entry. for_each_data_entry() for_each_cpu(cpu. cpumask) sum_events() flush() Otherwise it would be one atomic operation per data entry per cpu: for_each_cpu(cpu) for_each_data_entry() flush() Signed-off-by: Roman Gushchin --- mm/memcontrol.c | 61 ++++++++----------------------------------------- 1 file changed, 9 insertions(+), 52 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0f18bf2afea8..92c80275d5eb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2122,11 +2122,12 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) /* * Flush all per-cpu stats and events into atomics. * Try to minimize the number of atomic writes by gathering data from - * all cpus locally, and then make one atomic update. + * all cpus in cpumask locally, and then make one atomic update. * No locking is required, because no one has an access to * the offlined percpu data. */ -static void memcg_flush_offline_percpu(struct mem_cgroup *memcg) +static void memcg_flush_offline_percpu(struct mem_cgroup *memcg, + const struct cpumask *cpumask) { struct memcg_vmstats_percpu __percpu *vmstats_percpu; struct lruvec_stat __percpu *lruvec_stat_cpu; @@ -2140,7 +2141,7 @@ static void memcg_flush_offline_percpu(struct mem_cgroup *memcg) int nid; x = 0; - for_each_possible_cpu(cpu) + for_each_cpu(cpu, cpumask) x += per_cpu(vmstats_percpu->stat[i], cpu); if (x) atomic_long_add(x, &memcg->vmstats[i]); @@ -2153,7 +2154,7 @@ static void memcg_flush_offline_percpu(struct mem_cgroup *memcg) lruvec_stat_cpu = pn->lruvec_stat_cpu_offlined; x = 0; - for_each_possible_cpu(cpu) + for_each_cpu(cpu, cpumask) x += per_cpu(lruvec_stat_cpu->count[i], cpu); if (x) atomic_long_add(x, &pn->lruvec_stat[i]); @@ -2162,7 +2163,7 @@ static void memcg_flush_offline_percpu(struct mem_cgroup *memcg) for (i = 0; i < NR_VM_EVENT_ITEMS; i++) { x = 0; - for_each_possible_cpu(cpu) + for_each_cpu(cpu, cpumask) x += per_cpu(vmstats_percpu->events[i], cpu); if (x) atomic_long_add(x, &memcg->vmevents[i]); @@ -2171,8 +2172,6 @@ static void memcg_flush_offline_percpu(struct mem_cgroup *memcg) static int memcg_hotplug_cpu_dead(unsigned int cpu) { - struct memcg_vmstats_percpu __percpu *vmstats_percpu; - struct lruvec_stat __percpu *lruvec_stat_cpu; struct memcg_stock_pcp *stock; struct mem_cgroup *memcg; @@ -2180,50 +2179,8 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) drain_stock(stock); rcu_read_lock(); - for_each_mem_cgroup(memcg) { - int i; - - vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) - rcu_dereference(memcg->vmstats_percpu); - - for (i = 0; i < MEMCG_NR_STAT; i++) { - int nid; - long x; - - if (vmstats_percpu) { - x = this_cpu_xchg(vmstats_percpu->stat[i], 0); - if (x) - atomic_long_add(x, &memcg->vmstats[i]); - } - - if (i >= NR_VM_NODE_STAT_ITEMS) - continue; - - for_each_node(nid) { - struct mem_cgroup_per_node *pn; - - pn = mem_cgroup_nodeinfo(memcg, nid); - - lruvec_stat_cpu = (struct lruvec_stat __percpu*) - rcu_dereference(pn->lruvec_stat_cpu); - if (!lruvec_stat_cpu) - continue; - x = this_cpu_xchg(lruvec_stat_cpu->count[i], 0); - if (x) - atomic_long_add(x, &pn->lruvec_stat[i]); - } - } - - for (i = 0; i < NR_VM_EVENT_ITEMS; i++) { - long x; - - if (vmstats_percpu) { - x = this_cpu_xchg(vmstats_percpu->events[i], 0); - if (x) - atomic_long_add(x, &memcg->vmevents[i]); - } - } - } + for_each_mem_cgroup(memcg) + memcg_flush_offline_percpu(memcg, get_cpu_mask(cpu)); rcu_read_unlock(); return 0; @@ -4668,7 +4625,7 @@ static void percpu_rcu_free(struct rcu_head *rcu) struct mem_cgroup *memcg = container_of(rcu, struct mem_cgroup, rcu); int node; - memcg_flush_offline_percpu(memcg); + memcg_flush_offline_percpu(memcg, cpu_possible_mask); for_each_node(node) { struct mem_cgroup_per_node *pn = memcg->nodeinfo[node];