From patchwork Thu Mar 7 23:00:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10843943 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E9321139A for ; Thu, 7 Mar 2019 23:00:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D63082F672 for ; Thu, 7 Mar 2019 23:00:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CA7B22F66D; Thu, 7 Mar 2019 23:00:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B87F32F672 for ; Thu, 7 Mar 2019 23:00:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D52E28E0004; Thu, 7 Mar 2019 18:00:40 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CDBAF8E0002; Thu, 7 Mar 2019 18:00:40 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B075F8E0004; Thu, 7 Mar 2019 18:00:40 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 6550B8E0002 for ; Thu, 7 Mar 2019 18:00:40 -0500 (EST) Received: by mail-pg1-f197.google.com with SMTP id f6so17929987pgo.15 for ; Thu, 07 Mar 2019 15:00:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=2zD5nbQljCuVZEpjxLRZoYsTQjqMS5l1KBU8Q+Sdfe8=; b=r0P5tS96oy57mHdytocBnlPVNR93IeULEgneezapjFsXMGrSK2gJ9kCjkLq9VBi0LY l40DP9k1ecpqbC3Yma2UjDEC3cqWKNZV9OskQBHy//E1h3IMrno2f2fqAjVUSJFUMO+u 6QWePw6iMosHLHTSKm7qGfS8qtvML8mPa3kJ20FRvQreEnVQocMpuaCuGZBHQqDKguQl KgF5jGSuyUQ++I9AFKNWfGOgHbBfH/hezA+5xN6MagDOMk7eteT0nssrGghEodKZ4iyV VgyuIrYdoW5pACD8moKof3kXja8ExYsHyf0tWBTAQw0FT1IvsG02PNUq698MFTrmG3Q6 WKmw== X-Gm-Message-State: APjAAAVGbZHLfYixtoAv03MZNr2fimTMaSXQVsIZZZzSAFuhz0wEAW+P Gkuz27Vu+XQvp6MBsjPQIYOyRWDlb6DdoW+rIfHS9YBfwVdzlG+57NdpPX1TGnjb5e3qqM1x9a2 LG6FMxgbyscnmqLsz8BGCPqkrifhfMBlUe1avbqaWhUYg7nSfUeRUu/+UP9fo8ZElfJWNv2AZ9Q MYR3yMMqybTlQg4AfCDM2uT6EQWi0VW65sAbCIhfr8lD5dawt1JmXVkGTWX4zi5j/mE32R7vA9q 3BbI1ussZx0vrjjGh3145nDX9Kc/StU3MWlM2rTfu+GKRMfw/wai2MSDg8poSCDhiy/qxFLKO/d RiiLNzUph/oIDIpsJ0m9pc4RXibGC1mypUX1ZeNEDpQzuMl27YIIfRWVy7abIJOZDtZYQ3LXYvq Z X-Received: by 2002:a63:f648:: with SMTP id u8mr13374540pgj.91.1551999639933; Thu, 07 Mar 2019 15:00:39 -0800 (PST) X-Received: by 2002:a63:f648:: with SMTP id u8mr13374459pgj.91.1551999638738; Thu, 07 Mar 2019 15:00:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551999638; cv=none; d=google.com; s=arc-20160816; b=ua/QYeMTpjRpo59WBV6qL3nwU21quhQwxg9lkd9KBEIdv38nxMLuHkaiFCVJ9YLxSV j7JFJxlQZe0uP9OO62ySeX5Oj856m5ki3vgLJRAwroe5/VMx6uYZs3YSUPv36x+ZOPxH ank5HkVhd/JhhMaUaEWiOLH1c+6kh0MFVwY+cby2N3UiRoYuEpTVRE0/hfOHbB4o10Y0 YdnTjBxeODh01Eovr2t1v1Aeoc2UjuvI52+48lQeC3YfQgXq+fxJF/13H32IlBsf4IFH cMwhjTB30xNSxwL2xjU61kc1SCT/mLlf+wRYJRbcTF1xJYbrJUPiwDk5ExWv/F+SGMcu CUBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=2zD5nbQljCuVZEpjxLRZoYsTQjqMS5l1KBU8Q+Sdfe8=; b=embJrdm32xNs9YieZdeMWJbg7lPPDXUZ1OthwKmlLa3aItcqVl85QHBckuzlNFjLQX 4ys1A5hMVHKnWBGjVJWr1gZjXL1pcmUyW9HiAyOVcHEGZ8usnt7ZOCVMhGdlAfzwNfmk E4s0XhdIi5deMuFP6B8kaQL6EgWLphhk0AD5B/AC+89NH/OeLGJpB67TkM87FF2izOwl p05gPZpLhhFyNrlWqiLL4hx3wSMeXx36HAHwULVTbi4qYPEHzA3YtVqHbSKvEKvKF5D3 d72QbTA28MER1IppGi2C4A65NYx9d6oOkNCdCFQj5UdDN7Pqp2KOyl8IGYW5m62GkhYi R8KA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=V2nhvsUU; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id k190sor10086711pgc.32.2019.03.07.15.00.38 for (Google Transport Security); Thu, 07 Mar 2019 15:00:38 -0800 (PST) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=V2nhvsUU; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2zD5nbQljCuVZEpjxLRZoYsTQjqMS5l1KBU8Q+Sdfe8=; b=V2nhvsUUyX9TXoESjQOB5eMsm4xQAzZMmf0SgpDQU7WMnu5kBdLHv4JgEV90jf5GaP uLrVJnKQZ8gCQy3lXQGbFxG1kH6Z73WBa/x9i686DxKP3jrlEbvTTwherxEuLZByWTzX nadisGvGx7A5xom2xCSmhRoOfxmH8wlh5mbXW6Hg3mlOTrGa45fdNr1Ippn1B2ZD2QpU VC0ELQPKCVT6Wan0Qj5xK8EZevASG2q8+5N/HDZ8/O6H7arZTtoWzTyiGhyBwggpbdid /n/RTPzgam2gKExM2V7cwQSMLg41yt45EO4w/irg/VubtzXAhB5+O97LteDwWzCzNe9A WVTg== X-Google-Smtp-Source: APXvYqx/NgILDTTJRVUsh55o+xNSO6KhGs1zskXimET2dZyUOOxDdBosr71+ch0fXmf8EM8Wuep8xA== X-Received: by 2002:a65:4243:: with SMTP id d3mr13633502pgq.56.1551999638120; Thu, 07 Mar 2019 15:00:38 -0800 (PST) Received: from tower.thefacebook.com ([2620:10d:c090:200::2:d18b]) by smtp.gmail.com with ESMTPSA id i126sm11864806pfb.15.2019.03.07.15.00.37 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 07 Mar 2019 15:00:37 -0800 (PST) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH 1/5] mm: prepare to premature release of memcg->vmstats_percpu Date: Thu, 7 Mar 2019 15:00:29 -0800 Message-Id: <20190307230033.31975-2-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190307230033.31975-1-guro@fb.com> References: <20190307230033.31975-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Prepare to handle premature release of memcg->vmstats_percpu data. Currently it's a generic pointer which is expected to be non-NULL during the whole life time of a memcg. Switch over to the rcu-protected pointer, and carefully check it for being non-NULL. This change is a required step towards dynamic premature release of percpu memcg data. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 40 +++++++++++++++++------- mm/memcontrol.c | 62 +++++++++++++++++++++++++++++--------- 2 files changed, 77 insertions(+), 25 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 534267947664..05ca77767c6a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -274,7 +274,7 @@ struct mem_cgroup { struct task_struct *move_lock_task; /* memory.stat */ - struct memcg_vmstats_percpu __percpu *vmstats_percpu; + struct memcg_vmstats_percpu __rcu /* __percpu */ *vmstats_percpu; MEMCG_PADDING(_pad2_); @@ -597,17 +597,26 @@ static inline unsigned long memcg_page_state(struct mem_cgroup *memcg, static inline void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; long x; if (mem_cgroup_disabled()) return; - x = val + __this_cpu_read(memcg->vmstats_percpu->stat[idx]); - if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { - atomic_long_add(x, &memcg->vmstats[idx]); - x = 0; + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + if (likely(vmstats_percpu)) { + x = val + __this_cpu_read(vmstats_percpu->stat[idx]); + if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { + atomic_long_add(x, &memcg->vmstats[idx]); + x = 0; + } + __this_cpu_write(vmstats_percpu->stat[idx], x); + } else { + atomic_long_add(val, &memcg->vmstats[idx]); } - __this_cpu_write(memcg->vmstats_percpu->stat[idx], x); + rcu_read_unlock(); } /* idx can be of type enum memcg_stat_item or node_stat_item */ @@ -740,17 +749,26 @@ static inline void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, unsigned long count) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; unsigned long x; if (mem_cgroup_disabled()) return; - x = count + __this_cpu_read(memcg->vmstats_percpu->events[idx]); - if (unlikely(x > MEMCG_CHARGE_BATCH)) { - atomic_long_add(x, &memcg->vmevents[idx]); - x = 0; + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + if (likely(vmstats_percpu)) { + x = count + __this_cpu_read(vmstats_percpu->events[idx]); + if (unlikely(x > MEMCG_CHARGE_BATCH)) { + atomic_long_add(x, &memcg->vmevents[idx]); + x = 0; + } + __this_cpu_write(vmstats_percpu->events[idx], x); + } else { + atomic_long_add(count, &memcg->vmevents[idx]); } - __this_cpu_write(memcg->vmstats_percpu->events[idx], x); + rcu_read_unlock(); } static inline void count_memcg_events(struct mem_cgroup *memcg, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c532f8685aa3..803c772f354b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -697,6 +697,8 @@ static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, struct page *page, bool compound, int nr_pages) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; + /* * Here, RSS means 'mapped anon' and anon's SwapCache. Shmem/tmpfs is * counted as CACHE even if it's on ANON LRU. @@ -722,7 +724,12 @@ static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, nr_pages = -nr_pages; /* for event */ } - __this_cpu_add(memcg->vmstats_percpu->nr_page_events, nr_pages); + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + if (likely(vmstats_percpu)) + __this_cpu_add(vmstats_percpu->nr_page_events, nr_pages); + rcu_read_unlock(); } unsigned long mem_cgroup_node_nr_lru_pages(struct mem_cgroup *memcg, @@ -756,10 +763,18 @@ static unsigned long mem_cgroup_nr_lru_pages(struct mem_cgroup *memcg, static bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg, enum mem_cgroup_events_target target) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; unsigned long val, next; + bool ret = false; - val = __this_cpu_read(memcg->vmstats_percpu->nr_page_events); - next = __this_cpu_read(memcg->vmstats_percpu->targets[target]); + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + if (!vmstats_percpu) + goto out; + + val = __this_cpu_read(vmstats_percpu->nr_page_events); + next = __this_cpu_read(vmstats_percpu->targets[target]); /* from time_after() in jiffies.h */ if ((long)(next - val) < 0) { switch (target) { @@ -775,10 +790,12 @@ static bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg, default: break; } - __this_cpu_write(memcg->vmstats_percpu->targets[target], next); - return true; + __this_cpu_write(vmstats_percpu->targets[target], next); + ret = true; } - return false; +out: + rcu_read_unlock(); + return ret; } /* @@ -2104,22 +2121,29 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) static int memcg_hotplug_cpu_dead(unsigned int cpu) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; struct memcg_stock_pcp *stock; struct mem_cgroup *memcg; stock = &per_cpu(memcg_stock, cpu); drain_stock(stock); + rcu_read_lock(); for_each_mem_cgroup(memcg) { int i; + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + for (i = 0; i < MEMCG_NR_STAT; i++) { int nid; long x; - x = this_cpu_xchg(memcg->vmstats_percpu->stat[i], 0); - if (x) - atomic_long_add(x, &memcg->vmstats[i]); + if (vmstats_percpu) { + x = this_cpu_xchg(vmstats_percpu->stat[i], 0); + if (x) + atomic_long_add(x, &memcg->vmstats[i]); + } if (i >= NR_VM_NODE_STAT_ITEMS) continue; @@ -2137,11 +2161,14 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) for (i = 0; i < NR_VM_EVENT_ITEMS; i++) { long x; - x = this_cpu_xchg(memcg->vmstats_percpu->events[i], 0); - if (x) - atomic_long_add(x, &memcg->vmevents[i]); + if (vmstats_percpu) { + x = this_cpu_xchg(vmstats_percpu->events[i], 0); + if (x) + atomic_long_add(x, &memcg->vmevents[i]); + } } } + rcu_read_unlock(); return 0; } @@ -4464,7 +4491,8 @@ static struct mem_cgroup *mem_cgroup_alloc(void) if (memcg->id.id < 0) goto fail; - memcg->vmstats_percpu = alloc_percpu(struct memcg_vmstats_percpu); + rcu_assign_pointer(memcg->vmstats_percpu, + alloc_percpu(struct memcg_vmstats_percpu)); if (!memcg->vmstats_percpu) goto fail; @@ -6054,6 +6082,7 @@ static void uncharge_batch(const struct uncharge_gather *ug) { unsigned long nr_pages = ug->nr_anon + ug->nr_file + ug->nr_kmem; unsigned long flags; + struct memcg_vmstats_percpu __percpu *vmstats_percpu; if (!mem_cgroup_is_root(ug->memcg)) { page_counter_uncharge(&ug->memcg->memory, nr_pages); @@ -6070,7 +6099,12 @@ static void uncharge_batch(const struct uncharge_gather *ug) __mod_memcg_state(ug->memcg, MEMCG_RSS_HUGE, -ug->nr_huge); __mod_memcg_state(ug->memcg, NR_SHMEM, -ug->nr_shmem); __count_memcg_events(ug->memcg, PGPGOUT, ug->pgpgout); - __this_cpu_add(ug->memcg->vmstats_percpu->nr_page_events, nr_pages); + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(ug->memcg->vmstats_percpu); + if (likely(vmstats_percpu)) + __this_cpu_add(vmstats_percpu->nr_page_events, nr_pages); + rcu_read_unlock(); memcg_check_events(ug->memcg, ug->dummy_page); local_irq_restore(flags); From patchwork Thu Mar 7 23:00:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10843945 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 13D251575 for ; Thu, 7 Mar 2019 23:00:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 01AA82F664 for ; Thu, 7 Mar 2019 23:00:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E99342F67F; Thu, 7 Mar 2019 23:00:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5F0C92F664 for ; Thu, 7 Mar 2019 23:00:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 195378E0005; Thu, 7 Mar 2019 18:00:43 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0F5DD8E0002; Thu, 7 Mar 2019 18:00:42 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7A958E0005; Thu, 7 Mar 2019 18:00:42 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 951438E0002 for ; Thu, 7 Mar 2019 18:00:42 -0500 (EST) Received: by mail-pg1-f199.google.com with SMTP id o4so450014pgl.6 for ; Thu, 07 Mar 2019 15:00:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=XVAzLd6hdIPZj35NxoEZAwxg04Tr002qNrQDR1YF6Fc=; b=IaNMktEbM53lYajwIkJoEdFjgOFhVRENbH/3nL3COr7c9PKYhyeDLSHmdAap1rNs3q LiNYEF73FlEoGuTVKJeYjFoqDGLnY9788xru9a6OpQjX4M04YYxqdn4sIZAOgcncMrXb L8OV+NwJYCMWXqexipEKA1q9g8h8pT4I6V8k713/R1O+lZEkDMYTMUWqh3Mbdpfa8z05 5KIbJC5zo76+gNtMAhmlKeMQnFVFSX3Wem/yHSPT4ZTLWNtN5FOFCsrVm5JXjjYVPG+p Nu0shKfCH9E9Ijm5lmNtumSWTGNiUxbmQU6ii92s1AW7ijTjn9XYnzRPt0YRLiDbDFrn /UGw== X-Gm-Message-State: APjAAAV4j9YRu9KBP0YT2o6VujfK1kj6sYXLXXYKNbfha6KQSga3uPcB +HPuabq9JWsrWj7abJij8y5xIWlgukcwrNZpoYxuaWkuWW2Pi7nw+DK6pwmd8vyUsPj9WBEslBp TA+bFvqGufQMqUDi4t/r5Oqep7f0uBWYjqT+khfPcqqbrAtXK28dWrQFXZmAXO1rBjDR+m1d8TI d5dFWGnDxpaADB1VTcdvcGR3ja9C9GIifcx0mAPSg0B4sJ6yso2Brw7YslrLLmeDOFkndG9CYsS t/f7HOijrk7bv9u8KQPsRH+tblivWTEs5guZXz9bJp8KQZNXORLsMSDz3/f3K80a7B6xCopZ55y DPWjGUFW00cCgciPYOfzMyZVcn8dQePaaHGwx154golFh7KiTVaMl6xculCwzWJqmO7RJgEnEZD 1 X-Received: by 2002:a17:902:b493:: with SMTP id y19mr15773222plr.9.1551999641962; Thu, 07 Mar 2019 15:00:41 -0800 (PST) X-Received: by 2002:a17:902:b493:: with SMTP id y19mr15773054plr.9.1551999640273; Thu, 07 Mar 2019 15:00:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551999640; cv=none; d=google.com; s=arc-20160816; b=tFFi9uF3ExeZP8XCbOehNZ/Hj26WdOR5/BjCuKHe9emOWDkoWNJ685f/eRZir2xcLg /qtOIEPp9uILHDxon2+iRMUpp689W8gXBuDAH/WfRl1A8z+n8c2/N0UGHQbEfx7aY71S qL3WNlzgM3tQ8rMatCH1VX4sVFco3tzEzZYMTJ+hnUanN83IWwRO3HQXZ+qyveb2Axwy WzGIZ+2ZZ8R5lEw5lS1WEXsmrS+3CEoT7tfxpZkjrZH5M1PLpzsgL5qQ7AhhMnf1c/9T XvgHG5r8FiwLAeU5Azy5TwI1R1uB9OxORAA1tUd4iXNaXRLehUX3CtL9WA3N27+zCiCW TSoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=XVAzLd6hdIPZj35NxoEZAwxg04Tr002qNrQDR1YF6Fc=; b=zG+4bBnZ5X1eM8OnowMyI6IkrHlLycsYwf8ZfInCdj5b7kXttwGsrWQLWIv2I/5hse cSiL+BjhWTGcUndBtYI19pbHoQ5UmKh8TSpuj4S0Lf16UhMvJWfCVvlIL1fKzpwBPNxK KYx78Ls9jIF3NIkiT4vWsg6z2TmhQwozOlIaVLCTPNVih9uN3pD5epLZXEqbMXAvzMhp 7OilehA2MCW7+9cBqTsKeLP6MDo3BH7u7gohvpCmLKkJ5wzyivxyN+8gcu9m5jO0J3CJ FNpRn4JFiPalrZSXZPJywPrfR5NizvKDpggOpScDwWeKxQatIiqd6HbPpt5HNatYPJ+U mNlw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=cOsHXC8q; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id z4sor10087520pgr.3.2019.03.07.15.00.40 for (Google Transport Security); Thu, 07 Mar 2019 15:00:40 -0800 (PST) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=cOsHXC8q; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XVAzLd6hdIPZj35NxoEZAwxg04Tr002qNrQDR1YF6Fc=; b=cOsHXC8q1NlpYG2/9ohgQ48P9MJVgooAsBgk28QQbidN3suqAiF/mMOKqLcnQisWny D5RakrhfayswSpAIWwziHUMt52tfLfZKPBc/1xsRfVDrRIrqSdV++twQXPUiCwW8KvDa FfffGsXE2gkHQZPwi+1Kh3E4yORU25DiUZHtZhxNTkIiBGwmPNH38R/SFQP7Uv+ZDNiI 45Ptn3Amz54/mVGvbfQduB5lcVD8Qx9kFvrx+5XPyFShwlyP/VnzRKGtbQyUG9wbXLUs MPg4Z5uW7sOdJbJRFNjVgdZM99iSdhB3P6zXS1sDD9weT3lXJBR5Q956ZiY+/TEWY2nQ YCAg== X-Google-Smtp-Source: APXvYqzIHdXQsfrnBx1c44K71SD+IKizHnfv93In7LvP+Dos1vMHhkeSszM33wViMkWYf7DfSWojFQ== X-Received: by 2002:a63:6ecb:: with SMTP id j194mr14086143pgc.250.1551999639357; Thu, 07 Mar 2019 15:00:39 -0800 (PST) Received: from tower.thefacebook.com ([2620:10d:c090:200::2:d18b]) by smtp.gmail.com with ESMTPSA id i126sm11864806pfb.15.2019.03.07.15.00.38 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 07 Mar 2019 15:00:38 -0800 (PST) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH 2/5] mm: prepare to premature release of per-node lruvec_stat_cpu Date: Thu, 7 Mar 2019 15:00:30 -0800 Message-Id: <20190307230033.31975-3-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190307230033.31975-1-guro@fb.com> References: <20190307230033.31975-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Similar to the memcg's vmstats_percpu, per-memcg per-node stats consists of percpu- and atomic counterparts, and we do expect that both coexist during the whole life-cycle of the memcg. To prepare for a premature release of percpu per-node data, let's pretend that lruvec_stat_cpu is a rcu-protected pointer, which can be NULL. This patch adds corresponding checks whenever required. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 21 +++++++++++++++------ mm/memcontrol.c | 11 +++++++++-- 2 files changed, 24 insertions(+), 8 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 05ca77767c6a..8ac04632002a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -126,7 +126,7 @@ struct memcg_shrinker_map { struct mem_cgroup_per_node { struct lruvec lruvec; - struct lruvec_stat __percpu *lruvec_stat_cpu; + struct lruvec_stat __rcu /* __percpu */ *lruvec_stat_cpu; atomic_long_t lruvec_stat[NR_VM_NODE_STAT_ITEMS]; unsigned long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; @@ -682,6 +682,7 @@ static inline unsigned long lruvec_page_state(struct lruvec *lruvec, static inline void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val) { + struct lruvec_stat __percpu *lruvec_stat_cpu; struct mem_cgroup_per_node *pn; long x; @@ -697,12 +698,20 @@ static inline void __mod_lruvec_state(struct lruvec *lruvec, __mod_memcg_state(pn->memcg, idx, val); /* Update lruvec */ - x = val + __this_cpu_read(pn->lruvec_stat_cpu->count[idx]); - if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { - atomic_long_add(x, &pn->lruvec_stat[idx]); - x = 0; + rcu_read_lock(); + lruvec_stat_cpu = (struct lruvec_stat __percpu *) + rcu_dereference(pn->lruvec_stat_cpu); + if (likely(lruvec_stat_cpu)) { + x = val + __this_cpu_read(lruvec_stat_cpu->count[idx]); + if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { + atomic_long_add(x, &pn->lruvec_stat[idx]); + x = 0; + } + __this_cpu_write(lruvec_stat_cpu->count[idx], x); + } else { + atomic_long_add(val, &pn->lruvec_stat[idx]); } - __this_cpu_write(pn->lruvec_stat_cpu->count[idx], x); + rcu_read_unlock(); } static inline void mod_lruvec_state(struct lruvec *lruvec, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 803c772f354b..8f3cac02221a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2122,6 +2122,7 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) static int memcg_hotplug_cpu_dead(unsigned int cpu) { struct memcg_vmstats_percpu __percpu *vmstats_percpu; + struct lruvec_stat __percpu *lruvec_stat_cpu; struct memcg_stock_pcp *stock; struct mem_cgroup *memcg; @@ -2152,7 +2153,12 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) struct mem_cgroup_per_node *pn; pn = mem_cgroup_nodeinfo(memcg, nid); - x = this_cpu_xchg(pn->lruvec_stat_cpu->count[i], 0); + + lruvec_stat_cpu = (struct lruvec_stat __percpu*) + rcu_dereference(pn->lruvec_stat_cpu); + if (!lruvec_stat_cpu) + continue; + x = this_cpu_xchg(lruvec_stat_cpu->count[i], 0); if (x) atomic_long_add(x, &pn->lruvec_stat[i]); } @@ -4430,7 +4436,8 @@ static int alloc_mem_cgroup_per_node_info(struct mem_cgroup *memcg, int node) if (!pn) return 1; - pn->lruvec_stat_cpu = alloc_percpu(struct lruvec_stat); + rcu_assign_pointer(pn->lruvec_stat_cpu, + alloc_percpu(struct lruvec_stat)); if (!pn->lruvec_stat_cpu) { kfree(pn); return 1; From patchwork Thu Mar 7 23:00:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10843947 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8177D139A for ; Thu, 7 Mar 2019 23:00:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6F9692F665 for ; Thu, 7 Mar 2019 23:00:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 636EF2F692; Thu, 7 Mar 2019 23:00:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DBA102F665 for ; Thu, 7 Mar 2019 23:00:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C46E8E0006; Thu, 7 Mar 2019 18:00:44 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 64ACA8E0002; Thu, 7 Mar 2019 18:00:44 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C9FB8E0006; Thu, 7 Mar 2019 18:00:44 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 0C7668E0002 for ; Thu, 7 Mar 2019 18:00:44 -0500 (EST) Received: by mail-pg1-f199.google.com with SMTP id y1so17939400pgo.0 for ; Thu, 07 Mar 2019 15:00:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=SQTP2I1IGTAHWCfjeZNx8sn2B6cnWOl+ZJBV9SuBK5k=; b=hX+mdnF7A6Eklxnw4KLN4Qq5ZsEJeFjLEHMK9dZIl888bQTb+731WYDBQWLk5YDZAF aMsklQ77bStBd+USVb+nF887AmBOwBnvyKELkXFHSCHoApKIRXqzMPAZQj8KGVV3JxL4 b0kcIMOMWcUrSI7x8JJalWIKozZ2VGjo2VoQ+kzMgjmJX6PF2cdb2oj7mWvdTVf58Q44 clQG5NUMkVFm0+OIQChl+Ua78fDZgv/MTCPGKozw/x2B8odzasTehwqtSD61AlBMzRa/ mLX1NjhXyDe7sz55xMpBeHeOLQE7P0zDI6BO5f3NGgVmEP7exzg54MAm0ewRlPBYYEAm 7JFw== X-Gm-Message-State: APjAAAVGVKj1I3TdIclauitf7Tg3jPCz1sfG1qHUYsRdT55H158fOrJ7 6gk6cpEY99/PQCP3lLldEVMwzFAwsuLEt3dj6YZJHq7ngrSjCCg3d6C5muKSgMY2awNUGXT4BAA X2+xCBPZTCAiPl/DmDLLrSoBoGFwsagoooVhggrt2sjYrviSFW3czc3o+sDBEhvUtzGQJZChPGe Nxm1RXZ/w3pX6StZdcTsIv5bx+r7dnmgsSpO4sqdT35yExPhIIaQGzieQLEGUsEfw4tLdVe8oSC bMMTwT3KPWLoma3LXTUrmo4TBhokkFRDyhPNTB1MSCP10KLExVcNUoen3HvOBYmAp9Se/zbwwOK o0DGK2NdKsJ7aHNeXeoi9dE5GDQJrCYQVI1DlMyuzbKXnielDPgoXTNFJvRSXtcTb+wCy+zOBvi / X-Received: by 2002:a17:902:1105:: with SMTP id d5mr15611005pla.27.1551999643678; Thu, 07 Mar 2019 15:00:43 -0800 (PST) X-Received: by 2002:a17:902:1105:: with SMTP id d5mr15610797pla.27.1551999641318; Thu, 07 Mar 2019 15:00:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551999641; cv=none; d=google.com; s=arc-20160816; b=HWpGQcYOk4QKEj0v3/yZebub6YnXiIj988sTo/AFM/Sw8x1YsNtR6hai+V0inMGE/d IYexkv4bGieT6Ejm6vjW0arpWCmcuP10glkGCA5PfiVB2z4PYo5s5j55/leHHxxItsld f4TFdyOMySLNnp8bpBhroIMsroqSejo+NsRrOFVnevdIftC4iYPlRvNZ6icEX16mOgEg ZrqzZcTQT9eZEPRaVd6Fy8Hr6xEA9ziCTBSld6PP9OMsFXGsH3DMg2E5fu3RcJfBq18i KAxMl8Ww5aiVhKATjxTn3WP8rdH3MxrAcn6eWCpTfLdh+/7iqmBWu6T9wA+03uruFvdv O0nQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=SQTP2I1IGTAHWCfjeZNx8sn2B6cnWOl+ZJBV9SuBK5k=; b=caHEfG5CfoujyeuTq0tSoYcSy/w+87kN7eXMIfrjPM5m770mTUf1GAtqAnuuzMDyVE RWV61p451hyKGhvMz7Ru8CnN2DtdwpIYvlHl+xVkQ3giMTqIBoka1wsbSJgQtI3TwTnd DeGdWnE5ALF7dYio0MlRE4GdmVTFXYfT2sZ9dnY1CkJv6I6V4URHfov1hfcTeljPf1lO IfmaVr1bQTGWHJvbimRTxy4RqqsTbO7XUtkuIja17AY0jTFqMRvle4jRuz9VgX3EQmg6 9ZRPvfLHPat1UCz7qhySJxCrpjHsYZoDk+/BVky67M9RPWDswp4K3AfbIsHjkMRbh3x6 Qcfg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=os1d9NLE; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id z185sor9691992pgd.55.2019.03.07.15.00.41 for (Google Transport Security); Thu, 07 Mar 2019 15:00:41 -0800 (PST) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=os1d9NLE; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SQTP2I1IGTAHWCfjeZNx8sn2B6cnWOl+ZJBV9SuBK5k=; b=os1d9NLEH5YF0msLsK9q6RDbE3XUiJPCFohANPKSxtyy1x/weqPxw5h9t+FkSwtb+9 saeQFtoUHoBKFJasOMziLWJlyrqai9AU7rUCxmaiyWN5dE7t1mi2fYabAXmOvFRwbK/v dH9lNacjzQFRf+ia+gZQGZm8ZmRvl3FRXnt9eYMPesZ5xe6i/oC+USIAcfXUri7eXDCV bMWVeR5OD3oIIWdgoP/mz2x/zWuhPgEhzIJ1NiBqqtmSMCpZpsWbCakalGsqdNT/htmf dE7pCH1s6pZXaYOsvVhNaz6SWXisByM0sEPp9jVJEFN7bbWPi0Urx7ssMWxDxyGmVT3X jPlg== X-Google-Smtp-Source: APXvYqzuaBCXJLZFV1Hn/0UeWiWtkt2B6OPUoPpFb7ChIAkLJsa7GOYp5MTkoZaxbfQmXh2nd3BiVA== X-Received: by 2002:a63:4e1d:: with SMTP id c29mr13842615pgb.433.1551999640819; Thu, 07 Mar 2019 15:00:40 -0800 (PST) Received: from tower.thefacebook.com ([2620:10d:c090:200::2:d18b]) by smtp.gmail.com with ESMTPSA id i126sm11864806pfb.15.2019.03.07.15.00.39 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 07 Mar 2019 15:00:40 -0800 (PST) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH 3/5] mm: release memcg percpu data prematurely Date: Thu, 7 Mar 2019 15:00:31 -0800 Message-Id: <20190307230033.31975-4-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190307230033.31975-1-guro@fb.com> References: <20190307230033.31975-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP To reduce the memory footprint of a dying memory cgroup, let's release massive percpu data (vmstats_percpu) as early as possible, and use atomic counterparts instead. A dying cgroup can remain in the dying state for quite a long time, being pinned in memory by any reference. For example, if a page mlocked by some other cgroup, is charged to the dying cgroup, it won't go away until the page will be released. A dying memory cgroup can have some memory activity (e.g. dirty pages can be flushed after cgroup removal), but in general it's not expected to be very active in comparison to living cgroups. So reducing the memory footprint by releasing percpu data and switching over to atomics seems to be a good trade off. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 4 ++++ mm/memcontrol.c | 24 +++++++++++++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 8ac04632002a..569337514230 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -275,6 +275,10 @@ struct mem_cgroup { /* memory.stat */ struct memcg_vmstats_percpu __rcu /* __percpu */ *vmstats_percpu; + struct memcg_vmstats_percpu __percpu *vmstats_percpu_offlined; + + /* used to release non-used percpu memory */ + struct rcu_head rcu; MEMCG_PADDING(_pad2_); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 8f3cac02221a..8c55954e6f23 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4469,7 +4469,7 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg) for_each_node(node) free_mem_cgroup_per_node_info(memcg, node); - free_percpu(memcg->vmstats_percpu); + WARN_ON_ONCE(memcg->vmstats_percpu != NULL); kfree(memcg); } @@ -4612,6 +4612,26 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) return 0; } +static void mem_cgroup_free_percpu(struct rcu_head *rcu) +{ + struct mem_cgroup *memcg = container_of(rcu, struct mem_cgroup, rcu); + + free_percpu(memcg->vmstats_percpu_offlined); + WARN_ON_ONCE(memcg->vmstats_percpu); + + css_put(&memcg->css); +} + +static void mem_cgroup_offline_percpu(struct mem_cgroup *memcg) +{ + memcg->vmstats_percpu_offlined = (struct memcg_vmstats_percpu __percpu*) + rcu_dereference(memcg->vmstats_percpu); + rcu_assign_pointer(memcg->vmstats_percpu, NULL); + + css_get(&memcg->css); + call_rcu(&memcg->rcu, mem_cgroup_free_percpu); +} + static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) { struct mem_cgroup *memcg = mem_cgroup_from_css(css); @@ -4638,6 +4658,8 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) drain_all_stock(memcg); mem_cgroup_id_put(memcg); + + mem_cgroup_offline_percpu(memcg); } static void mem_cgroup_css_released(struct cgroup_subsys_state *css) From patchwork Thu Mar 7 23:00:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10843949 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 28998139A for ; Thu, 7 Mar 2019 23:00:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 15F602F630 for ; Thu, 7 Mar 2019 23:00:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0A2E12F617; Thu, 7 Mar 2019 23:00:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 87CB92F665 for ; Thu, 7 Mar 2019 23:00:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A24898E0008; Thu, 7 Mar 2019 18:00:45 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9367E8E0002; Thu, 7 Mar 2019 18:00:45 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7878A8E0008; Thu, 7 Mar 2019 18:00:45 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 3493A8E0002 for ; Thu, 7 Mar 2019 18:00:45 -0500 (EST) Received: by mail-pf1-f197.google.com with SMTP id h70so19603264pfd.11 for ; Thu, 07 Mar 2019 15:00:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=MfCrTGNmSaFUElZVoR94lwj3NK36d5qLYupM1vXTErA=; b=uknKFBkwCtE3+c+tI6mWin0sg2yTWKVajq+ycG/9mCce17lWOdCbD050lMprMvngzk Vwyv2KXJwJwNCbKgQkXKSIfMmd5kiRNMO43HDgA0KNYn5HYk+THpUFKf9L8XyIlxpYBp p7I54u8g7z0nwPw3/wBYzuq44/6eEjcQYHatwlqW8lLC4iYjprPc+9lOxjDsZ3bDKwCe mg/z9Qo2+fnVtD3bS9Mllb+4OVerm8fMxGsiNOlO/nD1hUJmPTrS8ISNsJfgo5CV4HQW 5Q8BAi/DBEuW35AbwjARkjZFjMF3lhDJxizkTQrLJ4OsFZgWveOyMiAMnUqbiniXJuW1 9DdA== X-Gm-Message-State: APjAAAX7EmoBx6+J0ism6UDE86EvpNK+gmNc3ZoesYtcAABg1EeEwy42 ZwAvRE59IyQGQsrsKE+50tkx6nG+aoc6NpPxCvy7v48D/Tp6Wg5OOxooulguZRggbZMp+73Pw5s HuDsdGigVnEa2ENQ/leKElFqtiScnJ7rGONFFgvY/d5jQA/1Po8sqEqEfmFYR1ZcuUZTjEmYjmt Vq7I4rewtVBACx0+Q5eRIWwytL0NIdS/gikdowZ+RdAxGGBUIAjS0cY3hdVXLKXiLQXBdZl6Z95 7FiiK8603mLDSdqOKh1c9z13YVB3i2Lu0wZSUatHKYHVK5X22ICqgrzskmturkrwBOcoZfHQDAQ GXsov7ufOJ4bwRm35Re+DVDquCg7dYcxwZvEgXUBSxsDwne9D6eYWK+g4odMDTN4+TFRYvkcMqv T X-Received: by 2002:a62:4743:: with SMTP id u64mr14908344pfa.95.1551999644869; Thu, 07 Mar 2019 15:00:44 -0800 (PST) X-Received: by 2002:a62:4743:: with SMTP id u64mr14908132pfa.95.1551999642323; Thu, 07 Mar 2019 15:00:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551999642; cv=none; d=google.com; s=arc-20160816; b=CCRoOR04RZoGVHYHpgU7cdAFdC4qCYwT9oGoDjPW+knQIj90ucnPIWDR1a0bjGv3Q0 ALXUaWo8Ks89ypqfXhPDPbcOmKVBu+tupud/UNg5qO7909lD7Y9B0Z8jLfyPRKgOU/2u w62aGxsxngjNFnYPFkpimIsx/YqI+Y92s/vvbQQqbRCoreFBEjMsYxqHnEVyFA3TMzQ4 XXfJwkDJHigL/z+jemsS+kmKiG2Ov1oP/lScdWzOs20cnJDwUF+HWbeCgTisoAdgDwM/ QL4tDNkXKvaMuWqG7uku83KbzRDJaLED+MnYYlswtCziMdyeEPi06j81iTwAQw2ReZik vEcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=MfCrTGNmSaFUElZVoR94lwj3NK36d5qLYupM1vXTErA=; b=RFYjwW/p3OTHjH9Z891A7jA5Zu4mZwY+hg0GA4nTh57Uwwd+IfrcQ4ppLefmN6iwun +0vqkNHapjJO6nR9x7+ZI9YwblhYA8wkRKJdF6vVn1zgV1LmvKzkqVY3FDmQrpUUufDg wV70FQKTBg2AVdnhdvIMTCbQVPwGOAhfsrijTyKXskfGkD4iazEL7J2zLDz29DMihAD/ x/hhbWAQzS7XxHqrk+34F7ZS/h/LvELe1Y4Qe/lSR4hGqM7X1QYO8/loUtfY6Az8uah3 egmQUfaIEJQzCBE/4rc7ZTVyMHg5RsW4ag95CeJduKmzf2hmK48wFmMs9sL6dJt4TXMW +6tw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="DckgoRO/"; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id u8sor9979896pgh.68.2019.03.07.15.00.42 for (Google Transport Security); Thu, 07 Mar 2019 15:00:42 -0800 (PST) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="DckgoRO/"; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MfCrTGNmSaFUElZVoR94lwj3NK36d5qLYupM1vXTErA=; b=DckgoRO/Og9fpzx5eO2KFbjkYdOnR1jrMLl012hR598K9U/OkK69u0GSHHoaGMPmDl S4buq5J+Tv3pZTi+Kdt+wCGz5NPe489QYj47U0E4z9qym9QV5kbKgACI6w2MBFcasnf5 lMenGSfadqVa9AEOW7wF8RYYuYhXOa54agdS1Q9xnVWSyU6ZnMBkcjFsl0w8dd4/CvO6 M2oo5gELJ2m+CL8xrrMFgdlvIqSY61S4ubbL10igebd/2s+pIC98lfsUCu0j5VO46PqJ k5YZmwMTIKRp3gKUaluH6M0WPUd96ydqN/tarPBdgXh/bLAmAUX9oimU/ZJTP4Kd0kPP Ikpg== X-Google-Smtp-Source: APXvYqyNrdFaTzkRpNlZuAWstlgn+HzqsgGmG/ZykiNpz92+GrArj5J170eRpoAbjVwIyx5oQFwreQ== X-Received: by 2002:a63:8542:: with SMTP id u63mr13751923pgd.323.1551999641771; Thu, 07 Mar 2019 15:00:41 -0800 (PST) Received: from tower.thefacebook.com ([2620:10d:c090:200::2:d18b]) by smtp.gmail.com with ESMTPSA id i126sm11864806pfb.15.2019.03.07.15.00.40 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 07 Mar 2019 15:00:41 -0800 (PST) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH 4/5] mm: release per-node memcg percpu data prematurely Date: Thu, 7 Mar 2019 15:00:32 -0800 Message-Id: <20190307230033.31975-5-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190307230033.31975-1-guro@fb.com> References: <20190307230033.31975-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Similar to memcg-level statistics, per-node data isn't expected to be hot after cgroup removal. Switching over to atomics and prematurely releasing percpu data helps to reduce the memory footprint of dying cgroups. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 24 +++++++++++++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 569337514230..f296693d102b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -127,6 +127,7 @@ struct mem_cgroup_per_node { struct lruvec lruvec; struct lruvec_stat __rcu /* __percpu */ *lruvec_stat_cpu; + struct lruvec_stat __percpu *lruvec_stat_cpu_offlined; atomic_long_t lruvec_stat[NR_VM_NODE_STAT_ITEMS]; unsigned long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 8c55954e6f23..18e863890392 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4459,7 +4459,7 @@ static void free_mem_cgroup_per_node_info(struct mem_cgroup *memcg, int node) if (!pn) return; - free_percpu(pn->lruvec_stat_cpu); + WARN_ON_ONCE(pn->lruvec_stat_cpu != NULL); kfree(pn); } @@ -4615,7 +4615,17 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) static void mem_cgroup_free_percpu(struct rcu_head *rcu) { struct mem_cgroup *memcg = container_of(rcu, struct mem_cgroup, rcu); + int node; + + for_each_node(node) { + struct mem_cgroup_per_node *pn = memcg->nodeinfo[node]; + if (!pn) + continue; + + free_percpu(pn->lruvec_stat_cpu_offlined); + WARN_ON_ONCE(pn->lruvec_stat_cpu != NULL); + } free_percpu(memcg->vmstats_percpu_offlined); WARN_ON_ONCE(memcg->vmstats_percpu); @@ -4624,6 +4634,18 @@ static void mem_cgroup_free_percpu(struct rcu_head *rcu) static void mem_cgroup_offline_percpu(struct mem_cgroup *memcg) { + int node; + + for_each_node(node) { + struct mem_cgroup_per_node *pn = memcg->nodeinfo[node]; + + if (!pn) + continue; + + pn->lruvec_stat_cpu_offlined = (struct lruvec_stat __percpu *) + rcu_dereference(pn->lruvec_stat_cpu); + rcu_assign_pointer(pn->lruvec_stat_cpu, NULL); + } memcg->vmstats_percpu_offlined = (struct memcg_vmstats_percpu __percpu*) rcu_dereference(memcg->vmstats_percpu); rcu_assign_pointer(memcg->vmstats_percpu, NULL); From patchwork Thu Mar 7 23:00:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10843951 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8315F14E1 for ; Thu, 7 Mar 2019 23:00:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 716842F67F for ; Thu, 7 Mar 2019 23:00:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 643F52F68F; Thu, 7 Mar 2019 23:00:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E435E2F66D for ; Thu, 7 Mar 2019 23:00:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB2978E0007; Thu, 7 Mar 2019 18:00:45 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E1A748E0002; Thu, 7 Mar 2019 18:00:45 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB7C38E0009; Thu, 7 Mar 2019 18:00:45 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 6604F8E0007 for ; Thu, 7 Mar 2019 18:00:45 -0500 (EST) Received: by mail-pf1-f200.google.com with SMTP id d5so19636665pfo.5 for ; Thu, 07 Mar 2019 15:00:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=JZX5VXWRV31e/dorMIyOAZjwZx1Lnus1V/4dOHz/P7o=; b=sKVHQ8P2PD1QG3uvp/Agrcv5Ef8alVxF3XuIhVlC+Z5S2coqtWAasX6VJOkFUOH48d E0qATS+6blzW58bX6zUXLGLs7Ov+6LnzZjk1XG3AoVKaQA3xYyE3zA5vLwswgJW2QHjM BF/He5Y0WbzjohcQpwDWZ8SpvRVgG10/rpX1bGK+3sF0im/gu3sd0aoLDJHxV+mYpIYD NSLNXahBeDrcziZXUDIq4B2HfEsPyzfc543LakooAvqlasYRo3irK8V1VUSOP86aLyCe bZp2bxwGZaDUcMtdpXBhGJKcBunqvbXu1cRj2/cZTLWDvJ87gI6m2SlpoYmySj31ZZXP 8vmA== X-Gm-Message-State: APjAAAXZpU9qcKnRpctE6qdE+45xBzCNBHg5dQsybEMxBDVFO0A94RP4 Ez2ctld4eit0VSu5PqdOAk5z3AfuDHYDrV98soat00cVrkJSre2npj12opz0Wqj1GOwPbomdCzc Y1+EToCTzDFK6NtXwZQYwjFNUIuuv3Nfm8y7QBjr0kw1+k0fs3lIXl3ZRP8S90Lm/MCO2y31cTm vKbjuHz5EZ+owBojRKu9a466gXNqMFigzrXGefCCe2/RINGOGbXDfKo25HGf3+TaV7bM08QRsB9 r41RIZWktNA89J4xS5yG3yLXAiI9rdVblBZ2n+vZ8ENDJCYtM4CBxx+AnCLS3pImBvEJY7GD3zH O1D3LrJyINV/f4B6eoaETTgaK83TP2DeDSjrlwpXf6WUBFbrprW5lYYn3gK+StmVZJxN3VsU9ZY k X-Received: by 2002:a62:f20d:: with SMTP id m13mr15051947pfh.174.1551999644982; Thu, 07 Mar 2019 15:00:44 -0800 (PST) X-Received: by 2002:a62:f20d:: with SMTP id m13mr15051816pfh.174.1551999643345; Thu, 07 Mar 2019 15:00:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551999643; cv=none; d=google.com; s=arc-20160816; b=SP/er3PRydp0E5l+z9LTKXlZM0KDimOMufn+WCbCBd4B0aRvYm2AI1aUe+4nmsmoEe DmxW5JXpUg9KMvEw6xdfiqh/nmHoa7REGhmDXlF58hQyMkwsnsIKA5YHgv5AHtGp6233 T8IciBaek01sNSPXYvW1FFCVtLmGoxyEB9mRkZKRuU/XD4lnozeifcUEhqDL0/2x5VIF w4yw2nmbMsWZKB3NGZlYG5OOA6kUQrHMTpaIBSBXXj1RF4lbMj+QsNZuHvnPJx/kjSB4 K6LlzzKlSPJlUDVtetbSKqqZAvY/18v+4frwHmrT2ma7R3dVPzXAXQnzZO+7cZecfX6s iQ4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=JZX5VXWRV31e/dorMIyOAZjwZx1Lnus1V/4dOHz/P7o=; b=uaoA5jlut1xLs+c6LAarnXKbGzm+UiCrUZ2YtFC+wxVIkbElbulzXrxBZ5mjkJeZq7 4vBO0Yw44KkME1AZvdPp0rj5o2IlpVucuAnZyNvKzsM0iQjwIB58IdAqZXRdSXvDzrSu cs/D8UHBbz07Wqp/4TQNOpMuhzO1PlWGEpogLqk4DsWQKNxB8uSHad+veUVHja0AgvZm 35P73fgk0Hssu/KagW/xMbE5XcVX/BwfTs28cG5nPuhhucj2H/IkjKdxvb+7jn5R9Iqi hFbRPwv8dHIATnheHafnTHZ7tX4CenQL/asnnP8lx5ojOI/SbeTSdKtx+J/rGZ0nGSEv 1xtQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Xm1kKMC9; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id a10sor9528123pgt.24.2019.03.07.15.00.43 for (Google Transport Security); Thu, 07 Mar 2019 15:00:43 -0800 (PST) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Xm1kKMC9; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JZX5VXWRV31e/dorMIyOAZjwZx1Lnus1V/4dOHz/P7o=; b=Xm1kKMC9B2w4t88wFEbcqoH6e3UjYFTd0TJSqsFikzco+7ogxo96EX0nP16sMJbA/D LLSVBZJBDEz2s/pXjjVn9UsVgWRbZVxLUbIf2ptRACQNUAGqUprEX6a1DRinHKw5cqeA saSPrh0ac7GIbDXM2tqBY5JabMVeKZK5pq2AM6gHHd2NgUEUM5jV35C6N+sm6BhlMSik OrbKHp1PzKc/8Lqk+TeUJEZBxlxgSWd6IjScBzR+zqvggjmPRH3lsIVbl0hk5XRSCeMU 7HEsMpuv84maXQ4wCXQUhIkvGmgg1GgNgoeCuvcEah9F6lF2KaEUozcRhvJrkXRIRfs3 bBLQ== X-Google-Smtp-Source: APXvYqyQ0WlWO2ShjokCb1rn3KTI4g8tpX1+qir97KWkQkaWwmJJtvRCFS7kkyFXX+SvJ87Cs4JUAQ== X-Received: by 2002:a63:8b42:: with SMTP id j63mr13351207pge.79.1551999642755; Thu, 07 Mar 2019 15:00:42 -0800 (PST) Received: from tower.thefacebook.com ([2620:10d:c090:200::2:d18b]) by smtp.gmail.com with ESMTPSA id i126sm11864806pfb.15.2019.03.07.15.00.41 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 07 Mar 2019 15:00:42 -0800 (PST) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH 5/5] mm: spill memcg percpu stats and events before releasing Date: Thu, 7 Mar 2019 15:00:33 -0800 Message-Id: <20190307230033.31975-6-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190307230033.31975-1-guro@fb.com> References: <20190307230033.31975-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Spill percpu stats and events data to corresponding before releasing percpu memory. Although per-cpu stats are never exactly precise, dropping them on floor regularly may lead to an accumulation of an error. So, it's safer to sync them before releasing. To minimize the number of atomic updates, let's sum all stats/events on all cpus locally, and then make a single update per entry. Signed-off-by: Roman Gushchin --- mm/memcontrol.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 18e863890392..b7eb6fac735e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4612,11 +4612,63 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) return 0; } +/* + * Spill all per-cpu stats and events into atomics. + * Try to minimize the number of atomic writes by gathering data from + * all cpus locally, and then make one atomic update. + * No locking is required, because no one has an access to + * the offlined percpu data. + */ +static void mem_cgroup_spill_offlined_percpu(struct mem_cgroup *memcg) +{ + struct memcg_vmstats_percpu __percpu *vmstats_percpu; + struct lruvec_stat __percpu *lruvec_stat_cpu; + struct mem_cgroup_per_node *pn; + int cpu, i; + long x; + + vmstats_percpu = memcg->vmstats_percpu_offlined; + + for (i = 0; i < MEMCG_NR_STAT; i++) { + int nid; + + x = 0; + for_each_possible_cpu(cpu) + x += per_cpu(vmstats_percpu->stat[i], cpu); + if (x) + atomic_long_add(x, &memcg->vmstats[i]); + + if (i >= NR_VM_NODE_STAT_ITEMS) + continue; + + for_each_node(nid) { + pn = mem_cgroup_nodeinfo(memcg, nid); + lruvec_stat_cpu = pn->lruvec_stat_cpu_offlined; + + x = 0; + for_each_possible_cpu(cpu) + x += per_cpu(lruvec_stat_cpu->count[i], cpu); + if (x) + atomic_long_add(x, &pn->lruvec_stat[i]); + } + } + + for (i = 0; i < NR_VM_EVENT_ITEMS; i++) { + x = 0; + for_each_possible_cpu(cpu) + x += per_cpu(vmstats_percpu->events[i], cpu); + if (x) + atomic_long_add(x, &memcg->vmevents[i]); + } +} + static void mem_cgroup_free_percpu(struct rcu_head *rcu) { struct mem_cgroup *memcg = container_of(rcu, struct mem_cgroup, rcu); int node; + mem_cgroup_spill_offlined_percpu(memcg); + for_each_node(node) { struct mem_cgroup_per_node *pn = memcg->nodeinfo[node];