From patchwork Wed Mar 13 18:39:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10851643 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 93E9B139A for ; Wed, 13 Mar 2019 18:40:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7E3CB29E1D for ; Wed, 13 Mar 2019 18:40:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 725A529E4D; Wed, 13 Mar 2019 18:40:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9699C29E1D for ; Wed, 13 Mar 2019 18:40:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8B1C8E0005; Wed, 13 Mar 2019 14:40:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A40508E0001; Wed, 13 Mar 2019 14:40:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 819F68E0005; Wed, 13 Mar 2019 14:40:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 437248E0001 for ; Wed, 13 Mar 2019 14:40:04 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id f12so3209902pgs.2 for ; Wed, 13 Mar 2019 11:40:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=CNCy2SGf01agfPIUoI0mNB3tC3JUsDAW7lCdNygAAfg=; b=GAfgxUUn3MY3mjLMtVVZQQmtEPZK//MmTg/N2zy72MvdTgIsUZAOQuInr7KMA6ji6B 6mYx9+f1goug3eU3Z9hoIBkGxqaOz4zL3nNez8rzNW/+ocCjZp8EhlGT6vuSQdtyYcuz aZ+ncV10ce8TTVWKfJZL5D41KN0TwCszHd/JGXmdlRtFTttTn81DJHH373HdjsQDo9HZ RDmH9xn+TG2/4HmgcPLtr8KBMkbYchm2bi2GSbHVTjzsETkUmoDehxILmvqRqGjjwWOq 9YzO3IsD2Cf6rNyPYKuYhiCDR6mPUKu2uc8oUGkAr0OFjxy4TvU207zGB8JSOySfTFI0 N/+A== X-Gm-Message-State: APjAAAUE69N4kBGhRChKQA9OCyAisvlAeKxv92dEa6NOOTbzVyJZoqtB NLpeCAq656bGNTUvrAAPxR9fzcng1kh1JMpK+pP+rZFKxHPY8owyRqZbjD7s74vB5Njrpx/3PkP wnJq3YIIgtcM7p6q3ULFT5Zgxf8t9qOX8zrwkz7IkbEuZFDcZKh45CQmcwESmoVBSylOd7hDzNS RDrrEWEGzY4fNHh1/PZsDfQZJs+ijMy0rYn0bI/VfWnCdsBvh72Bmv1POl6mbeHsS810S6PZU7P +b3JL5xSk36xcekcu59oayCLjatR0jcgJTtW6gtFqbjIo9g2P1vUfKbS95tJzAOPk4lnWvbMr// lamECFs8a7f1NhTOLFlBZ/HaeOxvtdH2kOyh7lI0NV6Ifp7JXGuPxBKoy4LVW15ZDbIVtUza8bk q X-Received: by 2002:a63:204d:: with SMTP id r13mr15642468pgm.63.1552502403916; Wed, 13 Mar 2019 11:40:03 -0700 (PDT) X-Received: by 2002:a63:204d:: with SMTP id r13mr15642379pgm.63.1552502402433; Wed, 13 Mar 2019 11:40:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552502402; cv=none; d=google.com; s=arc-20160816; b=kTfMTGjXeD+DN9o63mATU1B1N4rmeTVvLhr/nYGCKr0bDMpRZplIk2xeSZQruehPtX /WVSLyifi8Q7hrrm2+lb2V5mxYY6tIrmV3CfG7JmPvvzrLNNNuU5AKycWJCCw2Kd8xyK a75aRC0iHx5deQdwXM5p/AdXN1cLlR1GSkHQpelsUhGQVD0MvOxzIx7TxtqIpPmeXb/R AqyrT2QKYkSM4eO1uUn8VAcrh3g+IlsCPI1FbNXDY0jBHT+VI14nifsJsLKc5nox5ZpM 2NSOU9lDTAYIxvfP2E3RAtxjn29Py1cN62zspm1IXd1V/4L6WW0MHJYS01yhtdECnt0H ltHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=CNCy2SGf01agfPIUoI0mNB3tC3JUsDAW7lCdNygAAfg=; b=eI5mn2NzVhpAypM/ibGzS8d59dxKb+pO5VHg0qk6riXxj99UPuTJMDWAnzjCTlNstA uROLVYJMNQVvbPPs/BmN7yB045thTQoB8isOJIu5BgAsamXiFAmYhpn6VL0ZFIoqgomJ SkIDMXG3rxPcoXn6F7CQ1xFTE0jaFUr/S6qy/wfl58cLatXnQMkYSQza3Ta82RrZii/O +Iy/u/wV7mxNsBm2Rlw1Mlh3EvV2mlq9Iu9fryZF1D5GErcYbfRe7mYFXWOF+wNwu3so 2E806+MKSrEwKNxQUrFFOceTmX37cmh0Jvc9nesqEjVoPz2//JFOtjdPsUgQYAI998z2 gu6Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=mJoTELo+; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id p4sor19257616pgb.54.2019.03.13.11.40.02 for (Google Transport Security); Wed, 13 Mar 2019 11:40:02 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=mJoTELo+; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CNCy2SGf01agfPIUoI0mNB3tC3JUsDAW7lCdNygAAfg=; b=mJoTELo+iyROS0mFl9JODL/KJs5WbYUC7TBdhw+76tWgqH79EvhDRLhYvM6Sgp0Obr IIUaSmjkcLfafkzP8D998BLCCUbn29r22WjbrUISPJU1ke9uqeuNTiUmPf89QONLFc/F BFgCqN2hGPSuCMfgkRtu3quhTy0FgEQA/Bw+whplexOU2he9lOsCKsgTzz8JyU37U2Fr 3xLPXOOOuQi6EopT/hxlDOLaIkbd6SSnjKwDeHptJQg4ChWdB9ombVnwH1nIbrcuuFm1 3JSvgsQJlzFt90QwHP+m0zCqhkxrBOwdWCFKbCP8vcJM39cCBMgsEHHKjS2SVmTPoFHu tBVg== X-Google-Smtp-Source: APXvYqwOMrYCwRc7DDWC75xLvCPqxE473YDMEyQ6fsG8ge15uJdRhs1wjwu7FSlcZNM3ajMSVef6Ig== X-Received: by 2002:a65:424d:: with SMTP id d13mr27684845pgq.203.1552502401749; Wed, 13 Mar 2019 11:40:01 -0700 (PDT) Received: from castle.hsd1.ca.comcast.net ([2603:3024:1704:3e00::d657]) by smtp.gmail.com with ESMTPSA id i13sm15792562pgq.17.2019.03.13.11.40.00 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 13 Mar 2019 11:40:00 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH v3 1/6] mm: prepare to premature release of memcg->vmstats_percpu Date: Wed, 13 Mar 2019 11:39:48 -0700 Message-Id: <20190313183953.17854-2-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190313183953.17854-1-guro@fb.com> References: <20190313183953.17854-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Prepare to handle premature release of memcg->vmstats_percpu data. Currently it's a generic pointer which is expected to be non-NULL during the whole life time of a memcg. Switch over to the rcu-protected pointer, and carefully check it for being non-NULL. This change is a required step towards dynamic premature release of percpu memcg data. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 40 +++++++++++++++++------- mm/memcontrol.c | 62 +++++++++++++++++++++++++++++--------- 2 files changed, 77 insertions(+), 25 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 534267947664..05ca77767c6a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -274,7 +274,7 @@ struct mem_cgroup { struct task_struct *move_lock_task; /* memory.stat */ - struct memcg_vmstats_percpu __percpu *vmstats_percpu; + struct memcg_vmstats_percpu __rcu /* __percpu */ *vmstats_percpu; MEMCG_PADDING(_pad2_); @@ -597,17 +597,26 @@ static inline unsigned long memcg_page_state(struct mem_cgroup *memcg, static inline void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; long x; if (mem_cgroup_disabled()) return; - x = val + __this_cpu_read(memcg->vmstats_percpu->stat[idx]); - if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { - atomic_long_add(x, &memcg->vmstats[idx]); - x = 0; + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + if (likely(vmstats_percpu)) { + x = val + __this_cpu_read(vmstats_percpu->stat[idx]); + if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { + atomic_long_add(x, &memcg->vmstats[idx]); + x = 0; + } + __this_cpu_write(vmstats_percpu->stat[idx], x); + } else { + atomic_long_add(val, &memcg->vmstats[idx]); } - __this_cpu_write(memcg->vmstats_percpu->stat[idx], x); + rcu_read_unlock(); } /* idx can be of type enum memcg_stat_item or node_stat_item */ @@ -740,17 +749,26 @@ static inline void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, unsigned long count) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; unsigned long x; if (mem_cgroup_disabled()) return; - x = count + __this_cpu_read(memcg->vmstats_percpu->events[idx]); - if (unlikely(x > MEMCG_CHARGE_BATCH)) { - atomic_long_add(x, &memcg->vmevents[idx]); - x = 0; + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + if (likely(vmstats_percpu)) { + x = count + __this_cpu_read(vmstats_percpu->events[idx]); + if (unlikely(x > MEMCG_CHARGE_BATCH)) { + atomic_long_add(x, &memcg->vmevents[idx]); + x = 0; + } + __this_cpu_write(vmstats_percpu->events[idx], x); + } else { + atomic_long_add(count, &memcg->vmevents[idx]); } - __this_cpu_write(memcg->vmstats_percpu->events[idx], x); + rcu_read_unlock(); } static inline void count_memcg_events(struct mem_cgroup *memcg, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c532f8685aa3..803c772f354b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -697,6 +697,8 @@ static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, struct page *page, bool compound, int nr_pages) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; + /* * Here, RSS means 'mapped anon' and anon's SwapCache. Shmem/tmpfs is * counted as CACHE even if it's on ANON LRU. @@ -722,7 +724,12 @@ static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, nr_pages = -nr_pages; /* for event */ } - __this_cpu_add(memcg->vmstats_percpu->nr_page_events, nr_pages); + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + if (likely(vmstats_percpu)) + __this_cpu_add(vmstats_percpu->nr_page_events, nr_pages); + rcu_read_unlock(); } unsigned long mem_cgroup_node_nr_lru_pages(struct mem_cgroup *memcg, @@ -756,10 +763,18 @@ static unsigned long mem_cgroup_nr_lru_pages(struct mem_cgroup *memcg, static bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg, enum mem_cgroup_events_target target) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; unsigned long val, next; + bool ret = false; - val = __this_cpu_read(memcg->vmstats_percpu->nr_page_events); - next = __this_cpu_read(memcg->vmstats_percpu->targets[target]); + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + if (!vmstats_percpu) + goto out; + + val = __this_cpu_read(vmstats_percpu->nr_page_events); + next = __this_cpu_read(vmstats_percpu->targets[target]); /* from time_after() in jiffies.h */ if ((long)(next - val) < 0) { switch (target) { @@ -775,10 +790,12 @@ static bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg, default: break; } - __this_cpu_write(memcg->vmstats_percpu->targets[target], next); - return true; + __this_cpu_write(vmstats_percpu->targets[target], next); + ret = true; } - return false; +out: + rcu_read_unlock(); + return ret; } /* @@ -2104,22 +2121,29 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) static int memcg_hotplug_cpu_dead(unsigned int cpu) { + struct memcg_vmstats_percpu __percpu *vmstats_percpu; struct memcg_stock_pcp *stock; struct mem_cgroup *memcg; stock = &per_cpu(memcg_stock, cpu); drain_stock(stock); + rcu_read_lock(); for_each_mem_cgroup(memcg) { int i; + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(memcg->vmstats_percpu); + for (i = 0; i < MEMCG_NR_STAT; i++) { int nid; long x; - x = this_cpu_xchg(memcg->vmstats_percpu->stat[i], 0); - if (x) - atomic_long_add(x, &memcg->vmstats[i]); + if (vmstats_percpu) { + x = this_cpu_xchg(vmstats_percpu->stat[i], 0); + if (x) + atomic_long_add(x, &memcg->vmstats[i]); + } if (i >= NR_VM_NODE_STAT_ITEMS) continue; @@ -2137,11 +2161,14 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) for (i = 0; i < NR_VM_EVENT_ITEMS; i++) { long x; - x = this_cpu_xchg(memcg->vmstats_percpu->events[i], 0); - if (x) - atomic_long_add(x, &memcg->vmevents[i]); + if (vmstats_percpu) { + x = this_cpu_xchg(vmstats_percpu->events[i], 0); + if (x) + atomic_long_add(x, &memcg->vmevents[i]); + } } } + rcu_read_unlock(); return 0; } @@ -4464,7 +4491,8 @@ static struct mem_cgroup *mem_cgroup_alloc(void) if (memcg->id.id < 0) goto fail; - memcg->vmstats_percpu = alloc_percpu(struct memcg_vmstats_percpu); + rcu_assign_pointer(memcg->vmstats_percpu, + alloc_percpu(struct memcg_vmstats_percpu)); if (!memcg->vmstats_percpu) goto fail; @@ -6054,6 +6082,7 @@ static void uncharge_batch(const struct uncharge_gather *ug) { unsigned long nr_pages = ug->nr_anon + ug->nr_file + ug->nr_kmem; unsigned long flags; + struct memcg_vmstats_percpu __percpu *vmstats_percpu; if (!mem_cgroup_is_root(ug->memcg)) { page_counter_uncharge(&ug->memcg->memory, nr_pages); @@ -6070,7 +6099,12 @@ static void uncharge_batch(const struct uncharge_gather *ug) __mod_memcg_state(ug->memcg, MEMCG_RSS_HUGE, -ug->nr_huge); __mod_memcg_state(ug->memcg, NR_SHMEM, -ug->nr_shmem); __count_memcg_events(ug->memcg, PGPGOUT, ug->pgpgout); - __this_cpu_add(ug->memcg->vmstats_percpu->nr_page_events, nr_pages); + rcu_read_lock(); + vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) + rcu_dereference(ug->memcg->vmstats_percpu); + if (likely(vmstats_percpu)) + __this_cpu_add(vmstats_percpu->nr_page_events, nr_pages); + rcu_read_unlock(); memcg_check_events(ug->memcg, ug->dummy_page); local_irq_restore(flags); From patchwork Wed Mar 13 18:39:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10851645 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 31E7B139A for ; Wed, 13 Mar 2019 18:40:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1E86629FBE for ; Wed, 13 Mar 2019 18:40:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 12C7829F89; Wed, 13 Mar 2019 18:40:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 52C1929EEB for ; Wed, 13 Mar 2019 18:40:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 867EA8E0006; Wed, 13 Mar 2019 14:40:06 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 81A998E0001; Wed, 13 Mar 2019 14:40:06 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 644628E0006; Wed, 13 Mar 2019 14:40:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 2489C8E0001 for ; Wed, 13 Mar 2019 14:40:06 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id c15so3101200pfn.11 for ; Wed, 13 Mar 2019 11:40:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=oz/TqQdvSWg5Jr50nmxxA8VLsKG86DG+TYlsVRNN9G4=; b=Hc3jRM7WJVD8Wd3QjV/oCr7dwMAdRNsFH8o9q4FsOVlwL+gOs4wmgokeHfHpNkMZVi 0ayrwlgI7pzdRAu+BuPhdVavwNqOC8NnyobgiAT8298Kbh01ICHKHGvGGho9oYw/PLHM F7o0Uuu5nao6qQWScML2gqtxLqJGWsQr6Iv4LsMUqWsmK2b2z02NwBktMgcj2k21Q+YH kAl3KifeGQ/s8K0WWGehoRytlHjoZQwpH4Q47z+g/EIW2F2tyQh8U0OZPdJLOvypqFyP YMvblU8zjZaTn1BFcHzOTXfJltEOSVAe+V20IiAIXiogGJLYz9F60jovstBfzP2I3KO0 spOg== X-Gm-Message-State: APjAAAXMddKjMHm5UIAlnXTcDOubZHC0rViFezpHl+sNWVckJdsRwiH2 QNNyFuDd8lYemnq52jLETi5zbEpa3hq6/gveevIv2h0pWgia5+nYY91uM1OOp0CEYSf9w3ubQII qCpqzdiHN2w6zUFbAeGQAuNph1AhxuOvyTuA5H1y2KYPUSDZl7QmxvSmeuiLFieHQoryfbsqEzc YTsyU/IUZPbfBGyMxevRyEqZ3ESjtP4Y4IAc1QUDjh+BRXWUWSisadvGNVZsksOKfQEW10Xfnqk mfeiGn82eVWb6AcZ/LX+Uu7GxqudkMKBnSWisEn70f5u3mbymx4cFv5j8RNFKruqgmsWqnsBSJN OhqBg448q7umR8pJu8e3653uoP2vRddQwBZJ6QGMv/n3OUSygf6j5ijBfAVyCQWFe/OsA95gkK3 f X-Received: by 2002:a62:ee13:: with SMTP id e19mr45315253pfi.224.1552502405815; Wed, 13 Mar 2019 11:40:05 -0700 (PDT) X-Received: by 2002:a62:ee13:: with SMTP id e19mr45315137pfi.224.1552502404146; Wed, 13 Mar 2019 11:40:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552502404; cv=none; d=google.com; s=arc-20160816; b=CE6+oR28SDThB28SkdC4rdmK0xyCRqvg+/zAg4tNNBXC07e/igv7ZFmpMr0IX+pLPU 0lQ6YGcK3wjLovWGDAxVCYh0Utuy1TjAAexV/nt7vn4M+OEJmZ8nQEVJ3oWwnrqh935o jO62PeO5NzJsu2BIsOxIJXZqb3WR+B0IfJDeZyMH7IGEefkknQw+RpdqA8tg0xBWbzy3 GlERp1jpvx9/4Bn6de8/QHY8t6Tpjk18rg1de/+9NfMZ9lUFGj9KYkLgja+65VTKtjhP 5qFT+eoA8NXtC99lriDBM3viIh3LIz4F1/3B2XnMJadMuevZhHpnEWiK2DQfjd3TO4dA 7u0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=oz/TqQdvSWg5Jr50nmxxA8VLsKG86DG+TYlsVRNN9G4=; b=iMOPM5fS+qPLynA/QK+EPLVMSCeEZJnmGINcmxQINZvPB804ne07oHTZ/FUJ7pMi30 bBf/oh9dbmC7MW4iBnD8DFzsSkfY6qBx3a4bgfY1aOMBqfcHj3/aZqzlzChu52/TbESK smF+pFV1fgjonw/JSTbtWTfAgmFsSrndUJ0Xt9ZyBOxvm8got3UrGVVuzGlsPsaVrG5+ Klsd73NnZho3+GjydBnS/mnLwuFFaBtidmcYFD14V7hlumYnmsW9YlmLgHWeDEGP5Haz TqqbbZI6mUpEdz+NyiCKme4ZCaperYiNtuLZ/zuPRbpR3xj+xKAcMFseKy4d/kH4uXW5 wrqQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=hpA4uXez; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id gb2sor2442695plb.38.2019.03.13.11.40.04 for (Google Transport Security); Wed, 13 Mar 2019 11:40:04 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=hpA4uXez; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oz/TqQdvSWg5Jr50nmxxA8VLsKG86DG+TYlsVRNN9G4=; b=hpA4uXezJFytHX61offN86IKtbXpFLvm0HHYlX8PLBCp9odTDBdRU8iJWN3FkjG122 b3I5QU+GFy+ySN9iL/v1WbxdD4J5pM6RZhnbR/5Xy6yEWtUUb4SgbDfEL8AWQzU3ksc9 dxD0BT3xL9WkpAUrESpPq67O28TWu0JXAd6O2VkjjX5d4B+JUKYID4HLiDzUt007FtaP kcj3J+jPbkdIisv9YZiwBzrDl94TKH+SuAm4SQFwBLlZI5bgnTYIN6xU+op53OMku+3l ngLhXVK4SKh5wj9C6wp1kVoBQJwpZhaOJdgBC4HoUu/DX+kYq2axNWUAX6d2QOVCnjri 9lCw== X-Google-Smtp-Source: APXvYqx4HX+wI/C6J5UVMWvM/fRe8b3x/cIh7frkr+wyT2vN7/DzQXFBG11e1pn8YPu0Tg1SxzbrDw== X-Received: by 2002:a17:902:8c8a:: with SMTP id t10mr10452660plo.160.1552502403281; Wed, 13 Mar 2019 11:40:03 -0700 (PDT) Received: from castle.hsd1.ca.comcast.net ([2603:3024:1704:3e00::d657]) by smtp.gmail.com with ESMTPSA id i13sm15792562pgq.17.2019.03.13.11.40.01 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 13 Mar 2019 11:40:02 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH v3 2/6] mm: prepare to premature release of per-node lruvec_stat_cpu Date: Wed, 13 Mar 2019 11:39:49 -0700 Message-Id: <20190313183953.17854-3-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190313183953.17854-1-guro@fb.com> References: <20190313183953.17854-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Similar to the memcg's vmstats_percpu, per-memcg per-node stats consists of percpu- and atomic counterparts, and we do expect that both coexist during the whole life-cycle of the memcg. To prepare for a premature release of percpu per-node data, let's pretend that lruvec_stat_cpu is a rcu-protected pointer, which can be NULL. This patch adds corresponding checks whenever required. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 21 +++++++++++++++------ mm/memcontrol.c | 14 +++++++++++--- 2 files changed, 26 insertions(+), 9 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 05ca77767c6a..8ac04632002a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -126,7 +126,7 @@ struct memcg_shrinker_map { struct mem_cgroup_per_node { struct lruvec lruvec; - struct lruvec_stat __percpu *lruvec_stat_cpu; + struct lruvec_stat __rcu /* __percpu */ *lruvec_stat_cpu; atomic_long_t lruvec_stat[NR_VM_NODE_STAT_ITEMS]; unsigned long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; @@ -682,6 +682,7 @@ static inline unsigned long lruvec_page_state(struct lruvec *lruvec, static inline void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val) { + struct lruvec_stat __percpu *lruvec_stat_cpu; struct mem_cgroup_per_node *pn; long x; @@ -697,12 +698,20 @@ static inline void __mod_lruvec_state(struct lruvec *lruvec, __mod_memcg_state(pn->memcg, idx, val); /* Update lruvec */ - x = val + __this_cpu_read(pn->lruvec_stat_cpu->count[idx]); - if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { - atomic_long_add(x, &pn->lruvec_stat[idx]); - x = 0; + rcu_read_lock(); + lruvec_stat_cpu = (struct lruvec_stat __percpu *) + rcu_dereference(pn->lruvec_stat_cpu); + if (likely(lruvec_stat_cpu)) { + x = val + __this_cpu_read(lruvec_stat_cpu->count[idx]); + if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { + atomic_long_add(x, &pn->lruvec_stat[idx]); + x = 0; + } + __this_cpu_write(lruvec_stat_cpu->count[idx], x); + } else { + atomic_long_add(val, &pn->lruvec_stat[idx]); } - __this_cpu_write(pn->lruvec_stat_cpu->count[idx], x); + rcu_read_unlock(); } static inline void mod_lruvec_state(struct lruvec *lruvec, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 803c772f354b..5ef4098f3f8d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2122,6 +2122,7 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) static int memcg_hotplug_cpu_dead(unsigned int cpu) { struct memcg_vmstats_percpu __percpu *vmstats_percpu; + struct lruvec_stat __percpu *lruvec_stat_cpu; struct memcg_stock_pcp *stock; struct mem_cgroup *memcg; @@ -2152,7 +2153,12 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) struct mem_cgroup_per_node *pn; pn = mem_cgroup_nodeinfo(memcg, nid); - x = this_cpu_xchg(pn->lruvec_stat_cpu->count[i], 0); + + lruvec_stat_cpu = (struct lruvec_stat __percpu*) + rcu_dereference(pn->lruvec_stat_cpu); + if (!lruvec_stat_cpu) + continue; + x = this_cpu_xchg(lruvec_stat_cpu->count[i], 0); if (x) atomic_long_add(x, &pn->lruvec_stat[i]); } @@ -4414,6 +4420,7 @@ struct mem_cgroup *mem_cgroup_from_id(unsigned short id) static int alloc_mem_cgroup_per_node_info(struct mem_cgroup *memcg, int node) { + struct lruvec_stat __percpu *lruvec_stat_cpu; struct mem_cgroup_per_node *pn; int tmp = node; /* @@ -4430,11 +4437,12 @@ static int alloc_mem_cgroup_per_node_info(struct mem_cgroup *memcg, int node) if (!pn) return 1; - pn->lruvec_stat_cpu = alloc_percpu(struct lruvec_stat); - if (!pn->lruvec_stat_cpu) { + lruvec_stat_cpu = alloc_percpu(struct lruvec_stat); + if (!lruvec_stat_cpu) { kfree(pn); return 1; } + rcu_assign_pointer(pn->lruvec_stat_cpu, lruvec_stat_cpu); lruvec_init(&pn->lruvec); pn->usage_in_excess = 0; From patchwork Wed Mar 13 18:39:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10851647 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BCCEB6C2 for ; Wed, 13 Mar 2019 18:40:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A8F8C29F58 for ; Wed, 13 Mar 2019 18:40:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9D83929FC8; Wed, 13 Mar 2019 18:40:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1CFB429FCD for ; Wed, 13 Mar 2019 18:40:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D7D7A8E0007; Wed, 13 Mar 2019 14:40:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CD59F8E0001; Wed, 13 Mar 2019 14:40:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB32F8E0007; Wed, 13 Mar 2019 14:40:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 6CD998E0001 for ; Wed, 13 Mar 2019 14:40:07 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id n24so3156188pgm.17 for ; Wed, 13 Mar 2019 11:40:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=WITBUxeGSIQmxb1SdGnY+GHNprTyqfSQ+9QDULNZwIA=; b=AEdkU0OFrA0LhorZKSs3UxtpyoQyg1IP6LPi17H3t/E89Js5ggiJKx39chKVeBbj7O KxnvgAMx93PmyCE5Bu70x2zHAEXVpPnOnnSAEYLuxymA8DeS/HFTSlBuW8/F9RgeptbF rm47Uz8vSvXn7OHOE6vddq06VcUprO7ehu37hKjzSyGSa0bWU+oBJ47Rfs8J2IlQ6E3N QV4x/N6OlKuClNdpBCsHwLEkK3Fzfr7bT4JpWnsH9cCjZ0b0q96z5JKa6QgvCnR1gqds D9cJ6CULtmCTA+oZqpzrvBmBUG7QSGMQsRPn4YIZa01v9QP/mwGS5ic3FZMJ9cBi/tv+ 5ydA== X-Gm-Message-State: APjAAAV4vE6aSoEUJ8uwm2Xfd6XLWR0moKRkvV7yXdfLCGNE2/44gLVB cfcGUvEgvPPFFhrXj3n8WVXfXv36QGma6NHpnu1NPuPa6lzgjTah9kMtr4wt8pmULmz4Q2brb5D sNicppIxvq49TJ3DQ3huhJEQ71BtTUA6O1QCBTboNW/IXGow0H6/l0DmWSClkcUbDs7cQYZ3Mig vAlQIoNNj81LaQ1BcZobYzPSB3xJ052lAKEEuOvDP/7FVy3iO70swg/BdQn7ICrHyR6JDna1Oj/ n6C4+XLy6QS+MrHCt7Ay0+Q+JQxmjgumxoQm0hVIfg8JNR3xN63lUN0yrwryz3N4RXVGOy9m8X3 Rr+8TE/XZAO3/UnFZHeHNY0IWsK/tkHzKH0Lhd0FLLdi86+/9O6PoAgf91aRLZJwlsITreJ9QYF + X-Received: by 2002:a65:628f:: with SMTP id f15mr41805323pgv.410.1552502407094; Wed, 13 Mar 2019 11:40:07 -0700 (PDT) X-Received: by 2002:a65:628f:: with SMTP id f15mr41805220pgv.410.1552502405325; Wed, 13 Mar 2019 11:40:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552502405; cv=none; d=google.com; s=arc-20160816; b=IrHIBO+Q/EeJAcEqcFBkZhorO85S+KbyxTLQ0lYzvzmoLaAQZZS/CW7T2Q3bX4DZtu 7dt7j+vyozCsuMkLM52PQ0tiAal2jK2Xyc7Oe9jhyCqo7xJ2LEcf61NZz+4zCSPBDPqq HheBtQG++tlOq3W5YoX7qlvGd+/4Kp26K/obyrD19jjk47yrumzSsZwwNfMlkaTOMevB YKzuNzYcg8RwgpZG6Ni0+5M2YwIQ5stn3V7F/DdqOPiT/PEX7LeplxnS/69GDWP3FOGH DjFpNMpfWPicHMXTm3livyWJnpJBeYOTyqMYG5zwbji8KbpA61IgWKzo6VowItbWI3H/ qpIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=WITBUxeGSIQmxb1SdGnY+GHNprTyqfSQ+9QDULNZwIA=; b=ko1bc6rnkDgXCeJE7n/NFKtJ+YMQZcs684o49UPHFFJmFJqPQ0DZmDQRmjLpriHUxZ pvrP3P/BEdVjTwYTlnrIzVwtiWVU5HZqM0PooO2j6R9IMPrOBh4DURLZlAeMpmDAjd22 7cKoscn2rBBEPTpQMb12oY2ychGQQq8TpcPeg5osEHYV9p5O+9JARJyL/HSo8CUEp/jZ vGkuXvmrCsbFhVylMxUwAr81VqyQOVbv6WqYwsDqdWigNoLMrmPEB7rrBc8T6v32t4Rx xwjoEIM61geqqYlgDpnHRn3z0RoRifbVBLFnbYy//nKYQWCR48LycTL5iaezu/Bd0BMg G5NA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=hDD+JwIe; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id e15sor19428312pgv.66.2019.03.13.11.40.05 for (Google Transport Security); Wed, 13 Mar 2019 11:40:05 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=hDD+JwIe; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WITBUxeGSIQmxb1SdGnY+GHNprTyqfSQ+9QDULNZwIA=; b=hDD+JwIeip0gJspQJIlifBLo3RZkJBVwzeLhOJ71FgqywT6SXzvZP9DGDEs867pwdR WO8p8O0KNF1rt/jZhEK7qRkFmJuWonuupzsv25wmsIUlubrGq2+uI/WUOazBl55WoERs 6UGSTlw8yD5nIom+ZJg9sOINX3RB3BWpRWxKCIaOMpU3i5NEQIlcg9w7rqfEpTjqevdR Ld2PdY9k3jCJEkbbVafk5rGX8TJUQ4ZkcPAQoycBwKqG3ienRmFTdSplpjS6WxomRczi beS9hglAoeVCRuUSVrSHU2CeKJUnBfeBNkiuJhuizLUdgu8x9wOBdrgC0POHx/euvdoo Zy9Q== X-Google-Smtp-Source: APXvYqwpwjKd2OLLFUS/RUgUqKYXyrVqeFcG3DA+RFLUuCtoNO4f0ufr+hHUm2yNJIlTPoM2CaI/SQ== X-Received: by 2002:a63:cd02:: with SMTP id i2mr41652810pgg.111.1552502404731; Wed, 13 Mar 2019 11:40:04 -0700 (PDT) Received: from castle.hsd1.ca.comcast.net ([2603:3024:1704:3e00::d657]) by smtp.gmail.com with ESMTPSA id i13sm15792562pgq.17.2019.03.13.11.40.03 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 13 Mar 2019 11:40:03 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH v3 3/6] mm: release memcg percpu data prematurely Date: Wed, 13 Mar 2019 11:39:50 -0700 Message-Id: <20190313183953.17854-4-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190313183953.17854-1-guro@fb.com> References: <20190313183953.17854-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP To reduce the memory footprint of a dying memory cgroup, let's release massive percpu data (vmstats_percpu) as early as possible, and use atomic counterparts instead. A dying cgroup can remain in the dying state for quite a long time, being pinned in memory by any reference. For example, if a page mlocked by some other cgroup, is charged to the dying cgroup, it won't go away until the page will be released. A dying memory cgroup can have some memory activity (e.g. dirty pages can be flushed after cgroup removal), but in general it's not expected to be very active in comparison to living cgroups. So reducing the memory footprint by releasing percpu data and switching over to atomics seems to be a good trade off. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 4 ++++ mm/memcontrol.c | 24 +++++++++++++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 8ac04632002a..569337514230 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -275,6 +275,10 @@ struct mem_cgroup { /* memory.stat */ struct memcg_vmstats_percpu __rcu /* __percpu */ *vmstats_percpu; + struct memcg_vmstats_percpu __percpu *vmstats_percpu_offlined; + + /* used to release non-used percpu memory */ + struct rcu_head rcu; MEMCG_PADDING(_pad2_); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5ef4098f3f8d..efd5bc131a38 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4470,7 +4470,7 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg) for_each_node(node) free_mem_cgroup_per_node_info(memcg, node); - free_percpu(memcg->vmstats_percpu); + WARN_ON_ONCE(memcg->vmstats_percpu != NULL); kfree(memcg); } @@ -4613,6 +4613,26 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) return 0; } +static void percpu_rcu_free(struct rcu_head *rcu) +{ + struct mem_cgroup *memcg = container_of(rcu, struct mem_cgroup, rcu); + + free_percpu(memcg->vmstats_percpu_offlined); + WARN_ON_ONCE(memcg->vmstats_percpu); + + css_put(&memcg->css); +} + +static void mem_cgroup_offline_percpu(struct mem_cgroup *memcg) +{ + memcg->vmstats_percpu_offlined = (struct memcg_vmstats_percpu __percpu*) + rcu_dereference(memcg->vmstats_percpu); + rcu_assign_pointer(memcg->vmstats_percpu, NULL); + + css_get(&memcg->css); + call_rcu(&memcg->rcu, percpu_rcu_free); +} + static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) { struct mem_cgroup *memcg = mem_cgroup_from_css(css); @@ -4639,6 +4659,8 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) drain_all_stock(memcg); mem_cgroup_id_put(memcg); + + mem_cgroup_offline_percpu(memcg); } static void mem_cgroup_css_released(struct cgroup_subsys_state *css) From patchwork Wed Mar 13 18:39:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10851649 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 57644139A for ; Wed, 13 Mar 2019 18:40:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3FC1E29E1D for ; Wed, 13 Mar 2019 18:40:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 33EEC29E84; Wed, 13 Mar 2019 18:40:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B659529E1D for ; Wed, 13 Mar 2019 18:40:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D26F48E0008; Wed, 13 Mar 2019 14:40:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C63668E0001; Wed, 13 Mar 2019 14:40:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB35C8E0008; Wed, 13 Mar 2019 14:40:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 6DAC48E0001 for ; Wed, 13 Mar 2019 14:40:08 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id f12so3210089pgs.2 for ; Wed, 13 Mar 2019 11:40:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ZKgL4zeK+z0St01hNMYKSt6dKMvw92MkjAHduvDs3M4=; b=SO41XlPORO7O3EvAl3horS9/7rFS9PkggcjhrMH29Vm3l3L8Zg/Nd/xh/Gs9TyapkD T7RLthumZAvtlVmy1bhaZnqymrtvwiUTMYt2aHuCqAmrJbAmTfEqGtNcs+28jII7PG8x htqKwx5kGD1SGcnFnBmXkZ2zI4zhidL0B8ijVmQ1dZmM3ToEGMHd9jHNeTSKzuH4HHo7 OVKaE1DIEYs3yAxMj+/2XODFjG7tLbt42Cw77/p61N+HeRcY1wSPDxVF1wrkgJbA8kQ8 BMLQvk/MN7PnIMzFM6SVrK5ZTrC7N17tRuHyt/LidfbIscTbBgNVuF8/06gVjc/h4hyZ QwHA== X-Gm-Message-State: APjAAAU3Qs+sVKHmURVE9BtZsJnVu770/eGaNL5jJ6lcDhoFLpwqrOTJ IoHyay4x1fqFzPQOHtF8HlraInPdf6pTRMjA5RSm6pUTVeEydpLrNTmjPNaY9RyQBZL2nEedHb8 L44iX39HHFox3CRYjmRD66TvTsv2ej4AAQY5tmLUDqdxJ7tC+4LqvPBA+/25Wm8E3K47aG5xpPY Wqsku9KsYUiYU8mvaDpblB4v0/sceLLbEqbgccisThxLm36xk+4rrjl/eZFtF5ltyUw+SU+EGjg Z+aKDQbQgxqHKOcQIHhRCFmA4kqlrYOiIVWBg/S/kyB6Rrsr9hAOl6LUXAR8gO2dVbnfEXsZzdE JCXkGZlCcfa599rQzn55n9o1tiuMYMHGL4uUymNDV+Uto7K06dd+7aKl6z8+Wk4F/aQtItMghBl n X-Received: by 2002:a17:902:784c:: with SMTP id e12mr46569360pln.117.1552502408137; Wed, 13 Mar 2019 11:40:08 -0700 (PDT) X-Received: by 2002:a17:902:784c:: with SMTP id e12mr46569272pln.117.1552502406730; Wed, 13 Mar 2019 11:40:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552502406; cv=none; d=google.com; s=arc-20160816; b=jNcJglDab4GLDRPAOBsg88+qWIqYIMxNgFgsqvH6O4L+Lic9YOlI5ZD14JjkhgpI/u Dot5UDwz9G4N2xWFZtx5GgZxthIpfJvE3tLdxTgjZegnyVXTRtiYoNZPkdSolhAgiZ7v oJmzRSI0HruJ5flWqWVGBTxUvKC+0L9+91V9JuBHEsYvSE2AKlR8NUtzBxVvtvOHR4nI ljqo/odAB33V/lslx8eqe+yM2R/wKLzM290evm90HOifbhn0nKjOijWzy0qliMnJzmWR Vjko/ltSoiZGAaWBkjlEdmOcPo2eBQxqCFbf9KyZg7wCRfXkhOqRmkXDsAqIsYYb3dKh BCZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=ZKgL4zeK+z0St01hNMYKSt6dKMvw92MkjAHduvDs3M4=; b=pQ6M1C4C4+mndphaEj22eI812Ix7TsIwfqjMJoNWztuzuRQx3i46k9xgy0mUmmNeaQ U5duAan4EALxa3FaKuBblRtsz9idJpCvFHtIX07kkumYAQT3Mz1+gCCDHfu0YfY83B/J jMszeJdYtOhjC4pCVybyu8Y4OHP1I2hVtM89Rpldi5s0w9OZBj+KRafMVau+6Yrvw4vu DSBNTElzcIy4AWscW7KlhKsDjfiA5H3i7QRxXvecZ03UVqKF5p4u5D3m9/k3lKq8j4pj aOKzMTJksXXDUSO+T9Ybsz1gStFTr71gafdHX0y8UAaCFFQEjGM7wIYbPxMk8OoapDCa 8oDA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=TrX0uTyr; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id 34sor2430917plp.24.2019.03.13.11.40.06 for (Google Transport Security); Wed, 13 Mar 2019 11:40:06 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=TrX0uTyr; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZKgL4zeK+z0St01hNMYKSt6dKMvw92MkjAHduvDs3M4=; b=TrX0uTyrR8JGkApkgIGcwFbP61RnpY6vKQwkP3/WLW7+611dwmic3jm/UWC51wVBzv ZxmlfoRbLDOV/svdWScPzZ/35vHi6eAsW2cpY8KPHcmb/cksQIqgn3y0kW3HqrYsAViU PARJAv80Ak4rosKlqFWfUugSTU+uhn9jnGEMIaeC64cakf8i7ChAiCsDE+0A9ZwEGCge FYEJhOYXTTU3uD2MQVw2/thiMnBJdqoCRJOKTPFBfU/2O590YceAv1YBFw8evhN+Zv2A cTfzhAI9/zmULZ+hon4mYfTi8ukHQe/72jihu//A4YZaOWgfgWZWS/peYsKluEf3cG2l xV2g== X-Google-Smtp-Source: APXvYqxV2blt47QtOVx+WggwDQ01SnXjX2Y9PFVIr3pxDFvknWA+oQdov2HJkEBELj1nvrrU2TpWmg== X-Received: by 2002:a17:902:3:: with SMTP id 3mr41784620pla.114.1552502406117; Wed, 13 Mar 2019 11:40:06 -0700 (PDT) Received: from castle.hsd1.ca.comcast.net ([2603:3024:1704:3e00::d657]) by smtp.gmail.com with ESMTPSA id i13sm15792562pgq.17.2019.03.13.11.40.04 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 13 Mar 2019 11:40:05 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH v3 4/6] mm: release per-node memcg percpu data prematurely Date: Wed, 13 Mar 2019 11:39:51 -0700 Message-Id: <20190313183953.17854-5-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190313183953.17854-1-guro@fb.com> References: <20190313183953.17854-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Similar to memcg-level statistics, per-node data isn't expected to be hot after cgroup removal. Switching over to atomics and prematurely releasing percpu data helps to reduce the memory footprint of dying cgroups. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 24 +++++++++++++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 569337514230..f296693d102b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -127,6 +127,7 @@ struct mem_cgroup_per_node { struct lruvec lruvec; struct lruvec_stat __rcu /* __percpu */ *lruvec_stat_cpu; + struct lruvec_stat __percpu *lruvec_stat_cpu_offlined; atomic_long_t lruvec_stat[NR_VM_NODE_STAT_ITEMS]; unsigned long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index efd5bc131a38..1b5fe826d6d0 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4460,7 +4460,7 @@ static void free_mem_cgroup_per_node_info(struct mem_cgroup *memcg, int node) if (!pn) return; - free_percpu(pn->lruvec_stat_cpu); + WARN_ON_ONCE(pn->lruvec_stat_cpu != NULL); kfree(pn); } @@ -4616,7 +4616,17 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) static void percpu_rcu_free(struct rcu_head *rcu) { struct mem_cgroup *memcg = container_of(rcu, struct mem_cgroup, rcu); + int node; + + for_each_node(node) { + struct mem_cgroup_per_node *pn = memcg->nodeinfo[node]; + if (!pn) + continue; + + free_percpu(pn->lruvec_stat_cpu_offlined); + WARN_ON_ONCE(pn->lruvec_stat_cpu != NULL); + } free_percpu(memcg->vmstats_percpu_offlined); WARN_ON_ONCE(memcg->vmstats_percpu); @@ -4625,6 +4635,18 @@ static void percpu_rcu_free(struct rcu_head *rcu) static void mem_cgroup_offline_percpu(struct mem_cgroup *memcg) { + int node; + + for_each_node(node) { + struct mem_cgroup_per_node *pn = memcg->nodeinfo[node]; + + if (!pn) + continue; + + pn->lruvec_stat_cpu_offlined = (struct lruvec_stat __percpu *) + rcu_dereference(pn->lruvec_stat_cpu); + rcu_assign_pointer(pn->lruvec_stat_cpu, NULL); + } memcg->vmstats_percpu_offlined = (struct memcg_vmstats_percpu __percpu*) rcu_dereference(memcg->vmstats_percpu); rcu_assign_pointer(memcg->vmstats_percpu, NULL); From patchwork Wed Mar 13 18:39:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10851651 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3EA57139A for ; Wed, 13 Mar 2019 18:40:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2992029FC8 for ; Wed, 13 Mar 2019 18:40:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 24B2329FCD; Wed, 13 Mar 2019 18:40:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9A68E29FD6 for ; Wed, 13 Mar 2019 18:40:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 276538E0009; Wed, 13 Mar 2019 14:40:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 201C98E0001; Wed, 13 Mar 2019 14:40:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 053AA8E0009; Wed, 13 Mar 2019 14:40:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id BB8F28E0001 for ; Wed, 13 Mar 2019 14:40:09 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id f12so3210152pgs.2 for ; Wed, 13 Mar 2019 11:40:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=2cvW24YvYXEUFU3KxBgLfjLN1zP9HeM/T5I/fane4LI=; b=p4eldCgAt4tikuf+ZY2WUZ4Wai7ut7uHfocEPy1SxPcWeKPYU2dgapyAxcC3Tu3o+0 HXfY08sHsV6t2Q2T8Nvaq/28IbviBmcFGW1Lsdn0j6nNR3ht/ob+7UXDueRvfZrlcWhN tvaXonwEUzRmaqYqTOZBaMhyw9pG4YL7El29Q2Vt5UTkeeJkDbjlEzufpmtSFHKrUKMd McqXz8YdaPsHXH1v3ENI+uq/F2AK2XKeLPCDaThHij5ozmwnAJiU45igelB8d/+ENnA6 frEmonawqDWok3s/d670Unyh+Lot4u5uFQef9PJlytdkXVSXuLaifLe3FGF+Vlw0VBDG j7Tg== X-Gm-Message-State: APjAAAV4cbu3kAt7A7eeEjXOs91w54kXDJ/JDdfNqHz48jtg/qUFWt63 RSQXB4IgdaUgXM6XxDNpx40mSGvAaXte6PSMofBwnstw91pnTbHZ63VXUR++uCpWMDq2hRdwO4P P2S1IjMvJIOaOuZT7doIJyUoTyWK9pGvd38lR0HE63UjzCdaQw0hZIN/dD2gQaniG6EaYHbjjHG OYVKXKlVRuldqQSY+MM3mAOEIQMFOnTKxircYVgnZsyvkyX5S3F1wmIscK7+I+0wvuSjmlHQXE9 cqX+QWAPKqs6Y+JQ8DL0L72w6LPaqaXZSWINmru5VEQIX+bQVUvJXD6Zj5tV9vZ9QYU5w2BwHWV L1fGrRjhSTlMAjbty97645dzEd3nhBW0OILzuxx5iB1SththWkpmv6VInWiPA58uTo4DSGWw2j0 q X-Received: by 2002:a17:902:2e01:: with SMTP id q1mr463901plb.253.1552502409445; Wed, 13 Mar 2019 11:40:09 -0700 (PDT) X-Received: by 2002:a17:902:2e01:: with SMTP id q1mr463813plb.253.1552502408085; Wed, 13 Mar 2019 11:40:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552502408; cv=none; d=google.com; s=arc-20160816; b=lSyvu6D67Va+MvQCXoXp9Mu9ozXBwVwM03m1Jt+0Q0DBGGRi/NzuW5dcqjZtumFS3j Ha0Amchy8jXrXIWH6AcHkDxlTmsay5bjrQUV6LJSN192IO0SJOXOn1DL/4lDa5X69HuL S1Po3ILzSRSKCWazKIHRIDt3/r4dyZpYQoCvP3Ig1mR1GAyeFDMBMvRy1PM0bTj2ZseK pJFypA8fBLCPdzLobYaCtyUW2L1OsdaQjXbvk4FJhNxLu8yHckIYKmA41dX0GGlb1ANL 6H/6RTbsZ0D1JLRgubZTGxx2AqX1/35L7AYK11OJ8lMnpuxZmtfcB7O/dk3rPI/WDoob q9cg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=2cvW24YvYXEUFU3KxBgLfjLN1zP9HeM/T5I/fane4LI=; b=ZbyP9JQzShzojQ1wglly++0iitOoNhyZS8Y8ue5oHRJ8TuILbaka3879jtPwKcnk6S Dz4xmnJiXJA996R8ttUbbsfjzh+D0H4+99qu3JkhGlDap8U+EL/RXBjnWu6TOGsLN6Mk FudE9xcZkdyWwF1HOZx0UI8P7Bhp9Rhc84RfnZN6CD8akRjod1a4Zp9sl8c/d/SUMbjx G4lI3iU9FbWaKxwp6DhjbbGvrcvGwLjr7P/l0vad1yjOlvyMkf61GbRnSkoCAfGBKIFc sJqGFqtFTOc4ynUsOv4bao9l2tuKur1qP4ojCVVcH2NyLGi70vw6O3PqWaro4NPX0N49 k64Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Sj7e7KrT; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id d17sor2398843pll.50.2019.03.13.11.40.07 for (Google Transport Security); Wed, 13 Mar 2019 11:40:08 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Sj7e7KrT; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2cvW24YvYXEUFU3KxBgLfjLN1zP9HeM/T5I/fane4LI=; b=Sj7e7KrTxGgjBtBVwW3+pppFo1OluqFpgx9p+e0I2eslI0IX76GxI56FyMr0Ujn61G n1uwAi30e7xrRWMomWR9vdILS1jqnjUhGAy5SN5E3wk3gci8ouk2KGOxzM091kLH4psW caD1lAzHKoQUdqLU0neRW1iZ+UbJuolybIwG/RnT0S4Bf6qnocRv8R4aDfhhWpIcz/N2 VHXy5jPc34xUBgtkCA5Unh9g/nR69Htojaxs8hmdEN6sLd9DHj0vmv0f4bCjQYcTmKc1 pht4bgTrWHV8QEdJqc3QEU56m6n+9htJvRLPcJWs6IpSra+W7soIv9swTAFQo2B7cjbw gkpg== X-Google-Smtp-Source: APXvYqxN6T26cpU/i7gJrXRMqYa2D+TtQXlmh9inTy8ofpUF4HZSJ/PCcVKIq40FliL0BRe0lBbtlw== X-Received: by 2002:a17:902:bd97:: with SMTP id q23mr45933336pls.94.1552502407449; Wed, 13 Mar 2019 11:40:07 -0700 (PDT) Received: from castle.hsd1.ca.comcast.net ([2603:3024:1704:3e00::d657]) by smtp.gmail.com with ESMTPSA id i13sm15792562pgq.17.2019.03.13.11.40.06 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 13 Mar 2019 11:40:06 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH v3 5/6] mm: flush memcg percpu stats and events before releasing Date: Wed, 13 Mar 2019 11:39:52 -0700 Message-Id: <20190313183953.17854-6-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190313183953.17854-1-guro@fb.com> References: <20190313183953.17854-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Flush percpu stats and events data to corresponding before releasing percpu memory. Although per-cpu stats are never exactly precise, dropping them on floor regularly may lead to an accumulation of an error. So, it's safer to flush them before releasing. To minimize the number of atomic updates, let's sum all stats/events on all cpus locally, and then make a single update per entry. Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- mm/memcontrol.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1b5fe826d6d0..0f18bf2afea8 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2119,6 +2119,56 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) mutex_unlock(&percpu_charge_mutex); } +/* + * Flush all per-cpu stats and events into atomics. + * Try to minimize the number of atomic writes by gathering data from + * all cpus locally, and then make one atomic update. + * No locking is required, because no one has an access to + * the offlined percpu data. + */ +static void memcg_flush_offline_percpu(struct mem_cgroup *memcg) +{ + struct memcg_vmstats_percpu __percpu *vmstats_percpu; + struct lruvec_stat __percpu *lruvec_stat_cpu; + struct mem_cgroup_per_node *pn; + int cpu, i; + long x; + + vmstats_percpu = memcg->vmstats_percpu_offlined; + + for (i = 0; i < MEMCG_NR_STAT; i++) { + int nid; + + x = 0; + for_each_possible_cpu(cpu) + x += per_cpu(vmstats_percpu->stat[i], cpu); + if (x) + atomic_long_add(x, &memcg->vmstats[i]); + + if (i >= NR_VM_NODE_STAT_ITEMS) + continue; + + for_each_node(nid) { + pn = mem_cgroup_nodeinfo(memcg, nid); + lruvec_stat_cpu = pn->lruvec_stat_cpu_offlined; + + x = 0; + for_each_possible_cpu(cpu) + x += per_cpu(lruvec_stat_cpu->count[i], cpu); + if (x) + atomic_long_add(x, &pn->lruvec_stat[i]); + } + } + + for (i = 0; i < NR_VM_EVENT_ITEMS; i++) { + x = 0; + for_each_possible_cpu(cpu) + x += per_cpu(vmstats_percpu->events[i], cpu); + if (x) + atomic_long_add(x, &memcg->vmevents[i]); + } +} + static int memcg_hotplug_cpu_dead(unsigned int cpu) { struct memcg_vmstats_percpu __percpu *vmstats_percpu; @@ -4618,6 +4668,8 @@ static void percpu_rcu_free(struct rcu_head *rcu) struct mem_cgroup *memcg = container_of(rcu, struct mem_cgroup, rcu); int node; + memcg_flush_offline_percpu(memcg); + for_each_node(node) { struct mem_cgroup_per_node *pn = memcg->nodeinfo[node]; From patchwork Wed Mar 13 18:39:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10851653 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 810F3139A for ; Wed, 13 Mar 2019 18:40:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D3B029F89 for ; Wed, 13 Mar 2019 18:40:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 614D129F8C; Wed, 13 Mar 2019 18:40:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BF66E29F89 for ; Wed, 13 Mar 2019 18:40:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB4478E000A; Wed, 13 Mar 2019 14:40:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A63F88E0001; Wed, 13 Mar 2019 14:40:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 908B98E000A; Wed, 13 Mar 2019 14:40:11 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 441D48E0001 for ; Wed, 13 Mar 2019 14:40:11 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id c15so3101440pfn.11 for ; Wed, 13 Mar 2019 11:40:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=oYZ0d7IIZA63OVMPtucchTyEwGwVOr51nS/0GcejUsI=; b=b38nbgbfxXaOSr/JVFPfgw2IOUzZ62yrrE4jj/0k/3MYWYymL1DvRB/czyOJUAo4Sh UJBZR1oK6psCfhy4p8tOogxY3BchcakeGjzpPDmCG5AphKGFLonRRSzosNgsCbpY6rk3 bRi2oEqKV3yOjrX9LCXzGUU7ZYhipuDXnoKcnga34mHcPerraRrM9IokJWfVKcttARup zQPXdQInoWPEclXe/vazpvVBWfRqeid9efzKEXk9afihRHzSDD/zyb1JoX1GKDSzcek0 L/uDgDH/kK3rH5fG7UNXlEtqmXyYJ4FOCgkQD+hv2vc907mbOxzY5ZV5mYIAwzDmdexK e5XA== X-Gm-Message-State: APjAAAXRC4TBkkxDyiTmKqdPAv4FEwFwCaay55IP39cizaPS13BICzbM jLOzJ39UHKHQLMVKRRVCndztLasyK0Fdr2D1yiRqMwcQw3P27GUcmGKboMmD76WIPZ63KIru1qy kET0zIUPkmgpl0/9kW6ZN7pd2B7skKkpJn6tPqK8UtJJbKyWz+WMu8J17jRlp1Fi6UKStX3+Do7 hO0QhD6pd0xOOHRyAVnmwd///zxY1xyQohXkq3pxnma9AU09BP1JNs1jX2eNHA1/VqndiMIq/p7 Zoz4oH0g7ktwxuMI7/0nxmIas1d+qmen8XaTcAyzv6a38TQodX0f0n8ioU3a9NK8Ge0DJddrqz2 n2O9RxlkDv9hi+oa6PGhIZv0S0pBFanS5LPsKyPFku4vvKbz+UDGc8k+ALA8Su0Xf5huJWGmsml U X-Received: by 2002:a17:902:8ec1:: with SMTP id x1mr47664438plo.52.1552502410948; Wed, 13 Mar 2019 11:40:10 -0700 (PDT) X-Received: by 2002:a17:902:8ec1:: with SMTP id x1mr47664346plo.52.1552502409493; Wed, 13 Mar 2019 11:40:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552502409; cv=none; d=google.com; s=arc-20160816; b=xiZME+vqJRMokHqbNNlyaIcDPJhB8tUXHm/9r8SdbU+QSg/r1ZWjR3yYOWgqo11USs xgkH1T4x+fmTTQyCCsoO9cUvQCqXlvUnuo8R0jobBWj9M0kXHKLzV44nH9UoWmtk6gTS v3EpbHeRoDncym87K/bDFe9v886fAjqUFG8GMeS6q9BaEf2ZENWyaCsK7KD3jPTGnpaB nxahuwx7OIvMOGdBCZEqRz+NEG4LgodHegX+it4MGuv1ZCD6G7WGkElkm/L8z8X5UmwB drvQwa7N1pMxYoVj4Adbi+oqYpG82J6H/VEk3oxwCf2dn6/joa4tsJ8LxGpQg55s09Mg 7goA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=oYZ0d7IIZA63OVMPtucchTyEwGwVOr51nS/0GcejUsI=; b=fzWPnzwRuDCf4sss39BZtcDUurPqAKGThQuiNKOpjZOmfccJcJtqwcpOwHgaHxYdCM KNsQj1pX7iD/mbmQjBAryNaKPguae8CPTpDPC8GFNCHINTWl2TUuzbf7CZreQYxhE4G+ 86gsPtecxbsBB6NBOshH9GOjj1d1kWvyQUi1y19ei6P7HFU93almQPH8aUQlr68sF0tM /ffbMe+Fl6eCOMFp5tidTWqi+x+ncgTgiks6x+5nEwD1n4EDguWicpRzRrmOgceDrmq1 rYusN3ZUWIu6UiZpk4+KqEuZTpvY8VCCrthIpWvMv9ptH3gCzIjpNOLzHVwcIDhlXaRq ignQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=hjlmlrqo; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id z4sor2436157plk.2.2019.03.13.11.40.09 for (Google Transport Security); Wed, 13 Mar 2019 11:40:09 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=hjlmlrqo; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oYZ0d7IIZA63OVMPtucchTyEwGwVOr51nS/0GcejUsI=; b=hjlmlrqoyNqh8ZMDotoW00L/DiwAFvmdzuVBO5qEwnjoMnFwBtqz2v/L+DOuv1yKgE iFkL77zGQSG/AZc4wPNOQjIbqLrrRer/ITKaec1ZWBYULuL6i8fvKraEMqTdq1EMsv8o AgEdc2tWHxo31TIgegeSeM41gD89bZ7OO0hFTWtANImVHQnQsv00bzuBSO/ER/bycJeG ONJlxbJ8bQJwg2DEvUCj8jaI2x5boilM8pqZ2jIfebSy4iE4PwulXoZ7Hnt0r1zGorY4 BXUk1VJpzHHRIOLVSc1IxnGoKFjConEDOx+EayzAIzbbZPedYCaEE3HhZNC/mu7TT5QH Yjpw== X-Google-Smtp-Source: APXvYqwURWhkADd+qAQLVQN9HsSZU8WAqPGG3lUQMz73aqvzLNksvY2rMWiMAIlB7c+cWIpmfqzD0A== X-Received: by 2002:a17:902:1621:: with SMTP id g30mr47029359plg.116.1552502408883; Wed, 13 Mar 2019 11:40:08 -0700 (PDT) Received: from castle.hsd1.ca.comcast.net ([2603:3024:1704:3e00::d657]) by smtp.gmail.com with ESMTPSA id i13sm15792562pgq.17.2019.03.13.11.40.07 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 13 Mar 2019 11:40:07 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: linux-mm@kvack.org, kernel-team@fb.com Cc: linux-kernel@vger.kernel.org, Tejun Heo , Rik van Riel , Johannes Weiner , Michal Hocko , Roman Gushchin Subject: [PATCH v3 6/6] mm: refactor memcg_hotplug_cpu_dead() to use memcg_flush_offline_percpu() Date: Wed, 13 Mar 2019 11:39:53 -0700 Message-Id: <20190313183953.17854-7-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190313183953.17854-1-guro@fb.com> References: <20190313183953.17854-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It's possible to remove a big chunk of the redundant code by making memcg_flush_offline_percpu() to take cpumask as an argument and flush percpu data on all cpus belonging to the mask instead of all possible cpus. Then memcg_hotplug_cpu_dead() can call it with a single CPU bit set. This approach allows to remove all duplicated code, but safe the performance optimization made in memcg_flush_offline_percpu(): only one atomic operation per data entry. for_each_data_entry() for_each_cpu(cpu. cpumask) sum_events() flush() Otherwise it would be one atomic operation per data entry per cpu: for_each_cpu(cpu) for_each_data_entry() flush() Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner --- mm/memcontrol.c | 61 ++++++++----------------------------------------- 1 file changed, 9 insertions(+), 52 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0f18bf2afea8..5b6a2ea66774 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2122,11 +2122,12 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) /* * Flush all per-cpu stats and events into atomics. * Try to minimize the number of atomic writes by gathering data from - * all cpus locally, and then make one atomic update. + * all cpus in cpumask locally, and then make one atomic update. * No locking is required, because no one has an access to * the offlined percpu data. */ -static void memcg_flush_offline_percpu(struct mem_cgroup *memcg) +static void memcg_flush_offline_percpu(struct mem_cgroup *memcg, + const struct cpumask *cpumask) { struct memcg_vmstats_percpu __percpu *vmstats_percpu; struct lruvec_stat __percpu *lruvec_stat_cpu; @@ -2140,7 +2141,7 @@ static void memcg_flush_offline_percpu(struct mem_cgroup *memcg) int nid; x = 0; - for_each_possible_cpu(cpu) + for_each_cpu(cpu, cpumask) x += per_cpu(vmstats_percpu->stat[i], cpu); if (x) atomic_long_add(x, &memcg->vmstats[i]); @@ -2153,7 +2154,7 @@ static void memcg_flush_offline_percpu(struct mem_cgroup *memcg) lruvec_stat_cpu = pn->lruvec_stat_cpu_offlined; x = 0; - for_each_possible_cpu(cpu) + for_each_cpu(cpu, cpumask) x += per_cpu(lruvec_stat_cpu->count[i], cpu); if (x) atomic_long_add(x, &pn->lruvec_stat[i]); @@ -2162,7 +2163,7 @@ static void memcg_flush_offline_percpu(struct mem_cgroup *memcg) for (i = 0; i < NR_VM_EVENT_ITEMS; i++) { x = 0; - for_each_possible_cpu(cpu) + for_each_cpu(cpu, cpumask) x += per_cpu(vmstats_percpu->events[i], cpu); if (x) atomic_long_add(x, &memcg->vmevents[i]); @@ -2171,8 +2172,6 @@ static void memcg_flush_offline_percpu(struct mem_cgroup *memcg) static int memcg_hotplug_cpu_dead(unsigned int cpu) { - struct memcg_vmstats_percpu __percpu *vmstats_percpu; - struct lruvec_stat __percpu *lruvec_stat_cpu; struct memcg_stock_pcp *stock; struct mem_cgroup *memcg; @@ -2180,50 +2179,8 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) drain_stock(stock); rcu_read_lock(); - for_each_mem_cgroup(memcg) { - int i; - - vmstats_percpu = (struct memcg_vmstats_percpu __percpu *) - rcu_dereference(memcg->vmstats_percpu); - - for (i = 0; i < MEMCG_NR_STAT; i++) { - int nid; - long x; - - if (vmstats_percpu) { - x = this_cpu_xchg(vmstats_percpu->stat[i], 0); - if (x) - atomic_long_add(x, &memcg->vmstats[i]); - } - - if (i >= NR_VM_NODE_STAT_ITEMS) - continue; - - for_each_node(nid) { - struct mem_cgroup_per_node *pn; - - pn = mem_cgroup_nodeinfo(memcg, nid); - - lruvec_stat_cpu = (struct lruvec_stat __percpu*) - rcu_dereference(pn->lruvec_stat_cpu); - if (!lruvec_stat_cpu) - continue; - x = this_cpu_xchg(lruvec_stat_cpu->count[i], 0); - if (x) - atomic_long_add(x, &pn->lruvec_stat[i]); - } - } - - for (i = 0; i < NR_VM_EVENT_ITEMS; i++) { - long x; - - if (vmstats_percpu) { - x = this_cpu_xchg(vmstats_percpu->events[i], 0); - if (x) - atomic_long_add(x, &memcg->vmevents[i]); - } - } - } + for_each_mem_cgroup(memcg) + memcg_flush_offline_percpu(memcg, cpumask_of(cpu)); rcu_read_unlock(); return 0; @@ -4668,7 +4625,7 @@ static void percpu_rcu_free(struct rcu_head *rcu) struct mem_cgroup *memcg = container_of(rcu, struct mem_cgroup, rcu); int node; - memcg_flush_offline_percpu(memcg); + memcg_flush_offline_percpu(memcg, cpu_possible_mask); for_each_node(node) { struct mem_cgroup_per_node *pn = memcg->nodeinfo[node];