From patchwork Tue Jun 19 05:13:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 10473175 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7CE7660383 for ; Tue, 19 Jun 2018 05:13:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6C4022811E for ; Tue, 19 Jun 2018 05:13:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6031428470; Tue, 19 Jun 2018 05:13:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 47BD72811E for ; Tue, 19 Jun 2018 05:13:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F14916B000E; Tue, 19 Jun 2018 01:13:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E762E6B0010; Tue, 19 Jun 2018 01:13:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF09D6B0266; Tue, 19 Jun 2018 01:13:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f70.google.com (mail-pl0-f70.google.com [209.85.160.70]) by kanga.kvack.org (Postfix) with ESMTP id 893B96B000E for ; Tue, 19 Jun 2018 01:13:45 -0400 (EDT) Received: by mail-pl0-f70.google.com with SMTP id c6-v6so11421976pll.4 for ; Mon, 18 Jun 2018 22:13:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=MqmCD10RwYEL9UtGfpqsQpvbtnr+99X+2SVsogA+HbA=; b=ukPgUy7gvU7MNf88J7S7hruJttnD0IVdAIk8F2w6fZe+uz9yqpyfNsbYtzoCN4yWqI kEie2S0dOxw+ZXb47+g1DqJOikDpGPwnmGsaoy41YJpU/kM7SgkgV9ra3/rH2FN2L1ki K1tsrGlqbvRegul7YwSqnhxodV3wRt4ZVnDHGdcnnz0QdBEpI3KjYYdVyl6BtTrrcM+3 vvDHL2+CX8pMPjJ8OHr5YzOcex2C85kZMJaGiUpUcWQuFWxjN8D2OTcXEMyXeCjAynQY aYg7+Iq7mM4Z1SrIXS2h3mQI8VkdfyxQUWgJc/Kopl9uUlVW48nXQU3m2UsXfZeXRd3U W/cA== X-Gm-Message-State: APt69E19TW164TVXbEJ/I2iIvC2Yelg8flW0Bwd1+RU7Ye0Izj417GVo Z00ei2wWIlJBVLj33rktqYBYgjJroX73+KKsrJxC28K7suEhD5hE/8EJxNkCnqExHrEnph6dXLt ZRkBosMEUO9mbsy5rwRYnZnd3nxQI0O4L1MYs+XrVZn0F8jRkrjDQtQO9wxNtu9ZNbcR3vZPRln cXX82s+TFntNQ6pqelfqCMflYqPwQYIKZO9hs2fZ/jC3jV1qfpPHbVgCM8gxyOmRstvRH7Rdbqt Jl7or9eJbXYluBgGWot6u7GSGtCoDau+joE39thZqYerxIat331Ereialx4/R+WRlaDwMqTvYZr 3UwOvOIWrOYs83F8mFoPWAd5leVDg6IzMntdphob5QQ7giSjv6hRcyEpck1RSrojLyCv1RZpe37 6 X-Received: by 2002:a63:81c7:: with SMTP id t190-v6mr13368599pgd.378.1529385225199; Mon, 18 Jun 2018 22:13:45 -0700 (PDT) X-Received: by 2002:a63:81c7:: with SMTP id t190-v6mr13368565pgd.378.1529385224289; Mon, 18 Jun 2018 22:13:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529385224; cv=none; d=google.com; s=arc-20160816; b=WX++0QRlp5VoHRmqmAaVPEYw28J42OWPuX/iPfVD86VXw7dStluBd275Hw8u0KRTLE 5/2vW2hIdwzCyfbtjjg/cXpb7UmIg7aA2fc/5XtFqFRQDpPoDPk1gJenUDeh+Q4hpto9 6JUpjcRBCVYZ2qE3mMbJkxNi6dW6tUn25igJNkpBz6br3TzFqIJySvt14EDKFGRE33t8 fC4yz+cE1w6LIHMzLVC8QT7vvAhjl58rm81SSD6d2jHqlPTKcyWYPr3VbTSS0jMzKAmm //ApPk3fkIplVLcuRXQsnbO5f9axPP9onUP52WU9N5EDE8rw/VziQaV8+BSm4wcnWCwl n2ow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=MqmCD10RwYEL9UtGfpqsQpvbtnr+99X+2SVsogA+HbA=; b=PWsB72uSSTebvP8V1aXd6QqftQaARbgKLUMTFkguY+ivnEsUXmiaHJFEZ4aoA9QstD TTE1YfNtZxHPYAfbQWWMC6g6362if9PZVrmmI/0aSt1khTqzrC7L4GphLnrvZLtXNv8X pIki9eY5i16IEDtcNcxwvWbji06ViB6VAoFEFWpztvmWcy5TXWthxXdZMxyUngJ5D2HF rpHbOjtY5J4tKoL5Sf3vkDz+m0efCm1INH97/HvDwS1+aaPqpZJCQ3ydxO2pi1WNU9fQ byTh+DuRDdzGW7Yq2NqIo4nBRAYRugmj3DOfxJfSFAWHHd6Yqh2ISE6Xw07+RQIJKfyo BBLA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=UQKgqUza; spf=pass (google.com: domain of shakeelb@google.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id s11-v6sor2926434pgv.34.2018.06.18.22.13.44 for (Google Transport Security); Mon, 18 Jun 2018 22:13:44 -0700 (PDT) Received-SPF: pass (google.com: domain of shakeelb@google.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=UQKgqUza; spf=pass (google.com: domain of shakeelb@google.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MqmCD10RwYEL9UtGfpqsQpvbtnr+99X+2SVsogA+HbA=; b=UQKgqUzav1CUNvMMvmaU3NyeD13FZKaVcoxdmKfZgZV4KVaxPcnD3E0svKMRxThx1P rkATmYdPRDTipCukiN3Ma1jVohMWcsgJBi7Fra96GTPgYisP8dPWhjWxcB44VQiVXPkn /1UCOVXAVtTnU3JKOavvZ2FwrMFR0Jc3T9KsOtsX7D75zqgOSsC1vR/q/HfzlZTPdiWH QEcVG9660l/iuX7KmPEOLFhDfFRmmRyjknyNx5p7o6PqukAIebGZBw9DtMOGtUYUeYBm 9HdN3eqcfzvg2e3ihjxO/NzXYGy3msuK8Oqh44ZogfPDOwQXcYKaM7SFv3VF0x9rMWTB U4/w== X-Google-Smtp-Source: ADUXVKJNv+l8nqxG3So+pCP7h0SGLU6MaeeFdgFGKnXn1Cf3Anqmv31Ma6gtTlCKplnfpHLDVayMmg== X-Received: by 2002:a65:55c6:: with SMTP id k6-v6mr13406087pgs.6.1529385223576; Mon, 18 Jun 2018 22:13:43 -0700 (PDT) Received: from shakeelb.mtv.corp.google.com ([2620:15c:2cb:201:3a5f:3a4f:fa44:6b63]) by smtp.gmail.com with ESMTPSA id a23-v6sm28390504pfj.117.2018.06.18.22.13.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 18 Jun 2018 22:13:42 -0700 (PDT) From: Shakeel Butt To: Andrew Morton Cc: Michal Hocko , Johannes Weiner , Vladimir Davydov , Jan Kara , Greg Thelen , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, Shakeel Butt , Jan Kara , Amir Goldstein , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Mel Gorman , Vlastimil Babka , Alexander Viro Subject: [PATCH 1/3] mm: memcg: remote memcg charging for kmem allocations Date: Mon, 18 Jun 2018 22:13:25 -0700 Message-Id: <20180619051327.149716-2-shakeelb@google.com> X-Mailer: git-send-email 2.18.0.rc1.244.gcf134e6275-goog In-Reply-To: <20180619051327.149716-1-shakeelb@google.com> References: <20180619051327.149716-1-shakeelb@google.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Introduce the memcg variant for kmalloc[_node] and kmem_cache_alloc[_node]. For kmem_cache_alloc, the kernel switches the root kmem cache with the memcg specific kmem cache for __GFP_ACCOUNT allocations to charge those allocations to the memcg. However, the memcg to charge is extracted from the current task_struct. This patch introduces the variant of kmem cache allocation functions where the memcg can be provided explicitly by the caller instead of deducing the memcg from the current task. The kmalloc allocations are underlying served using the kmem caches unless the size of the allocation request is larger than KMALLOC_MAX_CACHE_SIZE, in which case, the kmem caches are bypassed and the request is routed directly to page allocator. So, for __GFP_ACCOUNT kmalloc allocations, the memcg of current task is charged. This patch introduces memcg variant of kmalloc functions to allow callers to provide memcg for charging. These functions are useful for use-cases where the allocations should be charged to the memcg different from the memcg of the caller. One such concrete use-case is the allocations for fsnotify event objects where the objects should be charged to the listener instead of the producer. One requirement to call these functions is that the caller must have the reference to the memcg provided to these functions. If reference acquition on the given memcg is failed (it can fail if memcg is offline) then the current's memcg is tried. These functions implicitly assumes that the caller wants a __GFP_ACCOUNT allocation. This patch also introduces scope API for targeted memcg charging. All the __GFP_ACCOUNT allocations between memalloc_memcg_save(target_memcg) and memalloc_memcg_restore(old_memcg) will be charged to target_memcg. Traditionally kmem charging is skipped for allocations by kthreads and allocations during interrupts. The reason is that the current's memcg might not be the right owner for such allocations. However targeted memcg charging does not have such limitation and can work even for allocations by kthreads and for allocations during interrupts. For now due to lack of actual use-case, targeted memcg charging for such allocations is not added. Though this might change in future. Signed-off-by: Shakeel Butt Cc: Michal Hocko Cc: Jan Kara Cc: Amir Goldstein Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Greg Thelen Cc: Johannes Weiner Cc: Vladimir Davydov Cc: Mel Gorman Cc: Vlastimil Babka Cc: Alexander Viro Cc: Andrew Morton --- Changelog sinve v5: - Added more explanation in commit message. - Added handling of NULL memcg for targeted memcg allocation functions. Changelog since v4: - Removed branch from hot path of memory charging. Changelog since v3: - Added node variant of directed kmem allocation functions. Changelog since v2: - Merge the kmalloc_memcg patch into this patch. - Instead of plumbing memcg throughout, use field in task_struct to pass the target_memcg. Changelog since v1: - Fixed build for SLOB include/linux/sched.h | 3 ++ include/linux/sched/mm.h | 24 ++++++++++++ include/linux/slab.h | 83 ++++++++++++++++++++++++++++++++++++++++ kernel/fork.c | 3 ++ mm/memcontrol.c | 18 ++++++++- 5 files changed, 129 insertions(+), 2 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 87bf02d93a27..cbd0def60fd4 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1149,6 +1149,9 @@ struct task_struct { /* Number of pages to reclaim on returning to userland: */ unsigned int memcg_nr_pages_over_high; + + /* Used by memcontrol for targeted memcg charge: */ + struct mem_cgroup *target_memcg; #endif #ifdef CONFIG_UPROBES diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 44d356f5e47c..2ffb194f3f32 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -248,6 +248,30 @@ static inline void memalloc_noreclaim_restore(unsigned int flags) current->flags = (current->flags & ~PF_MEMALLOC) | flags; } +#ifdef CONFIG_MEMCG +static inline struct mem_cgroup *memalloc_memcg_save(struct mem_cgroup *memcg) +{ + struct mem_cgroup *old_memcg = current->target_memcg; + + current->target_memcg = memcg; + return old_memcg; +} + +static inline void memalloc_memcg_restore(struct mem_cgroup *memcg) +{ + current->target_memcg = memcg; +} +#else +static inline struct mem_cgroup *memalloc_memcg_save(struct mem_cgroup *memcg) +{ + return NULL; +} + +static inline void memalloc_memcg_restore(struct mem_cgroup *memcg) +{ +} +#endif /* CONFIG_MEMCG */ + #ifdef CONFIG_MEMBARRIER enum { MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY = (1U << 0), diff --git a/include/linux/slab.h b/include/linux/slab.h index 14e3fe4bd6a1..2f6319fa0d3d 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -16,6 +16,7 @@ #include #include #include +#include /* @@ -375,6 +376,27 @@ static __always_inline void kfree_bulk(size_t size, void **p) kmem_cache_free_bulk(NULL, size, p); } +/* + * Calling kmem_cache_alloc_memcg implicitly assumes that the caller wants + * a __GFP_ACCOUNT allocation. However if memcg is NULL then + * kmem_cache_alloc_memcg is same as kmem_cache_alloc. + */ +static __always_inline void *kmem_cache_alloc_memcg(struct kmem_cache *cachep, + gfp_t flags, + struct mem_cgroup *memcg) +{ + struct mem_cgroup *old_memcg; + void *ptr; + + if (!memcg) + return kmem_cache_alloc(cachep, flags); + + old_memcg = memalloc_memcg_save(memcg); + ptr = kmem_cache_alloc(cachep, flags | __GFP_ACCOUNT); + memalloc_memcg_restore(old_memcg); + return ptr; +} + #ifdef CONFIG_NUMA void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment __malloc; void *kmem_cache_alloc_node(struct kmem_cache *, gfp_t flags, int node) __assume_slab_alignment __malloc; @@ -390,6 +412,27 @@ static __always_inline void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t f } #endif +/* + * Calling kmem_cache_alloc_node_memcg implicitly assumes that the caller + * wants a __GFP_ACCOUNT allocation. However if memcg is NULL then + * kmem_cache_alloc_node_memcg is same as kmem_cache_alloc_node. + */ +static __always_inline void * +kmem_cache_alloc_node_memcg(struct kmem_cache *cachep, gfp_t flags, int node, + struct mem_cgroup *memcg) +{ + struct mem_cgroup *old_memcg; + void *ptr; + + if (!memcg) + return kmem_cache_alloc_node(cachep, flags, node); + + old_memcg = memalloc_memcg_save(memcg); + ptr = kmem_cache_alloc_node(cachep, flags | __GFP_ACCOUNT, node); + memalloc_memcg_restore(old_memcg); + return ptr; +} + #ifdef CONFIG_TRACING extern void *kmem_cache_alloc_trace(struct kmem_cache *, gfp_t, size_t) __assume_slab_alignment __malloc; @@ -518,6 +561,26 @@ static __always_inline void *kmalloc(size_t size, gfp_t flags) return __kmalloc(size, flags); } +/* + * Calling kmalloc_memcg implicitly assumes that the caller wants a + * __GFP_ACCOUNT allocation. However if memcg is NULL then kmalloc_memcg + * is same as kmalloc. + */ +static __always_inline void *kmalloc_memcg(size_t size, gfp_t flags, + struct mem_cgroup *memcg) +{ + struct mem_cgroup *old_memcg; + void *ptr; + + if (!memcg) + return kmalloc(size, flags); + + old_memcg = memalloc_memcg_save(memcg); + ptr = kmalloc(size, flags | __GFP_ACCOUNT); + memalloc_memcg_restore(old_memcg); + return ptr; +} + /* * Determine size used for the nth kmalloc cache. * return size or 0 if a kmalloc cache for that @@ -555,6 +618,26 @@ static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node) return __kmalloc_node(size, flags, node); } +/* + * Calling kmalloc_node_memcg implicitly assumes that the caller wants a + * __GFP_ACCOUNT allocation. However if memcg is NULL then kmalloc_node_memcg + * is same as kmalloc_node. + */ +static __always_inline void * +kmalloc_node_memcg(size_t size, gfp_t flags, int node, struct mem_cgroup *memcg) +{ + struct mem_cgroup *old_memcg; + void *ptr; + + if (!memcg) + return kmalloc_node(size, flags, node); + + old_memcg = memalloc_memcg_save(memcg); + ptr = kmalloc_node(size, flags | __GFP_ACCOUNT, node); + memalloc_memcg_restore(old_memcg); + return ptr; +} + struct memcg_cache_array { struct rcu_head rcu; struct kmem_cache *entries[0]; diff --git a/kernel/fork.c b/kernel/fork.c index a64d0a19f174..5bf300015790 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -843,6 +843,9 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node) tsk->fail_nth = 0; #endif +#ifdef CONFIG_MEMCG + tsk->target_memcg = NULL; +#endif return tsk; free_stack: diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 02e77c88967a..08bfb8c2411b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -701,6 +701,20 @@ static struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) return memcg; } +static __always_inline struct mem_cgroup *get_mem_cgroup( + struct mem_cgroup *memcg, struct mm_struct *mm) +{ + if (unlikely(memcg)) { + rcu_read_lock(); + if (css_tryget_online(&memcg->css)) { + rcu_read_unlock(); + return memcg; + } + rcu_read_unlock(); + } + return get_mem_cgroup_from_mm(mm); +} + /** * mem_cgroup_iter - iterate over memory cgroup hierarchy * @root: hierarchy root @@ -2260,7 +2274,7 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) if (current->memcg_kmem_skip_account) return cachep; - memcg = get_mem_cgroup_from_mm(current->mm); + memcg = get_mem_cgroup(current->target_memcg, current->mm); kmemcg_id = READ_ONCE(memcg->kmemcg_id); if (kmemcg_id < 0) goto out; @@ -2344,7 +2358,7 @@ int memcg_kmem_charge(struct page *page, gfp_t gfp, int order) if (memcg_kmem_bypass()) return 0; - memcg = get_mem_cgroup_from_mm(current->mm); + memcg = get_mem_cgroup(current->target_memcg, current->mm); if (!mem_cgroup_is_root(memcg)) { ret = memcg_kmem_charge_memcg(page, gfp, order, memcg); if (!ret)