From patchwork Mon Jun 25 23:06:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 10487529 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E31A0601D5 for ; Mon, 25 Jun 2018 23:07:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D2CD8286A8 for ; Mon, 25 Jun 2018 23:07:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C5EF6286BD; Mon, 25 Jun 2018 23:07:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6F662286A8 for ; Mon, 25 Jun 2018 23:07:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 11F686B0006; Mon, 25 Jun 2018 19:07:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0A8876B0007; Mon, 25 Jun 2018 19:07:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E3C2A6B000A; Mon, 25 Jun 2018 19:07:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg0-f69.google.com (mail-pg0-f69.google.com [74.125.83.69]) by kanga.kvack.org (Postfix) with ESMTP id 9023D6B0006 for ; Mon, 25 Jun 2018 19:07:13 -0400 (EDT) Received: by mail-pg0-f69.google.com with SMTP id b7-v6so5526375pgv.5 for ; Mon, 25 Jun 2018 16:07:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=kMg3GPpgj+XppH7u6uP/AfWryHitotuZ03Sw2dV5Edk=; b=OHbtsnOd9PHEhEbyPbNH8QRckudou6pf55djDuNGpXmfPXQP2m0Pt8TWQ0FYR43QF+ JuU5k03eC3qzuSSBkvhlsF7mLalys1yhe3Jo84HhDThw2P+f1xgV5upRwl3kNSpmQ9MF GHXHh/ziJhcwXpSsW3KlMJvobAvRIvX1FimCftvfq3Os+JIWufe4vpyJSpeKtt73NRzZ pZyjFMDHPcJQ6BZdkM11gq/vxa9rDtOmZwR+boNVSKuc2V4VixnoFvNFJpHpB7i2IYkT /wbrSD/4n2dv1lYC4eoG9nMboGJBqjRTZYUqgdgCLXoApWVdcAUtQPA+8zIONMs1EMn+ Vesw== X-Gm-Message-State: APt69E39nH0TOft8Vm6GcrageCCHGO+K8B8uZpE1SdRucA/XhD1ZxOeQ b3w8ulvSg/0gjjpQ7eMBPq+evmyjL9QCp6QFpeBkkwYqXqVpWCs6juuKML7WZRlUHaKj7J3pn8F f/q/HfQl3ItmVNatILbbiVHNlT8Hl3dfZD/NDUWY81ci9U2YnIYPRFxzywDv7aV/3cysAt8uylC ggWji+E0qpk6I5Bcbp4KHXOt17Yi1NdtB+kqi4GmnJSes8Aq05xMFlIAKVVrTxCn4/CEuWCnHba qVUzQtbM95k7xKj6UssS9raKIofZiHlK1QU0lD0KsqeyS4mntlNVCG0O7D4E+fXcENeX4Fw/TXr dfYtJoO25qwek5aZlY08mJMMiFNo299IwecipNoZMkIb0OjjceUICIaqfFmJ6nF+/MOgWDA1rG9 V X-Received: by 2002:a17:902:70ca:: with SMTP id l10-v6mr14211485plt.174.1529968033215; Mon, 25 Jun 2018 16:07:13 -0700 (PDT) X-Received: by 2002:a17:902:70ca:: with SMTP id l10-v6mr14211435plt.174.1529968032152; Mon, 25 Jun 2018 16:07:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529968032; cv=none; d=google.com; s=arc-20160816; b=jXfOj7iznm5JKAM0Lqt6M9bExFruyQ6DjaIfdvvsv9WaiG81owh/CNjCSRWcItZbBD CJ80v3YyLol0vps3osCCbFQ7Dqk+MDVKydfS8MxJec719rN+8KIPYgKvCj3Ww1bE0RKp 6KNcK14tmWTPyrjwd2m3dsn6BjPlHLQbsQL34u5/hGZGqROEQ107nJW9726i4qRvE1ah YI5Kh7HHRdyVJ+w6YtB1weDDaueW7ol7TklxBakr++KR1pqS4wJWCwPVVaUcGOtf6BLG ND+QaJuQXg/fywqhloMoGGallLNz+TlOPhsu8v7tFwhfgtiY8W10CZsC9RC9wrkcqNdF 2YGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=kMg3GPpgj+XppH7u6uP/AfWryHitotuZ03Sw2dV5Edk=; b=TyhIDC7quq936kEQJHRkX3qUMwY28G+/wQ4GgjStn6JA9mlO0uqoK7hnglymKBNCSd QTf9MssoVeaOY2NX53fZ/HtN5koDe7ZTvVt5M06B09uM9eCgArH3+/PvM7XA71tmqbdm uGscFjl6RaaPrHd0v3wb4Opp1o1YBwChEsXyKCoSbrp/JSZI53KyXIhNx5lNfQVLSGKX jBu2mKFC01zwu5lx0lDE5W0AR539GVzkWSa4QR8ey3dNGB9NMmP36rU5KxAQwNTO9nn3 2xJSYpo5ryBT3d3GWoQoMzEXQpd3fHZU0qpocU+EgWH3ftH7AaIOYqY64now9lqakINa RI9g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b="WSmmSu/F"; spf=pass (google.com: domain of shakeelb@google.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id p22-v6sor35649pgv.75.2018.06.25.16.07.12 for (Google Transport Security); Mon, 25 Jun 2018 16:07:12 -0700 (PDT) Received-SPF: pass (google.com: domain of shakeelb@google.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b="WSmmSu/F"; spf=pass (google.com: domain of shakeelb@google.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=kMg3GPpgj+XppH7u6uP/AfWryHitotuZ03Sw2dV5Edk=; b=WSmmSu/Ff2LUNucgT6PEhaxN3yhSO57xDV+UUr7qsC7SzVz9swxnnza+oPBRKfx28H 0RfhKda7MOmEtPdYA02ONpnfJhsxH3olVYBDhxBfteBTF2i4Cpy/r72+kfVJOz3+veZe TO+WoQhew+E3ndhH33RpINxhPrMeaRPlF5dE7vE+2cjHEmTizSY9WiGgVjciBvaBcVzO 9oBT68zncvhULB0YB+N9jYK3LazjVcEjsAWB51CSTNsK0LH67/k/yraJZuhaA/+LTb7T vgQh40vKSHyGIwq1XytcekyASMSWuc2X49o9aWGP1ASjUkFhpbDIO4K55OxxO2qvTA+u 7c8Q== X-Google-Smtp-Source: ADUXVKKWOJacWWn1sIwcGpowFqMwcA4SvjoQLFvcz6BI8NWdA39l05wwrZmH5KH9ZeRu6H7vCFU9dg== X-Received: by 2002:a63:8648:: with SMTP id x69-v6mr8762131pgd.172.1529968031268; Mon, 25 Jun 2018 16:07:11 -0700 (PDT) Received: from shakeelb.mtv.corp.google.com ([2620:15c:2cb:201:3a5f:3a4f:fa44:6b63]) by smtp.gmail.com with ESMTPSA id p73-v6sm116657pfa.142.2018.06.25.16.07.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Jun 2018 16:07:09 -0700 (PDT) From: Shakeel Butt To: Andrew Morton Cc: Michal Hocko , Johannes Weiner , Vladimir Davydov , Jan Kara , Greg Thelen , Amir Goldstein , Roman Gushchin , Alexander Viro , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, Shakeel Butt , Jan Kara Subject: [PATCH 1/2] fs: fsnotify: account fsnotify metadata to kmemcg Date: Mon, 25 Jun 2018 16:06:58 -0700 Message-Id: <20180625230659.139822-2-shakeelb@google.com> X-Mailer: git-send-email 2.18.0.rc2.346.g013aa6912e-goog In-Reply-To: <20180625230659.139822-1-shakeelb@google.com> References: <20180625230659.139822-1-shakeelb@google.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP A lot of memory can be consumed by the events generated for the huge or unlimited queues if there is either no or slow listener. This can cause system level memory pressure or OOMs. So, it's better to account the fsnotify kmem caches to the memcg of the listener. However the listener can be in a different memcg than the memcg of the producer and these allocations happen in the context of the event producer. This patch introduces remote memcg charging scope API which the producer can use to charge the allocations to the memcg of the listener. There are seven fsnotify kmem caches and among them allocations from dnotify_struct_cache, dnotify_mark_cache, fanotify_mark_cache and inotify_inode_mark_cachep happens in the context of syscall from the listener. So, SLAB_ACCOUNT is enough for these caches. The objects from fsnotify_mark_connector_cachep are not accounted as they are small compared to the notification mark or events and it is unclear whom to account connector to since it is shared by all events attached to the inode. The allocations from the event caches happen in the context of the event producer. For such caches we will need to remote charge the allocations to the listener's memcg. Thus we save the memcg reference in the fsnotify_group structure of the listener. This patch has also moved the members of fsnotify_group to keep the size same, at least for 64 bit build, even with additional member by filling the holes. Signed-off-by: Shakeel Butt Cc: Michal Hocko Cc: Jan Kara Cc: Amir Goldstein Cc: Greg Thelen Cc: Johannes Weiner Cc: Vladimir Davydov Cc: Roman Gushchin --- Changelog since v6: - Removed Jan's ACK as the code has changed a lot - Squashed the separate remote charging API path into this one - Removed kmalloc* & kmem_cache_alloc* APIs and only kept the scope API - Changed fsnotify remote charging code to use scope API Changelog since v5: - None Changelog since v4: - Fixed the build for CONFIG_MEMCG=n Changelog since v3: - Rebased over Jan's patches. - Some cleanup based on Amir's comments. Changelog since v2: - None Changelog since v1: - no more charging fsnotify_mark_connector objects - Fixed the build for SLOB fs/notify/dnotify/dnotify.c | 5 ++-- fs/notify/fanotify/fanotify.c | 17 ++++++++++-- fs/notify/fanotify/fanotify_user.c | 5 +++- fs/notify/group.c | 4 +++ fs/notify/inotify/inotify_fsnotify.c | 15 +++++++++- fs/notify/inotify/inotify_user.c | 5 +++- include/linux/fsnotify_backend.h | 12 +++++--- include/linux/memcontrol.h | 7 +++++ include/linux/sched.h | 3 ++ include/linux/sched/mm.h | 41 ++++++++++++++++++++++++++++ kernel/fork.c | 3 ++ mm/memcontrol.c | 38 +++++++++++++++++++++++--- 12 files changed, 139 insertions(+), 16 deletions(-) diff --git a/fs/notify/dnotify/dnotify.c b/fs/notify/dnotify/dnotify.c index e2bea2ac5dfb..a6365e6bc047 100644 --- a/fs/notify/dnotify/dnotify.c +++ b/fs/notify/dnotify/dnotify.c @@ -384,8 +384,9 @@ int fcntl_dirnotify(int fd, struct file *filp, unsigned long arg) static int __init dnotify_init(void) { - dnotify_struct_cache = KMEM_CACHE(dnotify_struct, SLAB_PANIC); - dnotify_mark_cache = KMEM_CACHE(dnotify_mark, SLAB_PANIC); + dnotify_struct_cache = KMEM_CACHE(dnotify_struct, + SLAB_PANIC|SLAB_ACCOUNT); + dnotify_mark_cache = KMEM_CACHE(dnotify_mark, SLAB_PANIC|SLAB_ACCOUNT); dnotify_group = fsnotify_alloc_group(&dnotify_fsnotify_ops); if (IS_ERR(dnotify_group)) diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c index f90842efea13..d6dfcf0ec21f 100644 --- a/fs/notify/fanotify/fanotify.c +++ b/fs/notify/fanotify/fanotify.c @@ -11,6 +11,7 @@ #include #include #include +#include #include "fanotify.h" @@ -140,8 +141,9 @@ struct fanotify_event_info *fanotify_alloc_event(struct fsnotify_group *group, struct inode *inode, u32 mask, const struct path *path) { - struct fanotify_event_info *event; + struct fanotify_event_info *event = NULL; gfp_t gfp = GFP_KERNEL; + struct mem_cgroup *old_memcg = NULL; /* * For queues with unlimited length lost events are not expected and @@ -151,19 +153,25 @@ struct fanotify_event_info *fanotify_alloc_event(struct fsnotify_group *group, if (group->max_events == UINT_MAX) gfp |= __GFP_NOFAIL; + /* Whoever is interested in the event, pays for the allocation. */ + if (group->memcg) { + gfp |= __GFP_ACCOUNT; + old_memcg = memalloc_use_memcg(group->memcg); + } + if (fanotify_is_perm_event(mask)) { struct fanotify_perm_event_info *pevent; pevent = kmem_cache_alloc(fanotify_perm_event_cachep, gfp); if (!pevent) - return NULL; + goto out; event = &pevent->fae; pevent->response = 0; goto init; } event = kmem_cache_alloc(fanotify_event_cachep, gfp); if (!event) - return NULL; + goto out; init: __maybe_unused fsnotify_init_event(&event->fse, inode, mask); event->tgid = get_pid(task_tgid(current)); @@ -174,6 +182,9 @@ init: __maybe_unused event->path.mnt = NULL; event->path.dentry = NULL; } +out: + if (group->memcg) + memalloc_unuse_memcg(old_memcg); return event; } diff --git a/fs/notify/fanotify/fanotify_user.c b/fs/notify/fanotify/fanotify_user.c index ec4d8c59d0e3..0cf45041dc32 100644 --- a/fs/notify/fanotify/fanotify_user.c +++ b/fs/notify/fanotify/fanotify_user.c @@ -16,6 +16,7 @@ #include #include #include +#include #include @@ -756,6 +757,7 @@ SYSCALL_DEFINE2(fanotify_init, unsigned int, flags, unsigned int, event_f_flags) group->fanotify_data.user = user; atomic_inc(&user->fanotify_listeners); + group->memcg = get_mem_cgroup_from_mm(current->mm); oevent = fanotify_alloc_event(group, NULL, FS_Q_OVERFLOW, NULL); if (unlikely(!oevent)) { @@ -957,7 +959,8 @@ COMPAT_SYSCALL_DEFINE6(fanotify_mark, */ static int __init fanotify_user_setup(void) { - fanotify_mark_cache = KMEM_CACHE(fsnotify_mark, SLAB_PANIC); + fanotify_mark_cache = KMEM_CACHE(fsnotify_mark, + SLAB_PANIC|SLAB_ACCOUNT); fanotify_event_cachep = KMEM_CACHE(fanotify_event_info, SLAB_PANIC); if (IS_ENABLED(CONFIG_FANOTIFY_ACCESS_PERMISSIONS)) { fanotify_perm_event_cachep = diff --git a/fs/notify/group.c b/fs/notify/group.c index aa5468f23e45..cbcda1cb9a74 100644 --- a/fs/notify/group.c +++ b/fs/notify/group.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include "fsnotify.h" @@ -36,6 +37,9 @@ static void fsnotify_final_destroy_group(struct fsnotify_group *group) if (group->ops->free_group_priv) group->ops->free_group_priv(group); + if (group->memcg) + mem_cgroup_put(group->memcg); + kfree(group); } diff --git a/fs/notify/inotify/inotify_fsnotify.c b/fs/notify/inotify/inotify_fsnotify.c index 9ab6dde38a14..73b4d6c55497 100644 --- a/fs/notify/inotify/inotify_fsnotify.c +++ b/fs/notify/inotify/inotify_fsnotify.c @@ -31,6 +31,7 @@ #include #include #include +#include #include "inotify.h" @@ -73,9 +74,11 @@ int inotify_handle_event(struct fsnotify_group *group, struct inotify_inode_mark *i_mark; struct inotify_event_info *event; struct fsnotify_event *fsn_event; + struct mem_cgroup *old_memcg = NULL; int ret; int len = 0; int alloc_len = sizeof(struct inotify_event_info); + gfp_t gfp = GFP_KERNEL; if (WARN_ON(fsnotify_iter_vfsmount_mark(iter_info))) return 0; @@ -98,7 +101,17 @@ int inotify_handle_event(struct fsnotify_group *group, i_mark = container_of(inode_mark, struct inotify_inode_mark, fsn_mark); - event = kmalloc(alloc_len, GFP_KERNEL); + /* Whoever is interested in the event, pays for the allocation. */ + if (group->memcg) { + gfp |= __GFP_ACCOUNT; + old_memcg = memalloc_use_memcg(group->memcg); + } + + event = kmalloc(alloc_len, gfp); + + if (group->memcg) + memalloc_unuse_memcg(old_memcg); + if (unlikely(!event)) { /* * Treat lost event due to ENOMEM the same way as queue diff --git a/fs/notify/inotify/inotify_user.c b/fs/notify/inotify/inotify_user.c index 1cf5b779d862..749c46ababa0 100644 --- a/fs/notify/inotify/inotify_user.c +++ b/fs/notify/inotify/inotify_user.c @@ -38,6 +38,7 @@ #include #include #include +#include #include "inotify.h" #include "../fdinfo.h" @@ -636,6 +637,7 @@ static struct fsnotify_group *inotify_new_group(unsigned int max_events) oevent->name_len = 0; group->max_events = max_events; + group->memcg = get_mem_cgroup_from_mm(current->mm); spin_lock_init(&group->inotify_data.idr_lock); idr_init(&group->inotify_data.idr); @@ -808,7 +810,8 @@ static int __init inotify_user_setup(void) BUG_ON(hweight32(ALL_INOTIFY_BITS) != 21); - inotify_inode_mark_cachep = KMEM_CACHE(inotify_inode_mark, SLAB_PANIC); + inotify_inode_mark_cachep = KMEM_CACHE(inotify_inode_mark, + SLAB_PANIC|SLAB_ACCOUNT); inotify_max_queued_events = 16384; init_user_ns.ucount_max[UCOUNT_INOTIFY_INSTANCES] = 128; diff --git a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h index b38964a7a521..a0c4790c5302 100644 --- a/include/linux/fsnotify_backend.h +++ b/include/linux/fsnotify_backend.h @@ -84,6 +84,8 @@ struct fsnotify_event_private_data; struct fsnotify_fname; struct fsnotify_iter_info; +struct mem_cgroup; + /* * Each group much define these ops. The fsnotify infrastructure will call * these operations for each relevant group. @@ -127,6 +129,8 @@ struct fsnotify_event { * everything will be cleaned up. */ struct fsnotify_group { + const struct fsnotify_ops *ops; /* how this group handles things */ + /* * How the refcnt is used is up to each group. When the refcnt hits 0 * fsnotify will clean up all of the resources associated with this group. @@ -137,8 +141,6 @@ struct fsnotify_group { */ refcount_t refcnt; /* things with interest in this group */ - const struct fsnotify_ops *ops; /* how this group handles things */ - /* needed to send notification to userspace */ spinlock_t notification_lock; /* protect the notification_list */ struct list_head notification_list; /* list of event_holder this group needs to send to userspace */ @@ -160,6 +162,8 @@ struct fsnotify_group { atomic_t num_marks; /* 1 for each mark and 1 for not being * past the point of no return when freeing * a group */ + atomic_t user_waits; /* Number of tasks waiting for user + * response */ struct list_head marks_list; /* all inode marks for this group */ struct fasync_struct *fsn_fa; /* async notification */ @@ -167,8 +171,8 @@ struct fsnotify_group { struct fsnotify_event *overflow_event; /* Event we queue when the * notification list is too * full */ - atomic_t user_waits; /* Number of tasks waiting for user - * response */ + + struct mem_cgroup *memcg; /* memcg to charge allocations */ /* groups can define private fields here or use the void *private */ union { diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 3607913032be..6c857be8a9b7 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -378,6 +378,8 @@ struct lruvec *mem_cgroup_page_lruvec(struct page *, struct pglist_data *); bool task_in_mem_cgroup(struct task_struct *task, struct mem_cgroup *memcg); struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p); +struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm); + static inline struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL; @@ -857,6 +859,11 @@ static inline bool task_in_mem_cgroup(struct task_struct *task, return true; } +static inline struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) +{ + return NULL; +} + static inline void mem_cgroup_put(struct mem_cgroup *memcg) { } diff --git a/include/linux/sched.h b/include/linux/sched.h index 87bf02d93a27..9cba7f874443 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1149,6 +1149,9 @@ struct task_struct { /* Number of pages to reclaim on returning to userland: */ unsigned int memcg_nr_pages_over_high; + + /* Used by memcontrol for targeted memcg charge: */ + struct mem_cgroup *active_memcg; #endif #ifdef CONFIG_UPROBES diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 44d356f5e47c..75f2d6ee2b72 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -248,6 +248,47 @@ static inline void memalloc_noreclaim_restore(unsigned int flags) current->flags = (current->flags & ~PF_MEMALLOC) | flags; } +#ifdef CONFIG_MEMCG +/** + * memalloc_use_memcg - Starts the remote memcg charging scope. + * @memcg: memcg to charge. + * + * This function marks the beginning of the remote memcg charging scope. All the + * __GFP_ACCOUNT allocations till the end of the scope will be charged to the + * given memcg. Passing NULL will disable the remote memcg charging of the outer + * scope. + */ +static inline struct mem_cgroup *memalloc_use_memcg(struct mem_cgroup *memcg) +{ + struct mem_cgroup *old_memcg = current->active_memcg; + + current->active_memcg = memcg; + return old_memcg; +} + +/** + * memalloc_unuse_memcg - Ends the remote memcg charging scope. + * @memcg: outer scope memcg to restore. + * + * This function marks the end of the remote memcg charging scope started by + * memalloc_use_memcg(). Always make sure the given memcg is the return valure + * from the pairing memalloc_use_memcg call. + */ +static inline void memalloc_unuse_memcg(struct mem_cgroup *memcg) +{ + current->active_memcg = memcg; +} +#else +static inline struct mem_cgroup *memalloc_use_memcg(struct mem_cgroup *memcg) +{ + return NULL; +} + +static inline void memalloc_unuse_memcg(struct mem_cgroup *memcg) +{ +} +#endif + #ifdef CONFIG_MEMBARRIER enum { MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY = (1U << 0), diff --git a/kernel/fork.c b/kernel/fork.c index a64d0a19f174..0b07db6fef06 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -843,6 +843,9 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node) tsk->fail_nth = 0; #endif +#ifdef CONFIG_MEMCG + tsk->active_memcg = NULL; +#endif return tsk; free_stack: diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 02e77c88967a..6b1a8f8e0a82 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -678,9 +678,20 @@ struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p) } EXPORT_SYMBOL(mem_cgroup_from_task); -static struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) +/** + * get_mem_cgroup_from_mm: Obtain a reference on given mm_struct's memcg. + * @mm: mm from which memcg should be extracted. It can be NULL. + * + * Obtain a reference on mm->memcg and returns it if successful. Otherwise + * root_mem_cgroup is returned. However if mem_cgroup is disabled, NULL is + * returned. + */ +struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) { - struct mem_cgroup *memcg = NULL; + struct mem_cgroup *memcg; + + if (mem_cgroup_disabled()) + return NULL; rcu_read_lock(); do { @@ -700,6 +711,25 @@ static struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) rcu_read_unlock(); return memcg; } +EXPORT_SYMBOL(get_mem_cgroup_from_mm); + +/** + * First try to obtain reference on current->active_memcg. On failure, try to + * obtain reference on current->mm->memcg. On further failure root_mem_cgroup + * is returned. + */ +static __always_inline struct mem_cgroup *get_mem_cgroup_from_current(void) +{ + if (unlikely(current->active_memcg)) { + rcu_read_lock(); + if (css_tryget_online(¤t->active_memcg->css)) { + rcu_read_unlock(); + return current->active_memcg; + } + rcu_read_unlock(); + } + return get_mem_cgroup_from_mm(current->mm); +} /** * mem_cgroup_iter - iterate over memory cgroup hierarchy @@ -2260,7 +2290,7 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) if (current->memcg_kmem_skip_account) return cachep; - memcg = get_mem_cgroup_from_mm(current->mm); + memcg = get_mem_cgroup_from_current(); kmemcg_id = READ_ONCE(memcg->kmemcg_id); if (kmemcg_id < 0) goto out; @@ -2344,7 +2374,7 @@ int memcg_kmem_charge(struct page *page, gfp_t gfp, int order) if (memcg_kmem_bypass()) return 0; - memcg = get_mem_cgroup_from_mm(current->mm); + memcg = get_mem_cgroup_from_current(); if (!mem_cgroup_is_root(memcg)) { ret = memcg_kmem_charge_memcg(page, gfp, order, memcg); if (!ret)