From patchwork Wed Apr 17 21:54:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10906303 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CF82314DB for ; Wed, 17 Apr 2019 21:54:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ACE042880B for ; Wed, 17 Apr 2019 21:54:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9FF3428A27; Wed, 17 Apr 2019 21:54:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AEE0C2880B for ; Wed, 17 Apr 2019 21:54:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 663C16B0008; Wed, 17 Apr 2019 17:54:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 612E06B000A; Wed, 17 Apr 2019 17:54:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BA866B000C; Wed, 17 Apr 2019 17:54:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 0A3526B0008 for ; Wed, 17 Apr 2019 17:54:57 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id j18so15147pfi.20 for ; Wed, 17 Apr 2019 14:54:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=76gfXMG+0uRElSBO/APtur6FtcIZXTee5d1iEwxVGMw=; b=o/ZbxTo69x2VVNiCmvp7hGmKKBxQbm8AN6Hai0r+ZpBQcZyZ0xr7JPxqeN/w1YTXRZ h48GkuYIJfU42/AWzQKYEjoPTAzXsz93mHsd4XtB+DU/eNv0mDfnildCureMX3QPqr9B TB+SOoQag++Tz5QgBrRVlTcub8LiF88NSkB9e0zOouOwmZh5KqJHn8ekByOJKopyL+90 5hsmWBq3kWpqEnBv6Hp5KU+i0XGR6w0OvUT9yRuiNaN0cKKJAdIFaQnv/xLJa/6XBfrE mtNvkI9CMYYhsraS/u9qGHLZrmasdeL+wWeSn12Fy0ulTQNDR4RwL6UbtKUKtKB73q9y WMhA== X-Gm-Message-State: APjAAAXVzqAFC2z3MIXZsw5N0GLd+NKuLI3LonY7FFDkox3SJAiolOyz iwvVJB17NnXj7MWaGMwSsQ4nZSxDIOKKKGi11GVQSX7UxpBPwbqZDCkfsZZokd+Kyv/ofth2b9S Bx6ypGK1db5Mo5Rhu+d2NxwMcp0U2n5xLthLMcMu7e8s6T7+gCw4Hz+nw1nTlg3c3wQ== X-Received: by 2002:a17:902:1c1:: with SMTP id b59mr73452708plb.182.1555538096620; Wed, 17 Apr 2019 14:54:56 -0700 (PDT) X-Received: by 2002:a17:902:1c1:: with SMTP id b59mr73452609plb.182.1555538094945; Wed, 17 Apr 2019 14:54:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555538094; cv=none; d=google.com; s=arc-20160816; b=BrPTF3Pz0IAnubAx7KmR5gkjLGdO9rnzssGyq2amruA5fgCmHt1zK49TPlYo8EDgmT A5/fqhAOIvJeL+7jEpkX4aRRxm+qUZGvty/OuZKsGkpwJW8xSNqcVOgEc5oAok5Pa68f Fw6mRHcCpPh8CENL4qrIA4Yc8Y4oFDUQ90PNL5h9HQzgL18zZKcMNFM6m8o5cDV47dwQ 0KtvO1rZVIMB8K9ODOxrUyBCusSX/mL9CrJJ8DtWX6kdUvzUqnaBFao9s5u68DoAaeaY 11/x2xF3T8WVXlxL77l+J1kKfo61KWIW97EPx1qGsXS09v8TdYijtjnmEUcknNwfBpbf 5uZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=76gfXMG+0uRElSBO/APtur6FtcIZXTee5d1iEwxVGMw=; b=daCa32W1n58qEv1XcvbaxM/InYcKTOxUIzPOk2li6qo4RdaBUpKO5zZvNhaO3OFTdr ckmi5cnfbuIwNkY4xDTCw0Qnh8c6XTMHwMgtPfiSX6DkULe2rjZ1+IAGojm+JxCus5gO ZMI9nIZMskxu39VeupuZ4ZKEkZjTT529o0kDlCJ1WFTFaZXsl28Z1+T7vTPqjbXHr85A gpXXu5QW0EVqIHwwftkvxk2Xr/yknOBugn9rihnWh4VubtKWwzcL3Xv0oIyl9YX5XKLY awfjVQL09NPOubR4IYQThoWbkYCnUQdBGVC0FQKaZUGD1w+CaLqsHE8R176WGhKQfvJl eb+g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=gdI5EkLP; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id k1sor99658pfc.65.2019.04.17.14.54.54 for (Google Transport Security); Wed, 17 Apr 2019 14:54:54 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=gdI5EkLP; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=76gfXMG+0uRElSBO/APtur6FtcIZXTee5d1iEwxVGMw=; b=gdI5EkLPUs5z6EaRfXGw41UkfBRR+UKMoxX+s1rEhoXw64sOhR8jwzVaL3cG3TWfx5 o5aPe4KIU6ePKtPU2Uu5fTifU2uklWuSHVSrCpusHW4Ri0HIF5z2uMg1J/cR7CmROP0w lb6tUoAsX4j4BS6wtsZ5Qz7ICnhWgHnVqC++hCMJVymGGwketGMErK95ltTSPfj0xEE7 hHE3k6GXHf4IHciGhiSWFeogsipTbMCm3FzroabyrNDzl1aQG1rN+lh5lO0Onwy5spfT R/PejDH+aBhLKbhGoS8FqSORqec0RybvC2TFq3PCOPBGsC6MESABES8Q0QdksuPQbHyT h3UA== X-Google-Smtp-Source: APXvYqx8eaGaLpTxyhmCU99DvbZgOOvhza9jKdm621uZ7OVdDTLFJcxMXop75gq1ZtJxXug7JlIq/A== X-Received: by 2002:a62:1f92:: with SMTP id l18mr93491326pfj.180.1555538094660; Wed, 17 Apr 2019 14:54:54 -0700 (PDT) Received: from tower.thefacebook.com ([2620:10d:c090:200::5597]) by smtp.gmail.com with ESMTPSA id x6sm209024pfb.171.2019.04.17.14.54.53 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 17 Apr 2019 14:54:53 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@fb.com, Johannes Weiner , Michal Hocko , Rik van Riel , david@fromorbit.com, Christoph Lameter , Pekka Enberg , Vladimir Davydov , cgroups@vger.kernel.org, Roman Gushchin Subject: [PATCH 1/5] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache() Date: Wed, 17 Apr 2019 14:54:30 -0700 Message-Id: <20190417215434.25897-2-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190417215434.25897-1-guro@fb.com> References: <20190417215434.25897-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Initialize kmem_cache->memcg_params.memcg pointer in memcg_link_cache() rather than in init_memcg_params(). Once kmem_cache will hold a reference to the memory cgroup, it will simplify the refcounting. For non-root kmem_caches memcg_link_cache() is always called before the kmem_cache becomes visible to a user, so it's safe. Signed-off-by: Roman Gushchin --- mm/slab.c | 2 +- mm/slab.h | 5 +++-- mm/slab_common.c | 14 +++++++------- mm/slub.c | 2 +- 4 files changed, 12 insertions(+), 11 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index b1eefe751d2a..57a332f524cf 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1268,7 +1268,7 @@ void __init kmem_cache_init(void) nr_node_ids * sizeof(struct kmem_cache_node *), SLAB_HWCACHE_ALIGN, 0, 0); list_add(&kmem_cache->list, &slab_caches); - memcg_link_cache(kmem_cache); + memcg_link_cache(kmem_cache, NULL); slab_state = PARTIAL; /* diff --git a/mm/slab.h b/mm/slab.h index 43ac818b8592..6a562ca72bca 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -289,7 +289,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, } extern void slab_init_memcg_params(struct kmem_cache *); -extern void memcg_link_cache(struct kmem_cache *s); +extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, void (*deact_fn)(struct kmem_cache *)); @@ -344,7 +344,8 @@ static inline void slab_init_memcg_params(struct kmem_cache *s) { } -static inline void memcg_link_cache(struct kmem_cache *s) +static inline void memcg_link_cache(struct kmem_cache *s, + struct mem_cgroup *memcg) { } diff --git a/mm/slab_common.c b/mm/slab_common.c index 58251ba63e4a..6e00bdf8618d 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -140,13 +140,12 @@ void slab_init_memcg_params(struct kmem_cache *s) } static int init_memcg_params(struct kmem_cache *s, - struct mem_cgroup *memcg, struct kmem_cache *root_cache) + struct kmem_cache *root_cache) { struct memcg_cache_array *arr; if (root_cache) { s->memcg_params.root_cache = root_cache; - s->memcg_params.memcg = memcg; INIT_LIST_HEAD(&s->memcg_params.children_node); INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node); return 0; @@ -221,11 +220,12 @@ int memcg_update_all_caches(int num_memcgs) return ret; } -void memcg_link_cache(struct kmem_cache *s) +void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) { if (is_root_cache(s)) { list_add(&s->root_caches_node, &slab_root_caches); } else { + s->memcg_params.memcg = memcg; list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); list_add(&s->memcg_params.kmem_caches_node, @@ -244,7 +244,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) } #else static inline int init_memcg_params(struct kmem_cache *s, - struct mem_cgroup *memcg, struct kmem_cache *root_cache) + struct kmem_cache *root_cache) { return 0; } @@ -384,7 +384,7 @@ static struct kmem_cache *create_cache(const char *name, s->useroffset = useroffset; s->usersize = usersize; - err = init_memcg_params(s, memcg, root_cache); + err = init_memcg_params(s, root_cache); if (err) goto out_free_cache; @@ -394,7 +394,7 @@ static struct kmem_cache *create_cache(const char *name, s->refcount = 1; list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, memcg); out: if (err) return ERR_PTR(err); @@ -997,7 +997,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, create_boot_cache(s, name, size, flags, useroffset, usersize); list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, NULL); s->refcount = 1; return s; } diff --git a/mm/slub.c b/mm/slub.c index a34fbe1f6ede..2b9244529d76 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4224,7 +4224,7 @@ static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache) } slab_init_memcg_params(s); list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, NULL); return s; } From patchwork Wed Apr 17 21:54:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10906307 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6DC8214DB for ; Wed, 17 Apr 2019 21:55:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 56EF626255 for ; Wed, 17 Apr 2019 21:55:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4ADC52880B; Wed, 17 Apr 2019 21:55:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4260626255 for ; Wed, 17 Apr 2019 21:55:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D8EDB6B000A; Wed, 17 Apr 2019 17:54:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D41C26B000C; Wed, 17 Apr 2019 17:54:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B9A8A6B000D; Wed, 17 Apr 2019 17:54:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 750676B000A for ; Wed, 17 Apr 2019 17:54:58 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id g11so957501pgs.12 for ; Wed, 17 Apr 2019 14:54:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=MtkthRXb9xr3Z6+sHE+ddzmShZROkNDy015XAdhrxgc=; b=XKHkS9oN6/Kw3VwFE0IYAkOY1wqqpu1M6kCPKNOaCh2ShkDMnfEWLDbJPveNsQ3BZP aXZeeaMEHYWN+tm7HfMO47Thwgs78PEPAfSSSSFx8qdTImnQV8TjR+lUpW7fkf9JILK+ wC/W2dRLWSkoLXunUxdexZhkjXCBTwctjzv4oiTNOjDluMdJTr0JDSTZteYA7ROl7bJo H4MsJB44RMXSuJcn7pA1yugJe51xxIpKsSfKjP0gON/XORfJGpNRjuRl+aN/MaQrDlQa PqlseCsvHrejRZSBRoNGG8vVKKjgmEuwBINA9cqD9IbcocXW3a6+dWufH10z5k9Izxt+ AqLA== X-Gm-Message-State: APjAAAU9FwA71kvlRZ39OBCaXQcn5fYoQWsNEppvsbOo9O1aw+hM1N4F exNzNtB0rWv4EQcwL6ICwT5mLSYxwCvOkmfh+IT733mMgESIquQxRYgS1Kk/CG6gcmmMo4mbyRQ jHAp8PWrlqKWF6iVpY9Na+Az2gJdwnG8+DtbxBHo+Ycv/4bA0jxTWpA1rr0BYv1O1mQ== X-Received: by 2002:a63:185a:: with SMTP id 26mr58430656pgy.337.1555538098079; Wed, 17 Apr 2019 14:54:58 -0700 (PDT) X-Received: by 2002:a63:185a:: with SMTP id 26mr58430579pgy.337.1555538096633; Wed, 17 Apr 2019 14:54:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555538096; cv=none; d=google.com; s=arc-20160816; b=FRRH77Lov4x8G6yDik5AQGoH7oYIcCZeRakeQbVkSdXSGhcuDO3rpeb56VUkuaxT5Y XUwK8RYwICORfV5pgJulymN+HlKvZK42t9uriNcHyTV5Vg+gIk9OOXMW2PtzDXCWA/R0 hzpijl75B7FwaVGTaZqRRD1X00nHgpfaBZmbZmk2UOQpI9j6gXWNF8xe2gU+XoJAQKac 7Atkr9zReKz/aAd93yJrY7dTs7FaO6l4CgxgA222Gx4vIr6PPtihl302mlpJ9U+qhPbQ TUJaG7/6HKXumLgAmzYm7Ya9GYF4w/yTM9f92tLAC+LatKTTW4k8PWw3yUf0QO0kk0Z5 7Olg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=MtkthRXb9xr3Z6+sHE+ddzmShZROkNDy015XAdhrxgc=; b=CYUDFChjbj9busCGN5Vj92mBQJjLNrqGcYRV+bUoEEp4aRnNtKd99VRH6aEBJuX1HS 0WlGLyrnuMrJAh7bPiUmqV0Bd/fb1zEWRo28he1Q8vrhI3E5Vu36uueypt2ysPJ6PKly Qdi/w0TfU5U3BcOiIK/IQfG/t8rstQTMqGXtlAwPC4JxREo1cQO7584TUtxJKxLu7sDs zguytgkSS7qbe4xdxPghf2eFmxPVleSzeUqwG9syGu3YwSKjqbWe+ydUvQIBb6ZcECWq cMOkDVp7XfVdQwXxVMnjNV0rgAFZG0mW36ufc23yQnvuturAD6ZHWnBqncxCJmesa1NP Y9Rw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=bkJtX8C+; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id g95sor23379plb.5.2019.04.17.14.54.56 for (Google Transport Security); Wed, 17 Apr 2019 14:54:56 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=bkJtX8C+; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MtkthRXb9xr3Z6+sHE+ddzmShZROkNDy015XAdhrxgc=; b=bkJtX8C+3fj0Tjhyg4vCVv/IFmBzbGBkiaDAvFLmDb8Lq2CSpvTCu0DpsxNBzVRZE6 /KavKK6sZJp6q1on9j4P5GAND/bIlyXqkI64KWpz4q5A+ph+Wga2tCFOg5zBESw4qB6F 0a8qx/VE5UXWEj/OfguZ1W6Acn8pwoIJDzZu71ecJg94TK4ni4QRK9oElJ9DPSm60c+g K0zyATOpmJW3pcoANLB3OTOt8dtYU0H7WgFn7PoNVvhHacgEQcsm7GTnMLHoclASVfe4 LiW1YvyENbVaOxl0Qz3UxRV4EsfpG6OXUukFSX+jgB8C/tTLlnztGVBcSkJ+IPrW2RX7 G1Xg== X-Google-Smtp-Source: APXvYqwg/PODrPNFlRfYDoK4DEAkGsveLbL3EEOb4pDRkT0d8MTv1g9F41Dfa37BfIh/CXJNuXPr9w== X-Received: by 2002:a17:902:e382:: with SMTP id ch2mr89133541plb.94.1555538096288; Wed, 17 Apr 2019 14:54:56 -0700 (PDT) Received: from tower.thefacebook.com ([2620:10d:c090:200::5597]) by smtp.gmail.com with ESMTPSA id x6sm209024pfb.171.2019.04.17.14.54.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 17 Apr 2019 14:54:55 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@fb.com, Johannes Weiner , Michal Hocko , Rik van Riel , david@fromorbit.com, Christoph Lameter , Pekka Enberg , Vladimir Davydov , cgroups@vger.kernel.org, Roman Gushchin Subject: [PATCH 2/5] mm: generalize postponed non-root kmem_cache deactivation Date: Wed, 17 Apr 2019 14:54:31 -0700 Message-Id: <20190417215434.25897-3-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190417215434.25897-1-guro@fb.com> References: <20190417215434.25897-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently SLUB uses a work scheduled after an RCU grace period to deactivate a non-root kmem_cache. This mechanism can be reused for kmem_caches reparenting, but requires some generalization. Let's decouple all infrastructure (rcu callback, work callback) from the SLUB-specific code, so it can be used with SLAB as well. Also, let's rename some functions to make the code look simpler. All SLAB/SLUB-specific functions start with "__". Remove "deact_" prefix from the corresponding struct fields. Here is the graph of a new calling scheme: kmemcg_cache_deactivate() __kmemcg_cache_deactivate() SLAB/SLUB-specific kmemcg_schedule_work_after_rcu() rcu kmemcg_after_rcu_workfn() work kmemcg_cache_deactivate_after_rcu() __kmemcg_cache_deactivate_after_rcu() SLAB/SLUB-specific instead of: __kmemcg_cache_deactivate() SLAB/SLUB-specific slab_deactivate_memcg_cache_rcu_sched() SLUB-only kmemcg_deactivate_rcufn SLUB-only, rcu kmemcg_deactivate_workfn SLUB-only, work kmemcg_cache_deact_after_rcu() SLUB-only Signed-off-by: Roman Gushchin --- include/linux/slab.h | 6 ++--- mm/slab.c | 4 +++ mm/slab.h | 3 ++- mm/slab_common.c | 62 ++++++++++++++++++++------------------------ mm/slub.c | 8 +----- 5 files changed, 38 insertions(+), 45 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 9449b19c5f10..47923c173f30 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -642,10 +642,10 @@ struct memcg_cache_params { struct list_head children_node; struct list_head kmem_caches_node; - void (*deact_fn)(struct kmem_cache *); + void (*work_fn)(struct kmem_cache *); union { - struct rcu_head deact_rcu_head; - struct work_struct deact_work; + struct rcu_head rcu_head; + struct work_struct work; }; }; }; diff --git a/mm/slab.c b/mm/slab.c index 57a332f524cf..14466a73d057 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -2317,6 +2317,10 @@ void __kmemcg_cache_deactivate(struct kmem_cache *cachep) { __kmem_cache_shrink(cachep); } + +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) +{ +} #endif int __kmem_cache_shutdown(struct kmem_cache *cachep) diff --git a/mm/slab.h b/mm/slab.h index 6a562ca72bca..4a261c97c138 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -172,6 +172,7 @@ int __kmem_cache_shutdown(struct kmem_cache *); void __kmem_cache_release(struct kmem_cache *); int __kmem_cache_shrink(struct kmem_cache *); void __kmemcg_cache_deactivate(struct kmem_cache *s); +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s); void slab_kmem_cache_release(struct kmem_cache *); struct seq_file; @@ -291,7 +292,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, extern void slab_init_memcg_params(struct kmem_cache *); extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*deact_fn)(struct kmem_cache *)); + void (*work_fn)(struct kmem_cache *)); #else /* CONFIG_MEMCG_KMEM */ diff --git a/mm/slab_common.c b/mm/slab_common.c index 6e00bdf8618d..4e5b4292a763 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -691,17 +691,18 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg, put_online_cpus(); } -static void kmemcg_deactivate_workfn(struct work_struct *work) +static void kmemcg_after_rcu_workfn(struct work_struct *work) { struct kmem_cache *s = container_of(work, struct kmem_cache, - memcg_params.deact_work); + memcg_params.work); get_online_cpus(); get_online_mems(); mutex_lock(&slab_mutex); - s->memcg_params.deact_fn(s); + s->memcg_params.work_fn(s); + s->memcg_params.work_fn = NULL; mutex_unlock(&slab_mutex); @@ -712,37 +713,28 @@ static void kmemcg_deactivate_workfn(struct work_struct *work) css_put(&s->memcg_params.memcg->css); } -static void kmemcg_deactivate_rcufn(struct rcu_head *head) +/* + * We need to grab blocking locks. Bounce to ->work. The + * work item shares the space with the RCU head and can't be + * initialized eariler. +*/ +static void kmemcg_schedule_work_after_rcu(struct rcu_head *head) { struct kmem_cache *s = container_of(head, struct kmem_cache, - memcg_params.deact_rcu_head); + memcg_params.rcu_head); - /* - * We need to grab blocking locks. Bounce to ->deact_work. The - * work item shares the space with the RCU head and can't be - * initialized eariler. - */ - INIT_WORK(&s->memcg_params.deact_work, kmemcg_deactivate_workfn); - queue_work(memcg_kmem_cache_wq, &s->memcg_params.deact_work); + INIT_WORK(&s->memcg_params.work, kmemcg_after_rcu_workfn); + queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); } -/** - * slab_deactivate_memcg_cache_rcu_sched - schedule deactivation after a - * sched RCU grace period - * @s: target kmem_cache - * @deact_fn: deactivation function to call - * - * Schedule @deact_fn to be invoked with online cpus, mems and slab_mutex - * held after a sched RCU grace period. The slab is guaranteed to stay - * alive until @deact_fn is finished. This is to be used from - * __kmemcg_cache_deactivate(). - */ -void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*deact_fn)(struct kmem_cache *)) +static void kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { - if (WARN_ON_ONCE(is_root_cache(s)) || - WARN_ON_ONCE(s->memcg_params.deact_fn)) - return; + __kmemcg_cache_deactivate_after_rcu(s); +} + +static void kmemcg_cache_deactivate(struct kmem_cache *s) +{ + __kmemcg_cache_deactivate(s); if (s->memcg_params.root_cache->memcg_params.dying) return; @@ -750,8 +742,9 @@ void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, /* pin memcg so that @s doesn't get destroyed in the middle */ css_get(&s->memcg_params.memcg->css); - s->memcg_params.deact_fn = deact_fn; - call_rcu(&s->memcg_params.deact_rcu_head, kmemcg_deactivate_rcufn); + WARN_ON_ONCE(s->memcg_params.work_fn); + s->memcg_params.work_fn = kmemcg_cache_deactivate_after_rcu; + call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); } void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) @@ -773,7 +766,7 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) if (!c) continue; - __kmemcg_cache_deactivate(c); + kmemcg_cache_deactivate(c); arr->entries[idx] = NULL; } mutex_unlock(&slab_mutex); @@ -866,11 +859,12 @@ static void flush_memcg_workqueue(struct kmem_cache *s) mutex_unlock(&slab_mutex); /* - * SLUB deactivates the kmem_caches through call_rcu. Make + * SLAB and SLUB deactivate the kmem_caches through call_rcu. Make * sure all registered rcu callbacks have been invoked. */ - if (IS_ENABLED(CONFIG_SLUB)) - rcu_barrier(); +#ifndef CONFIG_SLOB + rcu_barrier(); +#endif /* * SLAB and SLUB create memcg kmem_caches through workqueue and SLUB diff --git a/mm/slub.c b/mm/slub.c index 2b9244529d76..195f61785c7d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4033,7 +4033,7 @@ int __kmem_cache_shrink(struct kmem_cache *s) } #ifdef CONFIG_MEMCG -static void kmemcg_cache_deact_after_rcu(struct kmem_cache *s) +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { /* * Called with all the locks held after a sched RCU grace period. @@ -4059,12 +4059,6 @@ void __kmemcg_cache_deactivate(struct kmem_cache *s) */ slub_set_cpu_partial(s, 0); s->min_partial = 0; - - /* - * s->cpu_partial is checked locklessly (see put_cpu_partial), so - * we have to make sure the change is visible before shrinking. - */ - slab_deactivate_memcg_cache_rcu_sched(s, kmemcg_cache_deact_after_rcu); } #endif /* CONFIG_MEMCG */ From patchwork Wed Apr 17 21:54:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10906309 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7FCD514DB for ; Wed, 17 Apr 2019 21:55:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6A3AC26255 for ; Wed, 17 Apr 2019 21:55:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5E9492880B; Wed, 17 Apr 2019 21:55:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D789126255 for ; Wed, 17 Apr 2019 21:55:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1FB186B000C; Wed, 17 Apr 2019 17:55:00 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 162EE6B000D; Wed, 17 Apr 2019 17:54:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2B4A6B000E; Wed, 17 Apr 2019 17:54:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id AB4756B000C for ; Wed, 17 Apr 2019 17:54:59 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id 33so15419964pgv.17 for ; Wed, 17 Apr 2019 14:54:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=cRmr36eXTaKgScK2fa3yPK15EeGvQLf0Lc+faeDoG3Q=; b=VEV3Li/9YuCNqydlSFEjpMflQPrYvqRYXUcLuce7mLrrcU/o7g9zaTqtFz9jxrHm1C eJvp9bDx04eHM3W+sfZKr3To6Ly6NGnCrtQc5qdyUQqmezzxvPzSM1JrtSW4WQVonGgk 2kX6AUxv3aDiE7G/0/2pCaPdvPZDF1bBaVpjt3d41DfefnABxya54XPBt8jVCzdBcgsf VzLQXLZe5xAyVa2UzfGpEfXPiyWQSac5wCcPMShUGpYKcjdhwW3xWAPk62cTKTnVFxOP V4/HtJu7feCCFgj1YYXkKBEGm59m8inQKVt28f4bKV0nwhU36ayUML4DZ25KuSd48gP3 vHgQ== X-Gm-Message-State: APjAAAUFHSFDEfFh6FwnWVMZvS3Gc3xeLz2wkBL0pO5shQ8MWD+o3AJ2 5vFFvlsoOewk8htM8bSWZfSZwoPg3/LHZdTVAHzmx+Qg7Tx5+EHKlHfWW6w2yS4olDoNmSV92BZ u2HijG9y6BNhlD2LHS9exRtagenQfHdmQ5JiQw/QyI/740mpjGzgNJvYpK0IXxZynmw== X-Received: by 2002:a63:e304:: with SMTP id f4mr80305927pgh.374.1555538099356; Wed, 17 Apr 2019 14:54:59 -0700 (PDT) X-Received: by 2002:a63:e304:: with SMTP id f4mr80305872pgh.374.1555538098078; Wed, 17 Apr 2019 14:54:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555538098; cv=none; d=google.com; s=arc-20160816; b=i+5bVRjePCRJFyp++vlpDFstjxV/eDy1IRwhns1isklCE8LSpuyA/2ntRWwpfFvx2V U683SCFASqCsyzncfZQ0UNEvYIrc5YyU+Xuotm4rg3iMUu39CofXuwL3JXNfzS0HnTPB QercExbpgbIvEgu0Xv72aRI9Jpi4S+Ol13Zep40vLd4wHKteuKuCCRZH3a4iYUQ5egdt 1OLWC7sBMuKGscB42SVMNSeom7OJ/2yOUYmJgWUs9/892o9qkKywrwWO460p3ojVjlFw PguyJ+tGekmNzO6wbpxpFzXNXzHZIf0kxEMWFYRmZUYSVyGjYJB8wHhU50ggnx5D7hY8 9Fow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=cRmr36eXTaKgScK2fa3yPK15EeGvQLf0Lc+faeDoG3Q=; b=h9U+cXwNvz8v5QQTO/aSA9h98DPM2GmAxLdjcyv19+is7UGQTNGpRyiYl8uiYxb4mJ dNes/KTDhJP0BvSMS58pYIJh4RzOQPjWiwbXQ6N8FeVqtEuEJAWkiljJ5xahAznE6FWg kVG/gjsJ9r4FGUZW9816T7Zs0JgwcP63SF5Sxii42X08Y2EYMWiP2hBfUfBuvApauVcC 94JRDMfQDlWQO24DbP/6to9xACmYACGoWo5Wq4wZztiX5wS+Y23yh/hvYD/p+h3Ab79f 3VzPKVMP+zs7luMjEYKxsl9HRSm2SwzSVkrkYSaxtQ2ldhC7GwbjVw6n2wk+Xc/vIeWp A5lQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=NTl4XRWo; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id c4sor50289pgh.26.2019.04.17.14.54.57 for (Google Transport Security); Wed, 17 Apr 2019 14:54:58 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=NTl4XRWo; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cRmr36eXTaKgScK2fa3yPK15EeGvQLf0Lc+faeDoG3Q=; b=NTl4XRWoAOp1/x1XLk1hvjEAxMAeJ6AzgfrCpOEahhVu5/jEYVGYy9d8pQy0Q15Q5U Af+lGn62qofRvBvBdvBMEbsEt60SVxN4kbLUVNa3p4G04O0b3J0+eI/IK/fQuiBqxiS6 WiuryM9qYvtgVaF+gjTGm1N4hsHjkIoMrNg+aPLTfQz5+EXgGtkXNKvE9FYVv7RBNw+3 //6RWegxUI3mqwV1pBVcvVWJrvdru3y0dvGNdNFFl8qK0pwHGJLQr9whaDrPGVDFzW8S 2zjbStnsuX0AEOR3NxF66n1IsvnQg5X6M9nueJXmXqIsgr7PkbG+bqIfS8Q3nPtlQuZA XMrA== X-Google-Smtp-Source: APXvYqxdPynQgSY+DGmrN9sj9aTAAj9DpThBayF5QAZBIIpe/kk5KnHKGpzPxv/HTkfDwV1hWwrARA== X-Received: by 2002:a63:d713:: with SMTP id d19mr28640839pgg.145.1555538097772; Wed, 17 Apr 2019 14:54:57 -0700 (PDT) Received: from tower.thefacebook.com ([2620:10d:c090:200::5597]) by smtp.gmail.com with ESMTPSA id x6sm209024pfb.171.2019.04.17.14.54.56 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 17 Apr 2019 14:54:57 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@fb.com, Johannes Weiner , Michal Hocko , Rik van Riel , david@fromorbit.com, Christoph Lameter , Pekka Enberg , Vladimir Davydov , cgroups@vger.kernel.org, Roman Gushchin Subject: [PATCH 3/5] mm: introduce __memcg_kmem_uncharge_memcg() Date: Wed, 17 Apr 2019 14:54:32 -0700 Message-Id: <20190417215434.25897-4-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190417215434.25897-1-guro@fb.com> References: <20190417215434.25897-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Let's separate the page counter modification code out of __memcg_kmem_uncharge() in a way similar to what __memcg_kmem_charge() and __memcg_kmem_charge_memcg() work. This will allow to reuse this code later using a new memcg_kmem_uncharge_memcg() wrapper, which calls __memcg_kmem_unchare_memcg() if memcg_kmem_enabled() check is passed. Signed-off-by: Roman Gushchin --- include/linux/memcontrol.h | 10 ++++++++++ mm/memcontrol.c | 25 +++++++++++++++++-------- 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 36bdfe8e5965..deb209510902 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1298,6 +1298,8 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge(struct page *page, int order); int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, struct mem_cgroup *memcg); +void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg, + unsigned int nr_pages); extern struct static_key_false memcg_kmem_enabled_key; extern struct workqueue_struct *memcg_kmem_cache_wq; @@ -1339,6 +1341,14 @@ static inline int memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, return __memcg_kmem_charge_memcg(page, gfp, order, memcg); return 0; } + +static inline void memcg_kmem_uncharge_memcg(struct page *page, int order, + struct mem_cgroup *memcg) +{ + if (memcg_kmem_enabled()) + __memcg_kmem_uncharge_memcg(memcg, 1 << order); +} + /* * helper for accessing a memcg's index. It will be used as an index in the * child cache array in kmem_cache, and also to derive its name. This function diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 48a8f1c35176..b2c39f187cbb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2750,6 +2750,22 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order) css_put(&memcg->css); return ret; } + +/** + * __memcg_kmem_uncharge_memcg: uncharge a kmem page + * @memcg: memcg to uncharge + * @nr_pages: number of pages to uncharge + */ +void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg, + unsigned int nr_pages) +{ + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + page_counter_uncharge(&memcg->kmem, nr_pages); + + page_counter_uncharge(&memcg->memory, nr_pages); + if (do_memsw_account()) + page_counter_uncharge(&memcg->memsw, nr_pages); +} /** * __memcg_kmem_uncharge: uncharge a kmem page * @page: page to uncharge @@ -2764,14 +2780,7 @@ void __memcg_kmem_uncharge(struct page *page, int order) return; VM_BUG_ON_PAGE(mem_cgroup_is_root(memcg), page); - - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) - page_counter_uncharge(&memcg->kmem, nr_pages); - - page_counter_uncharge(&memcg->memory, nr_pages); - if (do_memsw_account()) - page_counter_uncharge(&memcg->memsw, nr_pages); - + __memcg_kmem_uncharge_memcg(memcg, nr_pages); page->mem_cgroup = NULL; /* slab pages do not have PageKmemcg flag set */ From patchwork Wed Apr 17 21:54:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10906311 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0C9401390 for ; Wed, 17 Apr 2019 21:55:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E967E26255 for ; Wed, 17 Apr 2019 21:55:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DCE122880B; Wed, 17 Apr 2019 21:55:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B9BDB28756 for ; Wed, 17 Apr 2019 21:55:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9704B6B000D; Wed, 17 Apr 2019 17:55:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 923BF6B0266; Wed, 17 Apr 2019 17:55:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E6936B000D; Wed, 17 Apr 2019 17:55:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 19AE56B000D for ; Wed, 17 Apr 2019 17:55:02 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id f7so179806plr.10 for ; Wed, 17 Apr 2019 14:55:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=2qTSnTK6DrJDXJXnCNRZMj0fJcJ1BWzKc8LF04T53hg=; b=nCOzR4A2nu6Z84uqOUrtKXCriPWy/H+bgWEiSi/Y1utHVBYzjkiXPkEKTduR8ZWMcV iGRfp/6xJ7cGQax3Rr35CdvMXSz4AkMWf9dcaX2CjqVBUEdVBEtYmUn87U8sdkxsPqu/ e1E5AUm0I8ffENk8s/xe0OtgL8wOz0VYNNMdnY8FxP0VOhTqMXJQ+41oDSDhhZFI7RKE tb+Nsq/5lP3VHI0xYCJ+VeRNmedKQ/auDYd0h1Us2w3EKqmhfoRyaOO7guxLcOB+Sx9K kXlB1dSkhA/ZwDC5YO6mMDDKhalLTVuyug5j44t0FQ4FaCGKErJZ13Z5sPqlHxtSb/iB 57/g== X-Gm-Message-State: APjAAAVAE9I6UM7q7v03WzrOvN6ZUxUXAMZ21al1NkiTBAGTx4nEDwmq ycCl0J8uvdY5nnt/lpUso8xGTi76bP/GoFJ3R0m0AH5keC3hvKOzVAXVMz5Vr2qV7cl5qMERybr DSmT6bsbjDrdkZjaqJSWyI/yJMkU4OnCBH4hn2qTl6/bDdhqSguiE58wOR7yIXLl+fw== X-Received: by 2002:a63:180a:: with SMTP id y10mr2048844pgl.450.1555538101635; Wed, 17 Apr 2019 14:55:01 -0700 (PDT) X-Received: by 2002:a63:180a:: with SMTP id y10mr2048745pgl.450.1555538099769; Wed, 17 Apr 2019 14:54:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555538099; cv=none; d=google.com; s=arc-20160816; b=TsDKTqaa/W1uBS4VV7by8FLfCD7CJZIrpFZnerHOIXoOVDG8Ga+MpYtDdbM9lj6IzZ zULqEbxKa5OkWq/T0pnurDT3sFsfthAb+7+x5f6wmVbVPDNlBbTdsEaaKx2RJ9DQaxJ8 PGZ9JpEMMZkTROAfcGntj4nM5MHLng8YJMqYVZQ+cUCie7aKkfNs58e57ittzcVtyKAV 9UWdm/hIVTq/oNLShPPLUAWqkW34VKwUiNQ0TfCmtKsNMHKh0K0yRkNKsPAo/Y/+1QIu YJaN8Pt63txE2gVGfjY1/+meZnFoxjgSpvTRIUzba9jgBkyQjt5ne8PjfjAxhdb3g8pT 7xkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=2qTSnTK6DrJDXJXnCNRZMj0fJcJ1BWzKc8LF04T53hg=; b=jUlBgEWqDw3v4XWdgw+sKu844ND9QMkNxci2wji0M18j7uoLVBVuEU8q1m8dtOrPGh YP7LUwqgl7TNTVQcC18wT9sf17vZnz0KV4IvTWLp8KRGvyFSjizXUfAt9eY4OVEnC990 MyPOv1KHg4KHFlpbHB03P1S9N2P1LbrlgTvWI5yX6bNZycW5JHeX/1Qe1yHHjTFQXP7i VbhOPXfM8vQpv6BipetpOUmDW4hTA7FbjyRpdu8HL8HTyWuRmyBsVDt8qkzNjwqm4vDG qxoQ1T6Vts13wCV9+wNdHZ06MmSlvf3680sOuIzC1fFCRvMCkHYl8pdURdRjdHuS4ht1 3fhw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=WKg+XPP0; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id h2sor46944pgv.32.2019.04.17.14.54.59 for (Google Transport Security); Wed, 17 Apr 2019 14:54:59 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=WKg+XPP0; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2qTSnTK6DrJDXJXnCNRZMj0fJcJ1BWzKc8LF04T53hg=; b=WKg+XPP0EkFxCA1R9LDtszhfSNKpN3xu9VXo0hR8NaV9Hg5/d+F6vMYurdpDIaS8o3 TnHzRETH1byoyF/S8KLPzSPDTFfOUARS9AfsMbAl6obmJXjVtR/G3x4qzSqbSp6cgVlS XOYWUQAGr4Vnet0g5ylHUUb3TS4NtLDVkXaaHd7uWJ/FWSXtWWvzB6RRdXUI3gEubnoy w4jUCf2AIP17EQqYsyPO0YzK4JOYaH2lvR9ND7qX6w3Qb7G3+1ttB70ZY+kksY3vRr0+ 8Dp54cYDpYnwD6wjegW9aQBZXV/EGqqOIDQlW7R4aeIKjZfnke6yaJOyL+QjN6x2Zppc ecpg== X-Google-Smtp-Source: APXvYqz3rrrk7qPfYfvvzhHgMyqQGYm2yp5s+aBDKawTX9Uq/I5/BhGOZ5a1DLbYOKMEuo1Pd9uiJA== X-Received: by 2002:a63:4548:: with SMTP id u8mr2043991pgk.435.1555538099326; Wed, 17 Apr 2019 14:54:59 -0700 (PDT) Received: from tower.thefacebook.com ([2620:10d:c090:200::5597]) by smtp.gmail.com with ESMTPSA id x6sm209024pfb.171.2019.04.17.14.54.57 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 17 Apr 2019 14:54:58 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@fb.com, Johannes Weiner , Michal Hocko , Rik van Riel , david@fromorbit.com, Christoph Lameter , Pekka Enberg , Vladimir Davydov , cgroups@vger.kernel.org, Roman Gushchin Subject: [PATCH 4/5] mm: rework non-root kmem_cache lifecycle management Date: Wed, 17 Apr 2019 14:54:33 -0700 Message-Id: <20190417215434.25897-5-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190417215434.25897-1-guro@fb.com> References: <20190417215434.25897-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This commit makes several important changes in the lifecycle of a non-root kmem_cache, which also affect the lifecycle of a memory cgroup. Currently each charged slab page has a page->mem_cgroup pointer to the memory cgroup and holds a reference to it. Kmem_caches are held by the cgroup. On offlining empty kmem_caches are freed, all other are freed on cgroup release. So the current scheme can be illustrated as: page->mem_cgroup->kmem_cache. To implement the slab memory reparenting we need to invert the scheme into: page->kmem_cache->mem_cgroup. Let's make every page to hold a reference to the kmem_cache (we already have a stable pointer), and make kmem_caches to hold a single reference to the memory cgroup. To make this possible we need to introduce a new refcounter for non-root kmem_caches. It's atomic for now, but can be easily converted to a percpu counter, had we any performance penalty*. The initial value is set to 1, and it's decremented on deactivation, so we never shutdown an active cache. To shutdown non-active empty kmem_caches, let's reuse the infrastructure of the RCU-delayed work queue, used previously for the deactivation. After the generalization, it's perfectly suited for our needs. Since now we can release a kmem_cache at any moment after the deactivation, let's call sysfs_slab_remove() only from the shutdown path. It makes deactivation path simpler. Because we don't set the page->mem_cgroup pointer, we need to change the way how memcg-level stats is working for slab pages. We can't use mod_lruvec_page_state() helpers anymore, so switch over to mod_lruvec_state(). * I used the following simple approach to test the performance (stolen from another patchset by T. Harding): time find / -name fname-no-exist echo 2 > /proc/sys/vm/drop_caches repeat several times Results (I've chosen best results in several runs): orig patched real 0m0.712s 0m0.690s user 0m0.104s 0m0.101s sys 0m0.346s 0m0.340s real 0m0.728s 0m0.723s user 0m0.114s 0m0.115s sys 0m0.342s 0m0.338s real 0m0.685s 0m0.767s user 0m0.118s 0m0.114s sys 0m0.343s 0m0.336s So it looks like the difference is not noticeable in this test. Signed-off-by: Roman Gushchin --- include/linux/slab.h | 2 +- mm/memcontrol.c | 9 ++++---- mm/slab.c | 15 +----------- mm/slab.h | 54 +++++++++++++++++++++++++++++++++++++++++--- mm/slab_common.c | 51 +++++++++++++++++------------------------ mm/slub.c | 22 +----------------- 6 files changed, 79 insertions(+), 74 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 47923c173f30..4daaade76c63 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -152,7 +152,6 @@ int kmem_cache_shrink(struct kmem_cache *); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); void memcg_deactivate_kmem_caches(struct mem_cgroup *); -void memcg_destroy_kmem_caches(struct mem_cgroup *); /* * Please use this macro to create slab caches. Simply specify the @@ -641,6 +640,7 @@ struct memcg_cache_params { struct mem_cgroup *memcg; struct list_head children_node; struct list_head kmem_caches_node; + atomic_long_t refcnt; void (*work_fn)(struct kmem_cache *); union { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b2c39f187cbb..87c06e342e05 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2719,9 +2719,6 @@ int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, cancel_charge(memcg, nr_pages); return -ENOMEM; } - - page->mem_cgroup = memcg; - return 0; } @@ -2744,8 +2741,10 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order) memcg = get_mem_cgroup_from_current(); if (!mem_cgroup_is_root(memcg)) { ret = __memcg_kmem_charge_memcg(page, gfp, order, memcg); - if (!ret) + if (!ret) { + page->mem_cgroup = memcg; __SetPageKmemcg(page); + } } css_put(&memcg->css); return ret; @@ -3238,7 +3237,7 @@ static void memcg_free_kmem(struct mem_cgroup *memcg) memcg_offline_kmem(memcg); if (memcg->kmem_state == KMEM_ALLOCATED) { - memcg_destroy_kmem_caches(memcg); + WARN_ON(!list_empty(&memcg->kmem_caches)); static_branch_dec(&memcg_kmem_enabled_key); WARN_ON(page_counter_read(&memcg->kmem)); } diff --git a/mm/slab.c b/mm/slab.c index 14466a73d057..171b21ca617f 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1389,7 +1389,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, int nodeid) { struct page *page; - int nr_pages; flags |= cachep->allocflags; @@ -1404,12 +1403,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, return NULL; } - nr_pages = (1 << cachep->gfporder); - if (cachep->flags & SLAB_RECLAIM_ACCOUNT) - mod_lruvec_page_state(page, NR_SLAB_RECLAIMABLE, nr_pages); - else - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, nr_pages); - __SetPageSlab(page); /* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks() && page_is_pfmemalloc(page)) @@ -1424,12 +1417,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, static void kmem_freepages(struct kmem_cache *cachep, struct page *page) { int order = cachep->gfporder; - unsigned long nr_freed = (1 << order); - - if (cachep->flags & SLAB_RECLAIM_ACCOUNT) - mod_lruvec_page_state(page, NR_SLAB_RECLAIMABLE, -nr_freed); - else - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, -nr_freed); BUG_ON(!PageSlab(page)); __ClearPageSlabPfmemalloc(page); @@ -1438,7 +1425,7 @@ static void kmem_freepages(struct kmem_cache *cachep, struct page *page) page->mapping = NULL; if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += nr_freed; + current->reclaim_state->reclaimed_slab += 1 << order; memcg_uncharge_slab(page, order, cachep); __free_pages(page, order); } diff --git a/mm/slab.h b/mm/slab.h index 4a261c97c138..1f49945f5c1d 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -173,7 +173,9 @@ void __kmem_cache_release(struct kmem_cache *); int __kmem_cache_shrink(struct kmem_cache *); void __kmemcg_cache_deactivate(struct kmem_cache *s); void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s); +void kmemcg_cache_shutdown(struct kmem_cache *s); void slab_kmem_cache_release(struct kmem_cache *); +void kmemcg_queue_cache_shutdown(struct kmem_cache *s); struct seq_file; struct file; @@ -274,19 +276,65 @@ static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) return s->memcg_params.root_cache; } +static __always_inline void kmemcg_cache_get_many(struct kmem_cache *s, long n) +{ + atomic_long_add(n, &s->memcg_params.refcnt); +} + +static __always_inline void kmemcg_cache_put_many(struct kmem_cache *s, long n) +{ + if (atomic_long_sub_and_test(n, &s->memcg_params.refcnt)) + kmemcg_queue_cache_shutdown(s); +} + static __always_inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { - if (is_root_cache(s)) + int idx = (s->flags & SLAB_RECLAIM_ACCOUNT) ? + NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE; + struct mem_cgroup *memcg; + struct lruvec *lruvec; + int ret; + + if (is_root_cache(s)) { + mod_node_page_state(page_pgdat(page), idx, 1 << order); return 0; - return memcg_kmem_charge_memcg(page, gfp, order, s->memcg_params.memcg); + } + + memcg = s->memcg_params.memcg; + ret = memcg_kmem_charge_memcg(page, gfp, order, memcg); + if (!ret) { + lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); + mod_lruvec_state(lruvec, idx, 1 << order); + + /* transer try_charge() page references to kmem_cache */ + kmemcg_cache_get_many(s, 1 << order); + css_put_many(&memcg->css, 1 << order); + } + + return 0; } static __always_inline void memcg_uncharge_slab(struct page *page, int order, struct kmem_cache *s) { - memcg_kmem_uncharge(page, order); + int idx = (s->flags & SLAB_RECLAIM_ACCOUNT) ? + NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE; + struct mem_cgroup *memcg; + struct lruvec *lruvec; + + if (is_root_cache(s)) { + mod_node_page_state(page_pgdat(page), idx, -(1 << order)); + return; + } + + memcg = s->memcg_params.memcg; + lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); + mod_lruvec_state(lruvec, idx, -(1 << order)); + memcg_kmem_uncharge_memcg(page, order, memcg); + + kmemcg_cache_put_many(s, 1 << order); } extern void slab_init_memcg_params(struct kmem_cache *); diff --git a/mm/slab_common.c b/mm/slab_common.c index 4e5b4292a763..3fdd02979a1c 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -148,6 +148,7 @@ static int init_memcg_params(struct kmem_cache *s, s->memcg_params.root_cache = root_cache; INIT_LIST_HEAD(&s->memcg_params.children_node); INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node); + atomic_long_set(&s->memcg_params.refcnt, 1); return 0; } @@ -225,6 +226,7 @@ void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) if (is_root_cache(s)) { list_add(&s->root_caches_node, &slab_root_caches); } else { + css_get(&memcg->css); s->memcg_params.memcg = memcg; list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); @@ -240,6 +242,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) } else { list_del(&s->memcg_params.children_node); list_del(&s->memcg_params.kmem_caches_node); + css_put(&s->memcg_params.memcg->css); } } #else @@ -703,21 +706,19 @@ static void kmemcg_after_rcu_workfn(struct work_struct *work) s->memcg_params.work_fn(s); s->memcg_params.work_fn = NULL; + kmemcg_cache_put_many(s, 1); mutex_unlock(&slab_mutex); put_online_mems(); put_online_cpus(); - - /* done, put the ref from slab_deactivate_memcg_cache_rcu_sched() */ - css_put(&s->memcg_params.memcg->css); } /* * We need to grab blocking locks. Bounce to ->work. The * work item shares the space with the RCU head and can't be - * initialized eariler. -*/ + * initialized earlier. + */ static void kmemcg_schedule_work_after_rcu(struct rcu_head *head) { struct kmem_cache *s = container_of(head, struct kmem_cache, @@ -727,6 +728,21 @@ static void kmemcg_schedule_work_after_rcu(struct rcu_head *head) queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); } +static void kmemcg_cache_shutdown_after_rcu(struct kmem_cache *s) +{ + WARN_ON(shutdown_cache(s)); +} + +void kmemcg_queue_cache_shutdown(struct kmem_cache *s) +{ + if (s->memcg_params.root_cache->memcg_params.dying) + return; + + WARN_ON(s->memcg_params.work_fn); + s->memcg_params.work_fn = kmemcg_cache_shutdown_after_rcu; + call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); +} + static void kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { __kmemcg_cache_deactivate_after_rcu(s); @@ -739,9 +755,6 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s) if (s->memcg_params.root_cache->memcg_params.dying) return; - /* pin memcg so that @s doesn't get destroyed in the middle */ - css_get(&s->memcg_params.memcg->css); - WARN_ON_ONCE(s->memcg_params.work_fn); s->memcg_params.work_fn = kmemcg_cache_deactivate_after_rcu; call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); @@ -775,28 +788,6 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) put_online_cpus(); } -void memcg_destroy_kmem_caches(struct mem_cgroup *memcg) -{ - struct kmem_cache *s, *s2; - - get_online_cpus(); - get_online_mems(); - - mutex_lock(&slab_mutex); - list_for_each_entry_safe(s, s2, &memcg->kmem_caches, - memcg_params.kmem_caches_node) { - /* - * The cgroup is about to be freed and therefore has no charges - * left. Hence, all its caches must be empty by now. - */ - BUG_ON(shutdown_cache(s)); - } - mutex_unlock(&slab_mutex); - - put_online_mems(); - put_online_cpus(); -} - static int shutdown_memcg_caches(struct kmem_cache *s) { struct memcg_cache_array *arr; diff --git a/mm/slub.c b/mm/slub.c index 195f61785c7d..a28b3b3abf29 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1692,11 +1692,6 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) if (!page) return NULL; - mod_lruvec_page_state(page, - (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE, - 1 << oo_order(oo)); - inc_slabs_node(s, page_to_nid(page), page->objects); return page; @@ -1730,11 +1725,6 @@ static void __free_slab(struct kmem_cache *s, struct page *page) check_object(s, page, p, SLUB_RED_INACTIVE); } - mod_lruvec_page_state(page, - (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE, - -pages); - __ClearPageSlabPfmemalloc(page); __ClearPageSlab(page); @@ -4037,18 +4027,8 @@ void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { /* * Called with all the locks held after a sched RCU grace period. - * Even if @s becomes empty after shrinking, we can't know that @s - * doesn't have allocations already in-flight and thus can't - * destroy @s until the associated memcg is released. - * - * However, let's remove the sysfs files for empty caches here. - * Each cache has a lot of interface files which aren't - * particularly useful for empty draining caches; otherwise, we can - * easily end up with millions of unnecessary sysfs files on - * systems which have a lot of memory and transient cgroups. */ - if (!__kmem_cache_shrink(s)) - sysfs_slab_remove(s); + __kmem_cache_shrink(s); } void __kmemcg_cache_deactivate(struct kmem_cache *s) From patchwork Wed Apr 17 21:54:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10906313 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 89D141390 for ; Wed, 17 Apr 2019 21:55:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7432E26255 for ; Wed, 17 Apr 2019 21:55:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 680812880B; Wed, 17 Apr 2019 21:55:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D450126255 for ; Wed, 17 Apr 2019 21:55:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E55896B000E; Wed, 17 Apr 2019 17:55:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D8FA36B0010; Wed, 17 Apr 2019 17:55:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B1DF06B0269; Wed, 17 Apr 2019 17:55:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 600796B0010 for ; Wed, 17 Apr 2019 17:55:02 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id e128so13226pfc.22 for ; Wed, 17 Apr 2019 14:55:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=NUinwwoqQqqlnaj+hfWxAVvkl/hYr3H9Nu7bos7N4pE=; b=iEBCn/8jrLgU+CpGlp1bv05f8/qrt01GhXQ71N3fXCI+QN3GsNZ+jmqiIggLPRn37W vUa+z0bPWS0VnlwCfIvGZ8kdyPsrWyp+3siWz+9jmYawbLMSA52yuenjNXjHnoKMpl7S n3UYEkNZk+h8bElQjH8Zkl2GFug0aR9Nsp4zQs8/+KmkaGmiyKje1LCzn7bYKiVfeZGX 5q6LEXzcUDZAoP6foSX0Hv3yeHpP3pUwpw4LCqvhfFTtEDZ+u3Bse5wAOirTN9cX4Ic7 azrqqDNDk1LvJOkMqT8KavIYlsqvSnRICyYsMm7dmF/u9MM9WX5hfYCl7yFajes29Wm6 tttw== X-Gm-Message-State: APjAAAX58t4UBEIWjkURf5fh5NscA850udNf6prWsLgad3UdxtoULLCP 52hc0cy/cQqwTtGKWkSG22xLLqZqyZ1alql0zazE5AOH2xVA0ydPfqUxVIRDeok9oWaF0OChJcm JUpv6NWVfmhWQxBtmZHyeDTvh1qPA0biHG49PsoAMliOJG/c24ARuhJckOULC8gNRTA== X-Received: by 2002:a63:2c09:: with SMTP id s9mr81422641pgs.411.1555538101976; Wed, 17 Apr 2019 14:55:01 -0700 (PDT) X-Received: by 2002:a63:2c09:: with SMTP id s9mr81422595pgs.411.1555538101201; Wed, 17 Apr 2019 14:55:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555538101; cv=none; d=google.com; s=arc-20160816; b=B/ZBr0m7QmKpXYfZQyE8uHSrQLgNHz31kjUGDyVHfPJd6w/CNaImOkDvU5UawcO3d0 9GhLpNXrGj49yFL/6g6FJu3m581RTAObWux/G7dSPnVmsgIg520j7GSVAxgFp0XXAnNL z8RFsaYTxD8dx7OZrHLf01CseXrJsmgVKnP5VJxhfw9D/EZ9xcDVOAqJroPSYsf9Qcm5 oSjlkno/DB5zd2okO3Xuzr59iIkCQER1ABmTcNr7yTzki0pYS+h1taep1ovJ5S+CAveq M7af7gCh+ANaon1bLILK+QtM1eTt1pN/QyCngTHhqaE6GRsEU7Vus1a7i/3evpwqvgIW 1GVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=NUinwwoqQqqlnaj+hfWxAVvkl/hYr3H9Nu7bos7N4pE=; b=rMrDf1V7LEbpxJe//F2+RWJjpUM0g6HP6ZYOpN50LfUCTuXOHvjbWRbJXwijcVTtcM OvPwAOogUsmMVCom3hAVu+IpBniAqju91nrv7EtJtciiu4xCpmNwR+s0y3k661JVV0d5 wvvayGDemPmu1WGc4A608XwLYv0kHHhDcsPpXyMKFjaETbuVxjBjEgFjijuczGmnurVX /5ACU2G8rH1IeSQzYVoiGY7iTm01Ii1XTAahbqechbWiXYvFZn3a1mjvaGkMcLiWcbfz +2qYQv6f5GA/VwsYWawgI2qzD4L9g4YSqI9ZYBAqSRwQa7KJvP7os9zObWE02c8cIG7C aZsQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=TnQkXWhr; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id k65sor40412pge.41.2019.04.17.14.55.01 for (Google Transport Security); Wed, 17 Apr 2019 14:55:01 -0700 (PDT) Received-SPF: pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=TnQkXWhr; spf=pass (google.com: domain of guroan@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=guroan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NUinwwoqQqqlnaj+hfWxAVvkl/hYr3H9Nu7bos7N4pE=; b=TnQkXWhriERaZfgxnd6oMoknEpldpZY4+ddqcjT9GjKwVe5mHq0dOSP71ukwgXypvq gzYvkoj5vWdSnXSRCCXCNagTc11B45enD9TrF5IxKI/NmjZLJcINdRJXe//lfyxQlxpA g1QIAY14d3wugoG2GI6i9vqpSBFgidozl3m/zEJH2ZbtqH1G7kNtgeATO01BFvh1iAug txs59zhsh/UB2k+oPSuUnLxiP6LHaLRttbMVvFmVYUbYIx18owHtFaviC8Mb2QvyP44e D/8viFGHj8Ezgc3m9xb9MnAOcO2tPQYTC2AC7tt88lJ76EfU3h1K7a0HOPPF2HSCG/n1 td9Q== X-Google-Smtp-Source: APXvYqxo8fFyf4UjadiGuY16LsgpI7/aYDRL0AuqAX0+SEjmKOud84TnNLhLUufhyWE4dCeVlf+U9g== X-Received: by 2002:a63:6a44:: with SMTP id f65mr50910693pgc.354.1555538100889; Wed, 17 Apr 2019 14:55:00 -0700 (PDT) Received: from tower.thefacebook.com ([2620:10d:c090:200::5597]) by smtp.gmail.com with ESMTPSA id x6sm209024pfb.171.2019.04.17.14.54.59 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 17 Apr 2019 14:55:00 -0700 (PDT) From: Roman Gushchin X-Google-Original-From: Roman Gushchin To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@fb.com, Johannes Weiner , Michal Hocko , Rik van Riel , david@fromorbit.com, Christoph Lameter , Pekka Enberg , Vladimir Davydov , cgroups@vger.kernel.org, Roman Gushchin Subject: [PATCH 5/5] mm: reparent slab memory on cgroup removal Date: Wed, 17 Apr 2019 14:54:34 -0700 Message-Id: <20190417215434.25897-6-guro@fb.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190417215434.25897-1-guro@fb.com> References: <20190417215434.25897-1-guro@fb.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Let's reparent memcg slab memory on memcg offlining. This allows us to release the memory cgroup without waiting for the last outstanding kernel object (e.g. dentry used by another application). So instead of reparenting all accounted slab pages, let's do reparent a relatively small amount of kmem_caches. Reparenting is performed as the last part of the deactivation process, so it's guaranteed that all kmem_caches are not active at this moment. Since the parent cgroup is already charged, everything we need to do is to move the kmem_cache to the parent's kmem_caches list, swap the memcg pointer, bump parent's css refcounter and drop the cgroup's refcounter. Quite simple. We can't race with the slab allocation path, and if we race with deallocation path, it's not a big deal: parent's charge and slab stats are always correct*, and we don't care anymore about the child usage and stats. The child cgroup is already offline, so we don't use or show it anywhere. * please, look at the comment in kmemcg_cache_deactivate_after_rcu() for some additional details Signed-off-by: Roman Gushchin --- mm/memcontrol.c | 4 +++- mm/slab.h | 4 +++- mm/slab_common.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 34 insertions(+), 2 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 87c06e342e05..2f61d13df0c4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3239,7 +3239,6 @@ static void memcg_free_kmem(struct mem_cgroup *memcg) if (memcg->kmem_state == KMEM_ALLOCATED) { WARN_ON(!list_empty(&memcg->kmem_caches)); static_branch_dec(&memcg_kmem_enabled_key); - WARN_ON(page_counter_read(&memcg->kmem)); } } #else @@ -4651,6 +4650,9 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) /* The following stuff does not apply to the root */ if (!parent) { +#ifdef CONFIG_MEMCG_KMEM + INIT_LIST_HEAD(&memcg->kmem_caches); +#endif root_mem_cgroup = memcg; return &memcg->css; } diff --git a/mm/slab.h b/mm/slab.h index 1f49945f5c1d..be4f04ef65f9 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -329,10 +329,12 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, return; } - memcg = s->memcg_params.memcg; + rcu_read_lock(); + memcg = READ_ONCE(s->memcg_params.memcg); lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); mod_lruvec_state(lruvec, idx, -(1 << order)); memcg_kmem_uncharge_memcg(page, order, memcg); + rcu_read_unlock(); kmemcg_cache_put_many(s, 1 << order); } diff --git a/mm/slab_common.c b/mm/slab_common.c index 3fdd02979a1c..fc2e86de402f 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -745,7 +745,35 @@ void kmemcg_queue_cache_shutdown(struct kmem_cache *s) static void kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { + struct mem_cgroup *memcg, *parent; + __kmemcg_cache_deactivate_after_rcu(s); + + memcg = s->memcg_params.memcg; + parent = parent_mem_cgroup(memcg); + if (!parent) + parent = root_mem_cgroup; + + if (memcg == parent) + return; + + /* + * Let's reparent the kmem_cache. It's already deactivated, so we + * can't race with memcg_charge_slab(). We still can race with + * memcg_uncharge_slab(), but it's not a problem. The parent cgroup + * is already charged, so it's ok to uncharge either the parent cgroup + * directly, either recursively. + * The same is true for recursive vmstats. Local vmstats are not use + * anywhere, except count_shadow_nodes(). But reparenting will not + * cahnge anything for count_shadow_nodes(): on memcg removal + * shrinker lists are reparented, so it always returns SHRINK_EMPTY + * for non-leaf dead memcgs. For the parent memcgs local slab stats + * are always 0 now, so reparenting will not change anything. + */ + list_move(&s->memcg_params.kmem_caches_node, &parent->kmem_caches); + s->memcg_params.memcg = parent; + css_get(&parent->css); + css_put(&memcg->css); } static void kmemcg_cache_deactivate(struct kmem_cache *s)