From patchwork Wed May 8 20:24:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10936479 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C0823933 for ; Wed, 8 May 2019 20:31:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B0C992880D for ; Wed, 8 May 2019 20:31:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A4D7C288CE; Wed, 8 May 2019 20:31:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A71552880D for ; Wed, 8 May 2019 20:31:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 726486B000C; Wed, 8 May 2019 16:30:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6AEBD6B000D; Wed, 8 May 2019 16:30:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 54EAA6B000E; Wed, 8 May 2019 16:30:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f70.google.com (mail-yw1-f70.google.com [209.85.161.70]) by kanga.kvack.org (Postfix) with ESMTP id 2B6A76B000C for ; Wed, 8 May 2019 16:30:58 -0400 (EDT) Received: by mail-yw1-f70.google.com with SMTP id 11so40147805ywt.12 for ; Wed, 08 May 2019 13:30:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=ncEIzgzKw3O11kTP5Oohay5sLxt9wsLVCWoFPKhHn4M=; b=Wx/jRW4eF9mOPXl8Xmzs1yK8fWuoWcr7EPEGj6/Vu0KtL5smYBTX+MibGA/vl0aIrU /yhN5GbKU7M4touMBO/6vgVQv4/NExWN8Eik5LzZq4iYTGITMGf45cP2rHgQ1Qx/xp05 4IuElc34y5qZnztrO0GSeoEb4JpriZyDP/JY/giBZcSxJ1xuGPJV/21X0BrHoB25hYuv 8OgMuX9tad1JFct8Bx7qdCfOZMdw9Yu5QdXQjApXL3HC9F4QSiWmJNX38bIFRGR1seSE mKYl7kdQI+To9KnUTuyWjjGID097QrnsGA8XlMK5BI8wy0AI6w1FoFVR84KYmHFuFDld QtHA== X-Gm-Message-State: APjAAAV+dw9tdYSLfMbG2x+SdI68PbwLo8ZwLJ/toSeJz6CjXGGU6I7o MSxILoX250Dolp2nQddQ8e+exr2W0POrxmqzli+BRcVAreR9Ngphzp7dq8TBL2gsZFXpU5QGVSH D0eBEPKzwRD80RB4Y9MVdvftKK/yNj5NVUYvGPcz8YYNpTiez3gQw1PSfato/P2CpPQ== X-Received: by 2002:a25:268f:: with SMTP id m137mr25810921ybm.427.1557347457900; Wed, 08 May 2019 13:30:57 -0700 (PDT) X-Google-Smtp-Source: APXvYqz2DGUXgk6MFLZddRiT5Sd+1LnHV5bSTxH2y/QswIZeolWXLDWw08ebuUFsWveoumnYKJGS X-Received: by 2002:a25:268f:: with SMTP id m137mr25810879ybm.427.1557347457155; Wed, 08 May 2019 13:30:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557347457; cv=none; d=google.com; s=arc-20160816; b=GiXemcxYzMtejywxkjT1+N4hu5x18XTLBnRQdELu91CqPfycvoGzSb+HsraRylCazq lhwbdRa7Kzpi72Db5r0BzV8OTV4xT3porykzyvbx+3htn7on6wRevuuW9ggWt7Rsige6 bHuCinoYPNGaBUAMuL5LTYSV0HZ4s3xAIjm2zlyUCDW1/xiBy0D1Ngtua/+KWd4L3CXs hiG0pEwkZ2vxw7FCjJ0woccHAh2Gj8a559gUVdBO+KOnzfDJodC50T1ZOoMwZfOA/OQe bHNDjo7wdOQdcmCLZW8ZzrN2W5Xgeb3iyQ5sHIN9k1FMxg6iyabL7b7m7xIGysBAX+pe BCLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=ncEIzgzKw3O11kTP5Oohay5sLxt9wsLVCWoFPKhHn4M=; b=oce85ag7Aw1dRIjHivZI0hu247S0ntTI1uTGDBbHixe6890kDz0QydeaOmox10EbKz VwPLznDIc+B2f/WSzZ1tWNALGEl25w0o/wKB/XeE8cEKrZn5unHuSD2djznnjf7I9tM0 goB86wIiCI03zFOQ5hkM68f6R3ap7Pnac7Aee1u4JKU+pIJmtBdHTAMag1PtHBsdZ8Wl lbDBv5LQHJ4lQs6FYpDIe4LAai1FF72Rlw/4IriTfHRhsmrclGaK36HP+2hSUmgYx+wL 7oUkbUtrS9mJYatjIlpIZYt7uhWlsa1Y+RPi7SXE7a+aFUaiRKTKo7WB/gCWl9P5izGP UB/Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="k9Ah/ffe"; spf=pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=0031b2e447=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id v18si21659ywv.29.2019.05.08.13.30.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 May 2019 13:30:57 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="k9Ah/ffe"; spf=pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=0031b2e447=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x48KDoQJ029807 for ; Wed, 8 May 2019 13:30:56 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=ncEIzgzKw3O11kTP5Oohay5sLxt9wsLVCWoFPKhHn4M=; b=k9Ah/ffeMZAPhZxcCiKcPI2JxLGC6QAf+VitJG/8yft11cQy61xAMYXZRzOBR2yjRRxf i+oSOhGXZ18+qxkJcfF2cqU4a/CvTzEC98NMy8wcVYMkZZhZqJ8M8jObRlGDJkEuDhnZ H9xv3JmO4qkJXDN8EQXQZnE5O41JsGGUtXM= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2sc25590kt-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 08 May 2019 13:30:56 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 8 May 2019 13:30:49 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 0409111CDBCF2; Wed, 8 May 2019 13:25:00 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Shakeel Butt CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Christoph Lameter , Vladimir Davydov , , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v3 1/7] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache() Date: Wed, 8 May 2019 13:24:52 -0700 Message-ID: <20190508202458.550808-2-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190508202458.550808-1-guro@fb.com> References: <20190508202458.550808-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-08_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905080124 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Initialize kmem_cache->memcg_params.memcg pointer in memcg_link_cache() rather than in init_memcg_params(). Once kmem_cache will hold a reference to the memory cgroup, it will simplify the refcounting. For non-root kmem_caches memcg_link_cache() is always called before the kmem_cache becomes visible to a user, so it's safe. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt --- mm/slab.c | 2 +- mm/slab.h | 5 +++-- mm/slab_common.c | 14 +++++++------- mm/slub.c | 2 +- 4 files changed, 12 insertions(+), 11 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 2915d912e89a..f6eff59e018e 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1268,7 +1268,7 @@ void __init kmem_cache_init(void) nr_node_ids * sizeof(struct kmem_cache_node *), SLAB_HWCACHE_ALIGN, 0, 0); list_add(&kmem_cache->list, &slab_caches); - memcg_link_cache(kmem_cache); + memcg_link_cache(kmem_cache, NULL); slab_state = PARTIAL; /* diff --git a/mm/slab.h b/mm/slab.h index 43ac818b8592..6a562ca72bca 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -289,7 +289,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, } extern void slab_init_memcg_params(struct kmem_cache *); -extern void memcg_link_cache(struct kmem_cache *s); +extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, void (*deact_fn)(struct kmem_cache *)); @@ -344,7 +344,8 @@ static inline void slab_init_memcg_params(struct kmem_cache *s) { } -static inline void memcg_link_cache(struct kmem_cache *s) +static inline void memcg_link_cache(struct kmem_cache *s, + struct mem_cgroup *memcg) { } diff --git a/mm/slab_common.c b/mm/slab_common.c index 58251ba63e4a..6e00bdf8618d 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -140,13 +140,12 @@ void slab_init_memcg_params(struct kmem_cache *s) } static int init_memcg_params(struct kmem_cache *s, - struct mem_cgroup *memcg, struct kmem_cache *root_cache) + struct kmem_cache *root_cache) { struct memcg_cache_array *arr; if (root_cache) { s->memcg_params.root_cache = root_cache; - s->memcg_params.memcg = memcg; INIT_LIST_HEAD(&s->memcg_params.children_node); INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node); return 0; @@ -221,11 +220,12 @@ int memcg_update_all_caches(int num_memcgs) return ret; } -void memcg_link_cache(struct kmem_cache *s) +void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) { if (is_root_cache(s)) { list_add(&s->root_caches_node, &slab_root_caches); } else { + s->memcg_params.memcg = memcg; list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); list_add(&s->memcg_params.kmem_caches_node, @@ -244,7 +244,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) } #else static inline int init_memcg_params(struct kmem_cache *s, - struct mem_cgroup *memcg, struct kmem_cache *root_cache) + struct kmem_cache *root_cache) { return 0; } @@ -384,7 +384,7 @@ static struct kmem_cache *create_cache(const char *name, s->useroffset = useroffset; s->usersize = usersize; - err = init_memcg_params(s, memcg, root_cache); + err = init_memcg_params(s, root_cache); if (err) goto out_free_cache; @@ -394,7 +394,7 @@ static struct kmem_cache *create_cache(const char *name, s->refcount = 1; list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, memcg); out: if (err) return ERR_PTR(err); @@ -997,7 +997,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, create_boot_cache(s, name, size, flags, useroffset, usersize); list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, NULL); s->refcount = 1; return s; } diff --git a/mm/slub.c b/mm/slub.c index 5b2e364102e1..16f7e4f5a141 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4219,7 +4219,7 @@ static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache) } slab_init_memcg_params(s); list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, NULL); return s; } From patchwork Wed May 8 20:24:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10936475 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2D5B492A for ; Wed, 8 May 2019 20:30:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1AD852880D for ; Wed, 8 May 2019 20:30:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0C45F288CE; Wed, 8 May 2019 20:30:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2EC492880D for ; Wed, 8 May 2019 20:30:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B0D986B0008; Wed, 8 May 2019 16:30:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A7D526B000C; Wed, 8 May 2019 16:30:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E52D6B000D; Wed, 8 May 2019 16:30:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 337A96B0008 for ; Wed, 8 May 2019 16:30:52 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id q18so60756pll.16 for ; Wed, 08 May 2019 13:30:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=PF7epo97MxShKnhAVmvE6ZPniKssoQry9k+G7xvR+Wg=; b=rayIYHoZJ4Yi37xeW5JT8uYKDyL3w9CoIIJIjPGye1MUCfAP9UmHcx/zpNTV6BNJSj XNe0GpieaRyp4F68z2sltgHTwSGZ6mGHeVbdaHZOy3C9oX+zdZx6PJitCifWqYZi2PuM dRtguB94f7e4GL1RdH6YTefIWYq1+WDmoS+Y60XFOCTK4BcoU4+aVYLWleTkPOi2DYIu RNA8RGWNMUsj5u49Un41TgS/ncVMPd9/mNap+VHNzm+1A+WPzss6JHCPAHFlNALRdVrK 0eLfYSXxEQ8x7q8ahf7OH+6eoPLLLKrHTZFhc9xHo+PZ+SNemLfm5mpgg9IieH6tr67R 34Ow== X-Gm-Message-State: APjAAAX+5wxNzt/v3gOtcZNLJzufzZkivGDZUJIMPGkuQd33Cx+gdfTs zjV9WubeFZwOeJ4NqMJrdiQKLWlUo+DAWzrcciiU7x2bFvLoIJOWyyAOEOXJq+MSccCc/nJazXM /gKWebEAjlmrPliyO6hKM8aLcqQ1522l0datXMhI7m3NOkMA1tWeFbaBN12FR9qVzUQ== X-Received: by 2002:a65:60c7:: with SMTP id r7mr199490pgv.22.1557347451750; Wed, 08 May 2019 13:30:51 -0700 (PDT) X-Google-Smtp-Source: APXvYqx3wGEzssrZ96w1tIaMlP7xr7cVfCuw+fwTnAVrUvfxQuO7acGpLgg8MVrnzHbAEbjqVXH8 X-Received: by 2002:a65:60c7:: with SMTP id r7mr199376pgv.22.1557347450756; Wed, 08 May 2019 13:30:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557347450; cv=none; d=google.com; s=arc-20160816; b=b7fyG6SWNMuykgIrQdEA621CeU9LP6SGYdDdz6LqMRSc+Oh1Scgs+cH6yrnpg/ln/p KdRuNMoIPn3Nwj8SRNdHkpat8WD16C9KKbBZ81xL5u3taAmdQw6mZMEK6RlVPAse7xtl FUxIonLutqDG2rCQCQHwEBEttdw8e2HGvBinc0paW80DgH4gvX5qa561rSrIbr+qA40Y gOMPqPCssm/srXpgn+wFhwJ7NuBGTPxpmuoi272l6NbZ9eCLqiPdt7VzRH/cOnKqauCm IC9PJpjq2IpVIAj1HuSSyFU7plvgDch8aNfge7IoPUhkmtAe/FGkqk3qlgaGiyQ71+29 uiRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=PF7epo97MxShKnhAVmvE6ZPniKssoQry9k+G7xvR+Wg=; b=ar5vmHfPEgIOz2CurtHYdA24auuesdnJJ0pgvMKr8v3x/U0RYkQlbaznbO646JLpqb l5Yv6/ruyP3iSmo5fuZyLFXcyGiNNGrRM+Mki7NU/0MCTcOd/FEOIgZ4CWVXd9ff34L2 Oe6vdP+W6sLnUgONUlJaOMIWK32eYfo2VE3zBEvwfY4xA3QGWp5KPye806A9KtPJO3k7 nO3QWaFY8wMS5X1YMf3Eh2PqZ95xKJ1xB2RwpRTyBfvQ7SXSP4I0F5JONOrMtwW8r7Ha +Zq55e7/bbiTLmYSrYigQf2CRs9womBdMsPo1w7tREGCLE3cgchZTXvtqayqZT8IOppj g94w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=PdLT9fVP; spf=pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0031b2e447=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id t22si17118012pgg.292.2019.05.08.13.30.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 May 2019 13:30:50 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=PdLT9fVP; spf=pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0031b2e447=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x48KEfM6003391 for ; Wed, 8 May 2019 13:30:50 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=PF7epo97MxShKnhAVmvE6ZPniKssoQry9k+G7xvR+Wg=; b=PdLT9fVP79Ypex9FoewJmzx3IDUzS04bP65H3mEX5VyS0F11AcLsYBJCeetwwRY7ABGk tjn8ehydvbNMWE37KHnrBCPKXPamz9wMlBit2fds4Q1gtpdt1xnJYIO+6fY9hMruDGY3 3Kx9mjSAIe+hfmZmmmlHNnA4QgSf94ZiBnQ= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2sc2q3gv7c-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 08 May 2019 13:30:50 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::125) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Wed, 8 May 2019 13:30:48 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 0739611CDBCF4; Wed, 8 May 2019 13:25:00 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Shakeel Butt CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Christoph Lameter , Vladimir Davydov , , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v3 2/7] mm: generalize postponed non-root kmem_cache deactivation Date: Wed, 8 May 2019 13:24:53 -0700 Message-ID: <20190508202458.550808-3-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190508202458.550808-1-guro@fb.com> References: <20190508202458.550808-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-08_11:,, signatures=0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently SLUB uses a work scheduled after an RCU grace period to deactivate a non-root kmem_cache. This mechanism can be reused for kmem_caches reparenting, but requires some generalization. Let's decouple all infrastructure (rcu callback, work callback) from the SLUB-specific code, so it can be used with SLAB as well. Also, let's rename some functions to make the code look simpler. All SLAB/SLUB-specific functions start with "__". Remove "deact_" prefix from the corresponding struct fields. Here is the graph of a new calling scheme: kmemcg_cache_deactivate() __kmemcg_cache_deactivate() SLAB/SLUB-specific kmemcg_schedule_work_after_rcu() rcu kmemcg_after_rcu_workfn() work kmemcg_cache_deactivate_after_rcu() __kmemcg_cache_deactivate_after_rcu() SLAB/SLUB-specific instead of: __kmemcg_cache_deactivate() SLAB/SLUB-specific slab_deactivate_memcg_cache_rcu_sched() SLUB-only kmemcg_deactivate_rcufn SLUB-only, rcu kmemcg_deactivate_workfn SLUB-only, work kmemcg_cache_deact_after_rcu() SLUB-only Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt --- include/linux/slab.h | 6 ++--- mm/slab.c | 4 +++ mm/slab.h | 3 ++- mm/slab_common.c | 62 ++++++++++++++++++++------------------------ mm/slub.c | 8 +----- 5 files changed, 38 insertions(+), 45 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 9449b19c5f10..47923c173f30 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -642,10 +642,10 @@ struct memcg_cache_params { struct list_head children_node; struct list_head kmem_caches_node; - void (*deact_fn)(struct kmem_cache *); + void (*work_fn)(struct kmem_cache *); union { - struct rcu_head deact_rcu_head; - struct work_struct deact_work; + struct rcu_head rcu_head; + struct work_struct work; }; }; }; diff --git a/mm/slab.c b/mm/slab.c index f6eff59e018e..83000e46b870 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -2281,6 +2281,10 @@ void __kmemcg_cache_deactivate(struct kmem_cache *cachep) { __kmem_cache_shrink(cachep); } + +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) +{ +} #endif int __kmem_cache_shutdown(struct kmem_cache *cachep) diff --git a/mm/slab.h b/mm/slab.h index 6a562ca72bca..4a261c97c138 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -172,6 +172,7 @@ int __kmem_cache_shutdown(struct kmem_cache *); void __kmem_cache_release(struct kmem_cache *); int __kmem_cache_shrink(struct kmem_cache *); void __kmemcg_cache_deactivate(struct kmem_cache *s); +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s); void slab_kmem_cache_release(struct kmem_cache *); struct seq_file; @@ -291,7 +292,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, extern void slab_init_memcg_params(struct kmem_cache *); extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*deact_fn)(struct kmem_cache *)); + void (*work_fn)(struct kmem_cache *)); #else /* CONFIG_MEMCG_KMEM */ diff --git a/mm/slab_common.c b/mm/slab_common.c index 6e00bdf8618d..4e5b4292a763 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -691,17 +691,18 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg, put_online_cpus(); } -static void kmemcg_deactivate_workfn(struct work_struct *work) +static void kmemcg_after_rcu_workfn(struct work_struct *work) { struct kmem_cache *s = container_of(work, struct kmem_cache, - memcg_params.deact_work); + memcg_params.work); get_online_cpus(); get_online_mems(); mutex_lock(&slab_mutex); - s->memcg_params.deact_fn(s); + s->memcg_params.work_fn(s); + s->memcg_params.work_fn = NULL; mutex_unlock(&slab_mutex); @@ -712,37 +713,28 @@ static void kmemcg_deactivate_workfn(struct work_struct *work) css_put(&s->memcg_params.memcg->css); } -static void kmemcg_deactivate_rcufn(struct rcu_head *head) +/* + * We need to grab blocking locks. Bounce to ->work. The + * work item shares the space with the RCU head and can't be + * initialized eariler. +*/ +static void kmemcg_schedule_work_after_rcu(struct rcu_head *head) { struct kmem_cache *s = container_of(head, struct kmem_cache, - memcg_params.deact_rcu_head); + memcg_params.rcu_head); - /* - * We need to grab blocking locks. Bounce to ->deact_work. The - * work item shares the space with the RCU head and can't be - * initialized eariler. - */ - INIT_WORK(&s->memcg_params.deact_work, kmemcg_deactivate_workfn); - queue_work(memcg_kmem_cache_wq, &s->memcg_params.deact_work); + INIT_WORK(&s->memcg_params.work, kmemcg_after_rcu_workfn); + queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); } -/** - * slab_deactivate_memcg_cache_rcu_sched - schedule deactivation after a - * sched RCU grace period - * @s: target kmem_cache - * @deact_fn: deactivation function to call - * - * Schedule @deact_fn to be invoked with online cpus, mems and slab_mutex - * held after a sched RCU grace period. The slab is guaranteed to stay - * alive until @deact_fn is finished. This is to be used from - * __kmemcg_cache_deactivate(). - */ -void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*deact_fn)(struct kmem_cache *)) +static void kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { - if (WARN_ON_ONCE(is_root_cache(s)) || - WARN_ON_ONCE(s->memcg_params.deact_fn)) - return; + __kmemcg_cache_deactivate_after_rcu(s); +} + +static void kmemcg_cache_deactivate(struct kmem_cache *s) +{ + __kmemcg_cache_deactivate(s); if (s->memcg_params.root_cache->memcg_params.dying) return; @@ -750,8 +742,9 @@ void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, /* pin memcg so that @s doesn't get destroyed in the middle */ css_get(&s->memcg_params.memcg->css); - s->memcg_params.deact_fn = deact_fn; - call_rcu(&s->memcg_params.deact_rcu_head, kmemcg_deactivate_rcufn); + WARN_ON_ONCE(s->memcg_params.work_fn); + s->memcg_params.work_fn = kmemcg_cache_deactivate_after_rcu; + call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); } void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) @@ -773,7 +766,7 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) if (!c) continue; - __kmemcg_cache_deactivate(c); + kmemcg_cache_deactivate(c); arr->entries[idx] = NULL; } mutex_unlock(&slab_mutex); @@ -866,11 +859,12 @@ static void flush_memcg_workqueue(struct kmem_cache *s) mutex_unlock(&slab_mutex); /* - * SLUB deactivates the kmem_caches through call_rcu. Make + * SLAB and SLUB deactivate the kmem_caches through call_rcu. Make * sure all registered rcu callbacks have been invoked. */ - if (IS_ENABLED(CONFIG_SLUB)) - rcu_barrier(); +#ifndef CONFIG_SLOB + rcu_barrier(); +#endif /* * SLAB and SLUB create memcg kmem_caches through workqueue and SLUB diff --git a/mm/slub.c b/mm/slub.c index 16f7e4f5a141..43c34d54ad86 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4028,7 +4028,7 @@ int __kmem_cache_shrink(struct kmem_cache *s) } #ifdef CONFIG_MEMCG -static void kmemcg_cache_deact_after_rcu(struct kmem_cache *s) +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { /* * Called with all the locks held after a sched RCU grace period. @@ -4054,12 +4054,6 @@ void __kmemcg_cache_deactivate(struct kmem_cache *s) */ slub_set_cpu_partial(s, 0); s->min_partial = 0; - - /* - * s->cpu_partial is checked locklessly (see put_cpu_partial), so - * we have to make sure the change is visible before shrinking. - */ - slab_deactivate_memcg_cache_rcu_sched(s, kmemcg_cache_deact_after_rcu); } #endif /* CONFIG_MEMCG */ From patchwork Wed May 8 20:24:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10936477 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A0063933 for ; Wed, 8 May 2019 20:30:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 923ED2880D for ; Wed, 8 May 2019 20:30:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 856AE288CE; Wed, 8 May 2019 20:30:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0D6082880D for ; Wed, 8 May 2019 20:30:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B2D96B000A; Wed, 8 May 2019 16:30:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 466516B000C; Wed, 8 May 2019 16:30:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B8C76B000D; Wed, 8 May 2019 16:30:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f72.google.com (mail-yw1-f72.google.com [209.85.161.72]) by kanga.kvack.org (Postfix) with ESMTP id F27726B000A for ; Wed, 8 May 2019 16:30:56 -0400 (EDT) Received: by mail-yw1-f72.google.com with SMTP id a70so37253139ywe.21 for ; Wed, 08 May 2019 13:30:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=cRmr36eXTaKgScK2fa3yPK15EeGvQLf0Lc+faeDoG3Q=; b=mWVmTELQxPGjQ7/MbJqMCOBHIgdt97nk1ppa7b3S+SvfvhfAJPCsgBrxgPkrcSSmxD DbeJ4TzXjPrsA9rBnLVGQ4P5E7KmS2BXOzcGWhF2MQBYj2T+GEhX0F9dnzmRHOMbpKWy +iLWJV1ItkvSLnclM55UEbAgFB+XBG+xp3g0LgmF3HiVK8t98Lf8X1sJpN9rY5cIzy/5 oNOx59Ep1oCksLltOu4xjNPzlr8n3fBo/hTh0cDzoNcIpN5k3pmmR3uHfVaJon64guhc iZy740WysCAdGHSGNl0Sb3w8bUnkPw6svspFOkeEHjVR3L8Ku4n/lcu7N/ItDbsxb6Og aPRQ== X-Gm-Message-State: APjAAAWasgjxCnloM8rqs+Tn1HPr1aqLRAM+yb0kXENL9Luw0zSUXHuf KruYXWsDlugqMEG+0qQSSTQV1ELeNcbZvcAUi/wLfVbYYG/a2LFkh5qSw29SJ8yL5E2Uh1nzxDF YaMZAu+6wotOrajhLDYsEI8IFVaw7/a1BSzoMW22LbqP2OCbCRIZlXsVYgVosJRRWUA== X-Received: by 2002:a81:2fd6:: with SMTP id v205mr13388605ywv.205.1557347456586; Wed, 08 May 2019 13:30:56 -0700 (PDT) X-Google-Smtp-Source: APXvYqz8AmocoS/++ZcCVkF+yR0eOXAcO9BpRKF8m0xLsZbVcL1D+nOI5HZDcpb7naYoJ5/i3ZK/ X-Received: by 2002:a81:2fd6:: with SMTP id v205mr13388561ywv.205.1557347455984; Wed, 08 May 2019 13:30:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557347455; cv=none; d=google.com; s=arc-20160816; b=hj1S8Qd9ynNgXa97exe1eJxmlhTMm84tGrsH+SuGDBaZiaYw1yyNfUWydIAmeH7/ez byN+P6D/AK2tFr/z54hibQgxNh0LkLJTL+JsHPEyLk3s/G2cQdmi4H0CCd3Va2G7Y0ET xhimgCO8vgcV1Froo9vTVBbjKU/FCF503YOzh/AZBuO8m6phyEagxuCj+cFdxsJMyDmL CQUpVoP8K4VWRWNYuu4AkCOMqI2xtnm8EXW9ZyARG8GHH05DV9VgUhwpgMIAbPPFk48l DnFc7NavvlAl2zoK9pN4AZ9MiL8ffRSXf5d5itDgUPnB2tOuZ4/coKWvctIyJeIAuhfn btAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=cRmr36eXTaKgScK2fa3yPK15EeGvQLf0Lc+faeDoG3Q=; b=Oc2Ovb9SXiXPB0W1UI3s/qBsmSNHZzFX5c//JIF5MQp06+a7z1/TK3bMuT46Fg5SmW /7SQJ9NDRBHElgGnhdIldkXZucyJ4Z+Nl8FYRgEhL18yFkWmuar2pkMhOzRRah6NL3Qv hw0i0eQx6rTtfE3kXYmc+jDv/Mf0GMa+7yUG0zKlAawZ+p14SipfC0hCRV53DvtLBBjr o0zWc4Z3IKjxKYM4BVxov0UnEqjfOrTDEVj9GoyL6MuxHXU+3hZVxOnBhcqkhVUA4jh8 Jo6p/yPONNriAByugBsIP90PNDRjyw0K9FQWcCPS0Zc5C144O/7a/bXdkCvLY1TTWUKr N0Dw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=KxxyRoNW; spf=pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=0031b2e447=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id m79si42722ybm.359.2019.05.08.13.30.55 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 May 2019 13:30:55 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=KxxyRoNW; spf=pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=0031b2e447=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x48KDOmS029689 for ; Wed, 8 May 2019 13:30:55 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=cRmr36eXTaKgScK2fa3yPK15EeGvQLf0Lc+faeDoG3Q=; b=KxxyRoNWw5r0lsssspEynn5EEM+EXQJNywc6y79B8bU/5ri9UMYvV4KPL0gqvUbh3qjc eyZsnqk1OijDpHc3mMO4yry2z7lXWKyzKlmfKf9b5I1NRvrM55M4FOLBlit3Q88XlPua BAGRBadCHAldOTStaqAWBO6BlX9CFk5SHjc= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2sc25590kn-14 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 08 May 2019 13:30:55 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 8 May 2019 13:30:50 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 0B9FE11CDBCF6; Wed, 8 May 2019 13:25:00 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Shakeel Butt CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Christoph Lameter , Vladimir Davydov , , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v3 3/7] mm: introduce __memcg_kmem_uncharge_memcg() Date: Wed, 8 May 2019 13:24:54 -0700 Message-ID: <20190508202458.550808-4-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190508202458.550808-1-guro@fb.com> References: <20190508202458.550808-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-08_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=859 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905080124 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Let's separate the page counter modification code out of __memcg_kmem_uncharge() in a way similar to what __memcg_kmem_charge() and __memcg_kmem_charge_memcg() work. This will allow to reuse this code later using a new memcg_kmem_uncharge_memcg() wrapper, which calls __memcg_kmem_unchare_memcg() if memcg_kmem_enabled() check is passed. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt --- include/linux/memcontrol.h | 10 ++++++++++ mm/memcontrol.c | 25 +++++++++++++++++-------- 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 36bdfe8e5965..deb209510902 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1298,6 +1298,8 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge(struct page *page, int order); int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, struct mem_cgroup *memcg); +void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg, + unsigned int nr_pages); extern struct static_key_false memcg_kmem_enabled_key; extern struct workqueue_struct *memcg_kmem_cache_wq; @@ -1339,6 +1341,14 @@ static inline int memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, return __memcg_kmem_charge_memcg(page, gfp, order, memcg); return 0; } + +static inline void memcg_kmem_uncharge_memcg(struct page *page, int order, + struct mem_cgroup *memcg) +{ + if (memcg_kmem_enabled()) + __memcg_kmem_uncharge_memcg(memcg, 1 << order); +} + /* * helper for accessing a memcg's index. It will be used as an index in the * child cache array in kmem_cache, and also to derive its name. This function diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 48a8f1c35176..b2c39f187cbb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2750,6 +2750,22 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order) css_put(&memcg->css); return ret; } + +/** + * __memcg_kmem_uncharge_memcg: uncharge a kmem page + * @memcg: memcg to uncharge + * @nr_pages: number of pages to uncharge + */ +void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg, + unsigned int nr_pages) +{ + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + page_counter_uncharge(&memcg->kmem, nr_pages); + + page_counter_uncharge(&memcg->memory, nr_pages); + if (do_memsw_account()) + page_counter_uncharge(&memcg->memsw, nr_pages); +} /** * __memcg_kmem_uncharge: uncharge a kmem page * @page: page to uncharge @@ -2764,14 +2780,7 @@ void __memcg_kmem_uncharge(struct page *page, int order) return; VM_BUG_ON_PAGE(mem_cgroup_is_root(memcg), page); - - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) - page_counter_uncharge(&memcg->kmem, nr_pages); - - page_counter_uncharge(&memcg->memory, nr_pages); - if (do_memsw_account()) - page_counter_uncharge(&memcg->memsw, nr_pages); - + __memcg_kmem_uncharge_memcg(memcg, nr_pages); page->mem_cgroup = NULL; /* slab pages do not have PageKmemcg flag set */ From patchwork Wed May 8 20:24:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10936489 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 515881515 for ; Wed, 8 May 2019 20:40:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 432BE289A5 for ; Wed, 8 May 2019 20:40:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3706C289AA; Wed, 8 May 2019 20:40:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8DEC0289A5 for ; Wed, 8 May 2019 20:40:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12B2D6B000A; Wed, 8 May 2019 16:40:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0B44A6B000C; Wed, 8 May 2019 16:40:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E487B6B000E; Wed, 8 May 2019 16:40:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 882C26B000A for ; Wed, 8 May 2019 16:40:53 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id d10so80563plo.12 for ; Wed, 08 May 2019 13:40:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=waf7z8I248qGj1WPv3szkFNRuIjadGqCnJ4vjGJ4B5U=; b=dW3g/h8hrZMbRoTgaIIknYZa7FsLzP1XlEJobjnLaauJK5Q4Lx2t1/usw261FNsNGL Xgr1a/yzhm34uyL5XiNmyGTbPYq9WYPdt9B7wG+V+BUFmdGqCyWbQBj6srSo4hD6Xps/ ES2BDNhgFPttV8UOMmnag2nhhm8HwjwMdA8DOyRpJb2pguuuzjp86t9ebD3RBe/LM/rC 3GvyTr6Q/kx7GFLQzma7oDN+ZLkwYMfz4eY4M5V1bQR1rMkZhrz6ttoXpIc/UG7Gxnx+ iuKQkx8JclzEHjimH1Wq1E0SkUAzxMUmbnYHUZ59GqCiiscIpnwiqrrZMdixR0i9PcGK c8DQ== X-Gm-Message-State: APjAAAUzFsyTe13l6ZJFvyrjGYrx5Weuxdd0dIMLNuzkQUrilMYzmwR+ LUvMsbZhTlNMlhixfbXJorm+kseSX+YrqvyhkBWkRsJ0NxFbZHRhvzRhtS88mai9WnSDwMf0s8B AoVhsLMf2qan+Px1n47IRpz+QrEunEwK/m4SMuJE0gn+d4dmMf5LaA6lXS7wjOrh81g== X-Received: by 2002:a63:dd58:: with SMTP id g24mr177632pgj.161.1557348053193; Wed, 08 May 2019 13:40:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqyhrR25to8qGuaJCXMNGxmgI+bt9InYANCOrLfcbs5aSamzGvYvgR7UKMeuQH/BDeMINBVD X-Received: by 2002:a63:dd58:: with SMTP id g24mr177554pgj.161.1557348052288; Wed, 08 May 2019 13:40:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557348052; cv=none; d=google.com; s=arc-20160816; b=Lk8g80bJwEAZMFMyqNfbLauaYC76EXZf0xBlIK+dZ4db2BOyvziR5FJxqZ4oGNOj1+ Wke0nuXnGyMhLpTe9tx+kC1jgDiI65pDM5eXYS2yIIaojP4rxZ6pFoRWA06aY401GW+M +sOOjb4pGVjVvWpsJQhxS8lLrT3ehqRUEVG49YrhEX/BIuc0lheeM3XnOGqUOixZsHfD sp5LZoqFFnw5/VT9kmO2wMs5ODB9XZMlHBFmIbCiMzTFZWGZALk6LK/a4wS5T0QpIc7h 6tgEodBOFROG2g4gq5hlSvizMEkMbwMATitHtNTVB+Y59SrGjIJhpF9s/zzMzdSZxbx5 HU0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=waf7z8I248qGj1WPv3szkFNRuIjadGqCnJ4vjGJ4B5U=; b=q0HJWY5GVdkQBpU3NaA9RMnG1UJ2ZJQwzzvqc371yR9sRm6GnSVfODEByNXmnq0YSE lAb7hGAwDygWe7QsUAbKeRa1QjROh0B4G0CwAwx2qY+2aue1gmX/oYdZ0GlTG0TBqFI6 Z1RbQSvF3Dnjv7KwjbtJ6okxa1cRsuti7KZcPaZ2cRQlGMdwdUi3JjauOmqYTgS1fUA3 gMZ+iDWD60KVtYl8kU6iYUNcxBBI44XIKipa+kP/n77bCzVIynJ4bNNCSwWIHkHB7dtS y0po3Iw/uUjhOvP89nG0lCp2rrBI3bWbmJ8pyF/o8p0BKumfd8FAEGJAaM0gfclws00g 85JQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="mZw/5JT1"; spf=pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0031b2e447=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id s79si25120912pfa.69.2019.05.08.13.40.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 May 2019 13:40:52 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="mZw/5JT1"; spf=pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0031b2e447=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x48KeVEB023320 for ; Wed, 8 May 2019 13:40:51 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=waf7z8I248qGj1WPv3szkFNRuIjadGqCnJ4vjGJ4B5U=; b=mZw/5JT1BzFI2DXc0qEIgdhdIDH/8ub5Ar0QhPw9AdZhlvo4yw5L7NR1osh2eHJEQuXP CdJUvJPLWMol/aWq760mBflq4cL3A0529CUCuSe3q2vZn4ZaEHJhxDeD4zfZHZdAlHn4 tqQPk/UX6foQ3DwxTE48AQsOVhfEGWIuCkg= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2sc0p91d9f-9 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 08 May 2019 13:40:51 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::130) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Wed, 8 May 2019 13:40:48 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 10AA811CDBCFD; Wed, 8 May 2019 13:25:00 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Shakeel Butt CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Christoph Lameter , Vladimir Davydov , , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v3 4/7] mm: unify SLAB and SLUB page accounting Date: Wed, 8 May 2019 13:24:55 -0700 Message-ID: <20190508202458.550808-5-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190508202458.550808-1-guro@fb.com> References: <20190508202458.550808-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-08_11:,, signatures=0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently the page accounting code is duplicated in SLAB and SLUB internals. Let's move it into new (un)charge_slab_page helpers in the slab_common.c file. These helpers will be responsible for statistics (global and memcg-aware) and memcg charging. So they are replacing direct memcg_(un)charge_slab() calls. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Christoph Lameter --- mm/slab.c | 19 +++---------------- mm/slab.h | 25 +++++++++++++++++++++++++ mm/slub.c | 14 ++------------ 3 files changed, 30 insertions(+), 28 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 83000e46b870..32e6af9ed9af 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1389,7 +1389,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, int nodeid) { struct page *page; - int nr_pages; flags |= cachep->allocflags; @@ -1399,17 +1398,11 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, return NULL; } - if (memcg_charge_slab(page, flags, cachep->gfporder, cachep)) { + if (charge_slab_page(page, flags, cachep->gfporder, cachep)) { __free_pages(page, cachep->gfporder); return NULL; } - nr_pages = (1 << cachep->gfporder); - if (cachep->flags & SLAB_RECLAIM_ACCOUNT) - mod_lruvec_page_state(page, NR_SLAB_RECLAIMABLE, nr_pages); - else - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, nr_pages); - __SetPageSlab(page); /* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks() && page_is_pfmemalloc(page)) @@ -1424,12 +1417,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, static void kmem_freepages(struct kmem_cache *cachep, struct page *page) { int order = cachep->gfporder; - unsigned long nr_freed = (1 << order); - - if (cachep->flags & SLAB_RECLAIM_ACCOUNT) - mod_lruvec_page_state(page, NR_SLAB_RECLAIMABLE, -nr_freed); - else - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, -nr_freed); BUG_ON(!PageSlab(page)); __ClearPageSlabPfmemalloc(page); @@ -1438,8 +1425,8 @@ static void kmem_freepages(struct kmem_cache *cachep, struct page *page) page->mapping = NULL; if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += nr_freed; - memcg_uncharge_slab(page, order, cachep); + current->reclaim_state->reclaimed_slab += 1 << order; + uncharge_slab_page(page, order, cachep); __free_pages(page, order); } diff --git a/mm/slab.h b/mm/slab.h index 4a261c97c138..c9a31120fa1d 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -205,6 +205,12 @@ ssize_t slabinfo_write(struct file *file, const char __user *buffer, void __kmem_cache_free_bulk(struct kmem_cache *, size_t, void **); int __kmem_cache_alloc_bulk(struct kmem_cache *, gfp_t, size_t, void **); +static inline int cache_vmstat_idx(struct kmem_cache *s) +{ + return (s->flags & SLAB_RECLAIM_ACCOUNT) ? + NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE; +} + #ifdef CONFIG_MEMCG_KMEM /* List of all root caches. */ @@ -352,6 +358,25 @@ static inline void memcg_link_cache(struct kmem_cache *s, #endif /* CONFIG_MEMCG_KMEM */ +static __always_inline int charge_slab_page(struct page *page, + gfp_t gfp, int order, + struct kmem_cache *s) +{ + int ret = memcg_charge_slab(page, gfp, order, s); + + if (!ret) + mod_lruvec_page_state(page, cache_vmstat_idx(s), 1 << order); + + return ret; +} + +static __always_inline void uncharge_slab_page(struct page *page, int order, + struct kmem_cache *s) +{ + mod_lruvec_page_state(page, cache_vmstat_idx(s), -(1 << order)); + memcg_uncharge_slab(page, order, s); +} + static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) { struct kmem_cache *cachep; diff --git a/mm/slub.c b/mm/slub.c index 43c34d54ad86..9ec25a588bdd 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1494,7 +1494,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s, else page = __alloc_pages_node(node, flags, order); - if (page && memcg_charge_slab(page, flags, order, s)) { + if (page && charge_slab_page(page, flags, order, s)) { __free_pages(page, order); page = NULL; } @@ -1687,11 +1687,6 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) if (!page) return NULL; - mod_lruvec_page_state(page, - (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE, - 1 << oo_order(oo)); - inc_slabs_node(s, page_to_nid(page), page->objects); return page; @@ -1725,18 +1720,13 @@ static void __free_slab(struct kmem_cache *s, struct page *page) check_object(s, page, p, SLUB_RED_INACTIVE); } - mod_lruvec_page_state(page, - (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE, - -pages); - __ClearPageSlabPfmemalloc(page); __ClearPageSlab(page); page->mapping = NULL; if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages; - memcg_uncharge_slab(page, order, s); + uncharge_slab_page(page, order, s); __free_pages(page, order); } From patchwork Wed May 8 20:24:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10936487 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 443DE1515 for ; Wed, 8 May 2019 20:40:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 33460289A5 for ; Wed, 8 May 2019 20:40:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 24667289AA; Wed, 8 May 2019 20:40:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E158D289A5 for ; Wed, 8 May 2019 20:40:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D0F16B0008; Wed, 8 May 2019 16:40:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 659076B000A; Wed, 8 May 2019 16:40:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4FA0A6B000C; Wed, 8 May 2019 16:40:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id E6F2C6B0008 for ; Wed, 8 May 2019 16:40:51 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id y9so82676plt.11 for ; Wed, 08 May 2019 13:40:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=ilqSfy/vzYbKLUEqz87/KxpSAdY0RPIvj65efPUF4Qc=; b=pPVOFz9bl0ZCi0xd8jdQJOw4MV8D5y1bHxd6n+LaXiApYTnGpyGMMeoIw19NyUZO6Y soyAeLmShoYHhdp0xnvzwKkOj8FQJJ9CJiBH9Fd01tXlSrKs2YoPlKkRVQ7O8kVYKGaM IWwN7mD4QMEshCvhqnUcxP/hdjWBtNEdtraWujwJSmQftbcRfjD9/wsHEUhDHe6ABkcU Irsmxl//PrXNncI+oxJint8uF7Zv3sxvp4UjGojpAUY3/BwVkYief68qSkS4Sd61oZPz dRM+5xj3DdKcmFJvy+kap1Pko1uREYSKtqrZyW6X1PA/AaJg5AmKYMdJQEaPMHWSOsEk 1OgA== X-Gm-Message-State: APjAAAWMm+OslSbB5div+1AI01bc7qovUvou8vBlPn7SeaY5NOM1qVK9 MUw+4wrW5A5mguOSoH17m11MhniA3kAG7m/X+ZbgFGgFbzAhyo7Xiu2k8VTquBu2B+q2CL9ke+/ BMHhnBtIK2S2afuOHxWLT8fgWqh28gywzq8PcWZt0MjmM6MFTxwlmVHHA4hE+DgHCQA== X-Received: by 2002:a17:902:e305:: with SMTP id cg5mr50195740plb.112.1557348051478; Wed, 08 May 2019 13:40:51 -0700 (PDT) X-Google-Smtp-Source: APXvYqzQABqPfZTKtMfMnClQntPKkQ0sP+2akYf1GNmI3WzvQUeBXv9FAhJlH2ttqoH5Q04Qp1sy X-Received: by 2002:a17:902:e305:: with SMTP id cg5mr50195615plb.112.1557348050131; Wed, 08 May 2019 13:40:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557348050; cv=none; d=google.com; s=arc-20160816; b=1CQaaPEmTC0Eymns6NtZ+lw3X2KAfxn6t94rYuCw1Pgyp8WnngZn8TxoyRqa3W0ViG pd+CI12XXN3iDmm5Xzz+f5TNFXco/MRTRZHmUhe5cRGFDcukJ4Stalccfmb56/0KmA+6 G8qcApc/4KPR1vO4seFQe4/nH7X2tnhSJO1XGQW0CNxJCDRUUfPzdopLBYtiKRivjlNY IVkuAPv3wODtfG+pEfx16P35+mvEEYgFZU/cEOjuGYZEF6xYr6sXLnsyS1fqGpuIM7wd Kt5uoZLgPHL/f194Hf+1kQdam33ZlEfiKTQkJJuzRSTiIyeL8PUreEII+gv7sQVTv7Xl hD2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=ilqSfy/vzYbKLUEqz87/KxpSAdY0RPIvj65efPUF4Qc=; b=0CTc23D9Ephe6cwBdcU9Ibpy5iOTtW1hdTpBW+Gr9XQW61cy0FFh26s0P4SkPUn9Kg GzxmD99EA3nOiJPZHPUxj1rakEQfmiHvDwYMqvs4tCzJs2kVFLr5Hi+x57kpWdOAwCQy Uvv3Uj2cH5vdKU7w0sg4z8BHYbvlszuJRGc1LMGvob8suQFrCbqGSnEyA9jR3gsnOnsL 8hCMuhFbNC0q0UpyPkxpzMGB47fwwOFDwCpwdalL56gydm0Sm4cYhRdUbGz5QR3wF1em NOT/w3YpAvbxmP0dtLFlobHfE9TbdAuKuBf//SV4RKlU96YkGGh6QhChqLvd4GKfUeVv 80Gw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=o0f+49xS; spf=pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0031b2e447=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id k8si24826007pll.318.2019.05.08.13.40.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 May 2019 13:40:50 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=o0f+49xS; spf=pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0031b2e447=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x48KeVE5023320 for ; Wed, 8 May 2019 13:40:49 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=ilqSfy/vzYbKLUEqz87/KxpSAdY0RPIvj65efPUF4Qc=; b=o0f+49xS8bphtjt/Jn0Dvo5dx7chjIKwbFZt4wT+gKLVPWA4hr9hq2axPQBL1r1MHQIt /ez1MVbqAF5LF9fSG1B0lxOf7hPdy1I4MH74h+j2nZEOD0Yxc8hF+p4qy79fOPoJevNn vZ8YNmjNKwJ2PE0pYufRmCYRRRKnfmjH2Rk= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2sc0p91d9f-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 08 May 2019 13:40:49 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::130) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Wed, 8 May 2019 13:40:48 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 14C9C11CDBD00; Wed, 8 May 2019 13:25:00 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Shakeel Butt CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Christoph Lameter , Vladimir Davydov , , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v3 5/7] mm: rework non-root kmem_cache lifecycle management Date: Wed, 8 May 2019 13:24:56 -0700 Message-ID: <20190508202458.550808-6-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190508202458.550808-1-guro@fb.com> References: <20190508202458.550808-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-08_11:,, signatures=0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This commit makes several important changes in the lifecycle of a non-root kmem_cache, which also affect the lifecycle of a memory cgroup. Currently each charged slab page has a page->mem_cgroup pointer to the memory cgroup and holds a reference to it. Kmem_caches are held by the memcg and are released with it. It means that none of kmem_caches are released unless at least one reference to the memcg exists, which is not optimal. So the current scheme can be illustrated as: page->mem_cgroup->kmem_cache. To implement the slab memory reparenting we need to invert the scheme into: page->kmem_cache->mem_cgroup. Let's make every page to hold a reference to the kmem_cache (we already have a stable pointer), and make kmem_caches to hold a single reference to the memory cgroup. To make this possible we need to introduce a new percpu refcounter for non-root kmem_caches. The counter is initialized to the percpu mode, and is switched to atomic mode after deactivation, so we never shutdown an active cache. The counter is bumped for every charged page and also for every running allocation. So the kmem_cache can't be released unless all allocations complete. To shutdown non-active empty kmem_caches, let's reuse the infrastructure of the RCU-delayed work queue, used previously for the deactivation. After the generalization, it's perfectly suited for our needs. Since now we can release a kmem_cache at any moment after the deactivation, let's call sysfs_slab_remove() only from the shutdown path. It makes deactivation path simpler. Because we don't set the page->mem_cgroup pointer, we need to change the way how memcg-level stats is working for slab pages. We can't use mod_lruvec_page_state() helpers anymore, so switch over to mod_lruvec_state(). * I used the following simple approach to test the performance (stolen from another patchset by T. Harding): time find / -name fname-no-exist echo 2 > /proc/sys/vm/drop_caches repeat several times Results (I've chosen best results in several runs): orig patched real 0m0.700s 0m0.722s user 0m0.114s 0m0.120s sys 0m0.317s 0m0.324s real 0m0.729s 0m0.746s user 0m0.110s 0m0.139s sys 0m0.320s 0m0.317s real 0m0.745s 0m0.719s user 0m0.108s 0m0.124s sys 0m0.320s 0m0.323s So it looks like the difference is not noticeable in this test. Signed-off-by: Roman Gushchin --- include/linux/slab.h | 3 +- mm/memcontrol.c | 53 +++++++++++++++++++++---------- mm/slab.h | 74 +++++++++++++++++++++++--------------------- mm/slab_common.c | 63 +++++++++++++++++++------------------ mm/slub.c | 12 +------ 5 files changed, 112 insertions(+), 93 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 47923c173f30..1b54e5f83342 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -16,6 +16,7 @@ #include #include #include +#include /* @@ -152,7 +153,6 @@ int kmem_cache_shrink(struct kmem_cache *); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); void memcg_deactivate_kmem_caches(struct mem_cgroup *); -void memcg_destroy_kmem_caches(struct mem_cgroup *); /* * Please use this macro to create slab caches. Simply specify the @@ -641,6 +641,7 @@ struct memcg_cache_params { struct mem_cgroup *memcg; struct list_head children_node; struct list_head kmem_caches_node; + struct percpu_ref refcnt; void (*work_fn)(struct kmem_cache *); union { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b2c39f187cbb..9b27988c8969 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2610,12 +2610,13 @@ static void memcg_schedule_kmem_cache_create(struct mem_cgroup *memcg, { struct memcg_kmem_cache_create_work *cw; + if (!css_tryget_online(&memcg->css)) + return; + cw = kmalloc(sizeof(*cw), GFP_NOWAIT | __GFP_NOWARN); if (!cw) return; - css_get(&memcg->css); - cw->memcg = memcg; cw->cachep = cachep; INIT_WORK(&cw->work, memcg_kmem_cache_create_func); @@ -2651,20 +2652,35 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) struct mem_cgroup *memcg; struct kmem_cache *memcg_cachep; int kmemcg_id; + struct memcg_cache_array *arr; VM_BUG_ON(!is_root_cache(cachep)); if (memcg_kmem_bypass()) return cachep; - memcg = get_mem_cgroup_from_current(); + rcu_read_lock(); + + if (unlikely(current->active_memcg)) + memcg = current->active_memcg; + else + memcg = mem_cgroup_from_task(current); + + if (!memcg || memcg == root_mem_cgroup) + goto out_unlock; + kmemcg_id = READ_ONCE(memcg->kmemcg_id); if (kmemcg_id < 0) - goto out; + goto out_unlock; - memcg_cachep = cache_from_memcg_idx(cachep, kmemcg_id); - if (likely(memcg_cachep)) - return memcg_cachep; + arr = rcu_dereference(cachep->memcg_params.memcg_caches); + + /* + * Make sure we will access the up-to-date value. The code updating + * memcg_caches issues a write barrier to match this (see + * memcg_create_kmem_cache()). + */ + memcg_cachep = READ_ONCE(arr->entries[kmemcg_id]); /* * If we are in a safe context (can wait, and not in interrupt @@ -2677,10 +2693,16 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) * memcg_create_kmem_cache, this means no further allocation * could happen with the slab_mutex held. So it's better to * defer everything. + * + * If the memcg is dying or memcg_cache is about to be released, + * don't bother creating new kmem_caches. */ - memcg_schedule_kmem_cache_create(memcg, cachep); -out: - css_put(&memcg->css); + if (unlikely(!memcg_cachep)) + memcg_schedule_kmem_cache_create(memcg, cachep); + else if (percpu_ref_tryget(&memcg_cachep->memcg_params.refcnt)) + cachep = memcg_cachep; +out_unlock: + rcu_read_lock(); return cachep; } @@ -2691,7 +2713,7 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) void memcg_kmem_put_cache(struct kmem_cache *cachep) { if (!is_root_cache(cachep)) - css_put(&cachep->memcg_params.memcg->css); + percpu_ref_put(&cachep->memcg_params.refcnt); } /** @@ -2719,9 +2741,6 @@ int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, cancel_charge(memcg, nr_pages); return -ENOMEM; } - - page->mem_cgroup = memcg; - return 0; } @@ -2744,8 +2763,10 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order) memcg = get_mem_cgroup_from_current(); if (!mem_cgroup_is_root(memcg)) { ret = __memcg_kmem_charge_memcg(page, gfp, order, memcg); - if (!ret) + if (!ret) { + page->mem_cgroup = memcg; __SetPageKmemcg(page); + } } css_put(&memcg->css); return ret; @@ -3238,7 +3259,7 @@ static void memcg_free_kmem(struct mem_cgroup *memcg) memcg_offline_kmem(memcg); if (memcg->kmem_state == KMEM_ALLOCATED) { - memcg_destroy_kmem_caches(memcg); + WARN_ON(!list_empty(&memcg->kmem_caches)); static_branch_dec(&memcg_kmem_enabled_key); WARN_ON(page_counter_read(&memcg->kmem)); } diff --git a/mm/slab.h b/mm/slab.h index c9a31120fa1d..2acc68a7e0a0 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -173,6 +173,7 @@ void __kmem_cache_release(struct kmem_cache *); int __kmem_cache_shrink(struct kmem_cache *); void __kmemcg_cache_deactivate(struct kmem_cache *s); void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s); +void kmemcg_cache_shutdown(struct kmem_cache *s); void slab_kmem_cache_release(struct kmem_cache *); struct seq_file; @@ -248,31 +249,6 @@ static inline const char *cache_name(struct kmem_cache *s) return s->name; } -/* - * Note, we protect with RCU only the memcg_caches array, not per-memcg caches. - * That said the caller must assure the memcg's cache won't go away by either - * taking a css reference to the owner cgroup, or holding the slab_mutex. - */ -static inline struct kmem_cache * -cache_from_memcg_idx(struct kmem_cache *s, int idx) -{ - struct kmem_cache *cachep; - struct memcg_cache_array *arr; - - rcu_read_lock(); - arr = rcu_dereference(s->memcg_params.memcg_caches); - - /* - * Make sure we will access the up-to-date value. The code updating - * memcg_caches issues a write barrier to match this (see - * memcg_create_kmem_cache()). - */ - cachep = READ_ONCE(arr->entries[idx]); - rcu_read_unlock(); - - return cachep; -} - static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) { if (is_root_cache(s)) @@ -284,15 +260,37 @@ static __always_inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { - if (is_root_cache(s)) - return 0; - return memcg_kmem_charge_memcg(page, gfp, order, s->memcg_params.memcg); + struct mem_cgroup *memcg; + struct lruvec *lruvec; + int ret; + + memcg = s->memcg_params.memcg; + ret = memcg_kmem_charge_memcg(page, gfp, order, memcg); + if (ret) + return ret; + + lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); + mod_lruvec_state(lruvec, cache_vmstat_idx(s), 1 << order); + + /* transer try_charge() page references to kmem_cache */ + percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); + css_put_many(&memcg->css, 1 << order); + + return 0; } static __always_inline void memcg_uncharge_slab(struct page *page, int order, struct kmem_cache *s) { - memcg_kmem_uncharge(page, order); + struct mem_cgroup *memcg; + struct lruvec *lruvec; + + memcg = s->memcg_params.memcg; + lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); + mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order)); + memcg_kmem_uncharge_memcg(page, order, memcg); + + percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); } extern void slab_init_memcg_params(struct kmem_cache *); @@ -362,18 +360,24 @@ static __always_inline int charge_slab_page(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { - int ret = memcg_charge_slab(page, gfp, order, s); - - if (!ret) - mod_lruvec_page_state(page, cache_vmstat_idx(s), 1 << order); + if (is_root_cache(s)) { + mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), + 1 << order); + return 0; + } - return ret; + return memcg_charge_slab(page, gfp, order, s); } static __always_inline void uncharge_slab_page(struct page *page, int order, struct kmem_cache *s) { - mod_lruvec_page_state(page, cache_vmstat_idx(s), -(1 << order)); + if (is_root_cache(s)) { + mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), + -(1 << order)); + return; + } + memcg_uncharge_slab(page, order, s); } diff --git a/mm/slab_common.c b/mm/slab_common.c index 4e5b4292a763..995920222127 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -45,6 +45,8 @@ static void slab_caches_to_rcu_destroy_workfn(struct work_struct *work); static DECLARE_WORK(slab_caches_to_rcu_destroy_work, slab_caches_to_rcu_destroy_workfn); +static void kmemcg_queue_cache_shutdown(struct percpu_ref *percpu_ref); + /* * Set of flags that will prevent slab merging */ @@ -145,6 +147,12 @@ static int init_memcg_params(struct kmem_cache *s, struct memcg_cache_array *arr; if (root_cache) { + int ret = percpu_ref_init(&s->memcg_params.refcnt, + kmemcg_queue_cache_shutdown, + 0, GFP_KERNEL); + if (ret) + return ret; + s->memcg_params.root_cache = root_cache; INIT_LIST_HEAD(&s->memcg_params.children_node); INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node); @@ -170,6 +178,8 @@ static void destroy_memcg_params(struct kmem_cache *s) { if (is_root_cache(s)) kvfree(rcu_access_pointer(s->memcg_params.memcg_caches)); + else + percpu_ref_exit(&s->memcg_params.refcnt); } static void free_memcg_params(struct rcu_head *rcu) @@ -225,6 +235,7 @@ void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) if (is_root_cache(s)) { list_add(&s->root_caches_node, &slab_root_caches); } else { + css_get(&memcg->css); s->memcg_params.memcg = memcg; list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); @@ -240,6 +251,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) } else { list_del(&s->memcg_params.children_node); list_del(&s->memcg_params.kmem_caches_node); + css_put(&s->memcg_params.memcg->css); } } #else @@ -708,16 +720,13 @@ static void kmemcg_after_rcu_workfn(struct work_struct *work) put_online_mems(); put_online_cpus(); - - /* done, put the ref from slab_deactivate_memcg_cache_rcu_sched() */ - css_put(&s->memcg_params.memcg->css); } /* * We need to grab blocking locks. Bounce to ->work. The * work item shares the space with the RCU head and can't be - * initialized eariler. -*/ + * initialized earlier. + */ static void kmemcg_schedule_work_after_rcu(struct rcu_head *head) { struct kmem_cache *s = container_of(head, struct kmem_cache, @@ -727,9 +736,28 @@ static void kmemcg_schedule_work_after_rcu(struct rcu_head *head) queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); } +static void kmemcg_cache_shutdown_after_rcu(struct kmem_cache *s) +{ + WARN_ON(shutdown_cache(s)); +} + +static void kmemcg_queue_cache_shutdown(struct percpu_ref *percpu_ref) +{ + struct kmem_cache *s = container_of(percpu_ref, struct kmem_cache, + memcg_params.refcnt); + + if (s->memcg_params.root_cache->memcg_params.dying) + return; + + WARN_ON(s->memcg_params.work_fn); + s->memcg_params.work_fn = kmemcg_cache_shutdown_after_rcu; + call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); +} + static void kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { __kmemcg_cache_deactivate_after_rcu(s); + percpu_ref_kill(&s->memcg_params.refcnt); } static void kmemcg_cache_deactivate(struct kmem_cache *s) @@ -739,9 +767,6 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s) if (s->memcg_params.root_cache->memcg_params.dying) return; - /* pin memcg so that @s doesn't get destroyed in the middle */ - css_get(&s->memcg_params.memcg->css); - WARN_ON_ONCE(s->memcg_params.work_fn); s->memcg_params.work_fn = kmemcg_cache_deactivate_after_rcu; call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); @@ -775,28 +800,6 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) put_online_cpus(); } -void memcg_destroy_kmem_caches(struct mem_cgroup *memcg) -{ - struct kmem_cache *s, *s2; - - get_online_cpus(); - get_online_mems(); - - mutex_lock(&slab_mutex); - list_for_each_entry_safe(s, s2, &memcg->kmem_caches, - memcg_params.kmem_caches_node) { - /* - * The cgroup is about to be freed and therefore has no charges - * left. Hence, all its caches must be empty by now. - */ - BUG_ON(shutdown_cache(s)); - } - mutex_unlock(&slab_mutex); - - put_online_mems(); - put_online_cpus(); -} - static int shutdown_memcg_caches(struct kmem_cache *s) { struct memcg_cache_array *arr; diff --git a/mm/slub.c b/mm/slub.c index 9ec25a588bdd..e7ce810ebd02 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4022,18 +4022,8 @@ void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { /* * Called with all the locks held after a sched RCU grace period. - * Even if @s becomes empty after shrinking, we can't know that @s - * doesn't have allocations already in-flight and thus can't - * destroy @s until the associated memcg is released. - * - * However, let's remove the sysfs files for empty caches here. - * Each cache has a lot of interface files which aren't - * particularly useful for empty draining caches; otherwise, we can - * easily end up with millions of unnecessary sysfs files on - * systems which have a lot of memory and transient cgroups. */ - if (!__kmem_cache_shrink(s)) - sysfs_slab_remove(s); + __kmem_cache_shrink(s); } void __kmemcg_cache_deactivate(struct kmem_cache *s) From patchwork Wed May 8 20:24:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10936485 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2AC76924 for ; Wed, 8 May 2019 20:40:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 17390289A5 for ; Wed, 8 May 2019 20:40:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 08AF4289AA; Wed, 8 May 2019 20:40:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3F0E0289A5 for ; Wed, 8 May 2019 20:40:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28B226B0007; Wed, 8 May 2019 16:40:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1EB876B000A; Wed, 8 May 2019 16:40:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D9FB6B000C; Wed, 8 May 2019 16:40:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id C5E746B0007 for ; Wed, 8 May 2019 16:40:51 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id 17so13368374pfi.12 for ; Wed, 08 May 2019 13:40:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=LxIDu3UhYtNg1pn5UcXrVPh/HFhtDW9+IlbLZXygBUg=; b=bb/VlUe9K1JRKa27TfQWiWNU4G59ukuu55Am9Y3By9z/KQJ1rg1u1rzUXbHP3rYiXC R51P61RNTmweNcYebhpnOMlgS9/Q9CkEuGy76204bhcCDvviwUIv2X7TKn8JWqLyUNPF d4ny223hrzFI78vmOv5+ywvmSwOYnXw2gNG6VtUhTDSzr4MPQaajcjhQKCSo5Lps9PKH WIOf6HC1Sh6ItX9rsZ4JqYEwtJz9eOYj50Tr5craK7vdiU3n0XgYttMqmVM5ecOoPO6r i20byA+e+OcXAIbvSn0rK5ieEPS1xJU8qn7ay+roSlRHmV7H/a61ZCr48XiJ52eNI9b/ tIUg== X-Gm-Message-State: APjAAAWRdMjd1wJvahsmBDR9QCblYh165eZBLbi9ObWMIGlfMaRgl+Pk dAxb+KfE212dI28ELiLMtr0TkFndP34beEaWaHwIHKRwG8qVw58+vaVcT4PjDmAGRc2IJ+As4uL dMR/lTeIZHxv00OtArNeutFOVdz3fAKpyNeAc021pFs0gMv4MR+//SmLs5H27nnRWkQ== X-Received: by 2002:a65:4183:: with SMTP id a3mr180561pgq.121.1557348051445; Wed, 08 May 2019 13:40:51 -0700 (PDT) X-Google-Smtp-Source: APXvYqznDAX6v6Ih1S0/DqNVw8MEyljOeRhRd46LGx2QikqgRubO4pbFQX6O3uV+uVzdtAurOw2U X-Received: by 2002:a65:4183:: with SMTP id a3mr180482pgq.121.1557348050547; Wed, 08 May 2019 13:40:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557348050; cv=none; d=google.com; s=arc-20160816; b=YOO28x+Mc/foDkoXXMeFaQ984jiWBy4b1hVsl9GeImLnAVWr0XCRDVEVeZ6E/bjVik 29lClmQCABSc1SJH4ZxeqEQHkddWfw2GYM1WxqW5UAIrcEsb/hUkBWbxVck2gmBP+8qp ICUvCqQ1iFBxvUx7BxYe8pFwwleisR/uHCFZ6wJccMBscxVYsEtn+v6IY1Gp1FFyiugC 4VRXf8UtGfWQoHHHK0ugzfKVT36I39+vH3WHiXFtSR/9Due4T4YOBnN3Y9Sqp5suGlf+ osNCSg9xnNnn708/ijD72XCDY7RpM/h+AqmrPV79GEV3nduDB4M2ou30jEsW3Auw9uHU /g0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=LxIDu3UhYtNg1pn5UcXrVPh/HFhtDW9+IlbLZXygBUg=; b=TS6Bh1A4Q/T5M+wzyBUoa5PLmeKO+3vliLiREHdYt9XHM3gUOjILsbHM02zJTulkYK FW3pv3cDjqKYm5p8njDiThgwrofzeqLWPcSwWty13JbLaRdbwWayCe1cMv6Pbx4YfnQO K6JzXZU51qAoS+DndbEdioiIkr1ZY2UH5d7xApkhBlhGlphOiO2Hhy1kGPOebNjWohQx xSrqa6R6sGVx+ROb+wGX1g9qhRGQab7NHB/sWZiOurWN8aH9gu4+IFQUXhR5SHIoiR/0 cDdJGdvHlISuP0AeEEdNfDseIjFrVz2zZXOZVZXfIA21r0LhXgBszHKZdgaBNOKff2RP 3OBw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=haKVMOqt; spf=pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0031b2e447=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id v21si9826311pgb.560.2019.05.08.13.40.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 May 2019 13:40:50 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=haKVMOqt; spf=pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0031b2e447=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x48KeVE7023320 for ; Wed, 8 May 2019 13:40:50 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=LxIDu3UhYtNg1pn5UcXrVPh/HFhtDW9+IlbLZXygBUg=; b=haKVMOqtENtB2naPd2I/8wNLXd8v7C/VNmMn/GDfmaPUEDmhBJpVnKreS5QCc8kjaXjP o8xbw2CCqAFXzMv/tbUKWEesVYHXn4iXR2HW1KrH8iin8xdr+4UtnxKbIcSdnT/Zy0Mq zGk1hL/cezUfH0h8qWHljiOJyzZlYIPEzTs= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2sc0p91d9f-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 08 May 2019 13:40:50 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::130) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Wed, 8 May 2019 13:40:48 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 1895F11CDBD02; Wed, 8 May 2019 13:25:00 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Shakeel Butt CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Christoph Lameter , Vladimir Davydov , , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v3 6/7] mm: reparent slab memory on cgroup removal Date: Wed, 8 May 2019 13:24:57 -0700 Message-ID: <20190508202458.550808-7-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190508202458.550808-1-guro@fb.com> References: <20190508202458.550808-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-08_11:,, signatures=0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Let's reparent memcg slab memory on memcg offlining. This allows us to release the memory cgroup without waiting for the last outstanding kernel object (e.g. dentry used by another application). So instead of reparenting all accounted slab pages, let's do reparent a relatively small amount of kmem_caches. Reparenting is performed as a part of the deactivation process. Since the parent cgroup is already charged, everything we need to do is to splice the list of kmem_caches to the parent's kmem_caches list, swap the memcg pointer and drop the css refcounter for each kmem_cache and adjust the parent's css refcounter. Quite simple. Please, note that kmem_cache->memcg_params.memcg isn't a stable pointer anymore. It's safe to read it under rcu_read_lock() or with slab_mutex held. We can race with the slab allocation and deallocation paths. It's not a big problem: parent's charge and slab global stats are always correct, and we don't care anymore about the child usage and global stats. The child cgroup is already offline, so we don't use or show it anywhere. Local slab stats (NR_SLAB_RECLAIMABLE and NR_SLAB_UNRECLAIMABLE) aren't used anywhere except count_shadow_nodes(). But even there it won't break anything: after reparenting "nodes" will be 0 on child level (because we're already reparenting shrinker lists), and on parent level page stats always were 0, and this patch won't change anything. Signed-off-by: Roman Gushchin --- include/linux/slab.h | 4 ++-- mm/memcontrol.c | 14 ++++++++------ mm/slab.h | 14 +++++++++----- mm/slab_common.c | 23 ++++++++++++++++++++--- 4 files changed, 39 insertions(+), 16 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 1b54e5f83342..109cab2ad9b4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -152,7 +152,7 @@ void kmem_cache_destroy(struct kmem_cache *); int kmem_cache_shrink(struct kmem_cache *); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); -void memcg_deactivate_kmem_caches(struct mem_cgroup *); +void memcg_deactivate_kmem_caches(struct mem_cgroup *, struct mem_cgroup *); /* * Please use this macro to create slab caches. Simply specify the @@ -638,7 +638,7 @@ struct memcg_cache_params { bool dying; }; struct { - struct mem_cgroup *memcg; + struct mem_cgroup __rcu *memcg; struct list_head children_node; struct list_head kmem_caches_node; struct percpu_ref refcnt; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9b27988c8969..6e4d9ed16069 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3220,15 +3220,15 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) */ memcg->kmem_state = KMEM_ALLOCATED; - memcg_deactivate_kmem_caches(memcg); - - kmemcg_id = memcg->kmemcg_id; - BUG_ON(kmemcg_id < 0); - parent = parent_mem_cgroup(memcg); if (!parent) parent = root_mem_cgroup; + memcg_deactivate_kmem_caches(memcg, parent); + + kmemcg_id = memcg->kmemcg_id; + BUG_ON(kmemcg_id < 0); + /* * Change kmemcg_id of this cgroup and all its descendants to the * parent's id, and then move all entries from this cgroup's list_lrus @@ -3261,7 +3261,6 @@ static void memcg_free_kmem(struct mem_cgroup *memcg) if (memcg->kmem_state == KMEM_ALLOCATED) { WARN_ON(!list_empty(&memcg->kmem_caches)); static_branch_dec(&memcg_kmem_enabled_key); - WARN_ON(page_counter_read(&memcg->kmem)); } } #else @@ -4673,6 +4672,9 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) /* The following stuff does not apply to the root */ if (!parent) { +#ifdef CONFIG_MEMCG_KMEM + INIT_LIST_HEAD(&memcg->kmem_caches); +#endif root_mem_cgroup = memcg; return &memcg->css; } diff --git a/mm/slab.h b/mm/slab.h index 2acc68a7e0a0..acdc1810639d 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -264,10 +264,11 @@ static __always_inline int memcg_charge_slab(struct page *page, struct lruvec *lruvec; int ret; - memcg = s->memcg_params.memcg; + rcu_read_lock(); + memcg = rcu_dereference(s->memcg_params.memcg); ret = memcg_kmem_charge_memcg(page, gfp, order, memcg); if (ret) - return ret; + goto out; lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); mod_lruvec_state(lruvec, cache_vmstat_idx(s), 1 << order); @@ -275,8 +276,9 @@ static __always_inline int memcg_charge_slab(struct page *page, /* transer try_charge() page references to kmem_cache */ percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); css_put_many(&memcg->css, 1 << order); - - return 0; +out: + rcu_read_unlock(); + return ret; } static __always_inline void memcg_uncharge_slab(struct page *page, int order, @@ -285,10 +287,12 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, struct mem_cgroup *memcg; struct lruvec *lruvec; - memcg = s->memcg_params.memcg; + rcu_read_lock(); + memcg = rcu_dereference(s->memcg_params.memcg); lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order)); memcg_kmem_uncharge_memcg(page, order, memcg); + rcu_read_unlock(); percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); } diff --git a/mm/slab_common.c b/mm/slab_common.c index 995920222127..36673a43ed31 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -236,7 +236,7 @@ void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) list_add(&s->root_caches_node, &slab_root_caches); } else { css_get(&memcg->css); - s->memcg_params.memcg = memcg; + rcu_assign_pointer(s->memcg_params.memcg, memcg); list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); list_add(&s->memcg_params.kmem_caches_node, @@ -251,7 +251,8 @@ static void memcg_unlink_cache(struct kmem_cache *s) } else { list_del(&s->memcg_params.children_node); list_del(&s->memcg_params.kmem_caches_node); - css_put(&s->memcg_params.memcg->css); + mem_cgroup_put(rcu_dereference_protected(s->memcg_params.memcg, + lockdep_is_held(&slab_mutex))); } } #else @@ -772,11 +773,13 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s) call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); } -void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) +void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg, + struct mem_cgroup *parent) { int idx; struct memcg_cache_array *arr; struct kmem_cache *s, *c; + unsigned int nr_reparented; idx = memcg_cache_id(memcg); @@ -794,6 +797,20 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) kmemcg_cache_deactivate(c); arr->entries[idx] = NULL; } + if (memcg != parent) { + nr_reparented = 0; + list_for_each_entry(s, &memcg->kmem_caches, + memcg_params.kmem_caches_node) { + rcu_assign_pointer(s->memcg_params.memcg, parent); + css_put(&memcg->css); + nr_reparented++; + } + if (nr_reparented) { + list_splice_init(&memcg->kmem_caches, + &parent->kmem_caches); + css_get_many(&parent->css, nr_reparented); + } + } mutex_unlock(&slab_mutex); put_online_mems(); From patchwork Wed May 8 20:24:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10936491 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A3F79924 for ; Wed, 8 May 2019 20:41:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 95BBA289A5 for ; Wed, 8 May 2019 20:41:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8987B289AA; Wed, 8 May 2019 20:41:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 153E3289A5 for ; Wed, 8 May 2019 20:41:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B0686B000C; Wed, 8 May 2019 16:40:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 37B336B0010; Wed, 8 May 2019 16:40:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0C266B000D; Wed, 8 May 2019 16:40:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id A5E386B000C for ; Wed, 8 May 2019 16:40:53 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id a97so86089pla.9 for ; Wed, 08 May 2019 13:40:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=M3mL2Ve/kfONZivBQEHTZBNx9Zf4Go/ST+iIz38LOH4=; b=qWD+GngjRb0s2S3T3yzNayWd+5iBer9EMNC5+fFFRRtjuqToKAmJ1l+nYu6gKvD+R6 osoPzidW/ijhCEhcvKSOWiWOzNTe2LhkEv76byK0I5Jib1q82eNPeXnraFwcshVjY+BS 29WGGwl/CNUtRpInzTFMay8WOw6B3ROAZTHcWxYzyIhH+/uEErn6KXKeUzCBs1JMU+uA tc4k1QGnadFBuxpmwgyCKY1wdDB5nABZXrVAvPlUTOtKdbZ2dbncLudYR3vqRsMKMkoJ unmlZjdcmd6s6Ph7j8ZvMp7BFMxXxod1MgasPtHV3Id2KPuOnSGpAvbg3MiyLrrefOvf 9Tug== X-Gm-Message-State: APjAAAUgy2znQvrRKDnWBaZya9k/tmr3JRDKiVjvDKLtEfd6XmSYzCl5 Dq/8RpRhfsh7upmcWRBRg3KobCkC5rjdYXqBHjPEeCyHigbFhMGPsxM1ZnoXActADuBvU0qR1jn zpr4euijkqXUf4KW64ayNihJWopNa+My6ynvYSjjAY5l4ExVenQmsqRomOVPrHyclPw== X-Received: by 2002:a17:902:364:: with SMTP id 91mr49277203pld.72.1557348053319; Wed, 08 May 2019 13:40:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqzodJJczLtqFpiV6Fiy/uSkm8cA7m83+SIsNTp2Aod350w1P9TinySZcVTBR/7wd0qLO6sz X-Received: by 2002:a17:902:364:: with SMTP id 91mr49277136pld.72.1557348052530; Wed, 08 May 2019 13:40:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557348052; cv=none; d=google.com; s=arc-20160816; b=lwlIxw0awEMy/OzSRsapV8pgO1vIdk9RmqdNPn6moEI0MIVviyMzg8qandMemrfP4q lXP9ln3o6hOQ5vcMJcotZPc57qEaNUjPyAVoarUKNRiT0Xw4J+lwAxUEN6oVRAhKyOqh HLmRATX9UFhHEB48LW2luvOCrPefIg+duieEcbVGyPUziRWWE+0YwYOpVI+5Dv15ycef S0BqVFYfbR4SgYL179Zpp4Q6UwAU346AuBg/yfLxLW0zR4KN7kOOhkJK4/O69ppL/CaK qKDjf1YcypL/O4uuXXxwUzCms4rUSLoE5y4onMLiMrZvSRiskQCwKXgUW/hB+PsuPKZD jk6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=M3mL2Ve/kfONZivBQEHTZBNx9Zf4Go/ST+iIz38LOH4=; b=NhZwxLwDGKA8b8VUQMfDrGQsvGUrN0O9pQ1+Ullwh5UJR2Tb70uxeFjcZBZOfnSyJB faMCDJGym4a6DFWW9JYCBYY5S65kEe5Y522JlmQzePf1BB063RnhXJKGUlSQTIMq1zfo Dc835iUJMPlwGl9XEXzdM1pwvN8gmnyYjnEfX7ThL0nysEqglzNCvX3QEg11uLHARhmu zY2VDGCbURw42G2HW3pBbnhbpVrzTladJbuaV8j3iWWxsr72QCAT93z/3e0CHcrwAoek 7QRyHVA+R0YAhgIPWHt0lsX00OrdaVkxssCDgU7JP3LkDAacw7jjYp0vudSfABgsKFdi m93Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=q0QmNMek; spf=pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0031b2e447=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id q86si24499985pfi.197.2019.05.08.13.40.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 May 2019 13:40:52 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=q0QmNMek; spf=pass (google.com: domain of prvs=0031b2e447=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0031b2e447=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x48KeVEC023320 for ; Wed, 8 May 2019 13:40:51 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=M3mL2Ve/kfONZivBQEHTZBNx9Zf4Go/ST+iIz38LOH4=; b=q0QmNMek/9n6tqCcO+epeVgDd/whCZ15XILswuvC6FgxcQiz5zBckYB/zurQXID5IzkE 6nefIA2S9VUCRGzEvo6CoNnm+BUX7E68nCmIaJgU2t++Rh5c1W4xCo4nS/Bkik1Mwkdl t8cmZt8JtbJ2IBe6Eq+/C7ytksYoclIyOiA= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2sc0p91d9f-10 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 08 May 2019 13:40:51 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::130) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Wed, 8 May 2019 13:40:48 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 1CDB811CDBD04; Wed, 8 May 2019 13:25:00 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Shakeel Butt CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Christoph Lameter , Vladimir Davydov , , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v3 7/7] mm: fix /proc/kpagecgroup interface for slab pages Date: Wed, 8 May 2019 13:24:58 -0700 Message-ID: <20190508202458.550808-8-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190508202458.550808-1-guro@fb.com> References: <20190508202458.550808-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-08_11:,, signatures=0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Switching to an indirect scheme of getting mem_cgroup pointer for !root slab pages broke /proc/kpagecgroup interface for them. Let's fix it by learning page_cgroup_ino() how to get memcg pointer for slab pages. Reported-by: Shakeel Butt Signed-off-by: Roman Gushchin --- mm/memcontrol.c | 5 ++++- mm/slab.h | 21 +++++++++++++++++++++ mm/slab_common.c | 1 + 3 files changed, 26 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6e4d9ed16069..8114838759f6 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -494,7 +494,10 @@ ino_t page_cgroup_ino(struct page *page) unsigned long ino = 0; rcu_read_lock(); - memcg = READ_ONCE(page->mem_cgroup); + if (PageSlab(page)) + memcg = memcg_from_slab_page(page); + else + memcg = READ_ONCE(page->mem_cgroup); while (memcg && !(memcg->css.flags & CSS_ONLINE)) memcg = parent_mem_cgroup(memcg); if (memcg) diff --git a/mm/slab.h b/mm/slab.h index acdc1810639d..cb684fbe2cc2 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -256,6 +256,22 @@ static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) return s->memcg_params.root_cache; } +static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) +{ + struct kmem_cache *s; + + WARN_ON_ONCE(!rcu_read_lock_held()); + + if (PageTail(page)) + page = compound_head(page); + + s = READ_ONCE(page->slab_cache); + if (s && !is_root_cache(s)) + return rcu_dereference(s->memcg_params.memcg); + + return NULL; +} + static __always_inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) @@ -338,6 +354,11 @@ static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) return s; } +static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) +{ + return NULL; +} + static inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { diff --git a/mm/slab_common.c b/mm/slab_common.c index 36673a43ed31..0cfdad0a0aac 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -253,6 +253,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) list_del(&s->memcg_params.kmem_caches_node); mem_cgroup_put(rcu_dereference_protected(s->memcg_params.memcg, lockdep_is_held(&slab_mutex))); + rcu_assign_pointer(s->memcg_params.memcg, NULL); } } #else