From patchwork Wed Jun 5 02:44:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10976279 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 66CB813AD for ; Wed, 5 Jun 2019 02:45:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 56A8727FAC for ; Wed, 5 Jun 2019 02:45:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4AD4A28420; Wed, 5 Jun 2019 02:45:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AEBF727FAC for ; Wed, 5 Jun 2019 02:45:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 364906B0270; Tue, 4 Jun 2019 22:45:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0F1576B0276; Tue, 4 Jun 2019 22:45:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB65B6B0273; Tue, 4 Jun 2019 22:45:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id A94A66B026D for ; Tue, 4 Jun 2019 22:45:03 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id a5so15232027pla.3 for ; Tue, 04 Jun 2019 19:45:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=nRQ8E2xsdPMn6A3dTdY86dEXrfQWn/mxew8g7sZDO2Y=; b=IyxKWlQLsemCGKZbAxB/8ywU96iJ8rmOHAjRqDYdVdriaT1w1ekEhP8c15tFvU8jGr lQmMQzcvZvGFzD6fCdfb8TwPmo3qV80hIuLxhBiLoYWvedyU45I+b8KN5LEzVG/49eSv D6CAyy1FUZUV5CyA3NxI4d+pqJ6qT2W/Epicr6StL8XVLSO/q8uRflRaDtQu5U2MWsHD D0iMFSzoX1lKshJtlGVhw7F8ieU0TZWLyvbqriY+QUZCvAg3tSX3aTZ0uk44ZRpe5VBn 726XiF9GxmUyEOQ5II/apE4aJhnduSNhpswEDqgZ9G/l8v04E7CGuheCExwQ219CY+TU 19xg== X-Gm-Message-State: APjAAAVhXAK8ThPpD5LrYAf6ZjfYk5NG2cAgg+vZTlGJKQ2pSkQwQ09U JOTlCgUhAhGtSev2XUfSJvg8VLWtMJ+mUf5FWWvFxEMQK68fS02JMA1VhiI3NfuuULoCLwZwfXN KvAsC4RhHo6u2Ed2S9ird6wyc8joLYp6pOumXWcex86mu0C45xbxppzZASWIUggEHJg== X-Received: by 2002:a63:de4b:: with SMTP id y11mr920603pgi.301.1559702703156; Tue, 04 Jun 2019 19:45:03 -0700 (PDT) X-Google-Smtp-Source: APXvYqwFtMlKnYj+OXb9mj/lm85VoJqE4AVaEZ7U38uJ3o6a4G8UnLpvgsufmC3n11LBh391juMo X-Received: by 2002:a63:de4b:: with SMTP id y11mr920560pgi.301.1559702702336; Tue, 04 Jun 2019 19:45:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559702702; cv=none; d=google.com; s=arc-20160816; b=Jdu5mD8PiZneRUJ+3iMjPpwPhGjaR2S5QAxL93wlw53x5veCPcJ6PlqHyDmcdhqimv gcz/GF4HAVDMDZtXmzptJO3VtlVfKOKNWI15cq55Azto4HzajtFXS2OQWi/1NnNMxqn/ XzQBUQ83vm3L64C6SR2qK61RBRmxv2x8e0nSSmxFdKZJHqPudwIaRHawdn9YSiIkkjDV eAT+9eKklEh+DSZFXWw0uWM5yCOLM3srVpFyYG6e8R/1ybQ7zKFG5/cejw1AvT4ugYog EsnUl5+oUuBBM2Ohm3DzGHneD+IF7ucfb+ISPtGlYDZ7Ur+lw+Tyv6LudBtohj6YHtB3 LgwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=nRQ8E2xsdPMn6A3dTdY86dEXrfQWn/mxew8g7sZDO2Y=; b=THagxa5LAPlpUa+9WwPzxPICr9i7GC8zBRZW4w+eSCHco7q06YosKvKiYPZY/vmsbD dyVt0AsfRvDAx++mzDKfsS2Lnlw9NuKBS+G8GCPXvals4WYQf4/vviL5ZquKu43qq2SF 8VW3kFfgg4x59PeLbAsTPCpE3iVYZyF9kuUM7aE9D9SxMDpluMG2X5qpn329He5D6moT GE0Lq9oyvLA3rDkgM8o/BDb7gFhi8lCLu8v+XgcW5aFv0JxIfrgTL6h4mNyTKYH9a3zi TqLWxDwizlwkaw03M2Dx9Y8+5GBZVMAPiu9ayeqXkY8cgCvUKEZl7fMJT08tukSlYHO3 EN2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=ZH2TXFP5; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id i8si23338646pgs.220.2019.06.04.19.45.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 19:45:02 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=ZH2TXFP5; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x552j059031912 for ; Tue, 4 Jun 2019 19:45:01 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=nRQ8E2xsdPMn6A3dTdY86dEXrfQWn/mxew8g7sZDO2Y=; b=ZH2TXFP5HplCQcFb8fKTo705zDaomejANIkeVDEfi+eT9bkz/E/mo6EKWMQJWSm56Wop t6J0TxVTU+IuYGg522NC14setthkTEunGANNRnhXr+tpOQCz+p96U+h/61T3qcq7ATFn yhr6xk7Vm4ohJ6zLtKK9Z41q5ZPwiRPycHg= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2swx191ck2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 04 Jun 2019 19:45:01 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 4 Jun 2019 19:44:59 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 7928012C7FDC2; Tue, 4 Jun 2019 19:44:58 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v6 01/10] mm: add missing smp read barrier on getting memcg kmem_cache pointer Date: Tue, 4 Jun 2019 19:44:45 -0700 Message-ID: <20190605024454.1393507-2-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190605024454.1393507-1-guro@fb.com> References: <20190605024454.1393507-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-05_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906050015 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Johannes noticed that reading the memcg kmem_cache pointer in cache_from_memcg_idx() is performed using READ_ONCE() macro, which doesn't implement a SMP barrier, which is required by the logic. Add a proper smp_rmb() to be paired with smp_wmb() in memcg_create_kmem_cache(). The same applies to memcg_create_kmem_cache() itself, which reads the same value without barriers and READ_ONCE(). Suggested-by: Johannes Weiner Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/slab.h | 1 + mm/slab_common.c | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/slab.h b/mm/slab.h index 739099af6cbb..1176b61bb8fc 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -260,6 +260,7 @@ cache_from_memcg_idx(struct kmem_cache *s, int idx) * memcg_caches issues a write barrier to match this (see * memcg_create_kmem_cache()). */ + smp_rmb(); cachep = READ_ONCE(arr->entries[idx]); rcu_read_unlock(); diff --git a/mm/slab_common.c b/mm/slab_common.c index 58251ba63e4a..8092bdfc05d5 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -652,7 +652,8 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg, * allocation (see memcg_kmem_get_cache()), several threads can try to * create the same cache, but only one of them may succeed. */ - if (arr->entries[idx]) + smp_rmb(); + if (READ_ONCE(arr->entries[idx])) goto out_unlock; cgroup_name(css->cgroup, memcg_name_buf, sizeof(memcg_name_buf)); From patchwork Wed Jun 5 02:44:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10976269 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7673814B6 for ; Wed, 5 Jun 2019 02:45:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 64D952877B for ; Wed, 5 Jun 2019 02:45:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 593D428782; Wed, 5 Jun 2019 02:45:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BEC3A2877B for ; Wed, 5 Jun 2019 02:45:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E46886B0266; Tue, 4 Jun 2019 22:45:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C381B6B026D; Tue, 4 Jun 2019 22:45:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A14E56B026A; Tue, 4 Jun 2019 22:45:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 630306B0266 for ; Tue, 4 Jun 2019 22:45:02 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id y7so14244397pfy.9 for ; Tue, 04 Jun 2019 19:45:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=53XDDT3c53a/gFK121iyd3PvVTI7RPL9leWIgYOSFQQ=; b=H0HrtCiGqAKlFTl6GAI79MttFRhPvBqRZpITxAjF9f9IQYrRcEMaNATXpqfe6jQ0hp 5h8QRRjUVAVkwtFo11SGOxiQZRPrPqEcdjQOZcSRtXsieYnlgcIp/xo6jgPmGsOMaDzZ BGESzQPO/2WuZBZ9Bc7WDGzDtLiPaFiy7+j8vRnwRflGYPep2+CyvWmsOBEbBk+gZ1/U cAHFnYTve2CYwOtoobl7jbrM2z0qeeOmPhhmLjlFZhZOvbb3sBRlqYWVs/h9gMri7uOV rG9Kqfkk/ctQT8GPBClk7QrTYddL6vS7uAHYDkMfMBuF9+HLlxhcdzcWdLZm0XVlhI0m x2qA== X-Gm-Message-State: APjAAAXsiXoZpfABwB6TkQKU9UilNA7qnjvZJ55fEi4TIXljX/Wm7Uye zLbZEqSjTECkUJBiPze10Xc0O7aWHJz/0RKmsfxJNYvpjWKhGiVVZhRw/k9yUSkLIIozwAAK/pA 2W4GaoYjuuvQeQCxn+0G/DnLMqUuX2DdhBwU5uXOIJwkRZGE+FdrOkukRrsMJdPfD9Q== X-Received: by 2002:a17:902:4e:: with SMTP id 72mr41031850pla.80.1559702701796; Tue, 04 Jun 2019 19:45:01 -0700 (PDT) X-Google-Smtp-Source: APXvYqywXLWSxG1IQwNIjHbyTqQAuFffLCx1Kt4zHCz4Dza3mXmdQRyo6+2RbDmo+UNyY2mJEfe6 X-Received: by 2002:a17:902:4e:: with SMTP id 72mr41031796pla.80.1559702700572; Tue, 04 Jun 2019 19:45:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559702700; cv=none; d=google.com; s=arc-20160816; b=B0nFidKV4F/5hn384eQ0NyvZZs/TITpqZ6as1ib3Zx4Ou5klozs8lCycPWbCWBl5sh ig+qCM27GZJ/yA4ImyT94yP33tDti3Og2Qcq3Ju1JeOYEj1UTSjGxsalQ7KAHTCnxakk ehEeEkf8vueR6mIKJjHveSi1qOMfLSFDAS/OzHyXaGaWC3qi7zSlHtSIp6geEj769tdM 5rEA4eWQaP7T6nu73DVegal6Q8sLE4wEjllnOx5d9SLinmxe4XDw/4jI5ew6Rnh5+yUB f2CmP4BSk+HSm8dhtjobMJUHnIwjWnXHc9iq8WXU1Me2BCSx7Pw5QKY8n3t1XuKh/kyN ZCgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=53XDDT3c53a/gFK121iyd3PvVTI7RPL9leWIgYOSFQQ=; b=B35hjNVWAxexxenpOLXp/wg5Jdb8WBXcWAZ/skQ+jaC/R2/nErgqs9pg2uTSWuGnMd WNSfQ8XMrEUDDHdgFJFEnwsW8REYOqbuJEA8Ago+Vig1yxpZ8kEaLpK3o5GWsmgxc4h2 noroEtLy9DgH9mZtNUxK6WsGguQndxJAQnvD5TkHQrFYHejaxb7U73vNoydp831JN3FJ AuTHzw+T6NUQcU9jPEisbt628ORXZJFIaJbIHod4qfpj0PDrTYYmgB4+9/Uk3jyhmaFk RmiK2lEYOYm/F9Ze0Z7qVV93LonjRe9HudW1+Ut1HXfQPsqu4ulBI8KggtkhDD5SS5oW 2jmg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=d61Hm7uf; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id g95si25532010pje.41.2019.06.04.19.45.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 19:45:00 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=d61Hm7uf; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044008.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x552iAes001296 for ; Tue, 4 Jun 2019 19:45:00 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=53XDDT3c53a/gFK121iyd3PvVTI7RPL9leWIgYOSFQQ=; b=d61Hm7ufr0aAMhFrupy0vRGQZna0IgnPrswvu9MJSbyokT+710j+Ey5+hyHsLCv4RITQ 1vnoO0jPWyaKMilPry7ZQvuhlEOsJ7MzeqtWubdb1RecTe/cLJ/08+dfnvwEBSdxJxzy 5WKIsXeyeQMWhYxhX8IXYs+i5zhhSbsYUlk= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2swq1tjy3n-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 04 Jun 2019 19:44:59 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::125) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 4 Jun 2019 19:44:58 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 7B24512C7FDC4; Tue, 4 Jun 2019 19:44:58 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v6 02/10] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache() Date: Tue, 4 Jun 2019 19:44:46 -0700 Message-ID: <20190605024454.1393507-3-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190605024454.1393507-1-guro@fb.com> References: <20190605024454.1393507-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-05_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906050015 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Initialize kmem_cache->memcg_params.memcg pointer in memcg_link_cache() rather than in init_memcg_params(). Once kmem_cache will hold a reference to the memory cgroup, it will simplify the refcounting. For non-root kmem_caches memcg_link_cache() is always called before the kmem_cache becomes visible to a user, so it's safe. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Vladimir Davydov Acked-by: Johannes Weiner --- mm/slab.c | 2 +- mm/slab.h | 5 +++-- mm/slab_common.c | 14 +++++++------- mm/slub.c | 2 +- 4 files changed, 12 insertions(+), 11 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 9e3eee5568b6..a4091f8b3655 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1239,7 +1239,7 @@ void __init kmem_cache_init(void) nr_node_ids * sizeof(struct kmem_cache_node *), SLAB_HWCACHE_ALIGN, 0, 0); list_add(&kmem_cache->list, &slab_caches); - memcg_link_cache(kmem_cache); + memcg_link_cache(kmem_cache, NULL); slab_state = PARTIAL; /* diff --git a/mm/slab.h b/mm/slab.h index 1176b61bb8fc..c16e5af0fb59 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -290,7 +290,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, } extern void slab_init_memcg_params(struct kmem_cache *); -extern void memcg_link_cache(struct kmem_cache *s); +extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, void (*deact_fn)(struct kmem_cache *)); @@ -345,7 +345,8 @@ static inline void slab_init_memcg_params(struct kmem_cache *s) { } -static inline void memcg_link_cache(struct kmem_cache *s) +static inline void memcg_link_cache(struct kmem_cache *s, + struct mem_cgroup *memcg) { } diff --git a/mm/slab_common.c b/mm/slab_common.c index 8092bdfc05d5..77df6029de8e 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -140,13 +140,12 @@ void slab_init_memcg_params(struct kmem_cache *s) } static int init_memcg_params(struct kmem_cache *s, - struct mem_cgroup *memcg, struct kmem_cache *root_cache) + struct kmem_cache *root_cache) { struct memcg_cache_array *arr; if (root_cache) { s->memcg_params.root_cache = root_cache; - s->memcg_params.memcg = memcg; INIT_LIST_HEAD(&s->memcg_params.children_node); INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node); return 0; @@ -221,11 +220,12 @@ int memcg_update_all_caches(int num_memcgs) return ret; } -void memcg_link_cache(struct kmem_cache *s) +void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) { if (is_root_cache(s)) { list_add(&s->root_caches_node, &slab_root_caches); } else { + s->memcg_params.memcg = memcg; list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); list_add(&s->memcg_params.kmem_caches_node, @@ -244,7 +244,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) } #else static inline int init_memcg_params(struct kmem_cache *s, - struct mem_cgroup *memcg, struct kmem_cache *root_cache) + struct kmem_cache *root_cache) { return 0; } @@ -384,7 +384,7 @@ static struct kmem_cache *create_cache(const char *name, s->useroffset = useroffset; s->usersize = usersize; - err = init_memcg_params(s, memcg, root_cache); + err = init_memcg_params(s, root_cache); if (err) goto out_free_cache; @@ -394,7 +394,7 @@ static struct kmem_cache *create_cache(const char *name, s->refcount = 1; list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, memcg); out: if (err) return ERR_PTR(err); @@ -998,7 +998,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, create_boot_cache(s, name, size, flags, useroffset, usersize); list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, NULL); s->refcount = 1; return s; } diff --git a/mm/slub.c b/mm/slub.c index 1802c87799ff..9cb2eef62a37 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4213,7 +4213,7 @@ static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache) } slab_init_memcg_params(s); list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, NULL); return s; } From patchwork Wed Jun 5 02:44:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10976285 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 53B4413AD for ; Wed, 5 Jun 2019 02:45:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3ECA127FAC for ; Wed, 5 Jun 2019 02:45:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2F0BF28420; Wed, 5 Jun 2019 02:45:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 856E627FAC for ; Wed, 5 Jun 2019 02:45:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC7C66B0276; Tue, 4 Jun 2019 22:45:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B4F406B0277; Tue, 4 Jun 2019 22:45:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A405F6B0278; Tue, 4 Jun 2019 22:45:10 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f71.google.com (mail-yw1-f71.google.com [209.85.161.71]) by kanga.kvack.org (Postfix) with ESMTP id 6C6836B0276 for ; Tue, 4 Jun 2019 22:45:10 -0400 (EDT) Received: by mail-yw1-f71.google.com with SMTP id w127so21464926ywe.6 for ; Tue, 04 Jun 2019 19:45:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=F5q0Ous8JMckUZTK0P//WSX11GdeHQH8mlJbfSlHdWk=; b=sOPrx0y1t2UkRaFkO8GDMmFXOVzd8J7vSPHuAj6lOVciAItas3pXYEnA0KA2ZIMRUD eM/KsWCBevvDtQ5FzrQ7+qVfL2ySHGwbgEgZfTZHmG+xzqSigEU2fn3BGqsrVHc/8QRv 3usjDZOpSrr/zym422CTeWtGEg3Dtw8TIAItlN+xc4Rti3HtStiVig+2sEHvv5cXkZjL 8gOWXoa9bunF/T2ezkltLi5zefaI5wNvu2tTuGPI3PaxxJBZxnpFvRuKAjuNvDz9Fbpd ZTNjzC5sZCE4lgFyqygCT86dCP1AMN4m7Y6bdL9voBCaSe3mpWRLBx0Gvn2ac6URdEom cY8w== X-Gm-Message-State: APjAAAVldtCM+iL/wcLlamFVWsksYeg3myrEr+qINpFLAI1WyeL6hzLa 92ktvFLyExLWGmySMbBxbfqDKVHrd+jnPxpxPezz3HJ8otwhxvCdsJaQsPi8Gb8v4kb8G4epNh3 IkrgjBnH6n0Civ3j9ulw3zuAG3+MqbUdbm/x8aDsByNYFBZkT/fKdzYvh6dMmPz6vww== X-Received: by 2002:a81:195:: with SMTP id 143mr3001040ywb.147.1559702710183; Tue, 04 Jun 2019 19:45:10 -0700 (PDT) X-Google-Smtp-Source: APXvYqwYljq+4AdaxqZA6y/1sabcbDQdD9x8cA22hMKMG8hZnypL5xCeXsRqFbMrC8SzcF2F+SMB X-Received: by 2002:a81:195:: with SMTP id 143mr3000798ywb.147.1559702701793; Tue, 04 Jun 2019 19:45:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559702701; cv=none; d=google.com; s=arc-20160816; b=xNjcOF1ShBT7ffVx3n8wHN2LMmbQFSUYawOwFKrCfeDpr0MolCjtOy0XsFJ8l48mPA 0kTrswGzKf1/HeCJz7hYfGd4AIWFQQcxG90mh5Tte4z0T3hFFrz+RMPGoY7nBtJWDH+v 7I15M+cwDhsDcRRQ0+1deR7EtOAVE+AzDL0w2QfNrNco3DHPKC3d4BcV7xmE/9fKBCA6 NrmV10EPAVvifzyG5E2hqL2epXwGCZ+481olWtGhHhdCqo4xzJERVuYdfPmLK3gxKpwZ 2VHQFEurhgHD95Dq4VrmsPZe2NIdBU13Dpw+/e0zzGOv/0xvQCzEmJwZIWBdZI23gWKc jS7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=F5q0Ous8JMckUZTK0P//WSX11GdeHQH8mlJbfSlHdWk=; b=Rm0064ydAI8MBZnwh91nUTcoypYf4llHV7sOOfNtrSDf76+azA6LeL5Xa6NoGkrJA/ P4BddJQFNJOCsEz3I1aXDMaLi0utpv0Be09SzpAUBh3j+DxS1Lq8OGwNmV3d867zl/si QK0QLh4AYEgu8Z9F1tsBptKcQ0znw8ouqPwthCYC/QU17wxkVchu9zhWFY9sEN/tWJzb 1ImK4em9D4X9l63Ibk4gZeCpsXIzb05iUjSZlwnGcy45t28WQhvPvK1eo6dIcwx7FCi4 29w0xoorWGbCHJvuRoUwXQvUAEQiPVnT8PiPZtdGX+5KlnnlbQkqkeEP46hfGRwPjynG mYqQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=rGwlcvcn; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id q186si5644616ybq.473.2019.06.04.19.45.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 19:45:01 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=rGwlcvcn; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x552hZuX009253 for ; Tue, 4 Jun 2019 19:45:01 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=F5q0Ous8JMckUZTK0P//WSX11GdeHQH8mlJbfSlHdWk=; b=rGwlcvcnL/0ke9glhflkD/OtcJdh+z/2LjCHptAG7F8zR31PfCGh0rUOenA1vqf1cWBS 8M9UDjRc+tWJqZUSzyZU29ji1mlBYCu7A4DKB0wTNDS+9HHBMnOGksYaLVfq4nJpmkx2 70iuLK1pb9QrdvTjyUSoLU9r4H5GH2KYWBg= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2sx0j78sds-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 04 Jun 2019 19:45:01 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::c) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 4 Jun 2019 19:44:59 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 7FB4112C7FDC6; Tue, 4 Jun 2019 19:44:58 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v6 03/10] mm: rename slab delayed deactivation functions and fields Date: Tue, 4 Jun 2019 19:44:47 -0700 Message-ID: <20190605024454.1393507-4-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190605024454.1393507-1-guro@fb.com> References: <20190605024454.1393507-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-05_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906050015 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The delayed work/rcu deactivation infrastructure of non-root kmem_caches can be also used for asynchronous release of these objects. Let's get rid of the word "deactivation" in corresponding names to make the code look better after generalization. It's easier to make the renaming first, so that the generalized code will look consistent from scratch. Let's rename struct memcg_cache_params fields: deact_fn -> work_fn deact_rcu_head -> rcu_head deact_work -> work And RCU/delayed work callbacks in slab common code: kmemcg_deactivate_rcufn -> kmemcg_rcufn kmemcg_deactivate_workfn -> kmemcg_workfn This patch contains no functional changes, only renamings. Signed-off-by: Roman Gushchin Acked-by: Vladimir Davydov --- include/linux/slab.h | 6 +++--- mm/slab.h | 2 +- mm/slab_common.c | 30 +++++++++++++++--------------- 3 files changed, 19 insertions(+), 19 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 9449b19c5f10..47923c173f30 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -642,10 +642,10 @@ struct memcg_cache_params { struct list_head children_node; struct list_head kmem_caches_node; - void (*deact_fn)(struct kmem_cache *); + void (*work_fn)(struct kmem_cache *); union { - struct rcu_head deact_rcu_head; - struct work_struct deact_work; + struct rcu_head rcu_head; + struct work_struct work; }; }; }; diff --git a/mm/slab.h b/mm/slab.h index c16e5af0fb59..8ff90f42548a 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -292,7 +292,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, extern void slab_init_memcg_params(struct kmem_cache *); extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*deact_fn)(struct kmem_cache *)); + void (*work_fn)(struct kmem_cache *)); #else /* CONFIG_MEMCG_KMEM */ diff --git a/mm/slab_common.c b/mm/slab_common.c index 77df6029de8e..d019ee66bdc4 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -692,17 +692,17 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg, put_online_cpus(); } -static void kmemcg_deactivate_workfn(struct work_struct *work) +static void kmemcg_workfn(struct work_struct *work) { struct kmem_cache *s = container_of(work, struct kmem_cache, - memcg_params.deact_work); + memcg_params.work); get_online_cpus(); get_online_mems(); mutex_lock(&slab_mutex); - s->memcg_params.deact_fn(s); + s->memcg_params.work_fn(s); mutex_unlock(&slab_mutex); @@ -713,36 +713,36 @@ static void kmemcg_deactivate_workfn(struct work_struct *work) css_put(&s->memcg_params.memcg->css); } -static void kmemcg_deactivate_rcufn(struct rcu_head *head) +static void kmemcg_rcufn(struct rcu_head *head) { struct kmem_cache *s = container_of(head, struct kmem_cache, - memcg_params.deact_rcu_head); + memcg_params.rcu_head); /* - * We need to grab blocking locks. Bounce to ->deact_work. The + * We need to grab blocking locks. Bounce to ->work. The * work item shares the space with the RCU head and can't be * initialized eariler. */ - INIT_WORK(&s->memcg_params.deact_work, kmemcg_deactivate_workfn); - queue_work(memcg_kmem_cache_wq, &s->memcg_params.deact_work); + INIT_WORK(&s->memcg_params.work, kmemcg_workfn); + queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); } /** * slab_deactivate_memcg_cache_rcu_sched - schedule deactivation after a * sched RCU grace period * @s: target kmem_cache - * @deact_fn: deactivation function to call + * @work_fn: deactivation function to call * - * Schedule @deact_fn to be invoked with online cpus, mems and slab_mutex + * Schedule @work_fn to be invoked with online cpus, mems and slab_mutex * held after a sched RCU grace period. The slab is guaranteed to stay - * alive until @deact_fn is finished. This is to be used from + * alive until @work_fn is finished. This is to be used from * __kmemcg_cache_deactivate(). */ void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*deact_fn)(struct kmem_cache *)) + void (*work_fn)(struct kmem_cache *)) { if (WARN_ON_ONCE(is_root_cache(s)) || - WARN_ON_ONCE(s->memcg_params.deact_fn)) + WARN_ON_ONCE(s->memcg_params.work_fn)) return; if (s->memcg_params.root_cache->memcg_params.dying) @@ -751,8 +751,8 @@ void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, /* pin memcg so that @s doesn't get destroyed in the middle */ css_get(&s->memcg_params.memcg->css); - s->memcg_params.deact_fn = deact_fn; - call_rcu(&s->memcg_params.deact_rcu_head, kmemcg_deactivate_rcufn); + s->memcg_params.work_fn = work_fn; + call_rcu(&s->memcg_params.rcu_head, kmemcg_rcufn); } void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) From patchwork Wed Jun 5 02:44:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10976271 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3879713AD for ; Wed, 5 Jun 2019 02:45:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 288BD2877B for ; Wed, 5 Jun 2019 02:45:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1CE5F28782; Wed, 5 Jun 2019 02:45:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6AC3E2877B for ; Wed, 5 Jun 2019 02:45:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1A84D6B0269; Tue, 4 Jun 2019 22:45:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 07B0B6B026C; Tue, 4 Jun 2019 22:45:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD49C6B026F; Tue, 4 Jun 2019 22:45:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f199.google.com (mail-yb1-f199.google.com [209.85.219.199]) by kanga.kvack.org (Postfix) with ESMTP id A96E56B0266 for ; Tue, 4 Jun 2019 22:45:02 -0400 (EDT) Received: by mail-yb1-f199.google.com with SMTP id r142so13485057ybc.0 for ; Tue, 04 Jun 2019 19:45:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=Wv6PULr08vn6aMDCf+MGkZnKQqchIV9NxQ/kg/T87Dk=; b=WSIjup7FOcTC+Tmg2uQ2FW/C0OwKwpY4Tk3/5Pi9zBkm8oflEdQCYiXccHzVrsEOt+ LuznMPTzALwuXrQiJEIhXslJCkke8sXfSirdzK6jrCPylTjcCJ2sbXIoXpK1MuEW4A7f keOuzWfFrpXql0bv7eSLdA0UqvYiwfT9XRscMABonldf93URTDnKZfzSOzwUwdv4AJ0O mIsQVKe6ur3eEtXGSgoYL9Qw3uJogrdBt4Q0+YtKutdQz49TeoeT+0DMZf2zXDAxU58c 4YsxHutaHlvQSmxBwfjhoawSkM74hmZylfR5TmG5r7rEtfDVy1xlMuZ3XVppk1x9rhmg zKGQ== X-Gm-Message-State: APjAAAWxzaJ+SF6zHh/EpF09eP5E/WsPeWOxBbF6PflVzXc6NxxxhB2o 9uac7lSO2jZkBJdWZay5isiPj6944su/3TsRhQ/POc90tBuBswbuhPCiLBkvj56NnZnX3NVDVZD TzXCnsHjBgRDbPPYymfttqDa/0DhHh1W5XcUPWSC+fYB7gPa4rA4l1MQRVWaNL5PKhw== X-Received: by 2002:a25:7712:: with SMTP id s18mr16798462ybc.263.1559702702382; Tue, 04 Jun 2019 19:45:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqzVHgSK3pAWuth2y0YUg7NEIFi83mVc0Vat/UV4kTJDC8YKoYgZS2+QPYXn+lGIIoNgcXlS X-Received: by 2002:a25:7712:: with SMTP id s18mr16798443ybc.263.1559702701526; Tue, 04 Jun 2019 19:45:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559702701; cv=none; d=google.com; s=arc-20160816; b=iiTOjFCuSUiuDB2jkRcRRz3/5nYI7zQ6N+AgnKevESm678ASOjOiA2KEKA2dF7F/z5 dlPc64wuB7fJmAIH7eW64xjP64pbOwnbZMumDa5aNS4IR/ncJ4OLiHTzPO2o61GA5Ihx xjp0Ch2NhULvNh8i/f5Twy62QLFi09Ae6+hGFkh4Sib7wkVAE05e2f/eCNnawy/Kqdbd LuOs7Ts8K8t89fnmkSAD9Ob7Xpd80NWgfWINj6IVtLZQ318XutRnxzIY9VDQtwyE+LbH 7B4qvOdYB4Hjwb7Eflxt303PsLxtb3xFmKullu5skrl8ZOsagZcsDoP3qskkXYm7wBKQ mPzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=Wv6PULr08vn6aMDCf+MGkZnKQqchIV9NxQ/kg/T87Dk=; b=UzJ8VXKUmef4goUbGlEeLmnDLVOKQf++5HkLO6LEeOnjjYkq23FlbD+ARFPJ7brWyj 7HOI2/8I8ShPnLyxw1EX4poedwK1G1dmczWgFBdnl+4HJ6A2a0ac2K5HcEQ+jLXEIw43 gDHm4ehhnUD2tQRYKxeeFudnx6k0o0bE5vkVjrp5/V31U88MkfUdlb27qY5f1lFz6C8a 9tZVjXKi38JPy61sBoHqAS7etC4gXqNK/z5ZKuYX+nr0n98+dxP+rC1UwWPqkdsis/X3 OR7YE4Nfy0EmQMQCqG4MMkhYf67T5K+BFfD+k/9+aFFMaApjvWMJvDcAP5CvBkfSzKGQ /Nzg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="V/rfOhOR"; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id v12si5275432ybm.290.2019.06.04.19.45.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 19:45:01 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="V/rfOhOR"; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x552hgSe023104 for ; Tue, 4 Jun 2019 19:45:01 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=Wv6PULr08vn6aMDCf+MGkZnKQqchIV9NxQ/kg/T87Dk=; b=V/rfOhOR1cl30AcZIOb+eG4ipiYWLHfyERFMHlNPGsiZBivqkgrS4m8TeWLxuBcEBrhb bn0jJ8hPEKWkfBLiKHf+V2jlKxFpIYFrpbU4y2/Bv/+V/1ZOoQLbdiPZLFx0WnL4pKKA +zFdPCbi0Pbd17HY3nQV4JDVBeGyKiU2+n4= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2sx41n84vs-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 04 Jun 2019 19:45:01 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 4 Jun 2019 19:44:59 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 83CC012C7FDC8; Tue, 4 Jun 2019 19:44:58 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v6 04/10] mm: generalize postponed non-root kmem_cache deactivation Date: Tue, 4 Jun 2019 19:44:48 -0700 Message-ID: <20190605024454.1393507-5-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190605024454.1393507-1-guro@fb.com> References: <20190605024454.1393507-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-05_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906050015 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently SLUB uses a work scheduled after an RCU grace period to deactivate a non-root kmem_cache. This mechanism can be reused for kmem_caches release, but requires generalization for SLAB case. Introduce kmemcg_cache_deactivate() function, which calls allocator-specific __kmem_cache_deactivate() and schedules execution of __kmem_cache_deactivate_after_rcu() with all necessary locks in a worker context after an rcu grace period. Here is the new calling scheme: kmemcg_cache_deactivate() __kmemcg_cache_deactivate() SLAB/SLUB-specific kmemcg_rcufn() rcu kmemcg_workfn() work __kmemcg_cache_deactivate_after_rcu() SLAB/SLUB-specific instead of: __kmemcg_cache_deactivate() SLAB/SLUB-specific slab_deactivate_memcg_cache_rcu_sched() SLUB-only kmemcg_rcufn() rcu kmemcg_workfn() work kmemcg_cache_deact_after_rcu() SLUB-only For consistency, all allocator-specific functions start with "__". Signed-off-by: Roman Gushchin Acked-by: Vladimir Davydov --- mm/slab.c | 4 ++++ mm/slab.h | 3 +-- mm/slab_common.c | 27 ++++++++------------------- mm/slub.c | 8 +------- 4 files changed, 14 insertions(+), 28 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index a4091f8b3655..4b865393ebb4 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -2252,6 +2252,10 @@ void __kmemcg_cache_deactivate(struct kmem_cache *cachep) { __kmem_cache_shrink(cachep); } + +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) +{ +} #endif int __kmem_cache_shutdown(struct kmem_cache *cachep) diff --git a/mm/slab.h b/mm/slab.h index 8ff90f42548a..d35d85794247 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -172,6 +172,7 @@ int __kmem_cache_shutdown(struct kmem_cache *); void __kmem_cache_release(struct kmem_cache *); int __kmem_cache_shrink(struct kmem_cache *); void __kmemcg_cache_deactivate(struct kmem_cache *s); +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s); void slab_kmem_cache_release(struct kmem_cache *); struct seq_file; @@ -291,8 +292,6 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, extern void slab_init_memcg_params(struct kmem_cache *); extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); -extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*work_fn)(struct kmem_cache *)); #else /* CONFIG_MEMCG_KMEM */ diff --git a/mm/slab_common.c b/mm/slab_common.c index d019ee66bdc4..09b26673b63f 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -709,7 +709,7 @@ static void kmemcg_workfn(struct work_struct *work) put_online_mems(); put_online_cpus(); - /* done, put the ref from slab_deactivate_memcg_cache_rcu_sched() */ + /* done, put the ref from kmemcg_cache_deactivate() */ css_put(&s->memcg_params.memcg->css); } @@ -727,31 +727,21 @@ static void kmemcg_rcufn(struct rcu_head *head) queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); } -/** - * slab_deactivate_memcg_cache_rcu_sched - schedule deactivation after a - * sched RCU grace period - * @s: target kmem_cache - * @work_fn: deactivation function to call - * - * Schedule @work_fn to be invoked with online cpus, mems and slab_mutex - * held after a sched RCU grace period. The slab is guaranteed to stay - * alive until @work_fn is finished. This is to be used from - * __kmemcg_cache_deactivate(). - */ -void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*work_fn)(struct kmem_cache *)) +static void kmemcg_cache_deactivate(struct kmem_cache *s) { if (WARN_ON_ONCE(is_root_cache(s)) || WARN_ON_ONCE(s->memcg_params.work_fn)) return; + __kmemcg_cache_deactivate(s); + if (s->memcg_params.root_cache->memcg_params.dying) return; /* pin memcg so that @s doesn't get destroyed in the middle */ css_get(&s->memcg_params.memcg->css); - s->memcg_params.work_fn = work_fn; + s->memcg_params.work_fn = __kmemcg_cache_deactivate_after_rcu; call_rcu(&s->memcg_params.rcu_head, kmemcg_rcufn); } @@ -774,7 +764,7 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) if (!c) continue; - __kmemcg_cache_deactivate(c); + kmemcg_cache_deactivate(c); arr->entries[idx] = NULL; } mutex_unlock(&slab_mutex); @@ -867,11 +857,10 @@ static void flush_memcg_workqueue(struct kmem_cache *s) mutex_unlock(&slab_mutex); /* - * SLUB deactivates the kmem_caches through call_rcu. Make + * SLAB and SLUB deactivate the kmem_caches through call_rcu. Make * sure all registered rcu callbacks have been invoked. */ - if (IS_ENABLED(CONFIG_SLUB)) - rcu_barrier(); + rcu_barrier(); /* * SLAB and SLUB create memcg kmem_caches through workqueue and SLUB diff --git a/mm/slub.c b/mm/slub.c index 9cb2eef62a37..ae3b1e49ecec 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4022,7 +4022,7 @@ int __kmem_cache_shrink(struct kmem_cache *s) } #ifdef CONFIG_MEMCG -static void kmemcg_cache_deact_after_rcu(struct kmem_cache *s) +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { /* * Called with all the locks held after a sched RCU grace period. @@ -4048,12 +4048,6 @@ void __kmemcg_cache_deactivate(struct kmem_cache *s) */ slub_set_cpu_partial(s, 0); s->min_partial = 0; - - /* - * s->cpu_partial is checked locklessly (see put_cpu_partial), so - * we have to make sure the change is visible before shrinking. - */ - slab_deactivate_memcg_cache_rcu_sched(s, kmemcg_cache_deact_after_rcu); } #endif /* CONFIG_MEMCG */ From patchwork Wed Jun 5 02:44:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10976277 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 400E614B6 for ; Wed, 5 Jun 2019 02:45:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2FDED27FAC for ; Wed, 5 Jun 2019 02:45:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 23D2028420; Wed, 5 Jun 2019 02:45:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 99B4D27FAC for ; Wed, 5 Jun 2019 02:45:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0AE676B026D; Tue, 4 Jun 2019 22:45:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 003C56B0270; Tue, 4 Jun 2019 22:45:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE6A16B0274; Tue, 4 Jun 2019 22:45:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f199.google.com (mail-yb1-f199.google.com [209.85.219.199]) by kanga.kvack.org (Postfix) with ESMTP id B1D896B0270 for ; Tue, 4 Jun 2019 22:45:03 -0400 (EDT) Received: by mail-yb1-f199.google.com with SMTP id v6so18985181ybs.1 for ; Tue, 04 Jun 2019 19:45:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=kSMcIoZOy8bv6oPhIhSwMshcDK0XomRQ4uXD2a50jbU=; b=RneP9FLrfpb3wLstDrMbhiIoM2SIqmFsuS081dTLvuoEPKrHzxNzfn7Kmle3ZoPMey QVGD3SWCaaef0r7xwrM8YNgNX1taQTodgEYWmkLVZUIGeF+aS3B10o9J3ds8arc9BK9O 6fdLhoCzV3poZHqii3/cAkLGlk9BWiELpgeExgKS8Xk9eRy89wuVClTWao7DPuml6HYm mcE82mk4WuSHrWJIuMR5Gqg63s5fwFsiJg3BCE/nv7Kg7dtPd3ZRCZDsEaBV3Mf26B1r s/u23Be0NA5UGbOnPmI6jUsjqLAQbDE6/lDPRS3V4JkysRjJRAqWS2AAaSEI22enCllT Ujsw== X-Gm-Message-State: APjAAAWunJjHZiGU8vVZ+fnPm/Mvg+QW2TfNfFtCVOgq7DoQd569rmCd 8PrqRDRRmNON985UUUdi+DslRosM1sOMxR520EeKcKdVCdR/vpyewmMb/KHMELCXNubSMn8XspY foliXAMnVhro87JnAxexv8jg+upgqSXNxm33svf3wur8Ad+y32TVVDWBvnPj/LlnUbg== X-Received: by 2002:a5b:b4e:: with SMTP id b14mr14382624ybr.74.1559702703467; Tue, 04 Jun 2019 19:45:03 -0700 (PDT) X-Google-Smtp-Source: APXvYqyZfS1X6Ksyr75/1VSNGeYUaJEi4/pR7dYUPXi1AOp44KREi2o4M4kZZ9RM5KeD576oyDFV X-Received: by 2002:a5b:b4e:: with SMTP id b14mr14382615ybr.74.1559702702936; Tue, 04 Jun 2019 19:45:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559702702; cv=none; d=google.com; s=arc-20160816; b=EaSvLhgXwl5iBHsptBPPqUTjKTYEOm+DJvLAXjTh7Qcj+SOGuiQ2TkDCpryW1C5vye +7YDsdxMwvx/Gtg4UrGF3zbYI/9tFlCuhMe3UBVkwfbHo1n92Ao9vrGd2i44CIsWyaOI rO1v+VpAZRnWINy4s3KEPna+kZGV0m3PhLIqe7En828Hrj3GT7UQuxHUbAGLVMuegxnm F2YIaqwSBJY/PAhpKrxqztN/mux3QJomf/sFW2amxBR2LwCZ2EXICguDRClJy5UawI9i MyIPF86eH7AU/EsakkqhokOHNDbJdt1OwmSiLeW/U63sUrdHstut8DNZh4HXccbYxAWY noGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=kSMcIoZOy8bv6oPhIhSwMshcDK0XomRQ4uXD2a50jbU=; b=Ik8Kj1CRmBn7lAGDeKT/Lbzc1IDjZkKrYv9DVUpZbTTRBfl5sM/9uZpq2WypL7l+Tf LNTl/hE8i0tl/twsX1eKFQ1Eijk0SuSSUZ70V2OahipXW5Ym74xFi+9jxdTK3ciaqMFy pcAusT6YETag3AkJzXs6vZpKXrGyGMTHiMFTZGuj4AGQmQB8RGjD8R2+kvPP4kF/vw8E TjsmRXYlt8+Wt99qBwDXz66Ka199Pe7MfvbtFNtU1wIL1bIcMRoJxmfhL232atZWLK69 Sl7QgXdYZuy9nIClwvWrKLgEDa5bL4j7QoiJH0plUpy0T2gr80l/t36qEBuAecgzQyk0 IHUA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=oAw7uVY1; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id r1si5160361ybb.366.2019.06.04.19.45.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 19:45:02 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=oAw7uVY1; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x552i3Xw023200 for ; Tue, 4 Jun 2019 19:45:02 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=kSMcIoZOy8bv6oPhIhSwMshcDK0XomRQ4uXD2a50jbU=; b=oAw7uVY12LZ2xbms08qwfD8NE4cyI/1+y+8BI8DKv6UEG/gLTJUBKxEGxa2NWIu4HGq5 HRTcpwgTAToVDVUjrfe4gFCStwtKmP8QsTVU3TJ2F1LCaJ0JvFMjHb4OibE7hXY7EYfD ye9dcGmDdcu4Y7DbND0nfS9F49dQ+GIxU5Y= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2sx41n84vt-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 04 Jun 2019 19:45:02 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 4 Jun 2019 19:45:00 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 87DB812C7FDCA; Tue, 4 Jun 2019 19:44:58 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v6 05/10] mm: introduce __memcg_kmem_uncharge_memcg() Date: Tue, 4 Jun 2019 19:44:49 -0700 Message-ID: <20190605024454.1393507-6-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190605024454.1393507-1-guro@fb.com> References: <20190605024454.1393507-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-05_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=899 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906050015 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Let's separate the page counter modification code out of __memcg_kmem_uncharge() in a way similar to what __memcg_kmem_charge() and __memcg_kmem_charge_memcg() work. This will allow to reuse this code later using a new memcg_kmem_uncharge_memcg() wrapper, which calls __memcg_kmem_uncharge_memcg() if memcg_kmem_enabled() check is passed. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Vladimir Davydov --- include/linux/memcontrol.h | 10 ++++++++++ mm/memcontrol.c | 25 +++++++++++++++++-------- 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 3ca57bacfdd2..9abf31bbe53a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1304,6 +1304,8 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge(struct page *page, int order); int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, struct mem_cgroup *memcg); +void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg, + unsigned int nr_pages); extern struct static_key_false memcg_kmem_enabled_key; extern struct workqueue_struct *memcg_kmem_cache_wq; @@ -1345,6 +1347,14 @@ static inline int memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, return __memcg_kmem_charge_memcg(page, gfp, order, memcg); return 0; } + +static inline void memcg_kmem_uncharge_memcg(struct page *page, int order, + struct mem_cgroup *memcg) +{ + if (memcg_kmem_enabled()) + __memcg_kmem_uncharge_memcg(memcg, 1 << order); +} + /* * helper for accessing a memcg's index. It will be used as an index in the * child cache array in kmem_cache, and also to derive its name. This function diff --git a/mm/memcontrol.c b/mm/memcontrol.c index bdb66871cdec..3427396da612 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2731,6 +2731,22 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order) css_put(&memcg->css); return ret; } + +/** + * __memcg_kmem_uncharge_memcg: uncharge a kmem page + * @memcg: memcg to uncharge + * @nr_pages: number of pages to uncharge + */ +void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg, + unsigned int nr_pages) +{ + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + page_counter_uncharge(&memcg->kmem, nr_pages); + + page_counter_uncharge(&memcg->memory, nr_pages); + if (do_memsw_account()) + page_counter_uncharge(&memcg->memsw, nr_pages); +} /** * __memcg_kmem_uncharge: uncharge a kmem page * @page: page to uncharge @@ -2745,14 +2761,7 @@ void __memcg_kmem_uncharge(struct page *page, int order) return; VM_BUG_ON_PAGE(mem_cgroup_is_root(memcg), page); - - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) - page_counter_uncharge(&memcg->kmem, nr_pages); - - page_counter_uncharge(&memcg->memory, nr_pages); - if (do_memsw_account()) - page_counter_uncharge(&memcg->memsw, nr_pages); - + __memcg_kmem_uncharge_memcg(memcg, nr_pages); page->mem_cgroup = NULL; /* slab pages do not have PageKmemcg flag set */ From patchwork Wed Jun 5 02:44:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10976281 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9561714B6 for ; Wed, 5 Jun 2019 02:45:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8495827FAC for ; Wed, 5 Jun 2019 02:45:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 78A0128420; Wed, 5 Jun 2019 02:45:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C68EB27FAC for ; Wed, 5 Jun 2019 02:45:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E72D6B0273; Tue, 4 Jun 2019 22:45:06 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 474E76B0274; Tue, 4 Jun 2019 22:45:06 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3146A6B0277; Tue, 4 Jun 2019 22:45:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id D69F06B0273 for ; Tue, 4 Jun 2019 22:45:05 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id j36so13771318pgb.20 for ; Tue, 04 Jun 2019 19:45:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=oKYQ1N3o2byC0i/b69CpGwU+m8Lkbxo7avMVOowYKjc=; b=EzSLBcli36aat1JJ5gbfDgM/bHNsqf4VdyiCu8guDk6bulpKv1aDiZc6myEU3V/7G8 sMeAvBEcfftxhvJAaL96Y2PFq0X+p5HsCCd24y7PoEQLC8IfishfvV6moeZT+MyHMUHN Ru9QwDSdRGySV1VgYbIz74Xwc4p5xe4yzyoVERos41/uofcVBKUBPWdt6pAeGeO8zmRo o9vPxd3h31c8xyM5NILsOI6D8H59MlgjfVGG4NoeulZjjL22YSInuCZ44HezVa4/VQZJ //saQlRS15B/r6Zk4cVx7C0piJCF4ZCDhfE6hlwWYhD7zFaxbM5Rxtm5MnY/Hla+MBhW z80A== X-Gm-Message-State: APjAAAXZep/W99UsuQQW3n5kUNTOrS+zLVVWT44X6aYM6LVJ0XD4Qu0o wZ8o5olgIyxKZuPzPnUwAIS9kM0iFVfHmFZus2yTcn1sk2J/mCfOrMwGTPUEezibwjXoRQi5kD6 WoaJXFHYePjqQ5207F6gaY85IymO0o3+5vWTJwBHN9eXzg0Q97ht8YiXlBdO+rpv0wQ== X-Received: by 2002:aa7:8c10:: with SMTP id c16mr40980233pfd.89.1559702705262; Tue, 04 Jun 2019 19:45:05 -0700 (PDT) X-Google-Smtp-Source: APXvYqxdC5OB3n313hXuEp2yLHANHnsvI1C19CyWWs7HbTmI6gVHONwa9LFHKTb6LBvOplvTplOB X-Received: by 2002:aa7:8c10:: with SMTP id c16mr40980165pfd.89.1559702704214; Tue, 04 Jun 2019 19:45:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559702704; cv=none; d=google.com; s=arc-20160816; b=uMnfufF7efhMtbSlQM202wqHDNJhYdipP3yzbqFSEXITMlRVQ2JHdJ1YX66aw9IZsm cQUGTcZoNes866da/Ar74octSWIKDYN6zDSdc/yL38pFQXD+FUil6LjPs3zzum/Omfe+ 6HY6CtovpxxL+ISGp3qYz0+U7DX6EWBciwrMIBg/4Z9hVyrwni4I7DwEhGW0xorfAlSF u27VC7HgtiLgHBB+LKHO2W5Zl7S8fvuJMsjQ6Psf4ks+kf4fmFBcH+U1SS4jeiYobIej 9sphjJth6YC1LOJ/VPdyFJlZ/NCRH6ebw3L557/MRkCbxHuOsLCocVckdoteOPXn15pg qqDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=oKYQ1N3o2byC0i/b69CpGwU+m8Lkbxo7avMVOowYKjc=; b=SR7+sGzYRAc7Z7qdspBVDCVKejQKC8mi1qK/g0hH7cV8sSnyznaLurRxCQ+N06+25X vusplJSFtCoM0tgd6b8ifZtRsvPEdpbsx+I5N4vuCrJma+8cxwDtisRd7eqwY7T3GNYp 9D2AH1D57NwC0rzAJitCSCEsmQ1OzcbxRvpVMnxHWuDa4V5CHU+7utq1pfK4qSuACsNF CHAcpeotQSBUgqiYcwYOvfVsWMOPT98S1RuA6cG4oz1wmVp1GRSFxlYT5FcJhz8FzY4n 2d/L8nxCdf44oBHvSqEmUyjx+GZX3Gkm5vY1EaICd/QSGlHgZJMzFGGHiIP58uGEeoK/ KU4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=VOPrQ0JA; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id 2si26218293plc.228.2019.06.04.19.45.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 19:45:04 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=VOPrQ0JA; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044008.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x552iAP6001299 for ; Tue, 4 Jun 2019 19:45:03 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=oKYQ1N3o2byC0i/b69CpGwU+m8Lkbxo7avMVOowYKjc=; b=VOPrQ0JAg/nra9uRdO5LxJgcMWkDfH3HsL+8GNCOyljMjWWb+C2yk+MorV/9zb3xGOAL 7tN5UbvmTPnm5qvU2F/JS/9cmffrnu/PfF1YXdYQiVzSnxQEQ86jRc+BlsvuiP66dkFp noFLBbFEQ3fwQ+DTun+q7mTraroNGCyvOls= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2swq1tjy3t-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 04 Jun 2019 19:45:03 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 4 Jun 2019 19:44:59 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 8BB2312C7FDCC; Tue, 4 Jun 2019 19:44:58 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v6 06/10] mm: unify SLAB and SLUB page accounting Date: Tue, 4 Jun 2019 19:44:50 -0700 Message-ID: <20190605024454.1393507-7-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190605024454.1393507-1-guro@fb.com> References: <20190605024454.1393507-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-05_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906050015 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently the page accounting code is duplicated in SLAB and SLUB internals. Let's move it into new (un)charge_slab_page helpers in the slab_common.c file. These helpers will be responsible for statistics (global and memcg-aware) and memcg charging. So they are replacing direct memcg_(un)charge_slab() calls. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Christoph Lameter Acked-by: Vladimir Davydov Acked-by: Johannes Weiner --- mm/slab.c | 19 +++---------------- mm/slab.h | 25 +++++++++++++++++++++++++ mm/slub.c | 14 ++------------ 3 files changed, 30 insertions(+), 28 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 4b865393ebb4..b417824a9b15 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1360,7 +1360,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, int nodeid) { struct page *page; - int nr_pages; flags |= cachep->allocflags; @@ -1370,17 +1369,11 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, return NULL; } - if (memcg_charge_slab(page, flags, cachep->gfporder, cachep)) { + if (charge_slab_page(page, flags, cachep->gfporder, cachep)) { __free_pages(page, cachep->gfporder); return NULL; } - nr_pages = (1 << cachep->gfporder); - if (cachep->flags & SLAB_RECLAIM_ACCOUNT) - mod_lruvec_page_state(page, NR_SLAB_RECLAIMABLE, nr_pages); - else - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, nr_pages); - __SetPageSlab(page); /* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks() && page_is_pfmemalloc(page)) @@ -1395,12 +1388,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, static void kmem_freepages(struct kmem_cache *cachep, struct page *page) { int order = cachep->gfporder; - unsigned long nr_freed = (1 << order); - - if (cachep->flags & SLAB_RECLAIM_ACCOUNT) - mod_lruvec_page_state(page, NR_SLAB_RECLAIMABLE, -nr_freed); - else - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, -nr_freed); BUG_ON(!PageSlab(page)); __ClearPageSlabPfmemalloc(page); @@ -1409,8 +1396,8 @@ static void kmem_freepages(struct kmem_cache *cachep, struct page *page) page->mapping = NULL; if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += nr_freed; - memcg_uncharge_slab(page, order, cachep); + current->reclaim_state->reclaimed_slab += 1 << order; + uncharge_slab_page(page, order, cachep); __free_pages(page, order); } diff --git a/mm/slab.h b/mm/slab.h index d35d85794247..345fb59afb8f 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -205,6 +205,12 @@ ssize_t slabinfo_write(struct file *file, const char __user *buffer, void __kmem_cache_free_bulk(struct kmem_cache *, size_t, void **); int __kmem_cache_alloc_bulk(struct kmem_cache *, gfp_t, size_t, void **); +static inline int cache_vmstat_idx(struct kmem_cache *s) +{ + return (s->flags & SLAB_RECLAIM_ACCOUNT) ? + NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE; +} + #ifdef CONFIG_MEMCG_KMEM /* List of all root caches. */ @@ -362,6 +368,25 @@ static inline struct kmem_cache *virt_to_cache(const void *obj) return page->slab_cache; } +static __always_inline int charge_slab_page(struct page *page, + gfp_t gfp, int order, + struct kmem_cache *s) +{ + int ret = memcg_charge_slab(page, gfp, order, s); + + if (!ret) + mod_lruvec_page_state(page, cache_vmstat_idx(s), 1 << order); + + return ret; +} + +static __always_inline void uncharge_slab_page(struct page *page, int order, + struct kmem_cache *s) +{ + mod_lruvec_page_state(page, cache_vmstat_idx(s), -(1 << order)); + memcg_uncharge_slab(page, order, s); +} + static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) { struct kmem_cache *cachep; diff --git a/mm/slub.c b/mm/slub.c index ae3b1e49ecec..6a5174b51cd6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1488,7 +1488,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s, else page = __alloc_pages_node(node, flags, order); - if (page && memcg_charge_slab(page, flags, order, s)) { + if (page && charge_slab_page(page, flags, order, s)) { __free_pages(page, order); page = NULL; } @@ -1681,11 +1681,6 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) if (!page) return NULL; - mod_lruvec_page_state(page, - (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE, - 1 << oo_order(oo)); - inc_slabs_node(s, page_to_nid(page), page->objects); return page; @@ -1719,18 +1714,13 @@ static void __free_slab(struct kmem_cache *s, struct page *page) check_object(s, page, p, SLUB_RED_INACTIVE); } - mod_lruvec_page_state(page, - (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE, - -pages); - __ClearPageSlabPfmemalloc(page); __ClearPageSlab(page); page->mapping = NULL; if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages; - memcg_uncharge_slab(page, order, s); + uncharge_slab_page(page, order, s); __free_pages(page, order); } From patchwork Wed Jun 5 02:44:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10976275 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 49B8D14B6 for ; Wed, 5 Jun 2019 02:45:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2C77227FAC for ; Wed, 5 Jun 2019 02:45:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1F99528420; Wed, 5 Jun 2019 02:45:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9DA2D27FAC for ; Wed, 5 Jun 2019 02:45:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8BAB66B026C; Tue, 4 Jun 2019 22:45:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 81FBF6B0273; Tue, 4 Jun 2019 22:45:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A9146B026C; Tue, 4 Jun 2019 22:45:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 099D36B026F for ; Tue, 4 Jun 2019 22:45:03 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id f1so17652532pfb.0 for ; Tue, 04 Jun 2019 19:45:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=RAtQP8kdt+P/ff/UupI77hj0Bc++zM2YWzYBwAXPM6o=; b=oiO7sHPhVVXZZ1VgRQWwVsW2EsUgkRhhGfN7HvAqSp2Nj3RtXYqoEYFX3Bbwg9kOld F4xinnGjaIf1q3EXfwsdZ9X64erkYkzsjiPg5U4ZiS7MXFaiAtMEwS+IuPByCKrNaIWR v7wYK6z4YQ01K285DGccN1gq0110ynAzxEO0IfxfJA/0euoL5ct/VuvpfRJMkX7lImHi 5Khrq1BqnmBlsygSnkXvJhvZoCgCGptNaYgSVacns3sZ28dXcX1zstR+ZY/M8yb9qrM8 wmQjaDZ1kAMuc6PEQ/J8A/B5a8fCPLBWQNatJqcW5kw6viU2Y+olbImiAnss4+C8xDX8 MGuw== X-Gm-Message-State: APjAAAWd3KYsF1uVssvZNl65QQmvsYhkgJpaEDQ8khnoieJEHzLsBsz+ w4+vvK46FLdM6o9INsdqitxu3bvr+dV1UNEcnAZx8sDI7qdbRZYsBOnoP5CpcGUBT2m+lcBrMyx tjwX2hNtiwkpRUSqarICrchcBqGAfox9QgiFb15INkjZeRMMadh38SZQILs9TrD5D5w== X-Received: by 2002:a17:90a:2430:: with SMTP id h45mr20692505pje.14.1559702702500; Tue, 04 Jun 2019 19:45:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqxYyM8hOd559tYZfftEdIgrdCLUgEYngBp5A4sgeEdrOZap2vQAwY+SuXuwZzzOYLf+NCie X-Received: by 2002:a17:90a:2430:: with SMTP id h45mr20692438pje.14.1559702701355; Tue, 04 Jun 2019 19:45:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559702701; cv=none; d=google.com; s=arc-20160816; b=H+rJaIHf2HTW90kKFViyLaK1B4UsKQV2F4KlQlq6TimQDW3bi7tkLmHA7wytoVCdFk TFzdf4ywqhqUkbuAKlahpRMXx404L6TqMkYLuL+s05797ZURjdIZGxqpNpilnIezfMbz jhIBdm21xJQ+B82H4iZXc1tVZV2qoF302br5GbDno7JFENVEbkkt0JAWTBj50TLy1lPE 8N0+U0fr/3WJhfbUsE2g8sHgvTTN6qyaLEN9Z2pEVRIW9YVGzM5pukzvv+p2GIu/6kOf xwLia2Xt73hRDmCMqNnVvs2ij0mXbrdkjpJZseuEzPnLn7HeymT0ixwHNgfZ1muQZwTp JvDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=RAtQP8kdt+P/ff/UupI77hj0Bc++zM2YWzYBwAXPM6o=; b=bEXZOeyF8sYM56cn3RLhAeAsUc33fT7FXa/ol1nQplo/CEUPGKqdzPzsQzmS9YRKwS VMMNdGi2Cwhs61fGRSa7OWebp7ckfbVOEKWgjsfm+i/O8wSRbkTBqEtAmGp3RRBgDNBa rWYyOW84iFuCotToUMA4WjfyP/Ak7lQ9mpndQAd/qXmtR5vDV8eU3A7C3hlYxmHX1ikY V6LQnSQyeICLUMYYyA1vacMyyv5OSou+/fJ1loF9j7ZFRC5E2q1aTgOzZl2RLMd1zPvw lm0kbO+D+B8Z16FTHOAx9jfhf6tz3VYcT6t5wp/A+/C6JgTkrZxJqacoD3kosLmupgJO Oz6g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=N1ORyrtC; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id a14si22826742pgm.206.2019.06.04.19.45.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 19:45:01 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=N1ORyrtC; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x552hPgg011960 for ; Tue, 4 Jun 2019 19:45:00 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=RAtQP8kdt+P/ff/UupI77hj0Bc++zM2YWzYBwAXPM6o=; b=N1ORyrtCMpqM8ArTqyLILrOwr/rNYmMHMW9zOhG/NRyrlupsDadIRbH+Gt2zpNdLeTHX 6UchQ5coq6wyjYdFSjdBnNAWvDe9iDo/95rKavvBdoQXJ/GeLh+EHLugjQxPcLLLU5BB wHHKtT2bs17nxlMXfdM9vV2V0bZEmZnb0uA= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2sx00b8y22-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 04 Jun 2019 19:45:00 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::126) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 4 Jun 2019 19:44:59 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 8FE2012C7FDCE; Tue, 4 Jun 2019 19:44:58 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v6 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock Date: Tue, 4 Jun 2019 19:44:51 -0700 Message-ID: <20190605024454.1393507-8-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190605024454.1393507-1-guro@fb.com> References: <20190605024454.1393507-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-05_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=755 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906050015 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently the memcg_params.dying flag and the corresponding workqueue used for the asynchronous deactivation of kmem_caches is synchronized using the slab_mutex. It makes impossible to check this flag from the irq context, which will be required in order to implement asynchronous release of kmem_caches. So let's switch over to the irq-save flavor of the spinlock-based synchronization. Signed-off-by: Roman Gushchin --- mm/slab_common.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 09b26673b63f..2914a8f0aa85 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -130,6 +130,7 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr, #ifdef CONFIG_MEMCG_KMEM LIST_HEAD(slab_root_caches); +static DEFINE_SPINLOCK(memcg_kmem_wq_lock); void slab_init_memcg_params(struct kmem_cache *s) { @@ -629,6 +630,7 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg, struct memcg_cache_array *arr; struct kmem_cache *s = NULL; char *cache_name; + bool dying; int idx; get_online_cpus(); @@ -640,7 +642,13 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg, * The memory cgroup could have been offlined while the cache * creation work was pending. */ - if (memcg->kmem_state != KMEM_ONLINE || root_cache->memcg_params.dying) + if (memcg->kmem_state != KMEM_ONLINE) + goto out_unlock; + + spin_lock_irq(&memcg_kmem_wq_lock); + dying = root_cache->memcg_params.dying; + spin_unlock_irq(&memcg_kmem_wq_lock); + if (dying) goto out_unlock; idx = memcg_cache_id(memcg); @@ -735,14 +743,17 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s) __kmemcg_cache_deactivate(s); + spin_lock_irq(&memcg_kmem_wq_lock); if (s->memcg_params.root_cache->memcg_params.dying) - return; + goto unlock; /* pin memcg so that @s doesn't get destroyed in the middle */ css_get(&s->memcg_params.memcg->css); s->memcg_params.work_fn = __kmemcg_cache_deactivate_after_rcu; call_rcu(&s->memcg_params.rcu_head, kmemcg_rcufn); +unlock: + spin_unlock_irq(&memcg_kmem_wq_lock); } void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) @@ -852,9 +863,9 @@ static int shutdown_memcg_caches(struct kmem_cache *s) static void flush_memcg_workqueue(struct kmem_cache *s) { - mutex_lock(&slab_mutex); + spin_lock_irq(&memcg_kmem_wq_lock); s->memcg_params.dying = true; - mutex_unlock(&slab_mutex); + spin_unlock_irq(&memcg_kmem_wq_lock); /* * SLAB and SLUB deactivate the kmem_caches through call_rcu. Make From patchwork Wed Jun 5 02:44:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10976273 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9DF5B13AD for ; Wed, 5 Jun 2019 02:45:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8C6652877B for ; Wed, 5 Jun 2019 02:45:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 80CE128782; Wed, 5 Jun 2019 02:45:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 602272877B for ; Wed, 5 Jun 2019 02:45:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5AC426B026F; Tue, 4 Jun 2019 22:45:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 581F86B0270; Tue, 4 Jun 2019 22:45:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3650C6B0273; Tue, 4 Jun 2019 22:45:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f71.google.com (mail-yw1-f71.google.com [209.85.161.71]) by kanga.kvack.org (Postfix) with ESMTP id 0C11A6B0270 for ; Tue, 4 Jun 2019 22:45:03 -0400 (EDT) Received: by mail-yw1-f71.google.com with SMTP id v205so5739710ywb.11 for ; Tue, 04 Jun 2019 19:45:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=pBwyrsC3xS2PmE3Fb9BbKTWT0q0/P4Hp2VaXNFe4hs0=; b=IaKToa/RohHGbQnWssfoVKSXOr3VZV7GWFBd5k/s5cVRpgUIUPTb5X5uu05w6H5WA6 WZKUtyPViuG5ae7FlfuOg8nh9UyFgH3ojvBYt0P/r+gr66XnutofnlrrE7/MbAyMcBUg 9kYY6dztPI4bxKt/QFDUrlFKZ6U4nsmm/3fIcDSV/aADVNT1swCAEibdahGyVm5DWf5O ToB8eoW0kK3exfHAWHHWALinfl+XoA7nOigN3rltXPMQfJo9IxxdcVfGr+dyWmrz4t9D RCNcNteXMFRPs+KnTXHU6GN6740Z8NcoTQnEKRClqySNhJBdgH9ddlfPHAedgTyjem7n g2fg== X-Gm-Message-State: APjAAAUzi8ytEiZn1lo971Aavlsx2vtFi456WVmA44otGfrPzPaueNNC Mpv2RlyEuc464NJ3jtHKaHGpQMBqyhFZjM6ArtPMupV9dfCxIAHXnf7Ii/ykI7BgdA3euwCgiMF 1hWqOHkeM7QKyN15nprH9Kr5fUO1w0ka98d2HcPU9Yccuma4xVCXMf5xGo5WP6Lz2tA== X-Received: by 2002:a81:30c9:: with SMTP id w192mr10434761yww.371.1559702702744; Tue, 04 Jun 2019 19:45:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqyLhf6C8AF+dn4M3gYpLRFswQMLZFVQoKVt49+9hC1KoStIYtr7ckprSpZZa7Up0SDUdOqF X-Received: by 2002:a81:30c9:: with SMTP id w192mr10434728yww.371.1559702701458; Tue, 04 Jun 2019 19:45:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559702701; cv=none; d=google.com; s=arc-20160816; b=J1kGX/AQac6pIjTmA+BgtU+7LRO1m9Kt/qKkP6HHQ8Ph8thhKBfljx1wfNyAGe22PH NDTtJnQjThVKmshLd+7JavxDlFkoKS3Obg0rkl+akk6WAMCbWbAgHMWdMvLzdIAPQyJm VaF5g3Bk15BxcH01CgfhF8OHzD55YQtB/iY5GsM7lS9YMLcmzicY9VHjFOCP60DDKbtk D9K+6sr8SALf3e/XMrBHsjYc0z759opZ5Yg0mv319ziWjimeieXCeD8CcN3cpgO8nNmT vJypToxY28MhWjiUD0kDbEnCNJZNBHAfvuOC9q62wbDU7HjrWiKy9TXDYGdzxUJbCs/x orlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=pBwyrsC3xS2PmE3Fb9BbKTWT0q0/P4Hp2VaXNFe4hs0=; b=sTrEdzDbyQbuHucU3CQNbsxLddZcWtBMsEVAqvCxt0FeS1DrKGOciiwyvR7IRbDIxN 09xYjlq14kdiSzQsKok3hKIpFnUhoVpC08idoJK09KbSXssKzv/1l/5XF8LmcqdCupLJ AiwuPsDc9JCF2qiWUG6lox8VfkWuKItDmdW322DlnK08tm4i8OM2zWvN3qWFQ43C6iRJ gHDKh9FthHTFbu6JRqpoycjEOuHzMWlQD2w3SZafCnQzppZMt4uSDLjc9y3XR3NJlZaL JL64eahKNEuT48XJljY+AEPWQzIFhAdfZ/RbFOyf5/xphP/Bbld1ycJ2ZwZXNQiJBPWJ s2rw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="XG1fZDQ/"; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id e193si5953493ybh.416.2019.06.04.19.45.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 19:45:01 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="XG1fZDQ/"; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0001255.ppops.net [127.0.0.1]) by mx0b-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x552glB3011319 for ; Tue, 4 Jun 2019 19:45:01 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=pBwyrsC3xS2PmE3Fb9BbKTWT0q0/P4Hp2VaXNFe4hs0=; b=XG1fZDQ/VA/ty89WTMMiqHp0JJ3swFVe+4qf2QbZNoJY4fle/+lofAoPp4keSSC3Mapc JpBYgVQ6P405BKXa6ppg1BMRF57YiWhPmAGJo0eEajTbSyzr6bX8IuxQFvqwCKOdoyj4 Za0G0aaLyviWCKsu+XXybzejJ2fE/hJhXMY= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0b-00082601.pphosted.com with ESMTP id 2swxuq1615-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 04 Jun 2019 19:45:01 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::128) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 4 Jun 2019 19:44:59 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 947D512C7FDD0; Tue, 4 Jun 2019 19:44:58 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v6 08/10] mm: rework non-root kmem_cache lifecycle management Date: Tue, 4 Jun 2019 19:44:52 -0700 Message-ID: <20190605024454.1393507-9-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190605024454.1393507-1-guro@fb.com> References: <20190605024454.1393507-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-05_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906050015 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently each charged slab page holds a reference to the cgroup to which it's charged. Kmem_caches are held by the memcg and are released all together with the memory cgroup. It means that none of kmem_caches are released unless at least one reference to the memcg exists, which is very far from optimal. Let's rework it in a way that allows releasing individual kmem_caches as soon as the cgroup is offline, the kmem_cache is empty and there are no pending allocations. To make it possible, let's introduce a new percpu refcounter for non-root kmem caches. The counter is initialized to the percpu mode, and is switched to the atomic mode during kmem_cache deactivation. The counter is bumped for every charged page and also for every running allocation. So the kmem_cache can't be released unless all allocations complete. To shutdown non-active empty kmem_caches, let's reuse the work queue, previously used for the kmem_cache deactivation. Once the reference counter reaches 0, let's schedule an asynchronous kmem_cache release. * I used the following simple approach to test the performance (stolen from another patchset by T. Harding): time find / -name fname-no-exist echo 2 > /proc/sys/vm/drop_caches repeat 10 times Results: orig patched real 0m1.455s real 0m1.355s user 0m0.206s user 0m0.219s sys 0m0.855s sys 0m0.807s real 0m1.487s real 0m1.699s user 0m0.221s user 0m0.256s sys 0m0.806s sys 0m0.948s real 0m1.515s real 0m1.505s user 0m0.183s user 0m0.215s sys 0m0.876s sys 0m0.858s real 0m1.291s real 0m1.380s user 0m0.193s user 0m0.198s sys 0m0.843s sys 0m0.786s real 0m1.364s real 0m1.374s user 0m0.180s user 0m0.182s sys 0m0.868s sys 0m0.806s real 0m1.352s real 0m1.312s user 0m0.201s user 0m0.212s sys 0m0.820s sys 0m0.761s real 0m1.302s real 0m1.349s user 0m0.205s user 0m0.203s sys 0m0.803s sys 0m0.792s real 0m1.334s real 0m1.301s user 0m0.194s user 0m0.201s sys 0m0.806s sys 0m0.779s real 0m1.426s real 0m1.434s user 0m0.216s user 0m0.181s sys 0m0.824s sys 0m0.864s real 0m1.350s real 0m1.295s user 0m0.200s user 0m0.190s sys 0m0.842s sys 0m0.811s So it looks like the difference is not noticeable in this test. Signed-off-by: Roman Gushchin Acked-by: Vladimir Davydov --- include/linux/slab.h | 3 +- mm/memcontrol.c | 51 +++++++++++++++++++++------- mm/slab.h | 45 +++++++------------------ mm/slab_common.c | 79 ++++++++++++++++++++++++++------------------ 4 files changed, 100 insertions(+), 78 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 47923c173f30..1b54e5f83342 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -16,6 +16,7 @@ #include #include #include +#include /* @@ -152,7 +153,6 @@ int kmem_cache_shrink(struct kmem_cache *); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); void memcg_deactivate_kmem_caches(struct mem_cgroup *); -void memcg_destroy_kmem_caches(struct mem_cgroup *); /* * Please use this macro to create slab caches. Simply specify the @@ -641,6 +641,7 @@ struct memcg_cache_params { struct mem_cgroup *memcg; struct list_head children_node; struct list_head kmem_caches_node; + struct percpu_ref refcnt; void (*work_fn)(struct kmem_cache *); union { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 3427396da612..49084e2d81ff 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2591,12 +2591,13 @@ static void memcg_schedule_kmem_cache_create(struct mem_cgroup *memcg, { struct memcg_kmem_cache_create_work *cw; + if (!css_tryget_online(&memcg->css)) + return; + cw = kmalloc(sizeof(*cw), GFP_NOWAIT | __GFP_NOWARN); if (!cw) return; - css_get(&memcg->css); - cw->memcg = memcg; cw->cachep = cachep; INIT_WORK(&cw->work, memcg_kmem_cache_create_func); @@ -2631,6 +2632,7 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) { struct mem_cgroup *memcg; struct kmem_cache *memcg_cachep; + struct memcg_cache_array *arr; int kmemcg_id; VM_BUG_ON(!is_root_cache(cachep)); @@ -2638,14 +2640,29 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) if (memcg_kmem_bypass()) return cachep; - memcg = get_mem_cgroup_from_current(); + rcu_read_lock(); + + if (unlikely(current->active_memcg)) + memcg = current->active_memcg; + else + memcg = mem_cgroup_from_task(current); + + if (!memcg || memcg == root_mem_cgroup) + goto out_unlock; + kmemcg_id = READ_ONCE(memcg->kmemcg_id); if (kmemcg_id < 0) - goto out; + goto out_unlock; + + arr = rcu_dereference(cachep->memcg_params.memcg_caches); - memcg_cachep = cache_from_memcg_idx(cachep, kmemcg_id); - if (likely(memcg_cachep)) - return memcg_cachep; + /* + * Make sure we will access the up-to-date value. The code updating + * memcg_caches issues a write barrier to match this (see + * memcg_create_kmem_cache()). + */ + smp_rmb(); + memcg_cachep = READ_ONCE(arr->entries[kmemcg_id]); /* * If we are in a safe context (can wait, and not in interrupt @@ -2658,10 +2675,20 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) * memcg_create_kmem_cache, this means no further allocation * could happen with the slab_mutex held. So it's better to * defer everything. + * + * If the memcg is dying or memcg_cache is about to be released, + * don't bother creating new kmem_caches. Because memcg_cachep + * is ZEROed as the fist step of kmem offlining, we don't need + * percpu_ref_tryget_live() here. css_tryget_online() check in + * memcg_schedule_kmem_cache_create() will prevent us from + * creation of a new kmem_cache. */ - memcg_schedule_kmem_cache_create(memcg, cachep); -out: - css_put(&memcg->css); + if (unlikely(!memcg_cachep)) + memcg_schedule_kmem_cache_create(memcg, cachep); + else if (percpu_ref_tryget(&memcg_cachep->memcg_params.refcnt)) + cachep = memcg_cachep; +out_unlock: + rcu_read_unlock(); return cachep; } @@ -2672,7 +2699,7 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) void memcg_kmem_put_cache(struct kmem_cache *cachep) { if (!is_root_cache(cachep)) - css_put(&cachep->memcg_params.memcg->css); + percpu_ref_put(&cachep->memcg_params.refcnt); } /** @@ -3219,7 +3246,7 @@ static void memcg_free_kmem(struct mem_cgroup *memcg) memcg_offline_kmem(memcg); if (memcg->kmem_state == KMEM_ALLOCATED) { - memcg_destroy_kmem_caches(memcg); + WARN_ON(!list_empty(&memcg->kmem_caches)); static_branch_dec(&memcg_kmem_enabled_key); WARN_ON(page_counter_read(&memcg->kmem)); } diff --git a/mm/slab.h b/mm/slab.h index 345fb59afb8f..5d2b8511e6fb 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -248,32 +248,6 @@ static inline const char *cache_name(struct kmem_cache *s) return s->name; } -/* - * Note, we protect with RCU only the memcg_caches array, not per-memcg caches. - * That said the caller must assure the memcg's cache won't go away by either - * taking a css reference to the owner cgroup, or holding the slab_mutex. - */ -static inline struct kmem_cache * -cache_from_memcg_idx(struct kmem_cache *s, int idx) -{ - struct kmem_cache *cachep; - struct memcg_cache_array *arr; - - rcu_read_lock(); - arr = rcu_dereference(s->memcg_params.memcg_caches); - - /* - * Make sure we will access the up-to-date value. The code updating - * memcg_caches issues a write barrier to match this (see - * memcg_create_kmem_cache()). - */ - smp_rmb(); - cachep = READ_ONCE(arr->entries[idx]); - rcu_read_unlock(); - - return cachep; -} - static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) { if (is_root_cache(s)) @@ -285,14 +259,25 @@ static __always_inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { + int ret; + if (is_root_cache(s)) return 0; - return memcg_kmem_charge_memcg(page, gfp, order, s->memcg_params.memcg); + + ret = memcg_kmem_charge_memcg(page, gfp, order, s->memcg_params.memcg); + if (ret) + return ret; + + percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); + + return 0; } static __always_inline void memcg_uncharge_slab(struct page *page, int order, struct kmem_cache *s) { + if (!is_root_cache(s)) + percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); memcg_kmem_uncharge(page, order); } @@ -324,12 +309,6 @@ static inline const char *cache_name(struct kmem_cache *s) return s->name; } -static inline struct kmem_cache * -cache_from_memcg_idx(struct kmem_cache *s, int idx) -{ - return NULL; -} - static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) { return s; diff --git a/mm/slab_common.c b/mm/slab_common.c index 2914a8f0aa85..8255283025e3 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -132,6 +132,8 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr, LIST_HEAD(slab_root_caches); static DEFINE_SPINLOCK(memcg_kmem_wq_lock); +static void kmemcg_cache_shutdown(struct percpu_ref *percpu_ref); + void slab_init_memcg_params(struct kmem_cache *s) { s->memcg_params.root_cache = NULL; @@ -146,6 +148,12 @@ static int init_memcg_params(struct kmem_cache *s, struct memcg_cache_array *arr; if (root_cache) { + int ret = percpu_ref_init(&s->memcg_params.refcnt, + kmemcg_cache_shutdown, + 0, GFP_KERNEL); + if (ret) + return ret; + s->memcg_params.root_cache = root_cache; INIT_LIST_HEAD(&s->memcg_params.children_node); INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node); @@ -171,6 +179,8 @@ static void destroy_memcg_params(struct kmem_cache *s) { if (is_root_cache(s)) kvfree(rcu_access_pointer(s->memcg_params.memcg_caches)); + else + percpu_ref_exit(&s->memcg_params.refcnt); } static void free_memcg_params(struct rcu_head *rcu) @@ -226,6 +236,7 @@ void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) if (is_root_cache(s)) { list_add(&s->root_caches_node, &slab_root_caches); } else { + css_get(&memcg->css); s->memcg_params.memcg = memcg; list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); @@ -241,6 +252,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) } else { list_del(&s->memcg_params.children_node); list_del(&s->memcg_params.kmem_caches_node); + css_put(&s->memcg_params.memcg->css); } } #else @@ -686,7 +698,7 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg, } /* - * Since readers won't lock (see cache_from_memcg_idx()), we need a + * Since readers won't lock (see memcg_kmem_get_cache()), we need a * barrier here to ensure nobody will see the kmem_cache partially * initialized. */ @@ -711,14 +723,12 @@ static void kmemcg_workfn(struct work_struct *work) mutex_lock(&slab_mutex); s->memcg_params.work_fn(s); + s->memcg_params.work_fn = NULL; mutex_unlock(&slab_mutex); put_online_mems(); put_online_cpus(); - - /* done, put the ref from kmemcg_cache_deactivate() */ - css_put(&s->memcg_params.memcg->css); } static void kmemcg_rcufn(struct rcu_head *head) @@ -735,10 +745,39 @@ static void kmemcg_rcufn(struct rcu_head *head) queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); } +static void kmemcg_cache_shutdown_fn(struct kmem_cache *s) +{ + WARN_ON(shutdown_cache(s)); +} + +static void kmemcg_cache_shutdown(struct percpu_ref *percpu_ref) +{ + struct kmem_cache *s = container_of(percpu_ref, struct kmem_cache, + memcg_params.refcnt); + unsigned long flags; + + spin_lock_irqsave(&memcg_kmem_wq_lock, flags); + if (s->memcg_params.root_cache->memcg_params.dying) + goto unlock; + + WARN_ON(s->memcg_params.work_fn); + s->memcg_params.work_fn = kmemcg_cache_shutdown_fn; + INIT_WORK(&s->memcg_params.work, kmemcg_workfn); + queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); + +unlock: + spin_unlock_irqrestore(&memcg_kmem_wq_lock, flags); +} + +static void kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) +{ + __kmemcg_cache_deactivate_after_rcu(s); + percpu_ref_kill(&s->memcg_params.refcnt); +} + static void kmemcg_cache_deactivate(struct kmem_cache *s) { - if (WARN_ON_ONCE(is_root_cache(s)) || - WARN_ON_ONCE(s->memcg_params.work_fn)) + if (WARN_ON_ONCE(is_root_cache(s))) return; __kmemcg_cache_deactivate(s); @@ -747,10 +786,8 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s) if (s->memcg_params.root_cache->memcg_params.dying) goto unlock; - /* pin memcg so that @s doesn't get destroyed in the middle */ - css_get(&s->memcg_params.memcg->css); - - s->memcg_params.work_fn = __kmemcg_cache_deactivate_after_rcu; + WARN_ON_ONCE(s->memcg_params.work_fn); + s->memcg_params.work_fn = kmemcg_cache_deactivate_after_rcu; call_rcu(&s->memcg_params.rcu_head, kmemcg_rcufn); unlock: spin_unlock_irq(&memcg_kmem_wq_lock); @@ -784,28 +821,6 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) put_online_cpus(); } -void memcg_destroy_kmem_caches(struct mem_cgroup *memcg) -{ - struct kmem_cache *s, *s2; - - get_online_cpus(); - get_online_mems(); - - mutex_lock(&slab_mutex); - list_for_each_entry_safe(s, s2, &memcg->kmem_caches, - memcg_params.kmem_caches_node) { - /* - * The cgroup is about to be freed and therefore has no charges - * left. Hence, all its caches must be empty by now. - */ - BUG_ON(shutdown_cache(s)); - } - mutex_unlock(&slab_mutex); - - put_online_mems(); - put_online_cpus(); -} - static int shutdown_memcg_caches(struct kmem_cache *s) { struct memcg_cache_array *arr; From patchwork Wed Jun 5 02:44:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10976283 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5F94114B6 for ; Wed, 5 Jun 2019 02:45:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4F1AA27FAC for ; Wed, 5 Jun 2019 02:45:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 42F7628420; Wed, 5 Jun 2019 02:45:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0B38B27FAC for ; Wed, 5 Jun 2019 02:45:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 905626B0274; Tue, 4 Jun 2019 22:45:06 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8B88A6B0276; Tue, 4 Jun 2019 22:45:06 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E6EC6B0278; Tue, 4 Jun 2019 22:45:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 297F26B0276 for ; Tue, 4 Jun 2019 22:45:06 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id y187so5758560pgd.1 for ; Tue, 04 Jun 2019 19:45:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=n7BlWccP5e9E4x6aVqJtrHQl4d1EpfzcQz8Ne0PToZU=; b=hviQcPf1f8hdHk3Lwl0SLQrgA5eFVhcjDOqozsqesLUjPRjnhMMNpIfyWv+QQPTUPg SK6N/KoFXN+fEu/pECtcQgp9yWfecewIuBiLDd08q+mgS/6P2NPoSTsXLlTMT5hhWKs4 s0NSDQ7zlwfcpLHZg7oXr+Q7BjSksXSmhhPQH3q5EE/SqXhbzRNkVgKsUmTQajIPIsKe sTDPGW11Ogen3SZ6WRPkFUbGQsmXE5q9azJ+aLEwZFsyYhTbn0mqKZBXT3oT0U5MKp20 4uZgPD94LkSlVvd32uBVB/8yaEEaJZfCoZEA8atosgz6Bb1x1wAwNMuL3J8udygnTHWq v3Eg== X-Gm-Message-State: APjAAAUjWvZE4W86GIBnM7EtL9BQ6QOKvunf4eLWiqCVcjbWyWT0gzPg oHqAWVBEl9xnypatsQ7zSlxuwtnqeN/E3+azhXD+jixuZRun24hezlWkaOZwnCKkxChvKhmd7hh XhOTNo5sWPeS+fASrGhFe/3lxr/5dgoWeifDe1jz5EimtyeQNeYy3iO00UKA0bQv8Mg== X-Received: by 2002:a17:90a:b78b:: with SMTP id m11mr41027743pjr.106.1559702705655; Tue, 04 Jun 2019 19:45:05 -0700 (PDT) X-Google-Smtp-Source: APXvYqyEy+juNHCN4laE4+nhy0qaddeKfNYCcdyzv5yKAAHcBW4tCMpf7PM+CEyjEmsC/eZ10iSK X-Received: by 2002:a17:90a:b78b:: with SMTP id m11mr41027683pjr.106.1559702704558; Tue, 04 Jun 2019 19:45:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559702704; cv=none; d=google.com; s=arc-20160816; b=wt0y+BwYgz7ZpaqQEWpG9El3ExbLLerlkfib8qREASVeJur1OuuarQNn/TI/fNoJI/ fU6jaF9IgkqQsR0VsZsUJ9dHDlpOd8FUVxZ5YsEKvstajill03ObGHthM6KFS637g8W/ EEm2PwPU4kk/oEF7g0ubA2AQnt7Brqo3aadGMaPV7nsAhaoSukJdU/EIN76WCTUkKi1+ uRt+Wbe3s3SnpmhRt3trzlNKuH6IYFHWU+tEQdhwyoYBDAOjdm3IoDHcsNO+4qLUsBJD npFjePke5W/HTah2ssALKjb4FnaIYaCNOOGsSPAbohq8w8cfoBVTzffN7U1vxLGTN4+D C0LA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=n7BlWccP5e9E4x6aVqJtrHQl4d1EpfzcQz8Ne0PToZU=; b=kL6feF26boUYMxd8M0E1WPoJ5IaMNV5ewtW4OcKmNy8B8ljVMucXSqlQCDzj++egWN 02i7g+mRFef51yUODq1UwCe+KpSvCKRBBS1NOgAHkjrtO2KehgJqJD+BXgLNI2WDhKAm 2qlegDgOiAkm0Z4C2cEZcklkNa7qeJ+Tehu8u37rVV4TlG+nFzsBWjVbK0bBCd/8lUQt wkylIwtz0Kgw1C6P09UCsMa60t+W9cEbM8rSKfx5u72wKgibedd3h7gO24JkUa+isLic RlBI5C1Nylro22geNs7ihMBzSQfbxlK9eU7sLfVmsowQeONJRgNCH30h7DWjtIMQWDrN DuIw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=dWZpvwBN; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id x3si23317213pgr.22.2019.06.04.19.45.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 19:45:04 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=dWZpvwBN; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x552j05E031912 for ; Tue, 4 Jun 2019 19:45:04 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=n7BlWccP5e9E4x6aVqJtrHQl4d1EpfzcQz8Ne0PToZU=; b=dWZpvwBNdjJhXRH3Co8LeCEWmYGp7BZyMm2pKcsTPXZ7X75DWj9ViSkotXNM+IPfy9EN E08ucVEDNX1WHNtbEmUyeZHGzxktgV503nSgovL0VXK0aEQ4AKV2zb3dsNweZOs2U1Qk 6Z6nJ4UH6C4g0rBSbllLSze0zk4ewsf7hnQ= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2swx191ck2-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 04 Jun 2019 19:45:03 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 4 Jun 2019 19:45:00 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 986C112C7FDD2; Tue, 4 Jun 2019 19:44:58 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v6 09/10] mm: stop setting page->mem_cgroup pointer for slab pages Date: Tue, 4 Jun 2019 19:44:53 -0700 Message-ID: <20190605024454.1393507-10-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190605024454.1393507-1-guro@fb.com> References: <20190605024454.1393507-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-05_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=626 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906050015 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Every slab page charged to a non-root memory cgroup has a pointer to the memory cgroup and holds a reference to it, which protects a non-empty memory cgroup from being released. At the same time the page has a pointer to the corresponding kmem_cache, and also hold a reference to the kmem_cache. And kmem_cache by itself holds a reference to the cgroup. So there is clearly some redundancy, which allows to stop setting the page->mem_cgroup pointer and rely on getting memcg pointer indirectly via kmem_cache. Further it will allow to change this pointer easier, without a need to go over all charged pages. So let's stop setting page->mem_cgroup pointer for slab pages, and stop using the css refcounter directly for protecting the memory cgroup from going away. Instead rely on kmem_cache as an intermediate object. Make sure that vmstats and shrinker lists are working as previously, as well as /proc/kpagecgroup interface. Signed-off-by: Roman Gushchin Acked-by: Vladimir Davydov --- mm/list_lru.c | 3 +- mm/memcontrol.c | 12 ++++---- mm/slab.h | 74 ++++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 70 insertions(+), 19 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index 927d85be32f6..0f1f6b06b7f3 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -12,6 +12,7 @@ #include #include #include +#include "slab.h" #ifdef CONFIG_MEMCG_KMEM static LIST_HEAD(list_lrus); @@ -63,7 +64,7 @@ static __always_inline struct mem_cgroup *mem_cgroup_from_kmem(void *ptr) if (!memcg_kmem_enabled()) return NULL; page = virt_to_head_page(ptr); - return page->mem_cgroup; + return memcg_from_slab_page(page); } static inline struct list_lru_one * diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 49084e2d81ff..c097b1fc74ec 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -485,7 +485,10 @@ ino_t page_cgroup_ino(struct page *page) unsigned long ino = 0; rcu_read_lock(); - memcg = READ_ONCE(page->mem_cgroup); + if (PageHead(page) && PageSlab(page)) + memcg = memcg_from_slab_page(page); + else + memcg = READ_ONCE(page->mem_cgroup); while (memcg && !(memcg->css.flags & CSS_ONLINE)) memcg = parent_mem_cgroup(memcg); if (memcg) @@ -2727,9 +2730,6 @@ int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, cancel_charge(memcg, nr_pages); return -ENOMEM; } - - page->mem_cgroup = memcg; - return 0; } @@ -2752,8 +2752,10 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order) memcg = get_mem_cgroup_from_current(); if (!mem_cgroup_is_root(memcg)) { ret = __memcg_kmem_charge_memcg(page, gfp, order, memcg); - if (!ret) + if (!ret) { + page->mem_cgroup = memcg; __SetPageKmemcg(page); + } } css_put(&memcg->css); return ret; diff --git a/mm/slab.h b/mm/slab.h index 5d2b8511e6fb..7ead47cb9338 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -255,30 +255,67 @@ static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) return s->memcg_params.root_cache; } +/* + * Expects a pointer to a slab page. Please note, that PageSlab() check + * isn't sufficient, as it returns true also for tail compound slab pages, + * which do not have slab_cache pointer set. + * So this function assumes that the page can pass PageHead() and PageSlab() + * checks. + */ +static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) +{ + struct kmem_cache *s; + + s = READ_ONCE(page->slab_cache); + if (s && !is_root_cache(s)) + return s->memcg_params.memcg; + + return NULL; +} + +/* + * Charge the slab page belonging to the non-root kmem_cache. + * Can be called for non-root kmem_caches only. + */ static __always_inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { + struct mem_cgroup *memcg; + struct lruvec *lruvec; int ret; - if (is_root_cache(s)) - return 0; - - ret = memcg_kmem_charge_memcg(page, gfp, order, s->memcg_params.memcg); + memcg = s->memcg_params.memcg; + ret = memcg_kmem_charge_memcg(page, gfp, order, memcg); if (ret) return ret; + lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); + mod_lruvec_state(lruvec, cache_vmstat_idx(s), 1 << order); + + /* transer try_charge() page references to kmem_cache */ percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); + css_put_many(&memcg->css, 1 << order); return 0; } +/* + * Uncharge a slab page belonging to a non-root kmem_cache. + * Can be called for non-root kmem_caches only. + */ static __always_inline void memcg_uncharge_slab(struct page *page, int order, struct kmem_cache *s) { - if (!is_root_cache(s)) - percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); - memcg_kmem_uncharge(page, order); + struct mem_cgroup *memcg; + struct lruvec *lruvec; + + memcg = s->memcg_params.memcg; + lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); + mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order)); + memcg_kmem_uncharge_memcg(page, order, memcg); + + percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); } extern void slab_init_memcg_params(struct kmem_cache *); @@ -314,6 +351,11 @@ static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) return s; } +static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) +{ + return NULL; +} + static inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { @@ -351,18 +393,24 @@ static __always_inline int charge_slab_page(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { - int ret = memcg_charge_slab(page, gfp, order, s); - - if (!ret) - mod_lruvec_page_state(page, cache_vmstat_idx(s), 1 << order); + if (is_root_cache(s)) { + mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), + 1 << order); + return 0; + } - return ret; + return memcg_charge_slab(page, gfp, order, s); } static __always_inline void uncharge_slab_page(struct page *page, int order, struct kmem_cache *s) { - mod_lruvec_page_state(page, cache_vmstat_idx(s), -(1 << order)); + if (is_root_cache(s)) { + mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), + -(1 << order)); + return; + } + memcg_uncharge_slab(page, order, s); } From patchwork Wed Jun 5 02:44:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10976287 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4381F13AD for ; Wed, 5 Jun 2019 02:45:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3285827FAC for ; Wed, 5 Jun 2019 02:45:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 266D428420; Wed, 5 Jun 2019 02:45:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3735527FAC for ; Wed, 5 Jun 2019 02:45:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B9006B0277; Tue, 4 Jun 2019 22:45:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 86B3E6B0278; Tue, 4 Jun 2019 22:45:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 758EF6B0279; Tue, 4 Jun 2019 22:45:11 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f69.google.com (mail-yw1-f69.google.com [209.85.161.69]) by kanga.kvack.org (Postfix) with ESMTP id 4F08D6B0277 for ; Tue, 4 Jun 2019 22:45:11 -0400 (EDT) Received: by mail-yw1-f69.google.com with SMTP id w127so21464950ywe.6 for ; Tue, 04 Jun 2019 19:45:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=0SHoOhFKuZcfsmJPpk99bY607h8QTF2wVdFhMuuvmaA=; b=NRn496YzqaL3o3JMBP81+6v4lh66wgieR/Oi96w+K92UUuNDBgPOMGYFCzLjbciyoW 8aHzwEX75KXh1udVVAfou3P/lj1RaV6YQ3n0rF3sJjPBOdw6ttyZfiodQOg2qtAF1Z+J 2tHjOrqRG8epq+Kn0+Xgehcr38dDJIL5nzLUB9zn9WsvW1xB7iuG2c58HTKPORNudnx8 Dl1v6cg7BD/G9JUUyKumm/wxsh/zn9PpKpCxUGckuwgtwvx1LpUFxKTkET5yBNgzsm27 qSmoCgpF+m+xvs6EFmIFKOTwG6+UNYiWD/x+MDszxjCRxW0DDF2YX+OmLSVIkEaNlHwV zeyQ== X-Gm-Message-State: APjAAAVl8CwcvGKKUzNHzbhIZOzjyoqAmX6JGCWgH46v22/ZiyZNA0ug I88Qf0wZoVIhWeumLmXmiQTkDIsYW0gXtuLz8Nd6NZORBTihu8eVQFXbrMDGgl1WSyYeelLGe0W b/Bq2NP7mZl/xDB85oT0CYpqIjNRHqGQxVZDE62HiJNuW9kpm3sLtqp1SiqmI1dn+QQ== X-Received: by 2002:a25:d4c5:: with SMTP id m188mr15712586ybf.60.1559702711063; Tue, 04 Jun 2019 19:45:11 -0700 (PDT) X-Google-Smtp-Source: APXvYqyz1VrMTnwbJD/1P0t96s+BPKobgvIgoDC2EtaLmLw6JZfN4ZEneI+XQfo7gYPYGJ9joJp1 X-Received: by 2002:a25:d4c5:: with SMTP id m188mr15712545ybf.60.1559702710041; Tue, 04 Jun 2019 19:45:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559702710; cv=none; d=google.com; s=arc-20160816; b=VSHFjwyM9/6KQLuBVEW/Gqo7gGurySZURN0u03ny4TSMuHr67EOR5OqlByb+tzaiOv 4YOGltjK7S5ukHMnsHgX/ExUQ0p37YnIGu0djGjFSlOZNaWVeKTxL2nSlFFOy+1Ev7Q3 46S2kVKZMTq3IF97yT+WHO61WsarU/vPnMhuaXT1ry0RCBPsHegKQGyCzMbfarPfqZ2Z 5aolIJqjxdwQj1pWDgw0dtshOosLlJG3fOWPr4VHdPrxtoFQAsQscTkSG8sKtzFZDPrF afYdwFSq839RwdW3WbV1wMduq+rlPAAX7ApO0Ddapx8dOUM7qqtGgUx7o6lVkwxmOpJp T2YQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=0SHoOhFKuZcfsmJPpk99bY607h8QTF2wVdFhMuuvmaA=; b=RTIvnFj6nyrGFq4gGvUWJkjSbpEhdRngD/qOqbxkw9OuXhv2rY0hTAqPg9psQ1MQKe ws24fg57F2bGTWzP/bEMR9dGvrFF2utlFttYzMA1Vqt//sa2hKQPlnQhpd8la05l26vn RB0MNgOTCnHofTYOH53UgFsuA+W6vXD9vRpG4dTiUmF4ZSQVMKV8IafcuHhWsS9BSNmN qbQzyMViSQkqWdTUI6u7PzfbQx1SBz/xMQzOe7SlQ+oEwrgjaFsovVXqtSKjP2GIOlXV gwzgHqINxe0h0rB2eR5bipMlGT0Hlx/aO80dzbfpAZe7KlWihzy+HVqotXwwy/bSsrdo ESjg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=B02auHbM; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id g185si1244688ywa.288.2019.06.04.19.45.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2019 19:45:10 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=B02auHbM; spf=pass (google.com: domain of prvs=10599ee021=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=10599ee021=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0001255.ppops.net [127.0.0.1]) by mx0b-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x552gIO8010888 for ; Tue, 4 Jun 2019 19:45:09 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=0SHoOhFKuZcfsmJPpk99bY607h8QTF2wVdFhMuuvmaA=; b=B02auHbMGsowHj38Qm/IrhmwYpylRytVXJfuYYIB91fIK2dMBX9909eUYyEbzVKeDIzC Vy6cIv4IjBwS2DveCpUnjIS2iNrjz+t2TIJPO1Tol5Yzr/b9YK7hXFbQ9ibkyKwG8igm 2TLMUF9dPEzcaBnQxf2MA19BrdAZaL1u3FQ= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0b-00082601.pphosted.com with ESMTP id 2swxuq161e-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 04 Jun 2019 19:45:09 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::e) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 4 Jun 2019 19:45:00 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 9C91612C7FDD4; Tue, 4 Jun 2019 19:44:58 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Shakeel Butt , Vladimir Davydov , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v6 10/10] mm: reparent slab memory on cgroup removal Date: Tue, 4 Jun 2019 19:44:54 -0700 Message-ID: <20190605024454.1393507-11-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190605024454.1393507-1-guro@fb.com> References: <20190605024454.1393507-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-05_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906050015 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Let's reparent memcg slab memory on memcg offlining. This allows us to release the memory cgroup without waiting for the last outstanding kernel object (e.g. dentry used by another application). So instead of reparenting all accounted slab pages, let's do reparent a relatively small amount of kmem_caches. Reparenting is performed as a part of the deactivation process. Since the parent cgroup is already charged, everything we need to do is to splice the list of kmem_caches to the parent's kmem_caches list, swap the memcg pointer and drop the css refcounter for each kmem_cache and adjust the parent's css refcounter. Quite simple. Please, note that kmem_cache->memcg_params.memcg isn't a stable pointer anymore. It's safe to read it under rcu_read_lock() or with slab_mutex held. We can race with the slab allocation and deallocation paths. It's not a big problem: parent's charge and slab global stats are always correct, and we don't care anymore about the child usage and global stats. The child cgroup is already offline, so we don't use or show it anywhere. Local slab stats (NR_SLAB_RECLAIMABLE and NR_SLAB_UNRECLAIMABLE) aren't used anywhere except count_shadow_nodes(). But even there it won't break anything: after reparenting "nodes" will be 0 on child level (because we're already reparenting shrinker lists), and on parent level page stats always were 0, and this patch won't change anything. Signed-off-by: Roman Gushchin --- include/linux/slab.h | 4 ++-- mm/list_lru.c | 8 +++++++- mm/memcontrol.c | 14 ++++++++------ mm/slab.h | 23 +++++++++++++++++------ mm/slab_common.c | 22 +++++++++++++++++++--- 5 files changed, 53 insertions(+), 18 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 1b54e5f83342..109cab2ad9b4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -152,7 +152,7 @@ void kmem_cache_destroy(struct kmem_cache *); int kmem_cache_shrink(struct kmem_cache *); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); -void memcg_deactivate_kmem_caches(struct mem_cgroup *); +void memcg_deactivate_kmem_caches(struct mem_cgroup *, struct mem_cgroup *); /* * Please use this macro to create slab caches. Simply specify the @@ -638,7 +638,7 @@ struct memcg_cache_params { bool dying; }; struct { - struct mem_cgroup *memcg; + struct mem_cgroup __rcu *memcg; struct list_head children_node; struct list_head kmem_caches_node; struct percpu_ref refcnt; diff --git a/mm/list_lru.c b/mm/list_lru.c index 0f1f6b06b7f3..0b2319897e86 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -77,11 +77,15 @@ list_lru_from_kmem(struct list_lru_node *nlru, void *ptr, if (!nlru->memcg_lrus) goto out; + rcu_read_lock(); memcg = mem_cgroup_from_kmem(ptr); - if (!memcg) + if (!memcg) { + rcu_read_unlock(); goto out; + } l = list_lru_from_memcg_idx(nlru, memcg_cache_id(memcg)); + rcu_read_unlock(); out: if (memcg_ptr) *memcg_ptr = memcg; @@ -131,12 +135,14 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item) spin_lock(&nlru->lock); if (list_empty(item)) { + rcu_read_lock(); l = list_lru_from_kmem(nlru, item, &memcg); list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) memcg_set_shrinker_bit(memcg, nid, lru_shrinker_id(lru)); + rcu_read_unlock(); nlru->nr_items++; spin_unlock(&nlru->lock); return true; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c097b1fc74ec..0f64a2c06803 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3209,15 +3209,15 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) */ memcg->kmem_state = KMEM_ALLOCATED; - memcg_deactivate_kmem_caches(memcg); - - kmemcg_id = memcg->kmemcg_id; - BUG_ON(kmemcg_id < 0); - parent = parent_mem_cgroup(memcg); if (!parent) parent = root_mem_cgroup; + memcg_deactivate_kmem_caches(memcg, parent); + + kmemcg_id = memcg->kmemcg_id; + BUG_ON(kmemcg_id < 0); + /* * Change kmemcg_id of this cgroup and all its descendants to the * parent's id, and then move all entries from this cgroup's list_lrus @@ -3250,7 +3250,6 @@ static void memcg_free_kmem(struct mem_cgroup *memcg) if (memcg->kmem_state == KMEM_ALLOCATED) { WARN_ON(!list_empty(&memcg->kmem_caches)); static_branch_dec(&memcg_kmem_enabled_key); - WARN_ON(page_counter_read(&memcg->kmem)); } } #else @@ -4675,6 +4674,9 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) /* The following stuff does not apply to the root */ if (!parent) { +#ifdef CONFIG_MEMCG_KMEM + INIT_LIST_HEAD(&memcg->kmem_caches); +#endif root_mem_cgroup = memcg; return &memcg->css; } diff --git a/mm/slab.h b/mm/slab.h index 7ead47cb9338..34bf92382ecd 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -268,7 +268,7 @@ static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) s = READ_ONCE(page->slab_cache); if (s && !is_root_cache(s)) - return s->memcg_params.memcg; + return rcu_dereference(s->memcg_params.memcg); return NULL; } @@ -285,10 +285,18 @@ static __always_inline int memcg_charge_slab(struct page *page, struct lruvec *lruvec; int ret; - memcg = s->memcg_params.memcg; + rcu_read_lock(); + memcg = rcu_dereference(s->memcg_params.memcg); + while (memcg && !css_tryget_online(&memcg->css)) + memcg = parent_mem_cgroup(memcg); + rcu_read_unlock(); + + if (unlikely(!memcg)) + return true; + ret = memcg_kmem_charge_memcg(page, gfp, order, memcg); if (ret) - return ret; + goto out; lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); mod_lruvec_state(lruvec, cache_vmstat_idx(s), 1 << order); @@ -296,8 +304,9 @@ static __always_inline int memcg_charge_slab(struct page *page, /* transer try_charge() page references to kmem_cache */ percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); css_put_many(&memcg->css, 1 << order); - - return 0; +out: + css_put(&memcg->css); + return ret; } /* @@ -310,10 +319,12 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, struct mem_cgroup *memcg; struct lruvec *lruvec; - memcg = s->memcg_params.memcg; + rcu_read_lock(); + memcg = rcu_dereference(s->memcg_params.memcg); lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order)); memcg_kmem_uncharge_memcg(page, order, memcg); + rcu_read_unlock(); percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); } diff --git a/mm/slab_common.c b/mm/slab_common.c index 8255283025e3..00b380f5d467 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -237,7 +237,7 @@ void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) list_add(&s->root_caches_node, &slab_root_caches); } else { css_get(&memcg->css); - s->memcg_params.memcg = memcg; + rcu_assign_pointer(s->memcg_params.memcg, memcg); list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); list_add(&s->memcg_params.kmem_caches_node, @@ -252,7 +252,9 @@ static void memcg_unlink_cache(struct kmem_cache *s) } else { list_del(&s->memcg_params.children_node); list_del(&s->memcg_params.kmem_caches_node); - css_put(&s->memcg_params.memcg->css); + mem_cgroup_put(rcu_dereference_protected(s->memcg_params.memcg, + lockdep_is_held(&slab_mutex))); + rcu_assign_pointer(s->memcg_params.memcg, NULL); } } #else @@ -793,11 +795,13 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s) spin_unlock_irq(&memcg_kmem_wq_lock); } -void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) +void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg, + struct mem_cgroup *parent) { int idx; struct memcg_cache_array *arr; struct kmem_cache *s, *c; + unsigned int nr_reparented; idx = memcg_cache_id(memcg); @@ -815,6 +819,18 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) kmemcg_cache_deactivate(c); arr->entries[idx] = NULL; } + nr_reparented = 0; + list_for_each_entry(s, &memcg->kmem_caches, + memcg_params.kmem_caches_node) { + rcu_assign_pointer(s->memcg_params.memcg, parent); + css_put(&memcg->css); + nr_reparented++; + } + if (nr_reparented) { + list_splice_init(&memcg->kmem_caches, + &parent->kmem_caches); + css_get_many(&parent->css, nr_reparented); + } mutex_unlock(&slab_mutex); put_online_mems();