From patchwork Tue Sep 11 00:42:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Jordan X-Patchwork-Id: 10594947 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D2973921 for ; Tue, 11 Sep 2018 00:43:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BD2FE212DA for ; Tue, 11 Sep 2018 00:43:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AF598237A5; Tue, 11 Sep 2018 00:43:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 706D2212DA for ; Tue, 11 Sep 2018 00:42:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 804BA8E0004; Mon, 10 Sep 2018 20:42:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7D9D28E0001; Mon, 10 Sep 2018 20:42:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C72F8E0004; Mon, 10 Sep 2018 20:42:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f72.google.com (mail-yw1-f72.google.com [209.85.161.72]) by kanga.kvack.org (Postfix) with ESMTP id 3ADAA8E0001 for ; Mon, 10 Sep 2018 20:42:58 -0400 (EDT) Received: by mail-yw1-f72.google.com with SMTP id j71-v6so14393738ywb.22 for ; Mon, 10 Sep 2018 17:42:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=AYffcWNGUXpctZYcdeMQTwk8kPozUsCUImBciAV0wPQ=; b=s4wisBvjBaT8IO4J333ZhlGbKjEZXEHqcVsrJyCyWS5//H7srn53o+Ct6nzZVe6XTR XJxuWpWzIgdlJRBxHlI6tvjyYzCblT8FtKnJFRLiaOjZayeNeUPiPlOMKZ5+chwucCVD pTIJ1gW1NkhMxceJw7UvdC3DBOF5meF9nKLnL0+Q+sieGM/dy00gyZpHi/qTe9Ae+Uso Kb55M5y6MZdiMv7XeY0cBH0evbAKX1waabq9ODhknrjRxHIAzSpAxoTgeBQyv8fTGINY qjMqqds1Uu7M0DSjfhgKnlGLqcFnUPnNXP6ooA7P32cDyafhecnLwUiG7kK2bDAHuI3g oPpA== X-Gm-Message-State: APzg51BJZTw/pYyC0wFpAYXLQiNs/HkMH20bI37jR6GmVOiKhRAsQppf YCKBPIwune9/SBoddfQvVZ5v581i26HLUS7ZX199RIvf+6zr37oA0gxJPyulAS59tctEZb+vRQN i7x/AWiyLXr++1nfc8tvd26Xy2P0zmbUpc64Itnl/cHF81lpuLJs55SkvmPE75kYnkw== X-Received: by 2002:a0d:c385:: with SMTP id f127-v6mr11021479ywd.324.1536626577900; Mon, 10 Sep 2018 17:42:57 -0700 (PDT) X-Google-Smtp-Source: ANB0VdalmJIHOxJdaZ2ZJCt+z7hNYJXGTxztwZWEUYnOIBUc0G6+dLW0ypFiAsD6bgSNoWnvEo1h X-Received: by 2002:a0d:c385:: with SMTP id f127-v6mr11021458ywd.324.1536626577013; Mon, 10 Sep 2018 17:42:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536626576; cv=none; d=google.com; s=arc-20160816; b=K3wYGN+Nvi1RoB2El1fbKbdB5Xy1Q5Q7RqcNa7UukMBQ//yv5EgMVHZTYiZU5Yk/Ub MzUkkJz2HsTff/PT0ADgkdhFLnlveACynbJufczl0zF5AztuT8vP+2J349DBDQuelr4m /rW2xgc4KZQO5y0tGvq11gowj0M1aVD32LqqeIDEw9qpDzxp7pC1vhwnJ+6OdYDBTEf9 txWmCqQusCDoQwgV71XznOOkgjZUlPP5KayEg/WBph2f3rAxQAv9ZoMxjJgLNAV/DlqF s6j1hosC4XwvxHtiKIQsp9WN9b+9UCX7DHdmc/YykfLITWi/j8W8RuhGNyupJ2/kcx4J Ll9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=AYffcWNGUXpctZYcdeMQTwk8kPozUsCUImBciAV0wPQ=; b=G6jy1sEG28L/LV7QlGP5ICFHsnjZnXLWtcP8qVweUqXMkcUm2sSoQybBlmlGMBjkbB pvAIdaB8L/ydgrp4DSGMVGaZmjJ0Femhy9h0glqT/5+yk5rJWI2VgxRQXlXxQr2sY4v+ GF1Z2Md00twgjSHzFyCv4pIAZogPgW10KdTRW6EIWaBPdnq9qkGNUyvlzA2i+TslYIVk lJ9SGDbUpqlTZSNGvJctS/1QD4xys5hYDvzZlXXZF9SjzcYq6BlX3C9UHQfqVBb56Cg5 9zryNbcwXXxVe9tzKPBSJYpsuK1OeX2SwodRpf7BfEm4OX7piW4GzHPIpuL6enQB8op6 E8XA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=yE9q03bG; spf=pass (google.com: domain of daniel.m.jordan@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=daniel.m.jordan@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2120.oracle.com (aserp2120.oracle.com. [141.146.126.78]) by mx.google.com with ESMTPS id a133-v6si4797416ywb.15.2018.09.10.17.42.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Sep 2018 17:42:56 -0700 (PDT) Received-SPF: pass (google.com: domain of daniel.m.jordan@oracle.com designates 141.146.126.78 as permitted sender) client-ip=141.146.126.78; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=yE9q03bG; spf=pass (google.com: domain of daniel.m.jordan@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=daniel.m.jordan@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id w8B0clcb087637; Tue, 11 Sep 2018 00:42:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=AYffcWNGUXpctZYcdeMQTwk8kPozUsCUImBciAV0wPQ=; b=yE9q03bGg4DiTrzu//XkMz0lWNcWcMHJRdVcfQidV3VrlNKaKs91rjE0E5osBLaYN1QS 9/b5H7DspSjo409QRAhFRZBAm+3yiIEBCLldWsPSvUnAc1khkA8bjZyATQMs2y+RY0w4 Di3rFSVnxWPxbTzqocWWlm3ZPsPzQ1+PiErwTJp3HWKpcBJwSss0FajR+i7jXjm4cKGK D6UthFjwuorYfjBVp24QCf7XKFwj4lGDXlYx7iRdLPjewj8MTpAg5QtxXCFY7xNSZRSy W22Cu/tezk62jWB5yfW55DXrDx/wEFHhXRKUFvaMUY+0pWIBOaRyBXAhwAcM4Sw9tGOt Sw== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2120.oracle.com with ESMTP id 2mc6cph1fx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 11 Sep 2018 00:42:48 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id w8B0gmR3008962 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 11 Sep 2018 00:42:48 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id w8B0gmw1018658; Tue, 11 Sep 2018 00:42:48 GMT Received: from localhost.localdomain (/73.143.71.164) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 10 Sep 2018 17:42:47 -0700 From: Daniel Jordan To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org Cc: aaron.lu@intel.com, ak@linux.intel.com, akpm@linux-foundation.org, dave.dice@oracle.com, dave.hansen@linux.intel.com, hannes@cmpxchg.org, levyossi@icloud.com, ldufour@linux.vnet.ibm.com, mgorman@techsingularity.net, mhocko@kernel.org, Pavel.Tatashin@microsoft.com, steven.sistare@oracle.com, tim.c.chen@intel.com, vdavydov.dev@gmail.com, ying.huang@intel.com Subject: [RFC PATCH v2 1/8] mm, memcontrol.c: make memcg lru stats thread-safe without lru_lock Date: Mon, 10 Sep 2018 20:42:33 -0400 Message-Id: <20180911004240.4758-2-daniel.m.jordan@oracle.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180911004240.4758-1-daniel.m.jordan@oracle.com> References: <20180911004240.4758-1-daniel.m.jordan@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9012 signatures=668708 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1809110006 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP lru_lock needs to be held to update memcg LRU statistics. This requirement arises fairly naturally based on when the stats are updated because callers are holding lru_lock already. In preparation for allowing concurrent adds and removes from the LRU, however, make concurrent updates to these statistics safe without lru_lock. The lock continues to be held until later in the series, when it is replaced with a rwlock that also disables preemption, maintaining the assumption of __mod_lru_zone_size, which is introduced here. Follow the existing pattern for statistics in memcontrol.h by using a combination of per-cpu counters and atomics. Remove the negative statistics warning from ca707239e8a7 ("mm: update_lru_size warn and reset bad lru_size"). Although an earlier version of this patch updated the warning to account for the error introduced by the per-cpu counters, Hugh says this warning has not been seen in the wild and that for simplicity's sake it should probably just be removed. Signed-off-by: Daniel Jordan --- include/linux/memcontrol.h | 43 +++++++++++++++++++++++++++++--------- mm/memcontrol.c | 29 +++++++------------------ 2 files changed, 40 insertions(+), 32 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index d99b71bc2c66..6377dc76dc41 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -99,7 +99,8 @@ struct mem_cgroup_reclaim_iter { }; struct lruvec_stat { - long count[NR_VM_NODE_STAT_ITEMS]; + long node[NR_VM_NODE_STAT_ITEMS]; + long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; }; /* @@ -109,9 +110,8 @@ struct mem_cgroup_per_node { struct lruvec lruvec; struct lruvec_stat __percpu *lruvec_stat_cpu; - atomic_long_t lruvec_stat[NR_VM_NODE_STAT_ITEMS]; - - unsigned long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; + atomic_long_t node_stat[NR_VM_NODE_STAT_ITEMS]; + atomic_long_t lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; struct mem_cgroup_reclaim_iter iter[DEF_PRIORITY + 1]; @@ -446,7 +446,7 @@ unsigned long mem_cgroup_get_lru_size(struct lruvec *lruvec, enum lru_list lru) mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec); for (zid = 0; zid < MAX_NR_ZONES; zid++) - nr_pages += mz->lru_zone_size[zid][lru]; + nr_pages += atomic64_read(&mz->lru_zone_size[zid][lru]); return nr_pages; } @@ -457,7 +457,7 @@ unsigned long mem_cgroup_get_zone_lru_size(struct lruvec *lruvec, struct mem_cgroup_per_node *mz; mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - return mz->lru_zone_size[zone_idx][lru]; + return atomic64_read(&mz->lru_zone_size[zone_idx][lru]); } void mem_cgroup_handle_over_high(void); @@ -575,7 +575,7 @@ static inline unsigned long lruvec_page_state(struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - x = atomic_long_read(&pn->lruvec_stat[idx]); + x = atomic_long_read(&pn->node_stat[idx]); #ifdef CONFIG_SMP if (x < 0) x = 0; @@ -601,12 +601,12 @@ static inline void __mod_lruvec_state(struct lruvec *lruvec, __mod_memcg_state(pn->memcg, idx, val); /* Update lruvec */ - x = val + __this_cpu_read(pn->lruvec_stat_cpu->count[idx]); + x = val + __this_cpu_read(pn->lruvec_stat_cpu->node[idx]); if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { - atomic_long_add(x, &pn->lruvec_stat[idx]); + atomic_long_add(x, &pn->node_stat[idx]); x = 0; } - __this_cpu_write(pn->lruvec_stat_cpu->count[idx], x); + __this_cpu_write(pn->lruvec_stat_cpu->node[idx], x); } static inline void mod_lruvec_state(struct lruvec *lruvec, @@ -619,6 +619,29 @@ static inline void mod_lruvec_state(struct lruvec *lruvec, local_irq_restore(flags); } +/** + * __mod_lru_zone_size - update memcg lru statistics in batches + * + * Updates memcg lru statistics using per-cpu counters that spill into atomics + * above a threshold. + * + * Assumes that the caller has disabled preemption. IRQs may be enabled + * because this function is not called from irq context. + */ +static inline void __mod_lru_zone_size(struct mem_cgroup_per_node *pn, + enum lru_list lru, int zid, int val) +{ + long x; + struct lruvec_stat __percpu *lruvec_stat_cpu = pn->lruvec_stat_cpu; + + x = val + __this_cpu_read(lruvec_stat_cpu->lru_zone_size[zid][lru]); + if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { + atomic_long_add(x, &pn->lru_zone_size[zid][lru]); + x = 0; + } + __this_cpu_write(lruvec_stat_cpu->lru_zone_size[zid][lru], x); +} + static inline void __mod_lruvec_page_state(struct page *page, enum node_stat_item idx, int val) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2bd3df3d101a..5463ad160e10 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -962,36 +962,20 @@ struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgd * @zid: zone id of the accounted pages * @nr_pages: positive when adding or negative when removing * - * This function must be called under lru_lock, just before a page is added - * to or just after a page is removed from an lru list (that ordering being - * so as to allow it to check that lru_size 0 is consistent with list_empty). + * This function must be called just before a page is added to, or just after a + * page is removed from, an lru list. Callers aren't required to hold lru_lock + * because these statistics use per-cpu counters and atomics. */ void mem_cgroup_update_lru_size(struct lruvec *lruvec, enum lru_list lru, int zid, int nr_pages) { struct mem_cgroup_per_node *mz; - unsigned long *lru_size; - long size; if (mem_cgroup_disabled()) return; mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - lru_size = &mz->lru_zone_size[zid][lru]; - - if (nr_pages < 0) - *lru_size += nr_pages; - - size = *lru_size; - if (WARN_ONCE(size < 0, - "%s(%p, %d, %d): lru_size %ld\n", - __func__, lruvec, lru, nr_pages, size)) { - VM_BUG_ON(1); - *lru_size = 0; - } - - if (nr_pages > 0) - *lru_size += nr_pages; + __mod_lru_zone_size(mz, lru, zid, nr_pages); } bool task_in_mem_cgroup(struct task_struct *task, struct mem_cgroup *memcg) @@ -1833,9 +1817,10 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) struct mem_cgroup_per_node *pn; pn = mem_cgroup_nodeinfo(memcg, nid); - x = this_cpu_xchg(pn->lruvec_stat_cpu->count[i], 0); + x = this_cpu_xchg(pn->lruvec_stat_cpu->node[i], + 0); if (x) - atomic_long_add(x, &pn->lruvec_stat[i]); + atomic_long_add(x, &pn->node_stat[i]); } }