From patchwork Wed Feb 17 20:41:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Chen X-Patchwork-Id: 12092439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B97B6C433E6 for ; Wed, 17 Feb 2021 21:41:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 507116186A for ; Wed, 17 Feb 2021 21:41:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 507116186A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9485B6B006C; Wed, 17 Feb 2021 16:41:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 920DE8D0001; Wed, 17 Feb 2021 16:41:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E9636B0070; Wed, 17 Feb 2021 16:41:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0130.hostedemail.com [216.40.44.130]) by kanga.kvack.org (Postfix) with ESMTP id 6A9A06B006C for ; Wed, 17 Feb 2021 16:41:55 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 3312C283D for ; Wed, 17 Feb 2021 21:41:55 +0000 (UTC) X-FDA: 77829082590.22.E8519E1 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf09.hostedemail.com (Postfix) with ESMTP id D2748600249D for ; Wed, 17 Feb 2021 21:41:51 +0000 (UTC) IronPort-SDR: 0mvoRyU7woPeozKhtmZnSI4mqi3w9iZRf7GtTZhOXI1474p07RaACLkfVyYk/LxboAiH0SNf+k whwGcCj7PV8A== X-IronPort-AV: E=McAfee;i="6000,8403,9898"; a="182538753" X-IronPort-AV: E=Sophos;i="5.81,185,1610438400"; d="scan'208";a="182538753" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Feb 2021 13:41:52 -0800 IronPort-SDR: gQswESc8jFJCnGPHICQznAp1G92RTJTrzHTvajwRGCPDLpJzqZfkC8RbeDPAX+SmqhuWKVZuxY gvB7cnNEntVg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,185,1610438400"; d="scan'208";a="401430707" Received: from skl-02.jf.intel.com ([10.54.74.28]) by orsmga007.jf.intel.com with ESMTP; 17 Feb 2021 13:41:52 -0800 From: Tim Chen To: Andrew Morton , Johannes Weiner , Michal Hocko , Vladimir Davydov Cc: Tim Chen , Dave Hansen , Ying Huang , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 1/3] mm: Fix dropped memcg from mem cgroup soft limit tree Date: Wed, 17 Feb 2021 12:41:34 -0800 Message-Id: <8d35206601ccf0e1fe021d24405b2a0c2f4e052f.1613584277.git.tim.c.chen@linux.intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D2748600249D X-Stat-Signature: egrk6gqy9uuwmoyx68jhsq975ufq16xo Received-SPF: none (linux.intel.com>: No applicable sender policy available) receiver=imf09; identity=mailfrom; envelope-from=""; helo=mga14.intel.com; client-ip=192.55.52.115 X-HE-DKIM-Result: none/none X-HE-Tag: 1613598111-511557 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: During soft limit memory reclaim, we will temporarily remove the target mem cgroup from the cgroup soft limit tree. We then perform memory reclaim, update the memory usage excess count and re-insert the mem cgroup back into the mem cgroup soft limit tree according to the new memory usage excess count. However, when memory reclaim failed for a maximum number of attempts and we bail out of the reclaim loop, we forgot to put the target mem cgroup chosen for next reclaim back to the soft limit tree. This prevented pages in the mem cgroup from being reclaimed in the future even though the mem cgroup exceeded its soft limit. Fix the logic and put the mem cgroup back on the tree when page reclaim failed for the mem cgroup. Reviewed-by: Ying Huang Signed-off-by: Tim Chen Acked-by: Michal Hocko --- mm/memcontrol.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ed5cc78a8dbf..a51bf90732cb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3505,8 +3505,12 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, loop > MEM_CGROUP_MAX_SOFT_LIMIT_RECLAIM_LOOPS)) break; } while (!nr_reclaimed); - if (next_mz) + if (next_mz) { + spin_lock_irq(&mctz->lock); + __mem_cgroup_insert_exceeded(next_mz, mctz, excess); + spin_unlock_irq(&mctz->lock); css_put(&next_mz->memcg->css); + } return nr_reclaimed; } From patchwork Wed Feb 17 20:41:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Chen X-Patchwork-Id: 12092441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C726AC433E0 for ; Wed, 17 Feb 2021 21:41:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 356CC64E5F for ; Wed, 17 Feb 2021 21:41:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 356CC64E5F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 294B48D0002; Wed, 17 Feb 2021 16:41:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 21B588D0001; Wed, 17 Feb 2021 16:41:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 10A438D0002; Wed, 17 Feb 2021 16:41:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0193.hostedemail.com [216.40.44.193]) by kanga.kvack.org (Postfix) with ESMTP id EE4D08D0001 for ; Wed, 17 Feb 2021 16:41:56 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id BD64F18485042 for ; Wed, 17 Feb 2021 21:41:56 +0000 (UTC) X-FDA: 77829082632.05.0733372 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf09.hostedemail.com (Postfix) with ESMTP id 29D82600249C for ; Wed, 17 Feb 2021 21:41:52 +0000 (UTC) IronPort-SDR: zjS6ll4qNx/RC0JrekmY24E9LdHPHOQzCyYkxAAl7xjtjFMjoru/S/AhVmeyOARbbKBLaMf8tZ ilgxk0ESPiSg== X-IronPort-AV: E=McAfee;i="6000,8403,9898"; a="182538758" X-IronPort-AV: E=Sophos;i="5.81,185,1610438400"; d="scan'208";a="182538758" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Feb 2021 13:41:55 -0800 IronPort-SDR: 9Hq9KtM7miCiBDav7pvUqi5nyuDRGHZpaD7BBiidOVHP9uYoSc5++KYkN6vXYDuFE8r3evNcC+ tofPWpXB6Xqw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,185,1610438400"; d="scan'208";a="401430712" Received: from skl-02.jf.intel.com ([10.54.74.28]) by orsmga007.jf.intel.com with ESMTP; 17 Feb 2021 13:41:55 -0800 From: Tim Chen To: Andrew Morton , Johannes Weiner , Michal Hocko , Vladimir Davydov Cc: Tim Chen , Dave Hansen , Ying Huang , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/3] mm: Force update of mem cgroup soft limit tree on usage excess Date: Wed, 17 Feb 2021 12:41:35 -0800 Message-Id: <06f1f92f1f7d4e57c4e20c97f435252c16c60a27.1613584277.git.tim.c.chen@linux.intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 29D82600249C X-Stat-Signature: 6ay7ip3wn9esmis5xqyybhn6ktqeirim Received-SPF: none (linux.intel.com>: No applicable sender policy available) receiver=imf09; identity=mailfrom; envelope-from=""; helo=mga14.intel.com; client-ip=192.55.52.115 X-HE-DKIM-Result: none/none X-HE-Tag: 1613598112-27128 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To rate limit updates to the mem cgroup soft limit tree, we only perform updates every SOFTLIMIT_EVENTS_TARGET (defined as 1024) memory events. However, this sampling based updates may miss a critical update: i.e. when the mem cgroup first exceeded its limit but it was not on the soft limit tree. It should be on the tree at that point so it could be subjected to soft limit page reclaim. If the mem cgroup had few memory events compared with other mem cgroups, we may not update it and place in on the mem cgroup soft limit tree for many memory events. And this mem cgroup excess usage could creep up and the mem cgroup could be hidden from the soft limit page reclaim for a long time. Fix this issue by forcing an update to the mem cgroup soft limit tree if a mem cgroup has exceeded its memory soft limit but it is not on the mem cgroup soft limit tree. Reviewed-by: Ying Huang Signed-off-by: Tim Chen Reported-by: kernel test robot --- mm/memcontrol.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a51bf90732cb..d72449eeb85a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -985,15 +985,22 @@ static bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg, */ static void memcg_check_events(struct mem_cgroup *memcg, struct page *page) { + struct mem_cgroup_per_node *mz; + bool force_update = false; + + mz = mem_cgroup_nodeinfo(memcg, page_to_nid(page)); + if (mz && !mz->on_tree && soft_limit_excess(mz->memcg) > 0) + force_update = true; + /* threshold event is triggered in finer grain than soft limit */ - if (unlikely(mem_cgroup_event_ratelimit(memcg, + if (unlikely((force_update) || mem_cgroup_event_ratelimit(memcg, MEM_CGROUP_TARGET_THRESH))) { bool do_softlimit; do_softlimit = mem_cgroup_event_ratelimit(memcg, MEM_CGROUP_TARGET_SOFTLIMIT); mem_cgroup_threshold(memcg); - if (unlikely(do_softlimit)) + if (unlikely((force_update) || do_softlimit)) mem_cgroup_update_tree(memcg, page); } } From patchwork Wed Feb 17 20:41:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Chen X-Patchwork-Id: 12092443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57493C433E6 for ; Wed, 17 Feb 2021 21:42:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 07BA16186A for ; Wed, 17 Feb 2021 21:41:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 07BA16186A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 242EA8D0003; Wed, 17 Feb 2021 16:41:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F5FA8D0001; Wed, 17 Feb 2021 16:41:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3C2A8D0003; Wed, 17 Feb 2021 16:41:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0019.hostedemail.com [216.40.44.19]) by kanga.kvack.org (Postfix) with ESMTP id DE2D38D0001 for ; Wed, 17 Feb 2021 16:41:58 -0500 (EST) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id A6A1B18481CB6 for ; Wed, 17 Feb 2021 21:41:58 +0000 (UTC) X-FDA: 77829082716.13.C4CB1C6 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf09.hostedemail.com (Postfix) with ESMTP id 25F69600249C for ; Wed, 17 Feb 2021 21:41:54 +0000 (UTC) IronPort-SDR: rzLMIAB/XrG/gu5pGuHOB2m9vgEpxkt2GZgEjVeIzPmzTeuJrqo4QpZJxPuwrIrdSa1xn6f16M xQWtwQeTJUZA== X-IronPort-AV: E=McAfee;i="6000,8403,9898"; a="182538764" X-IronPort-AV: E=Sophos;i="5.81,185,1610438400"; d="scan'208";a="182538764" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Feb 2021 13:41:57 -0800 IronPort-SDR: fyXWGY+qTbQbuVmeCtCGViaNOmO5KLB9GCktnf6sqEPS4OOLdfqg8pm8qyxyXHVzxofdAq+crC i8lmAP2b6Ivw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,185,1610438400"; d="scan'208";a="401430721" Received: from skl-02.jf.intel.com ([10.54.74.28]) by orsmga007.jf.intel.com with ESMTP; 17 Feb 2021 13:41:57 -0800 From: Tim Chen To: Andrew Morton , Johannes Weiner , Michal Hocko , Vladimir Davydov Cc: Tim Chen , Dave Hansen , Ying Huang , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 3/3] mm: Fix missing mem cgroup soft limit tree updates Date: Wed, 17 Feb 2021 12:41:36 -0800 Message-Id: X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 25F69600249C X-Stat-Signature: n91t3bfex1bbr1kcqhhx5pzwrnqd7oy8 Received-SPF: none (linux.intel.com>: No applicable sender policy available) receiver=imf09; identity=mailfrom; envelope-from=""; helo=mga14.intel.com; client-ip=192.55.52.115 X-HE-DKIM-Result: none/none X-HE-Tag: 1613598114-115184 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On a per node basis, the mem cgroup soft limit tree on each node tracks how much a cgroup has exceeded its soft limit memory limit and sorts the cgroup by its excess usage. On page release, the trees are not updated right away, until we have gathered a batch of pages belonging to the same cgroup. This reduces the frequency of updating the soft limit tree and locking of the tree and associated cgroup. However, the batch of pages could contain pages from multiple nodes but only the soft limit tree from one node would get updated. Change the logic so that we update the tree in batch of pages, with each batch of pages all in the same mem cgroup and memory node. An update is issued for the batch of pages of a node collected till now whenever we encounter a page belonging to a different node. Note that this batching for the same node logic is only relevant for v1 cgroup that has a memory soft limit. Reviewed-by: Ying Huang Signed-off-by: Tim Chen --- mm/memcontrol.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d72449eeb85a..8bddee75f5cb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6804,6 +6804,7 @@ struct uncharge_gather { unsigned long pgpgout; unsigned long nr_kmem; struct page *dummy_page; + int nid; }; static inline void uncharge_gather_clear(struct uncharge_gather *ug) @@ -6849,7 +6850,13 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) * exclusive access to the page. */ - if (ug->memcg != page_memcg(page)) { + if (ug->memcg != page_memcg(page) || + /* + * Update soft limit tree used in v1 cgroup in page batch for + * the same node. Relevant only to v1 cgroup with a soft limit. + */ + (ug->dummy_page && ug->nid != page_to_nid(page) && + ug->memcg->soft_limit != PAGE_COUNTER_MAX)) { if (ug->memcg) { uncharge_batch(ug); uncharge_gather_clear(ug); @@ -6869,6 +6876,7 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) ug->pgpgout++; ug->dummy_page = page; + ug->nid = page_to_nid(page); page->memcg_data = 0; css_put(&ug->memcg->css); }