From patchwork Fri Sep 20 22:11:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kaiyang Zhao X-Patchwork-Id: 13808713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C6DBCF9C69 for ; Fri, 20 Sep 2024 22:12:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 090B56B008C; Fri, 20 Sep 2024 18:12:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 03F126B0092; Fri, 20 Sep 2024 18:12:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DFBA86B0093; Fri, 20 Sep 2024 18:12:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BE7746B008C for ; Fri, 20 Sep 2024 18:12:55 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5B4D4C06E7 for ; Fri, 20 Sep 2024 22:12:55 +0000 (UTC) X-FDA: 82586517510.18.19B4B34 Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) by imf22.hostedemail.com (Postfix) with ESMTP id 6E9BDC0007 for ; Fri, 20 Sep 2024 22:12:53 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=cs.cmu.edu header.s=google-2021 header.b=UnAhzWel; spf=pass (imf22.hostedemail.com: domain of kaiyang2@andrew.cmu.edu designates 209.85.219.53 as permitted sender) smtp.mailfrom=kaiyang2@andrew.cmu.edu; dmarc=pass (policy=none) header.from=cs.cmu.edu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726870258; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dua3EzvkneWM3TAvRq2/3u51msVEEUUg6Zhqmm/qfGM=; b=3Oa6LKQfcsEW+XH2uUMxVBZI/md/rX5gQzfppM2HY3Otm5wcDD32SUlB1XA399iHwMfekH xEJs+u4WiV/W5htQdPXZ1RvMAB3/pktNTB43Bh8FUR3V9k1TE4rIlXLO8zrLL97jhqXKLW Jfg/XFKy6A2gN2Iy9e/8qenslEx0NfU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726870258; a=rsa-sha256; cv=none; b=VQekYlAHTSTQMRo+ZTHWb829ptNO8+BFthDMftZBDwPGIp/mU75zc07T3/T+9ylfsCtbPH cY12jkEQgKxKxU9d/vr/7FF4dt85U8hqq6K5rza2Cpt4SVHTXSu2lPFn8SgpMm101kCgYr mGnzLiVZQ7f9cBy/QIvoO4e7U9DMR+c= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=cs.cmu.edu header.s=google-2021 header.b=UnAhzWel; spf=pass (imf22.hostedemail.com: domain of kaiyang2@andrew.cmu.edu designates 209.85.219.53 as permitted sender) smtp.mailfrom=kaiyang2@andrew.cmu.edu; dmarc=pass (policy=none) header.from=cs.cmu.edu Received: by mail-qv1-f53.google.com with SMTP id 6a1803df08f44-6c34c02ff1cso15810566d6.2 for ; Fri, 20 Sep 2024 15:12:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.cmu.edu; s=google-2021; t=1726870372; x=1727475172; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dua3EzvkneWM3TAvRq2/3u51msVEEUUg6Zhqmm/qfGM=; b=UnAhzWela+QLM4lEg4U+dD50PEXA0sTNAK+vnJTjlW2iy1I8a4NS02yaiyvhDeP0dm wYqKAJ5e2x+FB6p8cxLb5FG94LY1Vn22qAoRypiUl7EiWIdToL67fA4beVBiMGmkCcLK 2I0DB4ytCuqd6wSxb9p+eQx4L0+8lEOUHM96iviClQvcnI6SRS/irC4nA3CAivI5CeMX uIfhZwY1txgnLD4waZD598BPW5qq6MXypRS40b9KLpV7rEhQ8C65UijsCuI6hGqRH5l1 NGzL+ZzvlGo2xfOWGTV6z3CgPldJJ3Yzf7Z/65vPXaXWHrmT/KXWSFqxkdhQuKbM3s3J f6OA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726870372; x=1727475172; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dua3EzvkneWM3TAvRq2/3u51msVEEUUg6Zhqmm/qfGM=; b=p3kcoO5uqiaLmwvkbj4xfT6kFv9ss7uDxi7Ae1VdqE9/CXHnWeFQqVjudLHQ1CYthR fBUfub0eyGyLrWk0/TB0Eden+UarTSJkyaU4wRHjZ9vVSU/LAZgiiQKvyZ8grwB7lj6+ ffo0mvHYysDjTw5p3QfyLMvpbD4jIEssJE/atkgQrz3gZhfPu+3nRtQ4J+Z/awUMfeFS eHjJbHCZPNBrXa3hxNE6SOJWiLT4juushD+FpKwG/y1tDHRHoZpFg1tFBuLQXqUf6/Uw TjZ7jAO3lLsnuXPr4fSp2EaQ00bizPyl+dIQ9dimLDMY208XRZ19h9+riPMbMB7bVQv1 gaew== X-Gm-Message-State: AOJu0YwJ8S0l8QXTz8k74KsJsVPw5/qkD0uN0arbMvnCly9wXUHqwE/t 9nKLK0GDOJ7nXKv4N8jmGi9zWAfHjC6DUFPAd0HzBT4ena2y0Uey5X9gexhwwHP6qtx8kvMhn5M eKp10jym50KhAUusrx1x+A46Fyu6G1pTHjSESkFQHLEWaoNLo1CDTzz+qgIH7kp0dqHHPgBx6u1 qLSSLOGziVmQW/E1YLbzLWI+0Q3JOL8EIKjNkyjA== X-Google-Smtp-Source: AGHT+IFt0V61eP797QLmbHgiSqqaJMC8ZiLd28uIJxGM3kNR5lwGLx+JECFMYHwxRLmmCyhdNYC+KA== X-Received: by 2002:a05:6214:5c4a:b0:6c5:32a5:567a with SMTP id 6a1803df08f44-6c7bd486c6bmr56855866d6.1.1726870372363; Fri, 20 Sep 2024 15:12:52 -0700 (PDT) Received: from localhost (pool-74-98-231-160.pitbpa.fios.verizon.net. [74.98.231.160]) by smtp.gmail.com with UTF8SMTPSA id 6a1803df08f44-6c75e557e58sm23236106d6.81.2024.09.20.15.12.51 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 20 Sep 2024 15:12:52 -0700 (PDT) From: kaiyang2@cs.cmu.edu To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, mhocko@kernel.org, nehagholkar@meta.com, abhishekd@meta.com, hannes@cmpxchg.org, weixugc@google.com, rientjes@google.com, Kaiyang Zhao Subject: [RFC PATCH 3/4] use memory.low local node protection for local node reclaim Date: Fri, 20 Sep 2024 22:11:50 +0000 Message-ID: <20240920221202.1734227-4-kaiyang2@cs.cmu.edu> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240920221202.1734227-1-kaiyang2@cs.cmu.edu> References: <20240920221202.1734227-1-kaiyang2@cs.cmu.edu> MIME-Version: 1.0 X-Stat-Signature: rxerhrq7fbcd5socaf1ky1s9runfeike X-Rspamd-Queue-Id: 6E9BDC0007 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1726870373-121524 X-HE-Meta: U2FsdGVkX18kJ9v+2rOElhLkfMySS+KYl0uKPaXLNs8UkUi4XTkr+mKMj5Q/ophuiWrc+TNdHeWTXsj3B06tyjrY1JTUmC+WahcuvnNbapLTOIYMXDulzszGW3Hc1Rjdd5/j8JYRgKqkL/It0+dDnvoQrKCD+pq2CsEzXegY0OFaiYHf4TCwTMhQ6/86/ohbuqGLCVxkY40PhZZiSKLvhK4Upywso6T9h7T0MGME9C2G/v0mISEsRofITNa6dCfFFag3yuOBRjtL3KrZsFfn0MhMHBve6swdvOEBBEyuPb2niLFOVMT/0DyknQd0XG8p4f5aCbXHadwRjs2E4fAh86e21GnA8zOIHONBIswfMokkrFhlIdkLTIbbeGZDFTthOx5BaC1q0e28hKuDmH/tZG+ZU3Rd2BAwjM9KR2UlG2Ye0DBG04VXZ4YLz1plN5DLT7Nxo/m06/4R8dmJW7kL+1BmMXOBIhWX7PE9HpcJWpwSj8TGIjm6uKZf0+i0Ng/9sAn+/IEESp3ZxP91KYg9ZhcsMEzw/sTqEVyaBS8/xKJ5m4SxsJnojgb5+WvF57gLfl7L6xVC6j4SYORkGZ3FSqea052K97HHnGuqQvyrf0Y7v+gHLL+wdOhc6euGytX8ylpTaO7W871oZsAypWrTYt9KsU38KU/DcaO3sqWJ8KeaGax+M+wTe1sIbUf11Ga5BIfJ7iTkdrzDYml4Oi9K/P+qtx5KrPxsULt2+UmK5i0d4k2Z3bkjvudnWtxcyqBu7g2pZQtC8wV68mRG1EM1h9jAZ7a4d97Br5HE38H7vTiogzwqwH+7PgaEpXAmjW1FbmScd7cO1XKTfk/ntsXcn1fIH6oUY9EvoP4ZuyHK2JtjzqzwMgE5IjkeLPp9p28wGMQ77wiz00hF5Z/dWF54olMazojTNB/lckmbXWrcfvDTLgvE5wt3HjrmOpxqQ12tKSxbfCwdGIQImVofivd Xt9FGNcl i0pGPqbc3J49EtfMnMRiUAxK8YWWTx46+CsG9sP6HBATUjwVYAVFa5ZIONzCWl/T7Al38IiTl7YtS+jpuNVZ7RDS1YgOaHP7RIzGKRvJ6aqkiYOOwLKwqjdt59E6vYyEMspQ+rCYJiaRUOWfT0aBLIIef7t1CAKxaV0yziRN5jTGU4L4eXOKRrpmhP+OEGYkxbj2EK8TbBLEYCuP/iKNKVa0o1cjfpnrIGu7ds7/1B3RN/sYgm5Ph4hhQm5nqndmW3qx4qAWaQ0NpiS63KTnGk5Q4LwchC4BUnmFk1I+wuedRwn6K/FR5mhNvBS/BSRF8g1Hczehou1DQCh42duo9LJTl59tvnC2Z7pk4oZQ1Gu+5G0/fwWdPqN8QaAfCcEO4Uu/YbP5WxbHoKD4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kaiyang Zhao When reclaim targets the top-tier node usage by the root memcg, apply local memory.low protection instead of global protection. Signed-off-by: Kaiyang Zhao --- include/linux/memcontrol.h | 23 ++++++++++++++--------- mm/memcontrol.c | 4 ++-- mm/vmscan.c | 19 ++++++++++++++----- 3 files changed, 30 insertions(+), 16 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 94aba4498fca..256912b91922 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -586,9 +586,9 @@ static inline bool mem_cgroup_disabled(void) static inline void mem_cgroup_protection(struct mem_cgroup *root, struct mem_cgroup *memcg, unsigned long *min, - unsigned long *low) + unsigned long *low, unsigned long *locallow) { - *min = *low = 0; + *min = *low = *locallow = 0; if (mem_cgroup_disabled()) return; @@ -631,10 +631,11 @@ static inline void mem_cgroup_protection(struct mem_cgroup *root, *min = READ_ONCE(memcg->memory.emin); *low = READ_ONCE(memcg->memory.elow); + *locallow = READ_ONCE(memcg->memory.elocallow); } void mem_cgroup_calculate_protection(struct mem_cgroup *root, - struct mem_cgroup *memcg); + struct mem_cgroup *memcg, int is_local); static inline bool mem_cgroup_unprotected(struct mem_cgroup *target, struct mem_cgroup *memcg) @@ -651,13 +652,17 @@ static inline bool mem_cgroup_unprotected(struct mem_cgroup *target, unsigned long get_cgroup_local_usage(struct mem_cgroup *memcg, bool flush); static inline bool mem_cgroup_below_low(struct mem_cgroup *target, - struct mem_cgroup *memcg) + struct mem_cgroup *memcg, int is_local) { if (mem_cgroup_unprotected(target, memcg)) return false; - return READ_ONCE(memcg->memory.elow) >= - page_counter_read(&memcg->memory); + if (is_local) + return READ_ONCE(memcg->memory.elocallow) >= + get_cgroup_local_usage(memcg, true); + else + return READ_ONCE(memcg->memory.elow) >= + page_counter_read(&memcg->memory); } static inline bool mem_cgroup_below_min(struct mem_cgroup *target, @@ -1159,13 +1164,13 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm, static inline void mem_cgroup_protection(struct mem_cgroup *root, struct mem_cgroup *memcg, unsigned long *min, - unsigned long *low) + unsigned long *low, unsigned long *locallow) { *min = *low = 0; } static inline void mem_cgroup_calculate_protection(struct mem_cgroup *root, - struct mem_cgroup *memcg) + struct mem_cgroup *memcg, int is_local) { } @@ -1175,7 +1180,7 @@ static inline bool mem_cgroup_unprotected(struct mem_cgroup *target, return true; } static inline bool mem_cgroup_below_low(struct mem_cgroup *target, - struct mem_cgroup *memcg) + struct mem_cgroup *memcg, int is_local) { return false; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d7c5fff12105..61718ba998fe 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4495,7 +4495,7 @@ struct cgroup_subsys memory_cgrp_subsys = { * of a top-down tree iteration, not for isolated queries. */ void mem_cgroup_calculate_protection(struct mem_cgroup *root, - struct mem_cgroup *memcg) + struct mem_cgroup *memcg, int is_local) { bool recursive_protection = cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_RECURSIVE_PROT; @@ -4507,7 +4507,7 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root, root = root_mem_cgroup; page_counter_calculate_protection(&root->memory, &memcg->memory, - recursive_protection, false); + recursive_protection, is_local); } static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg, diff --git a/mm/vmscan.c b/mm/vmscan.c index ce471d686a88..a2681d52fc5f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2377,6 +2377,7 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, enum scan_balance scan_balance; unsigned long ap, fp; enum lru_list lru; + int is_local = (pgdat->node_id == 0) && root_reclaim(sc); /* If we have no swap space, do not bother scanning anon folios. */ if (!sc->may_swap || !can_reclaim_anon_pages(memcg, pgdat->node_id, sc)) { @@ -2457,12 +2458,14 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, for_each_evictable_lru(lru) { bool file = is_file_lru(lru); unsigned long lruvec_size; - unsigned long low, min; + unsigned long low, min, locallow; unsigned long scan; lruvec_size = lruvec_lru_size(lruvec, lru, sc->reclaim_idx); mem_cgroup_protection(sc->target_mem_cgroup, memcg, - &min, &low); + &min, &low, &locallow); + if (is_local) + low = locallow; if (min || low) { /* @@ -2494,7 +2497,12 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, * again by how much of the total memory used is under * hard protection. */ - unsigned long cgroup_size = mem_cgroup_size(memcg); + unsigned long cgroup_size; + + if (is_local) + cgroup_size = get_cgroup_local_usage(memcg, true); + else + cgroup_size = mem_cgroup_size(memcg); unsigned long protection; /* memory.low scaling, make sure we retry before OOM */ @@ -5869,6 +5877,7 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) }; struct mem_cgroup_reclaim_cookie *partial = &reclaim; struct mem_cgroup *memcg; + int is_local = (pgdat->node_id == 0) && root_reclaim(sc); /* * In most cases, direct reclaimers can do partial walks @@ -5896,7 +5905,7 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) */ cond_resched(); - mem_cgroup_calculate_protection(target_memcg, memcg); + mem_cgroup_calculate_protection(target_memcg, memcg, is_local); if (mem_cgroup_below_min(target_memcg, memcg)) { /* @@ -5904,7 +5913,7 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) * If there is no reclaimable memory, OOM. */ continue; - } else if (mem_cgroup_below_low(target_memcg, memcg)) { + } else if (mem_cgroup_below_low(target_memcg, memcg, is_local)) { /* * Soft protection. * Respect the protection only as long as