From patchwork Tue Aug 13 20:47:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kinsey Ho X-Patchwork-Id: 13762483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44911C52D7B for ; Tue, 13 Aug 2024 20:48:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 08E8D6B0095; Tue, 13 Aug 2024 16:48:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 018C96B0098; Tue, 13 Aug 2024 16:48:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D312B6B0099; Tue, 13 Aug 2024 16:48:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B52BA6B0095 for ; Tue, 13 Aug 2024 16:48:03 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 757DCA0A92 for ; Tue, 13 Aug 2024 20:48:03 +0000 (UTC) X-FDA: 82448409246.08.42D08EE Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf22.hostedemail.com (Postfix) with ESMTP id ADBB5C0027 for ; Tue, 13 Aug 2024 20:48:01 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4STg+RO4; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3gMa7ZggKCDkfdinZtcjbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--kinseyho.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3gMa7ZggKCDkfdinZtcjbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--kinseyho.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723582069; a=rsa-sha256; cv=none; b=3FKi0lmmuNxMup7yiZqUcrTrs/72ChRW6nf4RY3zjMz89eMvtZoiMMgg+qgthtXH2uQ3hi WoJV93P03UIxBl9TiczCPRRJ+4KNeTz7ljaLG5DeburttKqYgm/64yKknLETJ1DIUkoYRS 8OleKRb9KU/MMPaXztCs8f0PfCjkiWo= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4STg+RO4; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3gMa7ZggKCDkfdinZtcjbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--kinseyho.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3gMa7ZggKCDkfdinZtcjbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--kinseyho.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723582069; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AL9cOogrL47hO93mBXXtaHX7JJ+ZXjoDFTtox3tfTp8=; b=j2rVRCWAnDu8bfj7171tQSYD0jxuUgXVeVhHAfsYjcSdJ4u+H1aVVIwyxAOKNv3Al+n8CH nCavvaFROKIdJmO1SrENQLXqOyT8MhTyGhA9gyHf0s0T9rrIFPJifjnZS3Dug3i6yRyjbe Aj2CXjFooJy/eqCM/i7mpnRyr0qxR6k= Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-7c3da1ac936so1439107a12.2 for ; Tue, 13 Aug 2024 13:48:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723582080; x=1724186880; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AL9cOogrL47hO93mBXXtaHX7JJ+ZXjoDFTtox3tfTp8=; b=4STg+RO4NKn8gSuBe2QmAqxeAGrXceQmKKRK0X5XzlfVCTmgpG7wtmYPVa0i2NnKcy CLXupu4GQNCV8h+A15PlNfmx+0LMaGa/EvgGeONmoYuSJwpdaynAYJVgRkHVESd0JGZx yDBz9MEGgfOfIH5M8G1NBEDisLHhDR8+yDySdK5Fd7hnhCLxjfoDeigoRGq2sQBUh14G zHEmnAc9Wrqn3Mj3ZjcHuHwIeIcIvFL0i4nEA/kaxSyMF7VGafcz/gTin5MGIuVyeVn7 nfdQXg4gRux3av8wagLbYXKJ6oFWtThIuGw0bOG3yDgx9aJqDd3M1IyPrkkX1yeXpPpC R45A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723582080; x=1724186880; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AL9cOogrL47hO93mBXXtaHX7JJ+ZXjoDFTtox3tfTp8=; b=qfncWwFRTQclbuNL/N6nhl4x44U69pFdBdH3F1y3hMtutvMkUxdla3YYJeFntq9Ytp 0LM9tbuEDfQuRQZTsp1nwMYh3Ae99TNRZQqUEoboUAimmESd45n//R22+UymHH7GWWhG xlRSQgji/OShpxFeZxSYeOG9ejfcxB7f5f3twXmeWD1CUyTG/BTiDnElxNdNSaNJrNXI 4w9cH5niy19qJXzu83KC6Wp2QPKxvLKOoEFYR4SXG9XLG6Jf62keUAiq955qTPpVCFeI ILET+TibH1iqX3fhpTTjByEbu2cxYm/gZYqk21mQ6sFsfI9nxWjvdYHxGySuphsAxaDk KWpg== X-Gm-Message-State: AOJu0Yzfsip6yvey1JvkNlXRJk54wDeAqfjDeGQpKgi6znNx5SCleE+T aypb8gCS29UH/OidZaVgJx12heg/hYNqLtuNWbeoRifOQ6Bc7W6XryvMZwW+DjuFdsG7yDJImj7 wpr2Vz/e3Mg== X-Google-Smtp-Source: AGHT+IFYKAB9rOjPeQV1OvEWBj+/G0SON71UCsBpqeijGE33YYArHBwlllAogxZUus4RJ93ox/3eY4q8R0h4Eg== X-Received: from kinseyct.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:46b]) (user=kinseyho job=sendgmr) by 2002:a63:4045:0:b0:77d:336c:a857 with SMTP id 41be03b00d2f7-7c6a588a314mr766a12.11.1723582080284; Tue, 13 Aug 2024 13:48:00 -0700 (PDT) Date: Tue, 13 Aug 2024 20:47:14 +0000 In-Reply-To: <20240813204716.842811-1-kinseyho@google.com> Mime-Version: 1.0 References: <20240813204716.842811-1-kinseyho@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813204716.842811-5-kinseyho@google.com> Subject: [PATCH mm-unstable v2 4/5] mm: restart if multiple traversals raced From: Kinsey Ho To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Yosry Ahmed , Roman Gushchin , Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Tejun Heo , Zefan Li , mkoutny@suse.com, Kinsey Ho X-Rspam-User: X-Rspamd-Queue-Id: ADBB5C0027 X-Rspamd-Server: rspam01 X-Stat-Signature: uangn4krh4rx8x433mhdmeqcdhriht5z X-HE-Tag: 1723582081-818238 X-HE-Meta: U2FsdGVkX18PhKz52ae6DG2AbXrBWjSp5+8V4Ne8E++B49wgO4I13aIwRjLOZSuFzsDcc2R9Ba16T6No92kKi+SdJYLcG4j+MnqFQNET8FjbSdKT0LPqGdXjPc/KTB/jqoV5W+Q1q+FSCjop52KBziDkIsb3sGYO6bn9HmiuQVE8kSfphhvKT15Ef3IpJHNVDGDNSU1n9L3AbkSZoNrUziRvOf1KwtvCnyFLC1vU65tHQHAEGJGfE3ijzK9UTfg7hsze3EeUqj/zRHi9Ivd5W17KxIziH6M2+JlILOHhdUIzUlLNmPBGE0gz/kTpnknMj0VqOZ5GscW4/Fdo8qZ7YEdpCxgBn4Ge3nVD8o4EBa5YuJaczv+pSZz+1dgcVUlvVHotTa6w+wiH42Gn2N0hrXzFe9f1sfIhYxcxnOgq9Fmy+aF1tSsSlOmGYXRRArRQaJNRlVbWvNSlL3JzVeLW61pog2tXm1RW2q/zupBXCsp09Yrwq0w7LEDwDlWYj8yyAfrgueTnp+ZKM/r4QO7CQzbBH7Nbed9RKGKDob7rW28tpZ3nw3pFHNK0fuAkmHhh1PwdalxsjCTcB+UdwytiXPwjwLdexVkUOSx2jOXT5SBCHAgphvaQ4tXRgVLGDP1ikg8svsYi685VO0FHYuXifuahTmCYWU7AufATon25rbuvQ0iQgt/eZEYcaDaoB9KvZAUrz8Sjj3lKhSaPdQNl06JkuNGotJ4OYuq4B6KL97qJv8DONk9yOgiD2coqPtf2MT/3QsmQ7iabeCw3GCdYCOHr4q5L7B8f2GxFhJsmSBcRCCb6ZRba4BGg8IW2VKbYP1WmReQIdf+bQRSNTzZb+Nqubq9EHHrKORe7gfkkBauE+Eg+mOkBqcVdpUNDMzIjCx4Ng2+Z6imA9lc7hVZfaxlormISWebyNTQBUiup1NxNGHhONh5TwBBfNP61oA56Xr2u/TJbA5DfkObWcrK WqiQpDTg aUxryiK9vzaDL65FUwovd6C+ZADMtKutqC/RdAaOYsDWAH9H9g7P57Ztms4u3Xpn+1hU6cK2g8VKHhv7dj4n68bNsAoO8YhLMtNzIhi76rqw0l/LJhLuZZMp0/pl5eWjYmaGlhCshhKBMgRCQozqKhwTkx6AOfrw4S1eskBltNGxmSjFJCukyr7pARfIEP5be3M20E4py+Rx+qaHnCfsoZeUQvjm7KHvz8cpDB1qHSJ2htPw0aomUD8jZL9w73wQRka7pJHwn4FHmCfKBL3Er3GFkUWDDL2yxhS0hUwcK6tkB0QHqyPdPXS2dWmLMJe5+QoA4bo0vFup5vkWScZAC+Vxwu6d+pCAUcgG/q9clw3UPQ1hN0yu+cO1iB3HLY5yUIfHfdFQ3TdXh+pQZiDRpmAZ8CBWSYNviljZr+5AKjMJILn4qwxxzyD0+PuoE+NFA9Hc0BFXzx34/GS1vnoohFkg575FsmxyzRbtKwoqDEASA6a4HEd6q/Uhi3q7B269bkD5ACXOGbVQKKrzGpCb7mb5bkg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, if multiple reclaimers raced on the same position, the reclaimers which detect the race will still reclaim from the same memcg. Instead, the reclaimers which detect the race should move on to the next memcg in the hierarchy. So, in the case where multiple traversals race, jump back to the start of the mem_cgroup_iter() function to find the next memcg in the hierarchy to reclaim from. Signed-off-by: Kinsey Ho --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 22 ++++++++++++++-------- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 1aaed2f1f6ae..aada9ef3ca44 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -57,7 +57,7 @@ enum memcg_memory_event { struct mem_cgroup_reclaim_cookie { pg_data_t *pgdat; - unsigned int generation; + int generation; }; #ifdef CONFIG_MEMCG @@ -77,7 +77,7 @@ struct lruvec_stats; struct mem_cgroup_reclaim_iter { struct mem_cgroup __rcu *position; /* scan generation, increased every round-trip */ - unsigned int generation; + atomic_t generation; }; /* diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 937b7efc41ca..84de46ece9a9 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1025,7 +1025,7 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup *root, struct mem_cgroup_reclaim_cookie *reclaim) { struct mem_cgroup_reclaim_iter *iter; - struct cgroup_subsys_state *css = NULL; + struct cgroup_subsys_state *css; struct mem_cgroup *memcg = NULL; struct mem_cgroup *pos = NULL; @@ -1038,18 +1038,20 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup *root, rcu_read_lock(); restart: if (reclaim) { + int gen; struct mem_cgroup_per_node *mz; mz = root->nodeinfo[reclaim->pgdat->node_id]; iter = &mz->iter; + gen = atomic_read(&iter->generation); /* * On start, join the current reclaim iteration cycle. * Exit when a concurrent walker completes it. */ if (!prev) - reclaim->generation = iter->generation; - else if (reclaim->generation != iter->generation) + reclaim->generation = gen; + else if (reclaim->generation != gen) goto out_unlock; pos = rcu_dereference(iter->position); @@ -1057,8 +1059,7 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup *root, pos = prev; } - if (pos) - css = &pos->css; + css = pos ? &pos->css : NULL; for (;;) { css = css_next_descendant_pre(css, &root->css); @@ -1072,21 +1073,26 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup *root, * and kicking, and don't take an extra reference. */ if (css == &root->css || css_tryget(css)) { - memcg = mem_cgroup_from_css(css); break; } } + memcg = mem_cgroup_from_css(css); + if (reclaim) { /* * The position could have already been updated by a competing * thread, so check that the value hasn't changed since we read * it to avoid reclaiming from the same cgroup twice. */ - (void)cmpxchg(&iter->position, pos, memcg); + if (cmpxchg(&iter->position, pos, memcg) != pos) { + if (css && css != &root->css) + css_put(css); + goto restart; + } if (!memcg) { - iter->generation++; + atomic_inc(&iter->generation); /* * Reclaimers share the hierarchy walk, and a