From patchwork Tue Dec 31 04:35:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13923617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2539E77188 for ; Tue, 31 Dec 2024 04:36:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F9996B0095; Mon, 30 Dec 2024 23:36:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 284FD6B0096; Mon, 30 Dec 2024 23:36:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D4796B0098; Mon, 30 Dec 2024 23:36:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D34226B0095 for ; Mon, 30 Dec 2024 23:36:01 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 62CEB47B38 for ; Tue, 31 Dec 2024 04:36:01 +0000 (UTC) X-FDA: 82953990672.18.6BDD170 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf14.hostedemail.com (Postfix) with ESMTP id F1521100005 for ; Tue, 31 Dec 2024 04:35:06 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=N5wNiM41; spf=pass (imf14.hostedemail.com: domain of 3rnRzZwYKCKcfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3rnRzZwYKCKcfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735619715; a=rsa-sha256; cv=none; b=J4PcB+EEcs9+dUqq5KX4SuciIdqTz9H4YF30SWoFHfXJ/dWv9PPaEsfXLz54Uq6a9926SR hh9TfzIYlEebKnBgal4Fo4FMPB06/IlrerijUA86/ISXKDQwsdmFXMpEraiaFFn4vXYT1Z FXO64Nuy7qjaSfPEQ8sjNug114emVCc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=N5wNiM41; spf=pass (imf14.hostedemail.com: domain of 3rnRzZwYKCKcfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3rnRzZwYKCKcfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735619715; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o6XRtkbexDya+FE96V37S0c0XRTdkGfwnwLQlfXry/0=; b=AVEiqAHH44vRljO+QZbhPOWZ9mgoYcLdJz3Hr3mKto8jHF/ge4PnbODTVRIg93pvLcscns rQBX5HHkioIg6jvHp4vkXd2AWj3VxKkrkpVus/pGO5XbSNOodftoocQsg3do/s4fWw3Sc+ gtyPuKppH96CDAZ26bkKR/SwzFYpmt0= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-21681a2c0d5so105657775ad.2 for ; Mon, 30 Dec 2024 20:35:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1735619758; x=1736224558; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=o6XRtkbexDya+FE96V37S0c0XRTdkGfwnwLQlfXry/0=; b=N5wNiM41PdzNF6sQilN/Wupp5LXqdl9ypv81KIrF3qBTP6NBL6reJ7chdM+jXz3s7Z O8BdvUf2JES9BA0ApbJ4aN63RSGwSd0VVS/8AQpMYrtHYdCQIWXi+j8IxoupFl/vwYGt nOH79VaJoNvIhYCKN5UGbbYiXQ1A+4ciiPkrI/vQkJyuZhoZtt+OabeLgDWH36ltcZqB CidAJWzrjo3UTwlQe29TIWxEc1iSp0CPNrVN3/4RtkOD6bLrnX0luVmCajxjAf6eSsV2 QfVeyWttcmX6wyfF0+A2MxmGMTflzMjzfm9ig/O2OQqDhdi8pVppZDWXFZWfDcEHhHpu GG6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735619758; x=1736224558; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=o6XRtkbexDya+FE96V37S0c0XRTdkGfwnwLQlfXry/0=; b=n3cYMLI71exB+OvAx8vg5w+5AYeaiaHNb5OsQTUiKFMEvTr9DUKdXlAYLh8iSX4+N/ nQw1QTWzELgUvc77h/N7UjQu8gSlc4TVjqETSou2dkUiwZls1ducQHmeBzkWigMOiPys fjdVi4YN4HS0B3SvsGD5z4rBtEgZSsuKpMZJOFiP/yyKSqEsvAOY/7i3Fbjwx8S4CAvh VmDjo9yx+I5Zac+rQ22iGqkDhitidni5I/fsGvZG3lukzR+STUI6lmHzpw2IRkQTb3s0 PvB27dmU6LuL60R8+VjEcqicf3y+KxUX42STSg6sNetcM6gdD/0SzqsQVGbjO9l4/9MJ 236g== X-Gm-Message-State: AOJu0YxQvUV/Wn3TgD7JEg1vg/l1f3DdD7mwDYY0GC8y0ef6Sorg5bEk 8R9/ZWjDA1dzK273bk2vqc1jDFxnSs2SSjzxW1o9e6nNTel+HeEYlJsreUN0d9yv9a1upGfGo5r img== X-Google-Smtp-Source: AGHT+IFGxXIKapCxHqzF+SSkn6gmP8awPE8NmQhmaf+NKsYjxZAWuTACPo/hzKbtWVqcyzBxJYfvlIdIpEw= X-Received: from plmm9.prod.google.com ([2002:a17:902:c449:b0:215:6f59:ff5f]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:d502:b0:216:4348:149d with SMTP id d9443c01a7336-219e6f25d60mr601318685ad.53.1735619758185; Mon, 30 Dec 2024 20:35:58 -0800 (PST) Date: Mon, 30 Dec 2024 21:35:36 -0700 In-Reply-To: <20241231043538.4075764-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241231043538.4075764-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241231043538.4075764-6-yuzhao@google.com> Subject: [PATCH mm-unstable v4 5/7] mm/mglru: rework refault detection From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Kairui Song , Kalesh Singh X-Rspamd-Queue-Id: F1521100005 X-Stat-Signature: 4zfj1neupxhnzx3nkkwrkeqi4he1r1e8 X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1735619706-433142 X-HE-Meta: U2FsdGVkX197kM3DCsWQgLvdnmSp9mXiirH5pQFgkil/xYxvKWmCG3KM5vs3/G01iQ602OH++/gQ54muyg1vdpxpjQYprfQ08hsOPYxzwTG3SuHpvl6dupPmahx0+pWhkf0Z6gNQTR9yB072xu+Rmxt99rvj7T3uXpPX99uimGke5bb/Cu/bTLw4Jud+1nxQk9xFxOs/YK9JPSLKeQwvebIQyw4AMBs9U54bo7SM3wrmjiLdgLQQtr2jD+9EyA0dYWYdV9vTEjTpJgomaoFklYc/B0vJX8BTLEW3GurKuxfyViF7Wup2sOZUAEsZY0I1I0uy6mJ9QPz2I9BudT24P2pl9jHNC94h9Rl4RiiW7QAJCrkufSiOhar90V78gZ2E5wzajgRNLQ1cEmPT7Vq1k/1zm5xpE9AObghd8KdSTzQR0KyyfLvfTcdfY8YytPWflTR9I+A6/QOfqWNVEMSxTIYD3+FeO2p5anlZcZbyx8OPOutPciqyZ8RxLBevlkjN24Eq0oymLwdiEzmV2ZDaQYnmEqVkDO7i8+rAZPJXU+ZmTiwaPxyA+01tv/D6H168QQr5Z4iOd5o7NDevYCa1BhTrbqP6YERvi4JwiMg6bV1S2HjaPd8edPcSW9ZUWYmDDiydznPNHTrlieRThzH7AJhUycRMqWLe7ztohGZqWD1ReZMdlHR2negnwPTWFCbnydqSctPqtyXHgxjOJr9yLl87PNxB9S2gS2ZoGcGH5dLcFZcyXru5fb8zCM8K4pK1aJVnEJsYI6oV2PzGYSa1LZ+QeIyDrdUAqG8Sq5impO9PRbVjKRQnYubqVfqd2mkxcZp5dHW8+lXNx5f+g9xvJvKcSskaeJUpFb49GdDLEunpVcPuK3pQXulfVB+IVpRyZS4EHRIK8p/Ls422FzAj9T/N5sYxN3l0DlqUWfmBla7QErErcmIAcXEDgGYSIfPAIuXIcyjh+3GjNBsSqb9 p9dbq9lD eTiHcwhbgQ0NPFTOZdKoDxbvxSAMfgEhuNd1NGCXf9bINBKWa4wWD/os/Aiw27hvYc/5NRViisb16H34H6SxdVIL20Fojg2n9eK3p3XvtyrKbDDZY/MBxT3mJrq7jdxYtpu7xB412qcZWKue01yD8h7w7lMIGvjDsrBTzFTcd8TKyxaRdtqvg2RxsnmBVqn1Q2XUkBeTJa1Jk45FVCtWq0c3sQiuG5vcPIz6nckU/FEs43d7bVawrc6+dLDW547NuFpWRNyMlBI2ClmkGxq17SBD8BTYhZuYrhNXZVmXdlBJymA151+WMdCw6rm8K+jpxg1QcHFGIrd4k+nsUlhws9l8flfiKbYay8l1CiIsXh6i7TS/ppS/iUjYiSsLHU0Qx5zven+gqzto5marYOa8E3548rilC8dGcMfVTMrbUNzqOyd0+njLLalK4SKeclIKAbaj5mGjomYB819qX92907vnvLsyz+cJA7RpR1fMtXFFH4lWMhPvwpU4YzuSkz/Y6SBdGfbQ5GmHctaxBlKZ21pD1IGwSLIh9wb5Yksi3CRKanrMpGRG7OglC5iFjomOlNzMZ7Ynr5LYK5q+xykWTsKsW20jQezel5Khegq8tjefM0Ao= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000674, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With anon and file min_seq being able to move independently, rework workingset protection as well so that the comparison of refaults between anon and file is always on an equal footing. Specifically, make lru_gen_test_recent() return true for refaults happening within the distance of MAX_NR_GENS. For example, if min_seq of a type is max_seq-MIN_NR_GENS, refaults from min_seq-1, i.e., max_seq-MIN_NR_GENS-1, are also considered recent, since the distance max_seq-(max_seq-MIN_NR_GENS-1), i.e., MIN_NR_GENS+1 is less than MAX_NR_GENS. As an intermediate step to the final optimization, this change by itself should not have userspace-visiable effects beyond performance. Reported-by: Kairui Song Closes: https://lore.kernel.org/CAOUHufahuWcKf5f1Sg3emnqX+cODuR=2TQo7T4Gr-QYLujn4RA@mail.gmail.com/ Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/workingset.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index ad181d1b8cf1..2c310c29f51e 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -260,11 +260,11 @@ static void *lru_gen_eviction(struct folio *folio) * Tests if the shadow entry is for a folio that was recently evicted. * Fills in @lruvec, @token, @workingset with the values unpacked from shadow. */ -static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, +static bool lru_gen_test_recent(void *shadow, struct lruvec **lruvec, unsigned long *token, bool *workingset) { int memcg_id; - unsigned long min_seq; + unsigned long max_seq; struct mem_cgroup *memcg; struct pglist_data *pgdat; @@ -273,8 +273,10 @@ static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, memcg = mem_cgroup_from_id(memcg_id); *lruvec = mem_cgroup_lruvec(memcg, pgdat); - min_seq = READ_ONCE((*lruvec)->lrugen.min_seq[file]); - return (*token >> LRU_REFS_WIDTH) == (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH)); + max_seq = READ_ONCE((*lruvec)->lrugen.max_seq); + max_seq &= EVICTION_MASK >> LRU_REFS_WIDTH; + + return abs_diff(max_seq, *token >> LRU_REFS_WIDTH) < MAX_NR_GENS; } static void lru_gen_refault(struct folio *folio, void *shadow) @@ -290,7 +292,7 @@ static void lru_gen_refault(struct folio *folio, void *shadow) rcu_read_lock(); - recent = lru_gen_test_recent(shadow, type, &lruvec, &token, &workingset); + recent = lru_gen_test_recent(shadow, &lruvec, &token, &workingset); if (lruvec != folio_lruvec(folio)) goto unlock; @@ -331,7 +333,7 @@ static void *lru_gen_eviction(struct folio *folio) return NULL; } -static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec, +static bool lru_gen_test_recent(void *shadow, struct lruvec **lruvec, unsigned long *token, bool *workingset) { return false; @@ -432,8 +434,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset, bool recent; rcu_read_lock(); - recent = lru_gen_test_recent(shadow, file, &eviction_lruvec, - &eviction, workingset); + recent = lru_gen_test_recent(shadow, &eviction_lruvec, &eviction, workingset); rcu_read_unlock(); return recent; }