From patchwork Wed Dec 21 00:12:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13078274 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57520C4332F for ; Wed, 21 Dec 2022 00:12:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0E278E0006; Tue, 20 Dec 2022 19:12:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DBD908E0001; Tue, 20 Dec 2022 19:12:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C5F018E0006; Tue, 20 Dec 2022 19:12:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B73E18E0001 for ; Tue, 20 Dec 2022 19:12:40 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8ACCC1C3C44 for ; Wed, 21 Dec 2022 00:12:40 +0000 (UTC) X-FDA: 80264387280.14.C56D274 Received: from mail-il1-f201.google.com (mail-il1-f201.google.com [209.85.166.201]) by imf05.hostedemail.com (Postfix) with ESMTP id 790C5100006 for ; Wed, 21 Dec 2022 00:12:37 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Y99krAxU; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3dE-iYwYKCL83z4mftlttlqj.htrqnsz2-rrp0fhp.twl@flex--yuzhao.bounces.google.com designates 209.85.166.201 as permitted sender) smtp.mailfrom=3dE-iYwYKCL83z4mftlttlqj.htrqnsz2-rrp0fhp.twl@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671581557; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=x7hqwzZvz2lgYCROSGHPyAOjnAXLbIm00pvWJTo+FHY=; b=LwZfvdDI4jAi7yYclnnQShNE5ztBKae4BKGMV1Spdr/xuhc4+mHrJPeERC5kgAGE+2QVOt 0GOZ3pb7/rvuowyNYCPQp3j3v/NBfIi4H0r0RYi7jPFxYAUjsL7prZjEE4uohOkGgQsIhC P1odeiqOxARsB9Z0KRjW0q8/4nXSTWg= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Y99krAxU; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3dE-iYwYKCL83z4mftlttlqj.htrqnsz2-rrp0fhp.twl@flex--yuzhao.bounces.google.com designates 209.85.166.201 as permitted sender) smtp.mailfrom=3dE-iYwYKCL83z4mftlttlqj.htrqnsz2-rrp0fhp.twl@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671581557; a=rsa-sha256; cv=none; b=BRzv7lQ6vKvXEDNs8Z0Iyrfj7Al5mg5aaJrYM/H9JUCuRjtmVncUJQq/z3CY2AQI2vWZF7 qHbgUs1MnNNPbQapsWrw1tuHt4qFriAfPzmUNLdz0h4XGh+KL7fPV+Mgt3s5aNruga5LOs psX+cHCKhlCSJh+yOWSB/VOtTjJIoQc= Received: by mail-il1-f201.google.com with SMTP id e9-20020a056e020b2900b003036757d5caso9210647ilu.10 for ; Tue, 20 Dec 2022 16:12:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=x7hqwzZvz2lgYCROSGHPyAOjnAXLbIm00pvWJTo+FHY=; b=Y99krAxU+g6T25oGmPipeCWre53NFTuJA+DF0scSa6pVzx7mXz7QcRs4zj3Vl9HPUP tRBIUnIfjLKRhffplM9obWzr4hel+aA8KfE7YGeKIyJFgO8ieeYe689GCoDHNY2gpMbi zM4b5sdTiFZxfWyeTTDtHmyZ3KxDS29e9pBwXEAyARnePnRmobGsBdEh1yOqIqhgKc1a A6uogABkcc5cE5LeJndJvYQP3gvIMBmlhEP3b2yEq96hHoG+8AgIbcjB5BTclwI/Qy2B Dq4+gEK53lcx/Q8/KoZGnlUM5QopFlIwhp4GhqYF6UK/aB/azCbUMD3B1AEeV7oLuxja Qnrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=x7hqwzZvz2lgYCROSGHPyAOjnAXLbIm00pvWJTo+FHY=; b=LlKjqlwE/KzaRvMlZs+uNySbHLtlWHIbn9TO2bnQ6YY+SThRhnKiBSxz0QaEvH32x+ lLIKQxCfDXPwSTIEZ0vLiO/Ldrjgj/JveKMXTyZZOBIg1iNTLivVAuhJywFyjLtk7ylV bKk16TUn8JAejAZ/UCxdG/YfIFupMov7w6sNmNLtPVddneqyKHL2fv0T/QmOvCNd3Acl 4IqWoOj74oDGjUr9zXG+F3sKT83NIY4CagUvAhtwbtkw8vYxJJXk6/XnD0ixQsPwnWlN Co3bLm+db1+XcWRCLq9qHQJF9dQkGNDIQVyz3qz0nn2dO1OsCmyfMhaaLla0Cnn76DBk s/Bw== X-Gm-Message-State: ANoB5plZO38BCJY0syCshB3MZUfOHJajHtY4NyOfa9ie1GsTevGPpnNQ izee0/y6SodIcN7htEKhGfoW09POlYE= X-Google-Smtp-Source: AA0mqf6uw7KuVqvxq95bcw+r3WruBA/v608j3gUgEyvlJRm3jjlnvNsHzE5a8wwOhKy0j7TuaZOuPLg+a5E= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:a589:30f2:2daa:4ab7]) (user=yuzhao job=sendgmr) by 2002:a02:b01d:0:b0:38a:3421:be23 with SMTP id p29-20020a02b01d000000b0038a3421be23mr14395612jah.288.1671581556605; Tue, 20 Dec 2022 16:12:36 -0800 (PST) Date: Tue, 20 Dec 2022 17:12:01 -0700 In-Reply-To: <20221221001207.1376119-1-yuzhao@google.com> Message-Id: <20221221001207.1376119-2-yuzhao@google.com> Mime-Version: 1.0 References: <20221221001207.1376119-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Subject: [PATCH mm-unstable v2 1/8] mm: multi-gen LRU: rename lru_gen_struct to lru_gen_folio From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 790C5100006 X-Stat-Signature: kf5ex76q6t6ofqmqn3fqx5ji6gqecujo X-HE-Tag: 1671581557-907110 X-HE-Meta: U2FsdGVkX1+eaEln2fmgdtSan1fljKFDrGtd30q310xw8Hq0Ts1cPpEoz2CN5ip20C8UcJMI6OUH8Z9sIekvoqMFHC2bQ5g82jto0jNx8RjifVXuyYfZFmj3dfelds0X51el5PedPraoog2eMLhJK79A7Nxr9T2phQxZEyygvCVuYyeSXWNI0J2DuID+NbYRDtFu/84wZMsWcA9Sm8QMliNys3ElBdgj6qfbpdyqx9idOH6Ge60lYM/kW14b74zDdLRu0neTzyvO6tuXrPkssKoLWQ/bNfojj3N2A23xZtj8iLxBbznE1yloJmxNDmqMZBNG1pllyEa9ifLMYZOz5TLRLRLo3+PhFkyDkglK2crTUg91LaortBrJD+TiIPmReKF/q8XFrHT816birQV/PL0nTMQJKKxGgTq2/sqcBbUoRo38XfYOE6HsY1+jVFNiBUNkhrtIoCGLRTeGOOV03lVqYYc8IHiERGVNqr/DG4oSoeeCFSYYCQD5sbdLN8QwVN0dVvb/yiYuEvCTUL1NoBYXvSKgizFm6TrSq04aJ2FK5YYCA2g+GKdDSrvFKfCM2Ae35pBnf0tcstqrH3tzcaamEykboiGKeIVQrnMPrCG+1xY6cRfe7W4AQAZSPUd7RMZo0xfBDhFe0OuZ5Cl1NBZvYrJFYAMj8i+EXivE6xS8deTgUy5VPzHJ4pu6K9AZC9V8jH2J008HHJFiwlFRU2LOTml3L0GIr4/Ce37Du/MaZhwjjZKIbczNl97W0Junl4q3t1Hph09YdxOhQsxJMCqeD22dThpXViu7GvydDG0sE6B8m95V54bz0p/nf+oIn/HXWj6STFv6Bal9PUg0RDcA+wXiriBVVmb4w0KpIv8H35lBtc9InAxFLd+RwfTooRNQj3NnzmTAa3UCrAwavFSvCKNE5+nMBPMwh8Sb8D2e5GTmG6kp3O8h37PIFb1p6jfRjd+ex0Qe6f9Wa6y Dz0+CjCf rJrtUWESJf6YyI3o0qyilHjA8wTrShKlUYfhtaCfeC90VzKeenCegPk8w0FR3iKosDv9+hZUR2N9Lcp+Iq0oCVG0o0H/OvX+QEkwnVx/UoRDAxuY4v2gbzZXf0KeTp4bC7tEO4TBbMd6Fr7JkurhQcqUYCQsloxM1y/7oJrLVz7L3lPEQEx10+uGohW7N89jiZQdKj6R0jHQoiONt3KS7OQYBBuMdk4IKZ3K+GLm1LyOWQivXFo6Nvb0pJdB77wXGCgg8Z0lKdYcqxXLQgQVk4ixUVAQrZhdq5htKmHuzSbsoEMAggyYdrV2T75jx25MjA/mOTvV0ZSC4POJw23Uzwbi5+bN1JKGtRGQyHlpuH3RACPVOmUXyia4TgfBewlFzj26p9O1zkUuzUO1cd9Wrp38JSZQIn5gFVY8L X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The new name lru_gen_folio will be more distinct from the coming lru_gen_memcg. Signed-off-by: Yu Zhao Change-Id: Ic714d328df239f569fe37c3f1fac44a01549883c --- include/linux/mm_inline.h | 4 ++-- include/linux/mmzone.h | 6 +++--- mm/vmscan.c | 34 +++++++++++++++++----------------- mm/workingset.c | 4 ++-- 4 files changed, 24 insertions(+), 24 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index e8ed225d8f7c..f63968bd7de5 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -178,7 +178,7 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli int zone = folio_zonenum(folio); int delta = folio_nr_pages(folio); enum lru_list lru = type * LRU_INACTIVE_FILE; - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; VM_WARN_ON_ONCE(old_gen != -1 && old_gen >= MAX_NR_GENS); VM_WARN_ON_ONCE(new_gen != -1 && new_gen >= MAX_NR_GENS); @@ -224,7 +224,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, int gen = folio_lru_gen(folio); int type = folio_is_file_lru(folio); int zone = folio_zonenum(folio); - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; VM_WARN_ON_ONCE_FOLIO(gen != -1, folio); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index cd28a100d9e4..1686fcc4ed01 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -404,7 +404,7 @@ enum { * The number of pages in each generation is eventually consistent and therefore * can be transiently negative when reset_batch_size() is pending. */ -struct lru_gen_struct { +struct lru_gen_folio { /* the aging increments the youngest generation number */ unsigned long max_seq; /* the eviction increments the oldest generation numbers */ @@ -461,7 +461,7 @@ struct lru_gen_mm_state { struct lru_gen_mm_walk { /* the lruvec under reclaim */ struct lruvec *lruvec; - /* unstable max_seq from lru_gen_struct */ + /* unstable max_seq from lru_gen_folio */ unsigned long max_seq; /* the next address within an mm to scan */ unsigned long next_addr; @@ -524,7 +524,7 @@ struct lruvec { unsigned long flags; #ifdef CONFIG_LRU_GEN /* evictable pages divided into generations */ - struct lru_gen_struct lrugen; + struct lru_gen_folio lrugen; /* to concurrently iterate lru_gen_mm_list */ struct lru_gen_mm_state mm_state; #endif diff --git a/mm/vmscan.c b/mm/vmscan.c index 546540bc770a..c0c4b0cd2fe2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3215,7 +3215,7 @@ static int get_nr_gens(struct lruvec *lruvec, int type) static bool __maybe_unused seq_is_valid(struct lruvec *lruvec) { - /* see the comment on lru_gen_struct */ + /* see the comment on lru_gen_folio */ return get_nr_gens(lruvec, LRU_GEN_FILE) >= MIN_NR_GENS && get_nr_gens(lruvec, LRU_GEN_FILE) <= get_nr_gens(lruvec, LRU_GEN_ANON) && get_nr_gens(lruvec, LRU_GEN_ANON) <= MAX_NR_GENS; @@ -3612,7 +3612,7 @@ struct ctrl_pos { static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain, struct ctrl_pos *pos) { - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; int hist = lru_hist_from_seq(lrugen->min_seq[type]); pos->refaulted = lrugen->avg_refaulted[type][tier] + @@ -3627,7 +3627,7 @@ static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain, static void reset_ctrl_pos(struct lruvec *lruvec, int type, bool carryover) { int hist, tier; - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; bool clear = carryover ? NR_HIST_GENS == 1 : NR_HIST_GENS > 1; unsigned long seq = carryover ? lrugen->min_seq[type] : lrugen->max_seq + 1; @@ -3704,7 +3704,7 @@ static int folio_update_gen(struct folio *folio, int gen) static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclaiming) { int type = folio_is_file_lru(folio); - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); unsigned long new_flags, old_flags = READ_ONCE(folio->flags); @@ -3749,7 +3749,7 @@ static void update_batch_size(struct lru_gen_mm_walk *walk, struct folio *folio, static void reset_batch_size(struct lruvec *lruvec, struct lru_gen_mm_walk *walk) { int gen, type, zone; - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; walk->batched = 0; @@ -4266,7 +4266,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) { int zone; int remaining = MAX_LRU_BATCH; - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); if (type == LRU_GEN_ANON && !can_swap) @@ -4302,7 +4302,7 @@ static bool try_to_inc_min_seq(struct lruvec *lruvec, bool can_swap) { int gen, type, zone; bool success = false; - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; DEFINE_MIN_SEQ(lruvec); VM_WARN_ON_ONCE(!seq_is_valid(lruvec)); @@ -4323,7 +4323,7 @@ static bool try_to_inc_min_seq(struct lruvec *lruvec, bool can_swap) ; } - /* see the comment on lru_gen_struct */ + /* see the comment on lru_gen_folio */ if (can_swap) { min_seq[LRU_GEN_ANON] = min(min_seq[LRU_GEN_ANON], min_seq[LRU_GEN_FILE]); min_seq[LRU_GEN_FILE] = max(min_seq[LRU_GEN_ANON], lrugen->min_seq[LRU_GEN_FILE]); @@ -4345,7 +4345,7 @@ static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_scan) { int prev, next; int type, zone; - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; spin_lock_irq(&lruvec->lru_lock); @@ -4403,7 +4403,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, bool success; struct lru_gen_mm_walk *walk; struct mm_struct *mm = NULL; - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; VM_WARN_ON_ONCE(max_seq > READ_ONCE(lrugen->max_seq)); @@ -4468,7 +4468,7 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, unsig unsigned long old = 0; unsigned long young = 0; unsigned long total = 0; - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; struct mem_cgroup *memcg = lruvec_memcg(lruvec); for (type = !can_swap; type < ANON_AND_FILE; type++) { @@ -4753,7 +4753,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx) int delta = folio_nr_pages(folio); int refs = folio_lru_refs(folio); int tier = lru_tier_from_refs(refs); - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; VM_WARN_ON_ONCE_FOLIO(gen >= MAX_NR_GENS, folio); @@ -4853,7 +4853,7 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, int scanned = 0; int isolated = 0; int remaining = MAX_LRU_BATCH; - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; struct mem_cgroup *memcg = lruvec_memcg(lruvec); VM_WARN_ON_ONCE(!list_empty(list)); @@ -5254,7 +5254,7 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc static bool __maybe_unused state_is_valid(struct lruvec *lruvec) { - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; if (lrugen->enabled) { enum lru_list lru; @@ -5533,7 +5533,7 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec, int i; int type, tier; int hist = lru_hist_from_seq(seq); - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; for (tier = 0; tier < MAX_NR_TIERS; tier++) { seq_printf(m, " %10d", tier); @@ -5583,7 +5583,7 @@ static int lru_gen_seq_show(struct seq_file *m, void *v) unsigned long seq; bool full = !debugfs_real_fops(m->file)->write; struct lruvec *lruvec = v; - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; int nid = lruvec_pgdat(lruvec)->node_id; struct mem_cgroup *memcg = lruvec_memcg(lruvec); DEFINE_MAX_SEQ(lruvec); @@ -5837,7 +5837,7 @@ void lru_gen_init_lruvec(struct lruvec *lruvec) { int i; int gen, type, zone; - struct lru_gen_struct *lrugen = &lruvec->lrugen; + struct lru_gen_folio *lrugen = &lruvec->lrugen; lrugen->max_seq = MIN_NR_GENS + 1; lrugen->enabled = lru_gen_enabled(); diff --git a/mm/workingset.c b/mm/workingset.c index 1a86645b7b3c..fd666584515c 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -223,7 +223,7 @@ static void *lru_gen_eviction(struct folio *folio) unsigned long token; unsigned long min_seq; struct lruvec *lruvec; - struct lru_gen_struct *lrugen; + struct lru_gen_folio *lrugen; int type = folio_is_file_lru(folio); int delta = folio_nr_pages(folio); int refs = folio_lru_refs(folio); @@ -252,7 +252,7 @@ static void lru_gen_refault(struct folio *folio, void *shadow) unsigned long token; unsigned long min_seq; struct lruvec *lruvec; - struct lru_gen_struct *lrugen; + struct lru_gen_folio *lrugen; struct mem_cgroup *memcg; struct pglist_data *pgdat; int type = folio_is_file_lru(folio); From patchwork Wed Dec 21 00:12:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13078275 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31D19C3DA6E for ; Wed, 21 Dec 2022 00:12:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3888D8E0001; Tue, 20 Dec 2022 19:12:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 336558E0007; Tue, 20 Dec 2022 19:12:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D7A58E0001; Tue, 20 Dec 2022 19:12:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E93748E0007 for ; Tue, 20 Dec 2022 19:12:40 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B706BA4108 for ; Wed, 21 Dec 2022 00:12:40 +0000 (UTC) X-FDA: 80264387280.11.C30BF85 Received: from mail-oo1-f73.google.com (mail-oo1-f73.google.com [209.85.161.73]) by imf02.hostedemail.com (Postfix) with ESMTP id 1FF4D80009 for ; Wed, 21 Dec 2022 00:12:38 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="COBa/Wmi"; spf=pass (imf02.hostedemail.com: domain of 3dk-iYwYKCME516ohvnvvnsl.jvtspu14-ttr2hjr.vyn@flex--yuzhao.bounces.google.com designates 209.85.161.73 as permitted sender) smtp.mailfrom=3dk-iYwYKCME516ohvnvvnsl.jvtspu14-ttr2hjr.vyn@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671581559; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gUYK6iVE8NyMy+ADEXjGNSzP24X8eYlgIjWsftpIbpg=; b=hLFjnElksCXiBXr4XASBAaJoKdJeeu3tXcC2PrausFB1pMUTE2aA8lttBWnsT62B3FaO9m ldTXSEr6ecnfPv+t/2uyTqGlPBwJ9cG+IzOOIxn/Ja0sjOjm6MMDVHu/8DH7Wg6SrSFkzM Md4/TxTI2JBvW6FMSELZfLI5ZePckI0= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="COBa/Wmi"; spf=pass (imf02.hostedemail.com: domain of 3dk-iYwYKCME516ohvnvvnsl.jvtspu14-ttr2hjr.vyn@flex--yuzhao.bounces.google.com designates 209.85.161.73 as permitted sender) smtp.mailfrom=3dk-iYwYKCME516ohvnvvnsl.jvtspu14-ttr2hjr.vyn@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671581559; a=rsa-sha256; cv=none; b=Nsn4ZPjiEJ8zuW+2VhqKfn73C61I4cBcd1BqhdDma4Zu+6W6vf+99x+CWgLLU0YGLSFOwp cfoJn2ii8I9pM3jw1PujYkrE07aKxHO9AGjUT16JNHwIS+9JaxKLmB/z/xd6oEdf7KPpSs K0ywnh3B66qiVkAZzR3YAQ/gVPJFy5o= Received: by mail-oo1-f73.google.com with SMTP id b6-20020a4a3406000000b004a5ffc77240so6251608ooa.17 for ; Tue, 20 Dec 2022 16:12:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gUYK6iVE8NyMy+ADEXjGNSzP24X8eYlgIjWsftpIbpg=; b=COBa/Wmi0UcxSY0BUzCvzZSg4FdxLNKz22ZgMbaxMfNqGEwpnSzK+Ad+FmIUryr6eD hrvb+id173KoljBaCO6ipPfL22rnaj9lF2DyB8tH/BSRRrNnR2DEJAUpA4gNsiZ6FlUc 9HZPbwK4HzsTPIPRO9gNO96gKLjojmVvKA7YaNzfORXUGx/AUbGwKTYiOdvcUazPysOC RVTg6rZoSWlPvMoi58ZR0BbdLKUE04gIzuDoW6fpxTCkbueXJZsu7T+TVfsMZwmTtFgV N+JSbUk0w5cKXD/XclOTxPcIVANu4eZylI/VRfKjcol9oGuLvaY+QWoSz4BSy92hlMpb IPuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gUYK6iVE8NyMy+ADEXjGNSzP24X8eYlgIjWsftpIbpg=; b=nEjPWWATxLDOGBK3pTV7KMHqCElOM3o83CQoQVYg/1eYkhB8QD+8xlacZZUZDDqBRG tMo7vWCz8Ds+MiASxNndgN75gHMlmStTiHf4SHKMVuDaOs8m7HkrRoSCSt9y+KQ4MNnu viGOVoDZp8vXXXXfthL8Ncl1sJw709lPW4L5AimTsNxtzn3LZuyd2aV0Dy0prqEYiAZu 4yGnXgqHSWSf2AInMbqyKWMhHoGyPEGIlgQ3PkzdAwhtjRFAer3UXn5xq/362WQhUPXi GjqW217N97BeYa8M1d/7GV2bYT9ZcLxLeMg4x0EdiDk0ybg7s6zCIfrLvdRsolRKV5Sr oBHQ== X-Gm-Message-State: AFqh2ko2KZ8fn3M6VBs2w0oPSeqpEn66cZcRctRh1WoXPEmKOJt7gAL+ kXcEi+ZrD6iD4T+t+GElqeTwrlgyVrE= X-Google-Smtp-Source: AA0mqf4/rwvL7nj+AYw5nim+vzF9pRGefJw2rUsq83pO0l/l8ORY4gc6wDzcNVgSfgwxkSSHUd1IdMkHbhA= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:a589:30f2:2daa:4ab7]) (user=yuzhao job=sendgmr) by 2002:a05:6870:30e:b0:144:74f1:173d with SMTP id m14-20020a056870030e00b0014474f1173dmr2117657oaf.241.1671581558131; Tue, 20 Dec 2022 16:12:38 -0800 (PST) Date: Tue, 20 Dec 2022 17:12:02 -0700 In-Reply-To: <20221221001207.1376119-1-yuzhao@google.com> Message-Id: <20221221001207.1376119-3-yuzhao@google.com> Mime-Version: 1.0 References: <20221221001207.1376119-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Subject: [PATCH mm-unstable v2 2/8] mm: multi-gen LRU: rename lrugen->lists[] to lrugen->folios[] From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 1FF4D80009 X-Stat-Signature: z5fhgddar8w64z94izjybskcjpjmrp6f X-Rspam-User: X-HE-Tag: 1671581558-498567 X-HE-Meta: U2FsdGVkX18K/MSnHO5P17FGwVwx742AUKnI2yqiMbpj/d/yONHUs07oc9yuGDAW7MGjsPW51ejHgrS7gtl/wnFUjC/OAHi6lKtuRyy2fSF67FksgASl1Z1r15Lw7xLTOCl1Rc8MgnWBVgODSNcOO1Gcq1xryUMRQmxX4lhNGeE9Gh8NQGnhNnu4PQ5EpgK3JC8RF8dWtugyTNPcGI27/wLH4cgIm47nnBA0GxDzAHB0Q1A2GpSrsBfNoNLVCay9g4eks9wsbKQ5vdqk3l8amx6DRxkSTzlieJpWM5o7+/3DjHiZv47o9WL++X2FJ5ZHKnQfdxXLrTtUJ0leX48Z+AsarZ3EEhtWT7Gz69PEfVJ1uB+ZzDRWv5srAT1aWiZIV3nzG/gC5BTATdJe5mn/icpLGjRhUAraLiptAsWT9FWnBmoS82E4Chy6Zhlcu6MTtNK0TMkQCrF/db2l7jbiuUtTXrzKrK71a8sBhdihqxTSZHWMN6M6y6ehnNrqdXjOWP1F//hWPI2iKvnt39O3kZRT5CY7+I1Gt7rGSdsO3kxd3U8rRsYg9I6sSDV3vGbKlOA6PEkZpAgYmObaiXB8dhes5x1q94zXlSEMnJzy83973qv8X9KVH9dQC51JQfeOnTJTtp+y+xFRN+uWDAe0lt4wGPxKlXgd8cyxtNqHRR7drhD5QF14Mk+z18di5Zn55d2CJyYLi8Q3ORBzGOsYrcn28aa9SuiIERNV0DB07EeXdhy97VIgykrA+niEzVQMkVPyL32rtQtkANLhBjjEMg95qSL7k9MwebGCqXs5wmy9/dp4pBIG7g7wWqIsrmUShPnj5+QBfa/yo5WproMBc0muzekrVDC76R2PCvKWBHhSI7u8EWkVvcwXJ7B/3CQCcEqna1fOB+MXZGJ+FRS4X2kA7Kd20lfowPqeN05OkQekW+Beuxq7f4b+CAr3Nggg6yYEKvSiC5TJJyEx+/W 4BdtDYrP 7PuNAdeCIc9NGJhjM04DWLCd2FOSmXsFYOWMkXIPq19uFctQzVRCe9UhBgaPxSt2O0OhK1LsNsWjTdErSXlVheQ6YLes8yZ1c2i0IbKqwcazbdkeqIOn+NkzU5+eZeCic4lqTGkVIfNx3NIi6sWaL72WP3Jp3g+4E05OfL4pLiMPbp19m9TxQr+yrxFa4sr05xlsSDB668X/5kloef2Y6nXhOp1xCLShYZmUoLB9rpmz8shuF2vSlOmE8WbSdiHftGF68Z3AkwbO2tMiXaqK/A8pgOCiWTHI9F//VBZk7Df6k8WjyynB7PO/SikrvW25nglZDRukCIKFp9uv5xmsL/h2fKJMnm7MRZHVrvGxnVRnfQno= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: lru_gen_folio will be chained into per-node lists by the coming lrugen->list. Signed-off-by: Yu Zhao Change-Id: Ib1bee101fd6b73da5409d32e67409fa6475d8cc6 --- Documentation/mm/multigen_lru.rst | 8 ++++---- include/linux/mm_inline.h | 4 ++-- include/linux/mmzone.h | 8 ++++---- mm/vmscan.c | 20 ++++++++++---------- 4 files changed, 20 insertions(+), 20 deletions(-) diff --git a/Documentation/mm/multigen_lru.rst b/Documentation/mm/multigen_lru.rst index d7062c6a8946..d8f721f98868 100644 --- a/Documentation/mm/multigen_lru.rst +++ b/Documentation/mm/multigen_lru.rst @@ -89,15 +89,15 @@ variables are monotonically increasing. Generation numbers are truncated into ``order_base_2(MAX_NR_GENS+1)`` bits in order to fit into the gen counter in ``folio->flags``. Each -truncated generation number is an index to ``lrugen->lists[]``. The +truncated generation number is an index to ``lrugen->folios[]``. The sliding window technique is used to track at least ``MIN_NR_GENS`` and at most ``MAX_NR_GENS`` generations. The gen counter stores a value within ``[1, MAX_NR_GENS]`` while a page is on one of -``lrugen->lists[]``; otherwise it stores zero. +``lrugen->folios[]``; otherwise it stores zero. Each generation is divided into multiple tiers. A page accessed ``N`` times through file descriptors is in tier ``order_base_2(N)``. Unlike -generations, tiers do not have dedicated ``lrugen->lists[]``. In +generations, tiers do not have dedicated ``lrugen->folios[]``. In contrast to moving across generations, which requires the LRU lock, moving across tiers only involves atomic operations on ``folio->flags`` and therefore has a negligible cost. A feedback loop @@ -127,7 +127,7 @@ page mapped by this PTE to ``(max_seq%MAX_NR_GENS)+1``. Eviction -------- The eviction consumes old generations. Given an ``lruvec``, it -increments ``min_seq`` when ``lrugen->lists[]`` indexed by +increments ``min_seq`` when ``lrugen->folios[]`` indexed by ``min_seq%MAX_NR_GENS`` becomes empty. To select a type and a tier to evict from, it first compares ``min_seq[]`` to select the older type. If both types are equally old, it selects the one whose first tier has diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index f63968bd7de5..da38e3d962e2 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -256,9 +256,9 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, lru_gen_update_size(lruvec, folio, -1, gen); /* for folio_rotate_reclaimable() */ if (reclaiming) - list_add_tail(&folio->lru, &lrugen->lists[gen][type][zone]); + list_add_tail(&folio->lru, &lrugen->folios[gen][type][zone]); else - list_add(&folio->lru, &lrugen->lists[gen][type][zone]); + list_add(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 1686fcc4ed01..6c96ee823dbd 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -312,7 +312,7 @@ enum lruvec_flags { * They form a sliding window of a variable size [MIN_NR_GENS, MAX_NR_GENS]. An * offset within MAX_NR_GENS, i.e., gen, indexes the LRU list of the * corresponding generation. The gen counter in folio->flags stores gen+1 while - * a page is on one of lrugen->lists[]. Otherwise it stores 0. + * a page is on one of lrugen->folios[]. Otherwise it stores 0. * * A page is added to the youngest generation on faulting. The aging needs to * check the accessed bit at least twice before handing this page over to the @@ -324,8 +324,8 @@ enum lruvec_flags { * rest of generations, if they exist, are considered inactive. See * lru_gen_is_active(). * - * PG_active is always cleared while a page is on one of lrugen->lists[] so that - * the aging needs not to worry about it. And it's set again when a page + * PG_active is always cleared while a page is on one of lrugen->folios[] so + * that the aging needs not to worry about it. And it's set again when a page * considered active is isolated for non-reclaiming purposes, e.g., migration. * See lru_gen_add_folio() and lru_gen_del_folio(). * @@ -412,7 +412,7 @@ struct lru_gen_folio { /* the birth time of each generation in jiffies */ unsigned long timestamps[MAX_NR_GENS]; /* the multi-gen LRU lists, lazily sorted on eviction */ - struct list_head lists[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; + struct list_head folios[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; /* the multi-gen LRU sizes, eventually consistent */ long nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; /* the exponential moving average of refaulted */ diff --git a/mm/vmscan.c b/mm/vmscan.c index c0c4b0cd2fe2..94477894b226 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4274,7 +4274,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) /* prevent cold/hot inversion if force_scan is true */ for (zone = 0; zone < MAX_NR_ZONES; zone++) { - struct list_head *head = &lrugen->lists[old_gen][type][zone]; + struct list_head *head = &lrugen->folios[old_gen][type][zone]; while (!list_empty(head)) { struct folio *folio = lru_to_folio(head); @@ -4285,7 +4285,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); new_gen = folio_inc_gen(lruvec, folio, false); - list_move_tail(&folio->lru, &lrugen->lists[new_gen][type][zone]); + list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); if (!--remaining) return false; @@ -4313,7 +4313,7 @@ static bool try_to_inc_min_seq(struct lruvec *lruvec, bool can_swap) gen = lru_gen_from_seq(min_seq[type]); for (zone = 0; zone < MAX_NR_ZONES; zone++) { - if (!list_empty(&lrugen->lists[gen][type][zone])) + if (!list_empty(&lrugen->folios[gen][type][zone])) goto next; } @@ -4778,7 +4778,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx) /* promoted */ if (gen != lru_gen_from_seq(lrugen->min_seq[type])) { - list_move(&folio->lru, &lrugen->lists[gen][type][zone]); + list_move(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } @@ -4787,7 +4787,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx) int hist = lru_hist_from_seq(lrugen->min_seq[type]); gen = folio_inc_gen(lruvec, folio, false); - list_move_tail(&folio->lru, &lrugen->lists[gen][type][zone]); + list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); WRITE_ONCE(lrugen->protected[hist][type][tier - 1], lrugen->protected[hist][type][tier - 1] + delta); @@ -4799,7 +4799,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx) if (folio_test_locked(folio) || folio_test_writeback(folio) || (type == LRU_GEN_FILE && folio_test_dirty(folio))) { gen = folio_inc_gen(lruvec, folio, true); - list_move(&folio->lru, &lrugen->lists[gen][type][zone]); + list_move(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } @@ -4866,7 +4866,7 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, for (zone = sc->reclaim_idx; zone >= 0; zone--) { LIST_HEAD(moved); int skipped = 0; - struct list_head *head = &lrugen->lists[gen][type][zone]; + struct list_head *head = &lrugen->folios[gen][type][zone]; while (!list_empty(head)) { struct folio *folio = lru_to_folio(head); @@ -5267,7 +5267,7 @@ static bool __maybe_unused state_is_valid(struct lruvec *lruvec) int gen, type, zone; for_each_gen_type_zone(gen, type, zone) { - if (!list_empty(&lrugen->lists[gen][type][zone])) + if (!list_empty(&lrugen->folios[gen][type][zone])) return false; } } @@ -5312,7 +5312,7 @@ static bool drain_evictable(struct lruvec *lruvec) int remaining = MAX_LRU_BATCH; for_each_gen_type_zone(gen, type, zone) { - struct list_head *head = &lruvec->lrugen.lists[gen][type][zone]; + struct list_head *head = &lruvec->lrugen.folios[gen][type][zone]; while (!list_empty(head)) { bool success; @@ -5846,7 +5846,7 @@ void lru_gen_init_lruvec(struct lruvec *lruvec) lrugen->timestamps[i] = jiffies; for_each_gen_type_zone(gen, type, zone) - INIT_LIST_HEAD(&lrugen->lists[gen][type][zone]); + INIT_LIST_HEAD(&lrugen->folios[gen][type][zone]); lruvec->mm_state.seq = MIN_NR_GENS; init_waitqueue_head(&lruvec->mm_state.wait); From patchwork Wed Dec 21 00:12:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13078276 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7978C4167B for ; Wed, 21 Dec 2022 00:12:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8DDC78E0008; Tue, 20 Dec 2022 19:12:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8904A8E0007; Tue, 20 Dec 2022 19:12:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 708948E0008; Tue, 20 Dec 2022 19:12:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 57D208E0007 for ; Tue, 20 Dec 2022 19:12:42 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 22B33A0255 for ; Wed, 21 Dec 2022 00:12:42 +0000 (UTC) X-FDA: 80264387364.05.A319E01 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf30.hostedemail.com (Postfix) with ESMTP id 873F080006 for ; Wed, 21 Dec 2022 00:12:40 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GlPiUBTF; spf=pass (imf30.hostedemail.com: domain of 3d0-iYwYKCMI627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3d0-iYwYKCMI627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671581560; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=y306PH8Q0yp+xWizSY/ux8ywn6bxHcRrOk54D5HB2aU=; b=OGylBfRrFzStTyj0AlzS/TcDDW4JEwewZst2XJHnRxC/edqW2FnlgAEwkRvzKmI9dgW5K+ RcYfa+yy6iums/qZ5nN6HgXnceDk0don5ZOY3kJX1qpsRKgyqFjXQaayT9/jUl9M5XA9+X LO1Afrorc5HAQA2BwGNqTERmFwKhfYI= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GlPiUBTF; spf=pass (imf30.hostedemail.com: domain of 3d0-iYwYKCMI627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3d0-iYwYKCMI627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671581560; a=rsa-sha256; cv=none; b=TS9o9aWhsmzR9PQA+ftVZqswsaYplpmoKOsNJxrGWCbElGV43P8PUVYCqLYGtdnzq/UH6l hnqJILPX43f2k5g0jNPQbluFYZTaSzpajGZUrWOOqBShBe1FpECj4ZqPtkSd9SENYuqt/D Z6mE0J2Vcusi+qNpfGx5cuttFhhoEp8= Received: by mail-yb1-f201.google.com with SMTP id g9-20020a25bdc9000000b0073727a20239so15741819ybk.4 for ; Tue, 20 Dec 2022 16:12:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=y306PH8Q0yp+xWizSY/ux8ywn6bxHcRrOk54D5HB2aU=; b=GlPiUBTF+eOAUqdbrju+L1+Kh2wEO8CK+cMwhM7K6Y1IUVWqKXq6Srl0dPkE6NZEHM NvnzPFwFsuQL7HH8D4QeVAltwJBOXQAXNpJV2tlNz2kzmGWq9+RWxmJgjzKJ7BDlJTx8 BqlrCKIq6REydAxMtmiIG1Wk7yRm/0MIuq/sGKpU/qThTOiFIb8txt0eAaelMAUKVpmt xnWEPSQG7j+mL+CH22sZaicYWYgO9eV7VsqFV55qrg9aMH2v/OL9Z3KNrB4iBvpBGcTS 1auKzt2mFvRdMGB4k6qPXceTvJwVKzP3vKVe4g+/uaB/gSrnSFHF0RKsFAdGd7PY39hz SECQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=y306PH8Q0yp+xWizSY/ux8ywn6bxHcRrOk54D5HB2aU=; b=YSWea5/BcDwPJEcTkwlyFccIKnqc+NSmLEqq21gGSBdI+B13iWWdxhBVVhVDg2l/wE iFLRI9TrT5cZX2KjlwJxnmtEXAd2Tnw+/Pumd5HaESjod3zpUC7Q8//KZZF1zHg+NWUo aEg+I7Z9py+qweSLGM3cN9RZPNxloygo0NaGFt3QPozj1IOl1mx9WfSBNzL0uRaPPvEw tOuCbQZQghlQO84od/fkG4E35xuoVDHqO0L27F4oixUyOKxsVXLIQmCfXVJyxLEf7T7E 1S8bu9NnmPaqQQOXt3iebbAj8ZmuJ29n/YkUIW26W9mefh5fYsnD82caU+FmDtxsIfMG Xl0Q== X-Gm-Message-State: ANoB5pnGGUinmJU+FvcD/ZdHiOca0Hv8MEeHknMBGY00/TSB4FzyW+sD d57UI+LgX/ahmCA5SytSU9QxkzGB6wA= X-Google-Smtp-Source: AA0mqf4kmUm3Oml4SQcszfQyWi0BIkPzgCXt6tJS052cJtMI6HpR2ijMRjA//FRRHQ1i3YBOkd6DUZjJlrY= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:a589:30f2:2daa:4ab7]) (user=yuzhao job=sendgmr) by 2002:a05:690c:903:b0:3b7:e501:90cf with SMTP id cb3-20020a05690c090300b003b7e50190cfmr5497843ywb.501.1671581559741; Tue, 20 Dec 2022 16:12:39 -0800 (PST) Date: Tue, 20 Dec 2022 17:12:03 -0700 In-Reply-To: <20221221001207.1376119-1-yuzhao@google.com> Message-Id: <20221221001207.1376119-4-yuzhao@google.com> Mime-Version: 1.0 References: <20221221001207.1376119-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Subject: [PATCH mm-unstable v2 3/8] mm: multi-gen LRU: remove eviction fairness safeguard From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Queue-Id: 873F080006 X-Stat-Signature: 1auejh8wj36w6psjprdd6m4nsjaxenhe X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1671581560-140165 X-HE-Meta: U2FsdGVkX19Dp2Wd45cW7ixvr9CX/4vMjlHy6hrGLsS76npTVImPmA7YtNbbrJswKXKIapL4qSwA1U8euU07W9uVK5tZayrU8IiIvKj6AztNXTBe7kumRceOfkK+A4h99wk+FUIMv+EE5Ff7+5IoXH3paPa2ZqDvDRsxtuXDNG8BQdZPP7O2Cpdm1l5JlDlgt9hqpHOyB3/qOWNGK+HUod6KPJTIWTyXbFou9mFnGNdN7XvNBjwznn3UadghGOAv5Mz6ZfosPrnPAUR/CabRMRa1K0GRlPRJQLg+4X+ZYYdfhqNwx7HG7ZAanqruWfjdH8rmW9oK6mbJGe7zXA3jHUrvcpNRALLQc5U9C8qjyIwAtQMkw5wphwW1fuTrVSWYir4r/9LSffvmhoUopVdGY2i5F7yaO9o1lW/4l/id/np04l64PD+T6sDy0NejkzU0duCnIxB6Z9hvb23x2rdsgNx7VuAMnWdwXt7q1wWg/Tb4qrPzpNnOAWMnuNKXJo5uZH5aJBrcpa+NBDWE5lT7sJV/cKkRU8RVRVGRgZzFmPuex1ZH0XdFYu2TBF6lJh27JiGRgw1abOz0obZDH5vtt3XLEHbvI8Iy4sMV2P0+8/zZgxkJz0J+ndmA0fAmPXu25Z4XNHhwGKnZT7YU3ZAiT59X/eMgYE5Y0Nbc0+5jmhelW3iMqcmaEyvLJucLGicQBkXZRC6nyIFyIx7RiNqJ7t866RW1yj6GyFZMt2sD3o+1Eh5hdiuL/qJocsdYxj7aM2hkZeLvI7Ou1wd5cnKpQcoBILJefTrNs9EosAImVX9beQRJ72gy/f+zZzpgr15Xs9JxASqJ2rCWBiUgMU65qqOa2NWECJrS9A3VY+DXNh6aDWgG8lZpfW+Xs0c7SQot1C+FdF4ATJo0zAEW78ZWv7BpxxemPguztVF14qaMhtWIURpDHuGuPrD9CRuOoiMM/EyMkAshEAYc7k6JqJD knxRDg/g +oG7chJByrXib8u9rE/oNoNuvk/YlnARcmMao6ECt6nSkOiN8ZiBqAl+Bzckh9YjiQoVNZxStZqW4cP0h5v1Q5sVoMV8200H8+7g6RiA9mNwQm6jzDYD1Qj8JO/L3sgdqyripdeHWexdvW4Rf/Fr/jdWCUW45LuEGQ87k2ZH1P6bx+XEeWKCaQV68u+xOwesA+0BbGdi3zkbvGnFjlwni1qWUnfQ/6p2ZdNEeW8dJ+1lcILqGgfGiMNT7zTUDgjvZs51l6JW6ihtcQqEMW9VO/ExPdrt6eRKUzP/zOqYhOpHAOwHjFv0I45ujS9Fllzo3ID8H4mOv+fcGHUKufECEf1pZUeSKmDSAOitm2TQJij1/jOw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Recall that the eviction consumes the oldest generation: first it bucket-sorts folios whose gen counters were updated by the aging and reclaims the rest; then it increments lrugen->min_seq. The current eviction fairness safeguard for global reclaim has a dilemma: when there are multiple eligible memcgs, should it continue or stop upon meeting the reclaim goal? If it continues, it overshoots and increases direct reclaim latency; if it stops, it loses fairness between memcgs it has taken memory away from and those it has yet to. With memcg LRU, the eviction, while ensuring eventual fairness, will stop upon meeting its goal. Therefore the current eviction fairness safeguard for global reclaim will not be needed. Note that memcg LRU only applies to global reclaim. For memcg reclaim, the eviction will continue, even if it is overshooting. This becomes unconditional due to code simplification. Signed-off-by: Yu Zhao Change-Id: Ieb2a658c8d441ec11348d2985c7c4d6b72106f6a --- mm/vmscan.c | 81 +++++++++++++++-------------------------------------- 1 file changed, 23 insertions(+), 58 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 94477894b226..9655b3b3a95e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -449,6 +449,11 @@ static bool cgroup_reclaim(struct scan_control *sc) return sc->target_mem_cgroup; } +static bool global_reclaim(struct scan_control *sc) +{ + return !sc->target_mem_cgroup || mem_cgroup_is_root(sc->target_mem_cgroup); +} + /** * writeback_throttling_sane - is the usual dirty throttling mechanism available? * @sc: scan_control in question @@ -499,6 +504,11 @@ static bool cgroup_reclaim(struct scan_control *sc) return false; } +static bool global_reclaim(struct scan_control *sc) +{ + return true; +} + static bool writeback_throttling_sane(struct scan_control *sc) { return true; @@ -5009,8 +5019,7 @@ static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int sw return scanned; } -static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swappiness, - bool *need_swapping) +static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swappiness) { int type; int scanned; @@ -5099,9 +5108,6 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap goto retry; } - if (need_swapping && type == LRU_GEN_ANON) - *need_swapping = true; - return scanned; } @@ -5141,67 +5147,26 @@ static unsigned long get_nr_to_scan(struct lruvec *lruvec, struct scan_control * return min_seq[!can_swap] + MIN_NR_GENS <= max_seq ? nr_to_scan : 0; } -static bool should_abort_scan(struct lruvec *lruvec, unsigned long seq, - struct scan_control *sc, bool need_swapping) +static unsigned long get_nr_to_reclaim(struct scan_control *sc) { - int i; - DEFINE_MAX_SEQ(lruvec); + /* don't abort memcg reclaim to ensure fairness */ + if (!global_reclaim(sc)) + return -1; - if (!current_is_kswapd()) { - /* age each memcg at most once to ensure fairness */ - if (max_seq - seq > 1) - return true; + /* discount the previous progress for kswapd */ + if (current_is_kswapd()) + return sc->nr_to_reclaim + sc->last_reclaimed; - /* over-swapping can increase allocation latency */ - if (sc->nr_reclaimed >= sc->nr_to_reclaim && need_swapping) - return true; - - /* give this thread a chance to exit and free its memory */ - if (fatal_signal_pending(current)) { - sc->nr_reclaimed += MIN_LRU_BATCH; - return true; - } - - if (cgroup_reclaim(sc)) - return false; - } else if (sc->nr_reclaimed - sc->last_reclaimed < sc->nr_to_reclaim) - return false; - - /* keep scanning at low priorities to ensure fairness */ - if (sc->priority > DEF_PRIORITY - 2) - return false; - - /* - * A minimum amount of work was done under global memory pressure. For - * kswapd, it may be overshooting. For direct reclaim, the allocation - * may succeed if all suitable zones are somewhat safe. In either case, - * it's better to stop now, and restart later if necessary. - */ - for (i = 0; i <= sc->reclaim_idx; i++) { - unsigned long wmark; - struct zone *zone = lruvec_pgdat(lruvec)->node_zones + i; - - if (!managed_zone(zone)) - continue; - - wmark = current_is_kswapd() ? high_wmark_pages(zone) : low_wmark_pages(zone); - if (wmark > zone_page_state(zone, NR_FREE_PAGES)) - return false; - } - - sc->nr_reclaimed += MIN_LRU_BATCH; - - return true; + return max(sc->nr_to_reclaim, compact_gap(sc->order)); } static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) { struct blk_plug plug; bool need_aging = false; - bool need_swapping = false; unsigned long scanned = 0; unsigned long reclaimed = sc->nr_reclaimed; - DEFINE_MAX_SEQ(lruvec); + unsigned long nr_to_reclaim = get_nr_to_reclaim(sc); lru_add_drain(); @@ -5225,7 +5190,7 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc if (!nr_to_scan) goto done; - delta = evict_folios(lruvec, sc, swappiness, &need_swapping); + delta = evict_folios(lruvec, sc, swappiness); if (!delta) goto done; @@ -5233,7 +5198,7 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc if (scanned >= nr_to_scan) break; - if (should_abort_scan(lruvec, max_seq, sc, need_swapping)) + if (sc->nr_reclaimed >= nr_to_reclaim) break; cond_resched(); @@ -5680,7 +5645,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_co if (sc->nr_reclaimed >= nr_to_reclaim) return 0; - if (!evict_folios(lruvec, sc, swappiness, NULL)) + if (!evict_folios(lruvec, sc, swappiness)) return 0; cond_resched(); From patchwork Wed Dec 21 00:12:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13078277 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40568C4332F for ; Wed, 21 Dec 2022 00:12:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 07B408E0009; Tue, 20 Dec 2022 19:12:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 003F58E0007; Tue, 20 Dec 2022 19:12:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DBEFC8E0009; Tue, 20 Dec 2022 19:12:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C69CB8E0007 for ; Tue, 20 Dec 2022 19:12:43 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A17041C5C44 for ; Wed, 21 Dec 2022 00:12:43 +0000 (UTC) X-FDA: 80264387406.22.20A0CDC Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 0360CA0007 for ; Wed, 21 Dec 2022 00:12:41 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="eG/fUD4z"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3eU-iYwYKCMQ849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3eU-iYwYKCMQ849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671581562; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=W215O6nVZQSEsqpHqtb/vUKPzkuQ3oMopgbAZLI2Y6k=; b=URU2Hq3Qu7UyqnsFUGIMGwMPVUMs6L6n1WxGv6K6noXuO4inZn2sa93S0Ca1ENZO+AgThm ph6PJz8WBvAD/FrqDO6kXythfxCI9TM59YtJbldb99QCFOQVKb9dsZ+qQRoeqMLB0CbRHQ BL2rc/9VL1cj0esnusWwx4Gzo4K6UT8= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="eG/fUD4z"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3eU-iYwYKCMQ849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3eU-iYwYKCMQ849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671581562; a=rsa-sha256; cv=none; b=asB3v/OIc04seQ51egqP7mQoviaXldzFRl3SkKbA8ApO5g3VnBFGzQMr4gz9d0ly4O0CeH seIFsfV49Gg08f0Z420klr8Nn3rYXsIXX5XeAwvS8B169PFqH4m603qHw4nVO4C+BlXOFr LaN9ARybUsQJzPTvWvcLscg42O8M3wg= Received: by mail-yb1-f201.google.com with SMTP id 203-20020a2502d4000000b006f94ab02400so15812700ybc.2 for ; Tue, 20 Dec 2022 16:12:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=W215O6nVZQSEsqpHqtb/vUKPzkuQ3oMopgbAZLI2Y6k=; b=eG/fUD4zGkFUZK+ZF+HvzoNHr/U2E6OCu8wDseqLAQ7XetG2/d97LSGE+vjDKJW/8T 55c6gGFY7rbXkizJhVfunHV+jAKiBzVpXbBFN3tFXqv08tKo4j5tbJdoIFIVw8yuZZEd ze4WDTwcwXETWVT8Z2vGRdjfiisp9rSc1HLiCfPkE6vpTXTeOalRHGAvgdBLPNLdEZx2 jkxOB+JW7wkH/evopjYD7Zl4/gzUQ+yv9/VY+Ad2IDAebHIFWEaxMQI2vQiJqEidlS4M B0fpb9bK118a9RvoRkIDcjCWKNmiMIf1PgLrjubYfLSYbJNQvRTJ++80oy6hzbiOWW49 rdWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=W215O6nVZQSEsqpHqtb/vUKPzkuQ3oMopgbAZLI2Y6k=; b=Rls7tfaUMl7vA1fHzn4Kp4IMa5uZsLKHkpZjJN57tKfHUKBKRyL8MPhdTZA4xk5owZ Dj7w9m9rTPs//zoHnafGnqL1Hv3eYQ93qjUJBjtZrQDCOkaiRNi7lWr62R6zGVo6Q3fH FHNo1Q0GqUK/oTIA8Vg0iF7Fz83XWZr0HpsJbn/bnj7KroifKcsuW7E1dvS3WgW16G3Q Q5773Z95WalFTq92EVEU8XKfS07SNSlZxPRByQ9rplB72hL5GGi9Qe3PPmKybCnRyOaN jigFoQlh4DzIHmx9Mz5ZA+pyzfqxMywflVF2GaLo3SZitgkCpZZ9mIp8U3UzXUSrv7LG gSFw== X-Gm-Message-State: AFqh2kpCzhkvPvWs+DhqZhyXyEX1mU72kg3/LAIPUyJMYviTA+1K6TJ7 WiyW6cSgYBQL7UUdJuPj21NV+HLKQ10= X-Google-Smtp-Source: AMrXdXv0COwLPwkTS0Vl7hKGmMXAA1gr18YJ0e5zARRIGHI8mnjrVM8t0QekFFd4Y0N7jJ2180yJ1mZynA4= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:a589:30f2:2daa:4ab7]) (user=yuzhao job=sendgmr) by 2002:a25:9d8c:0:b0:733:4d8d:f01a with SMTP id v12-20020a259d8c000000b007334d8df01amr3223932ybp.399.1671581561148; Tue, 20 Dec 2022 16:12:41 -0800 (PST) Date: Tue, 20 Dec 2022 17:12:04 -0700 In-Reply-To: <20221221001207.1376119-1-yuzhao@google.com> Message-Id: <20221221001207.1376119-5-yuzhao@google.com> Mime-Version: 1.0 References: <20221221001207.1376119-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Subject: [PATCH mm-unstable v2 4/8] mm: multi-gen LRU: remove aging fairness safeguard From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Queue-Id: 0360CA0007 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: y9xp71o7fwmzottr46ppaws9eg5oehfa X-HE-Tag: 1671581561-187743 X-HE-Meta: U2FsdGVkX1+uvMmzUzvnDUFUIS4ws0QIDpjUU1NlRi/sA/MahgtdqRlA++Qf70tm370uIZQ7g5O60Xtfa5SC/ITIsAzHI+Xzfmmt4jpcSAzmc+KiH+yJFijuzjBoHkg5xLlIMIOBdY2XdnXboG20iVawIC+aHZa9ExUgFGwev87lyVopTGQ7TumKrQZ7KNzevDRz2LX2MofC8QObqEJ70ggwhUsesOKnBFCPyQjOtTj6qpkYyL+U8H8wOH+pUGAFWfH7M5ReU62eZHn2uG+BsbzV57feVyqzJYm6Rf6YXd4kwsn5iNXpLV3x77l2yDlqVWMnSQuYqqxKSXvbOTE3UJMxkBNjmDZ4bE5CTMaeHjVKB2CqDk2gDHmiMaOK2RvQ7Gu6uETU0VW2tx0wgfS9czYQtPAHCPQfqs5SQpvdvu6TuHAQYJSN2H+m7IsRcLCNrcRmR85eMI7qGx9/qOpAHCqFAGRz+HKPeizOzR4zltpa2Em18pduGYdy7yUr+vQNSJf2t2dyLul4owvSmlt5n3DrZf+9RC4a/YJoE58S1Bt2XD+nfJGMtnvC5Azdn6IugChENbpE6PH82jbN6cRBTwGPrPY+01IIUozVp2y3SK1rPV+4QAaKxTEWkDu63TnE8L5CbWGIdEZw5En04FpigLtwbcrFy5H1QSDS7u8WfP8D9p8yKCS/40w9bMXMb8lBCqj7P8olb8eCz4pt3lepCNgJPg0z5TFRTaQcKS0y/Lg9FrjXGoZ0raMAknRUCM9PyeJCHM5CoWfLih6psqmaD4fiPdxV6V6xHdzgZuMJFXYp+dpFki60Kt9iXKEL1vQutTsGGvuQrQsKJemnGLswLY5q2iQ2TJgQe+sdaUhhQZ2UbG6ElCnQOGNA6GDY64oEpF/p3MRXpjJPg/0PER2Y7YH7HnPR5ge5vq4qhDObgyI/pF9SXPD6bwgZwIQxsr+AZFd34gbP4LxMOWKXWul LiElmPdu cG7AjVXjQWbOofMUpN04W2LkBNWDLVlv1AF/IzEBi77Cpn1s4VjlBhioRHIv/nX4IFbPgMeguHu5z6qrwSEV6jjH4V04zNIT1vjsv4h3GhcGjoGywvg97iLgG+xbWVvxdBzFTqb4X4PrJH5u3A1gWSSK5Fdd4U+x6ygQFr+lCZDb2YqM2gKCvYB0TlViOG30/Fq/5Uea3n4nPUQzaSkSmMRjd/k+A6e0Oe8EOTocwb5ScEpOtVF0dn6ku7gekLgnuk1Atu+oAeUS1L4fnF4YbKAmxkmi/HIKYgDuJ6lfx6Y5Lv0KF1fNuqDnqGfue4sQs8eKVlMrbjscFgHDJF7ecz00F6OHaRRjX1CxsHwVu02VJDSk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Recall that the aging produces the youngest generation: first it scans for accessed folios and updates their gen counters; then it increments lrugen->max_seq. The current aging fairness safeguard for kswapd uses two passes to ensure the fairness to multiple eligible memcgs. On the first pass, which is shared with the eviction, it checks whether all eligible memcgs are low on cold folios. If so, it requires a second pass, on which it ages all those memcgs at the same time. With memcg LRU, the aging, while ensuring eventual fairness, will run when necessary. Therefore the current aging fairness safeguard for kswapd will not be needed. Note that memcg LRU only applies to global reclaim. For memcg reclaim, the aging can be unfair to different memcgs, i.e., their lrugen->max_seq can be incremented at different paces. Signed-off-by: Yu Zhao Change-Id: I66c70bd31d5276c710ad9209f0a74b1c24a0eda9 --- mm/vmscan.c | 150 +++++++++++++++++++++++++--------------------------- 1 file changed, 71 insertions(+), 79 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9655b3b3a95e..a2f71400b8be 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -137,7 +137,6 @@ struct scan_control { #ifdef CONFIG_LRU_GEN /* help kswapd make better choices among multiple memcgs */ - unsigned int memcgs_need_aging:1; unsigned long last_reclaimed; #endif @@ -4471,7 +4470,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, return true; } -static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, unsigned long *min_seq, +static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, struct scan_control *sc, bool can_swap, unsigned long *nr_to_scan) { int gen, type, zone; @@ -4480,6 +4479,13 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, unsig unsigned long total = 0; struct lru_gen_folio *lrugen = &lruvec->lrugen; struct mem_cgroup *memcg = lruvec_memcg(lruvec); + DEFINE_MIN_SEQ(lruvec); + + /* whether this lruvec is completely out of cold folios */ + if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) { + *nr_to_scan = 0; + return true; + } for (type = !can_swap; type < ANON_AND_FILE; type++) { unsigned long seq; @@ -4508,8 +4514,6 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, unsig * stalls when the number of generations reaches MIN_NR_GENS. Hence, the * ideal number of generations is MIN_NR_GENS+1. */ - if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) - return true; if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) return false; @@ -4528,40 +4532,54 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, unsig return false; } -static bool age_lruvec(struct lruvec *lruvec, struct scan_control *sc, unsigned long min_ttl) +static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc) { - bool need_aging; - unsigned long nr_to_scan; - int swappiness = get_swappiness(lruvec, sc); + int gen, type, zone; + unsigned long total = 0; + bool can_swap = get_swappiness(lruvec, sc); + struct lru_gen_folio *lrugen = &lruvec->lrugen; struct mem_cgroup *memcg = lruvec_memcg(lruvec); DEFINE_MAX_SEQ(lruvec); DEFINE_MIN_SEQ(lruvec); + for (type = !can_swap; type < ANON_AND_FILE; type++) { + unsigned long seq; + + for (seq = min_seq[type]; seq <= max_seq; seq++) { + gen = lru_gen_from_seq(seq); + + for (zone = 0; zone < MAX_NR_ZONES; zone++) + total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); + } + } + + /* whether the size is big enough to be helpful */ + return mem_cgroup_online(memcg) ? (total >> sc->priority) : total; +} + +static bool lruvec_is_reclaimable(struct lruvec *lruvec, struct scan_control *sc, + unsigned long min_ttl) +{ + int gen; + unsigned long birth; + struct mem_cgroup *memcg = lruvec_memcg(lruvec); + DEFINE_MIN_SEQ(lruvec); + VM_WARN_ON_ONCE(sc->memcg_low_reclaim); + /* see the comment on lru_gen_folio */ + gen = lru_gen_from_seq(min_seq[LRU_GEN_FILE]); + birth = READ_ONCE(lruvec->lrugen.timestamps[gen]); + + if (time_is_after_jiffies(birth + min_ttl)) + return false; + + if (!lruvec_is_sizable(lruvec, sc)) + return false; + mem_cgroup_calculate_protection(NULL, memcg); - if (mem_cgroup_below_min(NULL, memcg)) - return false; - - need_aging = should_run_aging(lruvec, max_seq, min_seq, sc, swappiness, &nr_to_scan); - - if (min_ttl) { - int gen = lru_gen_from_seq(min_seq[LRU_GEN_FILE]); - unsigned long birth = READ_ONCE(lruvec->lrugen.timestamps[gen]); - - if (time_is_after_jiffies(birth + min_ttl)) - return false; - - /* the size is likely too small to be helpful */ - if (!nr_to_scan && sc->priority != DEF_PRIORITY) - return false; - } - - if (need_aging) - try_to_inc_max_seq(lruvec, max_seq, sc, swappiness, false); - - return true; + return !mem_cgroup_below_min(NULL, memcg); } /* to protect the working set of the last N jiffies */ @@ -4570,46 +4588,32 @@ static unsigned long lru_gen_min_ttl __read_mostly; static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) { struct mem_cgroup *memcg; - bool success = false; unsigned long min_ttl = READ_ONCE(lru_gen_min_ttl); VM_WARN_ON_ONCE(!current_is_kswapd()); sc->last_reclaimed = sc->nr_reclaimed; - /* - * To reduce the chance of going into the aging path, which can be - * costly, optimistically skip it if the flag below was cleared in the - * eviction path. This improves the overall performance when multiple - * memcgs are available. - */ - if (!sc->memcgs_need_aging) { - sc->memcgs_need_aging = true; - return; - } - - set_mm_walk(pgdat); - - memcg = mem_cgroup_iter(NULL, NULL, NULL); - do { - struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); - - if (age_lruvec(lruvec, sc, min_ttl)) - success = true; - - cond_resched(); - } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL))); - - clear_mm_walk(); - /* check the order to exclude compaction-induced reclaim */ - if (success || !min_ttl || sc->order) + if (!min_ttl || sc->order || sc->priority == DEF_PRIORITY) return; + memcg = mem_cgroup_iter(NULL, NULL, NULL); + do { + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); + + if (lruvec_is_reclaimable(lruvec, sc, min_ttl)) { + mem_cgroup_iter_break(NULL, memcg); + return; + } + + cond_resched(); + } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL))); + /* * The main goal is to OOM kill if every generation from all memcgs is * younger than min_ttl. However, another possibility is all memcgs are - * either below min or empty. + * either too small or below min. */ if (mutex_trylock(&oom_lock)) { struct oom_control oc = { @@ -5117,34 +5121,28 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap * reclaim. */ static unsigned long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, - bool can_swap, bool *need_aging) + bool can_swap) { unsigned long nr_to_scan; struct mem_cgroup *memcg = lruvec_memcg(lruvec); DEFINE_MAX_SEQ(lruvec); - DEFINE_MIN_SEQ(lruvec); if (mem_cgroup_below_min(sc->target_mem_cgroup, memcg) || (mem_cgroup_below_low(sc->target_mem_cgroup, memcg) && !sc->memcg_low_reclaim)) return 0; - *need_aging = should_run_aging(lruvec, max_seq, min_seq, sc, can_swap, &nr_to_scan); - if (!*need_aging) + if (!should_run_aging(lruvec, max_seq, sc, can_swap, &nr_to_scan)) return nr_to_scan; /* skip the aging path at the default priority */ if (sc->priority == DEF_PRIORITY) - goto done; - - /* leave the work to lru_gen_age_node() */ - if (current_is_kswapd()) - return 0; - - if (try_to_inc_max_seq(lruvec, max_seq, sc, can_swap, false)) return nr_to_scan; -done: - return min_seq[!can_swap] + MIN_NR_GENS <= max_seq ? nr_to_scan : 0; + + try_to_inc_max_seq(lruvec, max_seq, sc, can_swap, false); + + /* skip this lruvec as it's low on cold folios */ + return 0; } static unsigned long get_nr_to_reclaim(struct scan_control *sc) @@ -5163,9 +5161,7 @@ static unsigned long get_nr_to_reclaim(struct scan_control *sc) static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) { struct blk_plug plug; - bool need_aging = false; unsigned long scanned = 0; - unsigned long reclaimed = sc->nr_reclaimed; unsigned long nr_to_reclaim = get_nr_to_reclaim(sc); lru_add_drain(); @@ -5186,13 +5182,13 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc else swappiness = 0; - nr_to_scan = get_nr_to_scan(lruvec, sc, swappiness, &need_aging); + nr_to_scan = get_nr_to_scan(lruvec, sc, swappiness); if (!nr_to_scan) - goto done; + break; delta = evict_folios(lruvec, sc, swappiness); if (!delta) - goto done; + break; scanned += delta; if (scanned >= nr_to_scan) @@ -5204,10 +5200,6 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc cond_resched(); } - /* see the comment in lru_gen_age_node() */ - if (sc->nr_reclaimed - reclaimed >= MIN_LRU_BATCH && !need_aging) - sc->memcgs_need_aging = false; -done: clear_mm_walk(); blk_finish_plug(&plug); From patchwork Wed Dec 21 00:12:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13078278 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A72A5C4167B for ; Wed, 21 Dec 2022 00:12:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 61FD88E000A; Tue, 20 Dec 2022 19:12:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5AB3E8E0007; Tue, 20 Dec 2022 19:12:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F9BB8E000A; Tue, 20 Dec 2022 19:12:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 27DCD8E0007 for ; Tue, 20 Dec 2022 19:12:45 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DE8AA4049A for ; Wed, 21 Dec 2022 00:12:44 +0000 (UTC) X-FDA: 80264387448.09.9EE804D Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf03.hostedemail.com (Postfix) with ESMTP id 48C4920005 for ; Wed, 21 Dec 2022 00:12:43 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=pYjtLo94; spf=pass (imf03.hostedemail.com: domain of 3ek-iYwYKCMU95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ek-iYwYKCMU95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671581563; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5gz0YgVLNDGFuqc3Lh4NnPiIlhtb3z+cq+FKhQZOoZk=; b=xtUiBQ45wV/jD8r4n3J8UozOTg4I1Vc0eL3DrYSFaroH7AGCmdZj0CdgSmgCc5OAmGrtJn X+/thDnjtwlWfIyi0DAvBuwwmpB4uJ02n2mcIRzD1wttszJTLnWbgmiMhiCaiOKDUoDfNs LbLFNiZsHGcOXvXqFNhrgpBEKEUkjPI= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=pYjtLo94; spf=pass (imf03.hostedemail.com: domain of 3ek-iYwYKCMU95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ek-iYwYKCMU95Aslzrzzrwp.nzxwty58-xxv6lnv.z2r@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671581563; a=rsa-sha256; cv=none; b=xQ14UdPyn/DWJ0fG9lzI2Nb4gGog7quBl0wA5xOekU+h54W58iU912fsk/AAJbvcHFD5Mq sRuc8SUaJtw/V1Una+OidZjZWWSqunviYsulDNRJwbjXuljKit2m1mFgWFQEtrm8KXPqik ByfIrpeNe1yjKCSivGLZUJa64eU/na8= Received: by mail-yb1-f201.google.com with SMTP id z17-20020a25e311000000b00719e04e59e1so15991219ybd.10 for ; Tue, 20 Dec 2022 16:12:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5gz0YgVLNDGFuqc3Lh4NnPiIlhtb3z+cq+FKhQZOoZk=; b=pYjtLo94emkX9HO5dNLwhnNl1q+/QkmPT4BV1Q2CYxKbEqjmuBFe+6E8hDAD23TmkJ rM8E5hVkIT1iJndDE4GFfnTHJmPy8rBNDiU6xKV12vavnbPR7PD/inFgUi8kelq6KFXs 0OyI9IPv8w8vh7ij2xerLvGSfzs7TWadUGqYgShpNGJ5FVlZQd8389NLfnqGGFtFYhZ5 yMAQh71Vn5/4k+ix3kKLG2LnpGcKhFJb6ylWem23nVTzY3PJF0dMSMUj/ohq835xZL4p HniHGo7EmzRDdAifMnyP6l1XmBjesx3FTwryw+4a+i46NpHmyyWubfVZzsndi55Y/SZd SgAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5gz0YgVLNDGFuqc3Lh4NnPiIlhtb3z+cq+FKhQZOoZk=; b=0muzJXK0JGfizLWKe+EN3pfSAZtqO3wU736PUqGLXPQ4in39rDa+kDVtFxK7AXCSJl C3a0VdZ7yW1FTgE8ieqxrwBap+B9FEQbhAs1TWjcMnmuvOVOuMcXvv7sJho4n2JbnqrK 9UQ2SKbsw8xoFwlRRBm8ESBvIM/BHsKZTyd6YCNyl6dEjccoyqXAqMcx43SWJ5ulX21O Q8Oi5hE9MAtLQKGs6pL32y1ZNOz3ypB6ljl/3x4Q3kG9NbWTA2t1l0AZsTcjom2iL0O4 PDImu5rfEBGe+60ADbISSHRuO2cuK6QJe/vPCvucBZ7U+eHAJ5yCwnKkpBh1is8eTkNZ v/AA== X-Gm-Message-State: ANoB5pmO4ZiCm0+Fe/ruxP0sZJguqnflIkLlzCE6kycwTNT+CMKQl8MX EgHT8Y/bEOzCrO4t5muElW5PAvokcIk= X-Google-Smtp-Source: AA0mqf5J92WPjbmjq0S0/8L8nqyD/H5zHMKEWJbsreI8pJ3tp81Dmd741SGe/PrsNv8NhFNSwOv39dKlDJo= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:a589:30f2:2daa:4ab7]) (user=yuzhao job=sendgmr) by 2002:a81:4d07:0:b0:427:c319:1abe with SMTP id a7-20020a814d07000000b00427c3191abemr3853654ywb.328.1671581562522; Tue, 20 Dec 2022 16:12:42 -0800 (PST) Date: Tue, 20 Dec 2022 17:12:05 -0700 In-Reply-To: <20221221001207.1376119-1-yuzhao@google.com> Message-Id: <20221221001207.1376119-6-yuzhao@google.com> Mime-Version: 1.0 References: <20221221001207.1376119-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Subject: [PATCH mm-unstable v2 5/8] mm: multi-gen LRU: shuffle should_run_aging() From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 48C4920005 X-Stat-Signature: tzw43uh7b8kaydg3kmkap7oye5c4mjx4 X-Rspam-User: X-HE-Tag: 1671581563-663207 X-HE-Meta: U2FsdGVkX19ShBiq8CtWXBi4+/u6JeQHmojKcd+6KH4a8+LvIZhpS9ZFKuG/Qe7Wo3oCkB5s7CdJJq+V0P5x/RnB2rhEyq/jyusOcEvLBjOCVU3jFseim5LriiM74cIRWJeJ1UOvnPXCdl5VNDYX1HzJ90ZsYTZ5z1GM5toFcoDYx8zVBTBdtjth3beoGEA6ZQQb8DILItGfR4xmCiy0X3AlqTgAqVogr6gK49j3mbvIi944RNqETYjLhOg8bl+04tL0WjjgAZQGrW0tGC4hOlxMLsk5ZKZP0D7gN1+5gbFwU9NJjHomOGboIXHbMO5cObn587+s2TD9hjHJBFCYDs7s617wktU6fymD/C4M/5hR3cOBCl9S85QZSHbMtk8q7XCvxUdnnVoR+Rzt9mqKK2ER9DbOxwwGdYmXAro/K6eFF4AkyHiu3wRn4/LdEbdPr2uF/TeSTL1Ujp+TOT9ty3S7TZkNDxcR/SYMb2HfDO7wU4PK/fDKT1ZvUV8F+h0Oi2X01JafR4YBkSXDfDXPw2axlwGLPkxjzr9OIJkB8CM42ygaQxgNMKP5PxFGvzBdLufMxnjtsFfUO3M+5jRKYM9axJH4saRLcUxZMADePiIcS77pww6NQzxnk0mDO4biP3xCZwZlL++j6O/BSrQNFyWl3xk27AlOQAS04TCtVTnsEmR/GMMn/6mi+9+0rvs752afS1bROqwRcNSX4t/bxWdqv/55nO9qcq7pt9UF4L4/THHuFkzWGj6A6TZ22ZRIET+M2AQiIvw3PyW1ZRSbfFhvx1bMAWAk+VYG8kHMiKfINUcZlvLaNRMNXgsFxlGjk+OYy3Bq7wdMagQQt6pcHWl/zLR97blVBld1704ecspfmfTDSH/eV9pfNYAgRnJn0PNjT0eo7Hbt3jvQUzenF7Blbuwp3GgpR0ZyP5xZIxIvJTINAFEZ6+wbvOOdzJGsTkab+v3xa7HLhSUyiyV Ipnt2iWH YC3QB1ejnEWFnw26cXuLxMj93yFqvlJVHTnYo3to8agV8ykjwdoTYiOxtbazwI3f9XB/qoL/PwqtoYecI4K+7Ou2ipBNLp6bpLpfV0E3ontQE+Gb0L2LfXyD9EWEUcbpxyRhro0wfmKXpP0mmYb5ff7WhH3/VRqIMtcBEbPaeJEpqtgrbG4CStFiTmwoD1vUAO+XfED2q1H1CDBKoO82qb7bk9ifilzCj+jhw+XFwl4Z6dP+bGndK2padfbV8NLIjAjuC5dRqkAIgg+tRzPmAOrED+rlMxnKtB9UzWDtp28lZRQDebAt9gAVUUExedbNT14iVuJo+/iFsWW5GT61YlUz6RcoOkijz2kXtAO6na0NS9sVpsS19VC8eWpRxBpKFcdtMJWYULtv9G2f4qoLfcDG4zWOe2fCz3Mq4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move should_run_aging() next to its only caller left. Signed-off-by: Yu Zhao Change-Id: I07e5372b98ca28c003861fdeddadde4304abcfe4 --- mm/vmscan.c | 124 ++++++++++++++++++++++++++-------------------------- 1 file changed, 62 insertions(+), 62 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a2f71400b8be..c424cc06f8c6 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4470,68 +4470,6 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, return true; } -static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, - struct scan_control *sc, bool can_swap, unsigned long *nr_to_scan) -{ - int gen, type, zone; - unsigned long old = 0; - unsigned long young = 0; - unsigned long total = 0; - struct lru_gen_folio *lrugen = &lruvec->lrugen; - struct mem_cgroup *memcg = lruvec_memcg(lruvec); - DEFINE_MIN_SEQ(lruvec); - - /* whether this lruvec is completely out of cold folios */ - if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) { - *nr_to_scan = 0; - return true; - } - - for (type = !can_swap; type < ANON_AND_FILE; type++) { - unsigned long seq; - - for (seq = min_seq[type]; seq <= max_seq; seq++) { - unsigned long size = 0; - - gen = lru_gen_from_seq(seq); - - for (zone = 0; zone < MAX_NR_ZONES; zone++) - size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); - - total += size; - if (seq == max_seq) - young += size; - else if (seq + MIN_NR_GENS == max_seq) - old += size; - } - } - - /* try to scrape all its memory if this memcg was deleted */ - *nr_to_scan = mem_cgroup_online(memcg) ? (total >> sc->priority) : total; - - /* - * The aging tries to be lazy to reduce the overhead, while the eviction - * stalls when the number of generations reaches MIN_NR_GENS. Hence, the - * ideal number of generations is MIN_NR_GENS+1. - */ - if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) - return false; - - /* - * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1) - * of the total number of pages for each generation. A reasonable range - * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The - * aging cares about the upper bound of hot pages, while the eviction - * cares about the lower bound of cold pages. - */ - if (young * MIN_NR_GENS > total) - return true; - if (old * (MIN_NR_GENS + 2) < total) - return true; - - return false; -} - static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc) { int gen, type, zone; @@ -5115,6 +5053,68 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap return scanned; } +static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, + struct scan_control *sc, bool can_swap, unsigned long *nr_to_scan) +{ + int gen, type, zone; + unsigned long old = 0; + unsigned long young = 0; + unsigned long total = 0; + struct lru_gen_folio *lrugen = &lruvec->lrugen; + struct mem_cgroup *memcg = lruvec_memcg(lruvec); + DEFINE_MIN_SEQ(lruvec); + + /* whether this lruvec is completely out of cold folios */ + if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) { + *nr_to_scan = 0; + return true; + } + + for (type = !can_swap; type < ANON_AND_FILE; type++) { + unsigned long seq; + + for (seq = min_seq[type]; seq <= max_seq; seq++) { + unsigned long size = 0; + + gen = lru_gen_from_seq(seq); + + for (zone = 0; zone < MAX_NR_ZONES; zone++) + size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); + + total += size; + if (seq == max_seq) + young += size; + else if (seq + MIN_NR_GENS == max_seq) + old += size; + } + } + + /* try to scrape all its memory if this memcg was deleted */ + *nr_to_scan = mem_cgroup_online(memcg) ? (total >> sc->priority) : total; + + /* + * The aging tries to be lazy to reduce the overhead, while the eviction + * stalls when the number of generations reaches MIN_NR_GENS. Hence, the + * ideal number of generations is MIN_NR_GENS+1. + */ + if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) + return false; + + /* + * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1) + * of the total number of pages for each generation. A reasonable range + * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The + * aging cares about the upper bound of hot pages, while the eviction + * cares about the lower bound of cold pages. + */ + if (young * MIN_NR_GENS > total) + return true; + if (old * (MIN_NR_GENS + 2) < total) + return true; + + return false; +} + /* * For future optimizations: * 1. Defer try_to_inc_max_seq() to workqueues to reduce latency for memcg From patchwork Wed Dec 21 00:12:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13078279 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E528DC3DA6E for ; Wed, 21 Dec 2022 00:12:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A3E618E0007; Tue, 20 Dec 2022 19:12:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9EF4F900002; Tue, 20 Dec 2022 19:12:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 755FB8E000B; Tue, 20 Dec 2022 19:12:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5EC7E8E0007 for ; Tue, 20 Dec 2022 19:12:46 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3ECEDA0274 for ; Wed, 21 Dec 2022 00:12:46 +0000 (UTC) X-FDA: 80264387532.01.81D09AC Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf07.hostedemail.com (Postfix) with ESMTP id 960FE40002 for ; Wed, 21 Dec 2022 00:12:44 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=codeYwy+; spf=pass (imf07.hostedemail.com: domain of 3e0-iYwYKCMYA6Btm0s00sxq.o0yxuz69-yyw7mow.03s@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3e0-iYwYKCMYA6Btm0s00sxq.o0yxuz69-yyw7mow.03s@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671581564; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9RaF+jsgl78F4GBrxUC5NFR9ONl7CxKbRAdYv2n4r6w=; b=8qHC5mhCZpjARd9MGdk0vPwNniWzdA1OuT75P7ucRgkMRk3bupaPrsIrNP/IK0T+u9Yh4I 4tWfRVBCE3ajK2/ySmqsFleX4zzAkpqxqcawH/HuWefraZLutLng+rHc9VnUujHzbZ1En3 Dt9bz9pb/rmbqJXtGXe/utUWmJ9uvQc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=codeYwy+; spf=pass (imf07.hostedemail.com: domain of 3e0-iYwYKCMYA6Btm0s00sxq.o0yxuz69-yyw7mow.03s@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3e0-iYwYKCMYA6Btm0s00sxq.o0yxuz69-yyw7mow.03s@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671581564; a=rsa-sha256; cv=none; b=rMM9SYSMii1C8FOnNNL7XC6FrWGzFuLJiSiGpmsUlBSRXsLzZJ6tgh5YmByUMAssxuwNAX 06lsUtYkJUojS5lvo8SGnj1wv/itekpHAAcGDOpbeiRGRSmCdnBvIi7hR+XHC00Y5YKpHR Os8fxYx31mf0KNC8FAjUPGqklD/H8Iw= Received: by mail-yb1-f202.google.com with SMTP id t186-20020a2546c3000000b0071e2aff109dso15933629yba.16 for ; Tue, 20 Dec 2022 16:12:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9RaF+jsgl78F4GBrxUC5NFR9ONl7CxKbRAdYv2n4r6w=; b=codeYwy+3WhegyKvjhf4lUEgs7xctF0KP/3pv3spz94RHy1bRHbB8jKCb97UGUOalS KKDVWrvMFW2rphd+ZI0JSoJY9Y7mTw0WGdchgryFQADd9AGx4vcMwqYPhTI8/OXwHnbP H5Qbt6g84Cl44k6CS3i+ry19zUKF3Fsxejw5SRrV79PFh5FHR7Z8Qz/EOWGtPb9uKNj3 zwwel3UXeFTLAbral6+S+Oi9qDmV7cjTKdXqmWGxKYUoBdVy/YrHwpE2s25PKDc2T3ao BFE2IZiPiTrFSn9apFczgMUF+GWJQAt4FznOi//kGIHTXJlUDg3lzzanVGGRrZsrko/s mDEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9RaF+jsgl78F4GBrxUC5NFR9ONl7CxKbRAdYv2n4r6w=; b=lBAc+HcSFxAjU5V1MhWUrlZ3Xo2cdNnMve6L18SmuyvgxM0WrFu2+DAI4GgX4x92Yd m3Bjs65Cb5VV9OdYyVN7NQGpWawHf0l0n68WQR7ZxhyCzjK4FRUe2eFxqIkK6ojBwPGd s2zj7V+sHuiNdXRuUCCNu9UQfImjOyQqz3LkI/+a5QH0WXBZP8cQftFTsVZ9iENf/jw3 YqHxaSviNbcMPanuBzEfa9G0QcJ4mPcO/bvGotLxvBf0H6EX1cOK31RCTp4FhZdOlQGv F+L/rkweqXKRXIUWTsmIr2CXVzYicaZno4S0l1kSPeAmK5P1dRMcPEEk4OmUn5CLh1uk N+hg== X-Gm-Message-State: ANoB5pnqXYqq3D+y5X2svqNgtv1oyoe73B4BHdF2vfWFD8VN6YUeO5Mm tocCKUIoz77cZO1xVmuw/reLC4i5eXU= X-Google-Smtp-Source: AA0mqf7uAAg/m7pbYV9kmIBLSnvfL8glYEfVXInZT2H+UAcmDJBY8hVnUYkB5BKQtF8O+eTGM7YJG+bxu80= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:a589:30f2:2daa:4ab7]) (user=yuzhao job=sendgmr) by 2002:a0d:dd8a:0:b0:391:c415:f872 with SMTP id g132-20020a0ddd8a000000b00391c415f872mr12456349ywe.318.1671581563815; Tue, 20 Dec 2022 16:12:43 -0800 (PST) Date: Tue, 20 Dec 2022 17:12:06 -0700 In-Reply-To: <20221221001207.1376119-1-yuzhao@google.com> Message-Id: <20221221001207.1376119-7-yuzhao@google.com> Mime-Version: 1.0 References: <20221221001207.1376119-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Subject: [PATCH mm-unstable v2 6/8] mm: multi-gen LRU: per-node lru_gen_folio lists From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 960FE40002 X-Rspam-User: X-Stat-Signature: hjpmopaj5a9h5pg5fec5q1c9zr7thyrf X-HE-Tag: 1671581564-628708 X-HE-Meta: U2FsdGVkX19Q70woGUjuh811Q8N76+TCD/cyvPEQ8ZuZSRijTIMdLu9n1a9j/Y54ONoCG4f/QWyVOQh01wy1QOaHA/PNmBexP5LIX87fftnjwUWvR7YhnIg727E5E9LdU5BQpH2bzJ03slf4Is7ya/lDp1G7JOD4iXIRbSzXB8fEr+J74ngvqx2GojJfMv30nyF/eqlC6vcNyL8qaQmz9D02gykVhQWtbW5AcGR8+jO0R+rV+iKO3AgSzN6pQfZuciwr1i7ZpN8K0T7e1BCAMD3N0ZIVyGBQCYH1dZvirxZ8h1wEE/+fbyilcw1+c0tGq21Fc5F5Vej5ouPFihgIOMb+MMzdvBHhGkRBkbcPOKwaQyZASP+8w+wr2+cs0XbdKV8Rs6wPTlUINSI65T8pjK9PnGuNPAB/DmwqmdtXn0bQNKHsPgvQqX3GyCCryztvD6CUpSodFuaHU+pq05FR80pK42iHSsZu+PLDDeWW+T8Zm4MAXj8tuVwlfEU+1+VrEeWUImAIaeGaxI5dAUFrNDpPdDqYMqs+ivGrv+uTrmFZkln4oZXYthCCpt/fWbEkb1BDco1IkLccAl3VSGTjwUovIAkzzyw3A6AxdaTa2oXwFhzC/HXbi0GQDDZzEoBEjVV1tz/nNyp8jJrkPUWHtjlLkOuFlvrELcFTZU29CcYTYwgL1x36g2zjSpdTkWm7bYgevD7MreJDJLDOF0Y7Uy4ArSXsTUlo+0yslLmI1sTBZxfsEa9hTBLAIkYeui5N2fOx9YGTBSs7CMiLFn9UWfxgTzhQcGLUMIdF7kSWn018FjtXhVxWhQWGaqXQepqYP14l9NafoigQP+CQhvfmbHiqrM1m8C+tVysr6SrJ+uUwMnCOzcnjxkQgpGNs8LbS5L83XgWTGlotmzTqbNLSRGvkFarqU3H63dznB4x2EyNb35kHAEXE7F5+xjoaxmlpP+7+nuzfx9nbujS+hg1 enB52zi/ 7woD78GBULsPZvmeZNEvtwC2VKRsVkf/6KwdDhUqdqzVBrDw8Kdpp4PMXIzOn/P2ILdlrQO5vTZNxltdJNv5rgS34imjYPbpeIjeVnWJcIa5wjtI/RkQXsMScoqwIIjGx6BxIcAfp8LpSWPjBnuMx8lSps2d3FpRalVUilAH8ChAsyHODKCTHFW9+RnCGf+UyxbC45NjhI7QMAdRUh6nnEgQ5NOL4cr/9v31HaYS5vnTRnO7lgnUdgVtDhXqdPurBFiV1f0sJfx7l+qUFJ6kgu1l8M3JIDuhJ7NOxoNlKGfamP9x8fMNwLbPibmPkPPHFvVAH/RgEqUY37s1OBFkUE785aMbpXq+5GJwv1s+XZCgaWijW3FdKLplgaz+h+RvesIh1+/MTgOW/6A8oOrN89j1feLqL0QSZAmja X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For each node, memcgs are divided into two generations: the old and the young. For each generation, memcgs are randomly sharded into multiple bins to improve scalability. For each bin, an RCU hlist_nulls is virtually divided into three segments: the head, the tail and the default. An onlining memcg is added to the tail of a random bin in the old generation. The eviction starts at the head of a random bin in the old generation. The per-node memcg generation counter, whose reminder (mod 2) indexes the old generation, is incremented when all its bins become empty. There are four operations: 1. MEMCG_LRU_HEAD, which moves an memcg to the head of a random bin in its current generation (old or young) and updates its "seg" to "head"; 2. MEMCG_LRU_TAIL, which moves an memcg to the tail of a random bin in its current generation (old or young) and updates its "seg" to "tail"; 3. MEMCG_LRU_OLD, which moves an memcg to the head of a random bin in the old generation, updates its "gen" to "old" and resets its "seg" to "default"; 4. MEMCG_LRU_YOUNG, which moves an memcg to the tail of a random bin in the young generation, updates its "gen" to "young" and resets its "seg" to "default". The events that trigger the above operations are: 1. Exceeding the soft limit, which triggers MEMCG_LRU_HEAD; 2. The first attempt to reclaim an memcg below low, which triggers MEMCG_LRU_TAIL; 3. The first attempt to reclaim an memcg below reclaimable size threshold, which triggers MEMCG_LRU_TAIL; 4. The second attempt to reclaim an memcg below reclaimable size threshold, which triggers MEMCG_LRU_YOUNG; 5. Attempting to reclaim an memcg below min, which triggers MEMCG_LRU_YOUNG; 6. Finishing the aging on the eviction path, which triggers MEMCG_LRU_YOUNG; 7. Offlining an memcg, which triggers MEMCG_LRU_OLD. Note that memcg LRU only applies to global reclaim, and the round-robin incrementing of their max_seq counters ensures the eventual fairness to all eligible memcgs. For memcg reclaim, it still relies on mem_cgroup_iter(). Signed-off-by: Yu Zhao Change-Id: I65cba82fc667920a3cfac8870d1f6ca250fc3804 --- include/linux/memcontrol.h | 10 + include/linux/mm_inline.h | 17 ++ include/linux/mmzone.h | 117 +++++++++++- mm/memcontrol.c | 16 ++ mm/page_alloc.c | 1 + mm/vmscan.c | 373 +++++++++++++++++++++++++++++++++---- 6 files changed, 499 insertions(+), 35 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index d3c8203cab6c..2e08b05bc6bf 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -794,6 +794,11 @@ static inline void obj_cgroup_put(struct obj_cgroup *objcg) percpu_ref_put(&objcg->refcnt); } +static inline bool mem_cgroup_tryget(struct mem_cgroup *memcg) +{ + return !memcg || css_tryget(&memcg->css); +} + static inline void mem_cgroup_put(struct mem_cgroup *memcg) { if (memcg) @@ -1301,6 +1306,11 @@ static inline void obj_cgroup_put(struct obj_cgroup *objcg) { } +static inline bool mem_cgroup_tryget(struct mem_cgroup *memcg) +{ + return true; +} + static inline void mem_cgroup_put(struct mem_cgroup *memcg) { } diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index da38e3d962e2..c1fd3922dc5d 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -122,6 +122,18 @@ static inline bool lru_gen_in_fault(void) return current->in_lru_fault; } +#ifdef CONFIG_MEMCG +static inline int lru_gen_memcg_seg(struct lruvec *lruvec) +{ + return READ_ONCE(lruvec->lrugen.seg); +} +#else +static inline int lru_gen_memcg_seg(struct lruvec *lruvec) +{ + return 0; +} +#endif + static inline int lru_gen_from_seq(unsigned long seq) { return seq % MAX_NR_GENS; @@ -297,6 +309,11 @@ static inline bool lru_gen_in_fault(void) return false; } +static inline int lru_gen_memcg_seg(struct lruvec *lruvec) +{ + return 0; +} + static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming) { return false; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 6c96ee823dbd..815c7c2edf45 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -7,6 +7,7 @@ #include #include +#include #include #include #include @@ -367,6 +368,15 @@ struct page_vma_mapped_walk; #define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF) #define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF) +/* see the comment on MEMCG_NR_GENS */ +enum { + MEMCG_LRU_NOP, + MEMCG_LRU_HEAD, + MEMCG_LRU_TAIL, + MEMCG_LRU_OLD, + MEMCG_LRU_YOUNG, +}; + #ifdef CONFIG_LRU_GEN enum { @@ -426,6 +436,14 @@ struct lru_gen_folio { atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; /* whether the multi-gen LRU is enabled */ bool enabled; +#ifdef CONFIG_MEMCG + /* the memcg generation this lru_gen_folio belongs to */ + u8 gen; + /* the list segment this lru_gen_folio belongs to */ + u8 seg; + /* per-node lru_gen_folio list for global reclaim */ + struct hlist_nulls_node list; +#endif }; enum { @@ -479,12 +497,87 @@ void lru_gen_init_lruvec(struct lruvec *lruvec); void lru_gen_look_around(struct page_vma_mapped_walk *pvmw); #ifdef CONFIG_MEMCG + +/* + * For each node, memcgs are divided into two generations: the old and the + * young. For each generation, memcgs are randomly sharded into multiple bins + * to improve scalability. For each bin, the hlist_nulls is virtually divided + * into three segments: the head, the tail and the default. + * + * An onlining memcg is added to the tail of a random bin in the old generation. + * The eviction starts at the head of a random bin in the old generation. The + * per-node memcg generation counter, whose reminder (mod MEMCG_NR_GENS) indexes + * the old generation, is incremented when all its bins become empty. + * + * There are four operations: + * 1. MEMCG_LRU_HEAD, which moves an memcg to the head of a random bin in its + * current generation (old or young) and updates its "seg" to "head"; + * 2. MEMCG_LRU_TAIL, which moves an memcg to the tail of a random bin in its + * current generation (old or young) and updates its "seg" to "tail"; + * 3. MEMCG_LRU_OLD, which moves an memcg to the head of a random bin in the old + * generation, updates its "gen" to "old" and resets its "seg" to "default"; + * 4. MEMCG_LRU_YOUNG, which moves an memcg to the tail of a random bin in the + * young generation, updates its "gen" to "young" and resets its "seg" to + * "default". + * + * The events that trigger the above operations are: + * 1. Exceeding the soft limit, which triggers MEMCG_LRU_HEAD; + * 2. The first attempt to reclaim an memcg below low, which triggers + * MEMCG_LRU_TAIL; + * 3. The first attempt to reclaim an memcg below reclaimable size threshold, + * which triggers MEMCG_LRU_TAIL; + * 4. The second attempt to reclaim an memcg below reclaimable size threshold, + * which triggers MEMCG_LRU_YOUNG; + * 5. Attempting to reclaim an memcg below min, which triggers MEMCG_LRU_YOUNG; + * 6. Finishing the aging on the eviction path, which triggers MEMCG_LRU_YOUNG; + * 7. Offlining an memcg, which triggers MEMCG_LRU_OLD. + * + * Note that memcg LRU only applies to global reclaim, and the round-robin + * incrementing of their max_seq counters ensures the eventual fairness to all + * eligible memcgs. For memcg reclaim, it still relies on mem_cgroup_iter(). + */ +#define MEMCG_NR_GENS 2 +#define MEMCG_NR_BINS 8 + +struct lru_gen_memcg { + /* the per-node memcg generation counter */ + unsigned long seq; + /* each memcg has one lru_gen_folio per node */ + unsigned long nr_memcgs[MEMCG_NR_GENS]; + /* per-node lru_gen_folio list for global reclaim */ + struct hlist_nulls_head fifo[MEMCG_NR_GENS][MEMCG_NR_BINS]; + /* protects the above */ + spinlock_t lock; +}; + +void lru_gen_init_pgdat(struct pglist_data *pgdat); + void lru_gen_init_memcg(struct mem_cgroup *memcg); void lru_gen_exit_memcg(struct mem_cgroup *memcg); -#endif +void lru_gen_online_memcg(struct mem_cgroup *memcg); +void lru_gen_offline_memcg(struct mem_cgroup *memcg); +void lru_gen_release_memcg(struct mem_cgroup *memcg); +void lru_gen_rotate_memcg(struct lruvec *lruvec, int op); + +#else /* !CONFIG_MEMCG */ + +#define MEMCG_NR_GENS 1 + +struct lru_gen_memcg { +}; + +static inline void lru_gen_init_pgdat(struct pglist_data *pgdat) +{ +} + +#endif /* CONFIG_MEMCG */ #else /* !CONFIG_LRU_GEN */ +static inline void lru_gen_init_pgdat(struct pglist_data *pgdat) +{ +} + static inline void lru_gen_init_lruvec(struct lruvec *lruvec) { } @@ -494,6 +587,7 @@ static inline void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) } #ifdef CONFIG_MEMCG + static inline void lru_gen_init_memcg(struct mem_cgroup *memcg) { } @@ -501,7 +595,24 @@ static inline void lru_gen_init_memcg(struct mem_cgroup *memcg) static inline void lru_gen_exit_memcg(struct mem_cgroup *memcg) { } -#endif + +static inline void lru_gen_online_memcg(struct mem_cgroup *memcg) +{ +} + +static inline void lru_gen_offline_memcg(struct mem_cgroup *memcg) +{ +} + +static inline void lru_gen_release_memcg(struct mem_cgroup *memcg) +{ +} + +static inline void lru_gen_rotate_memcg(struct lruvec *lruvec, int op) +{ +} + +#endif /* CONFIG_MEMCG */ #endif /* CONFIG_LRU_GEN */ @@ -1243,6 +1354,8 @@ typedef struct pglist_data { #ifdef CONFIG_LRU_GEN /* kswap mm walk data */ struct lru_gen_mm_walk mm_walk; + /* lru_gen_folio list */ + struct lru_gen_memcg memcg_lru; #endif CACHELINE_PADDING(_pad2_); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 92f319ef6c99..36200a1a448f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -477,6 +477,16 @@ static void mem_cgroup_update_tree(struct mem_cgroup *memcg, int nid) struct mem_cgroup_per_node *mz; struct mem_cgroup_tree_per_node *mctz; + if (lru_gen_enabled()) { + struct lruvec *lruvec = &memcg->nodeinfo[nid]->lruvec; + + /* see the comment on MEMCG_NR_GENS */ + if (soft_limit_excess(memcg) && lru_gen_memcg_seg(lruvec) != MEMCG_LRU_HEAD) + lru_gen_rotate_memcg(lruvec, MEMCG_LRU_HEAD); + + return; + } + mctz = soft_limit_tree.rb_tree_per_node[nid]; if (!mctz) return; @@ -3526,6 +3536,9 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, struct mem_cgroup_tree_per_node *mctz; unsigned long excess; + if (lru_gen_enabled()) + return 0; + if (order > 0) return 0; @@ -5386,6 +5399,7 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) if (unlikely(mem_cgroup_is_root(memcg))) queue_delayed_work(system_unbound_wq, &stats_flush_dwork, 2UL*HZ); + lru_gen_online_memcg(memcg); return 0; offline_kmem: memcg_offline_kmem(memcg); @@ -5417,6 +5431,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) memcg_offline_kmem(memcg); reparent_shrinker_deferred(memcg); wb_memcg_offline(memcg); + lru_gen_offline_memcg(memcg); drain_all_stock(memcg); @@ -5428,6 +5443,7 @@ static void mem_cgroup_css_released(struct cgroup_subsys_state *css) struct mem_cgroup *memcg = mem_cgroup_from_css(css); invalidate_reclaim_iterators(memcg); + lru_gen_release_memcg(memcg); } static void mem_cgroup_css_free(struct cgroup_subsys_state *css) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 7d980dc0000e..5668c1a2de49 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7941,6 +7941,7 @@ static void __init free_area_init_node(int nid) pgdat_set_deferred_range(pgdat); free_area_init_core(pgdat); + lru_gen_init_pgdat(pgdat); } static void __init free_area_init_memoryless_node(int nid) diff --git a/mm/vmscan.c b/mm/vmscan.c index c424cc06f8c6..fd837a13617c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -55,6 +55,7 @@ #include #include #include +#include #include #include @@ -135,11 +136,6 @@ struct scan_control { /* Always discard instead of demoting to lower tier memory */ unsigned int no_demotion:1; -#ifdef CONFIG_LRU_GEN - /* help kswapd make better choices among multiple memcgs */ - unsigned long last_reclaimed; -#endif - /* Allocation order */ s8 order; @@ -3185,6 +3181,9 @@ DEFINE_STATIC_KEY_ARRAY_FALSE(lru_gen_caps, NR_LRU_GEN_CAPS); for ((type) = 0; (type) < ANON_AND_FILE; (type)++) \ for ((zone) = 0; (zone) < MAX_NR_ZONES; (zone)++) +#define get_memcg_gen(seq) ((seq) % MEMCG_NR_GENS) +#define get_memcg_bin(bin) ((bin) % MEMCG_NR_BINS) + static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) { struct pglist_data *pgdat = NODE_DATA(nid); @@ -4456,8 +4455,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, if (sc->priority <= DEF_PRIORITY - 2) wait_event_killable(lruvec->mm_state.wait, max_seq < READ_ONCE(lrugen->max_seq)); - - return max_seq < READ_ONCE(lrugen->max_seq); + return false; } VM_WARN_ON_ONCE(max_seq != READ_ONCE(lrugen->max_seq)); @@ -4530,8 +4528,6 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) VM_WARN_ON_ONCE(!current_is_kswapd()); - sc->last_reclaimed = sc->nr_reclaimed; - /* check the order to exclude compaction-induced reclaim */ if (!min_ttl || sc->order || sc->priority == DEF_PRIORITY) return; @@ -5120,8 +5116,7 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, * 1. Defer try_to_inc_max_seq() to workqueues to reduce latency for memcg * reclaim. */ -static unsigned long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, - bool can_swap) +static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, bool can_swap) { unsigned long nr_to_scan; struct mem_cgroup *memcg = lruvec_memcg(lruvec); @@ -5139,10 +5134,8 @@ static unsigned long get_nr_to_scan(struct lruvec *lruvec, struct scan_control * if (sc->priority == DEF_PRIORITY) return nr_to_scan; - try_to_inc_max_seq(lruvec, max_seq, sc, can_swap, false); - /* skip this lruvec as it's low on cold folios */ - return 0; + return try_to_inc_max_seq(lruvec, max_seq, sc, can_swap, false) ? -1 : 0; } static unsigned long get_nr_to_reclaim(struct scan_control *sc) @@ -5151,29 +5144,18 @@ static unsigned long get_nr_to_reclaim(struct scan_control *sc) if (!global_reclaim(sc)) return -1; - /* discount the previous progress for kswapd */ - if (current_is_kswapd()) - return sc->nr_to_reclaim + sc->last_reclaimed; - return max(sc->nr_to_reclaim, compact_gap(sc->order)); } -static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) +static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) { - struct blk_plug plug; + long nr_to_scan; unsigned long scanned = 0; unsigned long nr_to_reclaim = get_nr_to_reclaim(sc); - lru_add_drain(); - - blk_start_plug(&plug); - - set_mm_walk(lruvec_pgdat(lruvec)); - while (true) { int delta; int swappiness; - unsigned long nr_to_scan; if (sc->may_swap) swappiness = get_swappiness(lruvec, sc); @@ -5183,7 +5165,7 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc swappiness = 0; nr_to_scan = get_nr_to_scan(lruvec, sc, swappiness); - if (!nr_to_scan) + if (nr_to_scan <= 0) break; delta = evict_folios(lruvec, sc, swappiness); @@ -5200,10 +5182,251 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc cond_resched(); } + /* whether try_to_inc_max_seq() was successful */ + return nr_to_scan < 0; +} + +static int shrink_one(struct lruvec *lruvec, struct scan_control *sc) +{ + bool success; + unsigned long scanned = sc->nr_scanned; + unsigned long reclaimed = sc->nr_reclaimed; + int seg = lru_gen_memcg_seg(lruvec); + struct mem_cgroup *memcg = lruvec_memcg(lruvec); + struct pglist_data *pgdat = lruvec_pgdat(lruvec); + + /* see the comment on MEMCG_NR_GENS */ + if (!lruvec_is_sizable(lruvec, sc)) + return seg != MEMCG_LRU_TAIL ? MEMCG_LRU_TAIL : MEMCG_LRU_YOUNG; + + mem_cgroup_calculate_protection(NULL, memcg); + + if (mem_cgroup_below_min(NULL, memcg)) + return MEMCG_LRU_YOUNG; + + if (mem_cgroup_below_low(NULL, memcg)) { + /* see the comment on MEMCG_NR_GENS */ + if (seg != MEMCG_LRU_TAIL) + return MEMCG_LRU_TAIL; + + memcg_memory_event(memcg, MEMCG_LOW); + } + + success = try_to_shrink_lruvec(lruvec, sc); + + shrink_slab(sc->gfp_mask, pgdat->node_id, memcg, sc->priority); + + if (!sc->proactive) + vmpressure(sc->gfp_mask, memcg, false, sc->nr_scanned - scanned, + sc->nr_reclaimed - reclaimed); + + sc->nr_reclaimed += current->reclaim_state->reclaimed_slab; + current->reclaim_state->reclaimed_slab = 0; + + return success ? MEMCG_LRU_YOUNG : 0; +} + +#ifdef CONFIG_MEMCG + +static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc) +{ + int gen; + int bin; + int first_bin; + struct lruvec *lruvec; + struct lru_gen_folio *lrugen; + const struct hlist_nulls_node *pos; + int op = 0; + struct mem_cgroup *memcg = NULL; + unsigned long nr_to_reclaim = get_nr_to_reclaim(sc); + + bin = first_bin = prandom_u32_max(MEMCG_NR_BINS); +restart: + gen = get_memcg_gen(READ_ONCE(pgdat->memcg_lru.seq)); + + rcu_read_lock(); + + hlist_nulls_for_each_entry_rcu(lrugen, pos, &pgdat->memcg_lru.fifo[gen][bin], list) { + if (op) + lru_gen_rotate_memcg(lruvec, op); + + mem_cgroup_put(memcg); + + lruvec = container_of(lrugen, struct lruvec, lrugen); + memcg = lruvec_memcg(lruvec); + + if (!mem_cgroup_tryget(memcg)) { + op = 0; + memcg = NULL; + continue; + } + + rcu_read_unlock(); + + op = shrink_one(lruvec, sc); + + if (sc->nr_reclaimed >= nr_to_reclaim) + goto success; + + rcu_read_lock(); + } + + rcu_read_unlock(); + + /* restart if raced with lru_gen_rotate_memcg() */ + if (gen != get_nulls_value(pos)) + goto restart; + + /* try the rest of the bins of the current generation */ + bin = get_memcg_bin(bin + 1); + if (bin != first_bin) + goto restart; +success: + if (op) + lru_gen_rotate_memcg(lruvec, op); + + mem_cgroup_put(memcg); +} + +static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) +{ + struct blk_plug plug; + + VM_WARN_ON_ONCE(global_reclaim(sc)); + + lru_add_drain(); + + blk_start_plug(&plug); + + set_mm_walk(lruvec_pgdat(lruvec)); + + if (try_to_shrink_lruvec(lruvec, sc)) + lru_gen_rotate_memcg(lruvec, MEMCG_LRU_YOUNG); + + clear_mm_walk(); + + blk_finish_plug(&plug); +} + +#else /* !CONFIG_MEMCG */ + +static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc) +{ + BUILD_BUG(); +} + +static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) +{ + BUILD_BUG(); +} + +#endif + +static void set_initial_priority(struct pglist_data *pgdat, struct scan_control *sc) +{ + int priority; + unsigned long reclaimable; + struct lruvec *lruvec = mem_cgroup_lruvec(NULL, pgdat); + + if (sc->priority != DEF_PRIORITY || sc->nr_to_reclaim < MIN_LRU_BATCH) + return; + /* + * Determine the initial priority based on ((total / MEMCG_NR_GENS) >> + * priority) * reclaimed_to_scanned_ratio = nr_to_reclaim, where the + * estimated reclaimed_to_scanned_ratio = inactive / total. + */ + reclaimable = node_page_state(pgdat, NR_INACTIVE_FILE); + if (get_swappiness(lruvec, sc)) + reclaimable += node_page_state(pgdat, NR_INACTIVE_ANON); + + reclaimable /= MEMCG_NR_GENS; + + /* round down reclaimable and round up sc->nr_to_reclaim */ + priority = fls_long(reclaimable) - 1 - fls_long(sc->nr_to_reclaim - 1); + + sc->priority = clamp(priority, 0, DEF_PRIORITY); +} + +static void lru_gen_shrink_node(struct pglist_data *pgdat, struct scan_control *sc) +{ + struct blk_plug plug; + unsigned long reclaimed = sc->nr_reclaimed; + + VM_WARN_ON_ONCE(!global_reclaim(sc)); + + lru_add_drain(); + + blk_start_plug(&plug); + + set_mm_walk(pgdat); + + set_initial_priority(pgdat, sc); + + if (current_is_kswapd()) + sc->nr_reclaimed = 0; + + if (mem_cgroup_disabled()) + shrink_one(&pgdat->__lruvec, sc); + else + shrink_many(pgdat, sc); + + if (current_is_kswapd()) + sc->nr_reclaimed += reclaimed; + clear_mm_walk(); blk_finish_plug(&plug); + + /* kswapd should never fail */ + pgdat->kswapd_failures = 0; +} + +#ifdef CONFIG_MEMCG +void lru_gen_rotate_memcg(struct lruvec *lruvec, int op) +{ + int seg; + int old, new; + int bin = prandom_u32_max(MEMCG_NR_BINS); + struct pglist_data *pgdat = lruvec_pgdat(lruvec); + + spin_lock(&pgdat->memcg_lru.lock); + + VM_WARN_ON_ONCE(hlist_nulls_unhashed(&lruvec->lrugen.list)); + + seg = 0; + new = old = lruvec->lrugen.gen; + + /* see the comment on MEMCG_NR_GENS */ + if (op == MEMCG_LRU_HEAD) + seg = MEMCG_LRU_HEAD; + else if (op == MEMCG_LRU_TAIL) + seg = MEMCG_LRU_TAIL; + else if (op == MEMCG_LRU_OLD) + new = get_memcg_gen(pgdat->memcg_lru.seq); + else if (op == MEMCG_LRU_YOUNG) + new = get_memcg_gen(pgdat->memcg_lru.seq + 1); + else + VM_WARN_ON_ONCE(true); + + hlist_nulls_del_rcu(&lruvec->lrugen.list); + + if (op == MEMCG_LRU_HEAD || op == MEMCG_LRU_OLD) + hlist_nulls_add_head_rcu(&lruvec->lrugen.list, &pgdat->memcg_lru.fifo[new][bin]); + else + hlist_nulls_add_tail_rcu(&lruvec->lrugen.list, &pgdat->memcg_lru.fifo[new][bin]); + + pgdat->memcg_lru.nr_memcgs[old]--; + pgdat->memcg_lru.nr_memcgs[new]++; + + lruvec->lrugen.gen = new; + WRITE_ONCE(lruvec->lrugen.seg, seg); + + if (!pgdat->memcg_lru.nr_memcgs[old] && old == get_memcg_gen(pgdat->memcg_lru.seq)) + WRITE_ONCE(pgdat->memcg_lru.seq, pgdat->memcg_lru.seq + 1); + + spin_unlock(&pgdat->memcg_lru.lock); } +#endif /****************************************************************************** * state change @@ -5658,11 +5881,11 @@ static int run_cmd(char cmd, int memcg_id, int nid, unsigned long seq, if (!mem_cgroup_disabled()) { rcu_read_lock(); + memcg = mem_cgroup_from_id(memcg_id); -#ifdef CONFIG_MEMCG - if (memcg && !css_tryget(&memcg->css)) + if (!mem_cgroup_tryget(memcg)) memcg = NULL; -#endif + rcu_read_unlock(); if (!memcg) @@ -5810,6 +6033,19 @@ void lru_gen_init_lruvec(struct lruvec *lruvec) } #ifdef CONFIG_MEMCG + +void lru_gen_init_pgdat(struct pglist_data *pgdat) +{ + int i, j; + + spin_lock_init(&pgdat->memcg_lru.lock); + + for (i = 0; i < MEMCG_NR_GENS; i++) { + for (j = 0; j < MEMCG_NR_BINS; j++) + INIT_HLIST_NULLS_HEAD(&pgdat->memcg_lru.fifo[i][j], i); + } +} + void lru_gen_init_memcg(struct mem_cgroup *memcg) { INIT_LIST_HEAD(&memcg->mm_list.fifo); @@ -5833,7 +6069,69 @@ void lru_gen_exit_memcg(struct mem_cgroup *memcg) } } } -#endif + +void lru_gen_online_memcg(struct mem_cgroup *memcg) +{ + int gen; + int nid; + int bin = prandom_u32_max(MEMCG_NR_BINS); + + for_each_node(nid) { + struct pglist_data *pgdat = NODE_DATA(nid); + struct lruvec *lruvec = get_lruvec(memcg, nid); + + spin_lock(&pgdat->memcg_lru.lock); + + VM_WARN_ON_ONCE(!hlist_nulls_unhashed(&lruvec->lrugen.list)); + + gen = get_memcg_gen(pgdat->memcg_lru.seq); + + hlist_nulls_add_tail_rcu(&lruvec->lrugen.list, &pgdat->memcg_lru.fifo[gen][bin]); + pgdat->memcg_lru.nr_memcgs[gen]++; + + lruvec->lrugen.gen = gen; + + spin_unlock(&pgdat->memcg_lru.lock); + } +} + +void lru_gen_offline_memcg(struct mem_cgroup *memcg) +{ + int nid; + + for_each_node(nid) { + struct lruvec *lruvec = get_lruvec(memcg, nid); + + lru_gen_rotate_memcg(lruvec, MEMCG_LRU_OLD); + } +} + +void lru_gen_release_memcg(struct mem_cgroup *memcg) +{ + int gen; + int nid; + + for_each_node(nid) { + struct pglist_data *pgdat = NODE_DATA(nid); + struct lruvec *lruvec = get_lruvec(memcg, nid); + + spin_lock(&pgdat->memcg_lru.lock); + + VM_WARN_ON_ONCE(hlist_nulls_unhashed(&lruvec->lrugen.list)); + + gen = lruvec->lrugen.gen; + + hlist_nulls_del_rcu(&lruvec->lrugen.list); + pgdat->memcg_lru.nr_memcgs[gen]--; + + if (!pgdat->memcg_lru.nr_memcgs[gen] && gen == get_memcg_gen(pgdat->memcg_lru.seq)) + WRITE_ONCE(pgdat->memcg_lru.seq, pgdat->memcg_lru.seq + 1); + + spin_unlock(&pgdat->memcg_lru.lock); + } +} + +#endif /* CONFIG_MEMCG */ static int __init init_lru_gen(void) { @@ -5860,6 +6158,10 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc { } +static void lru_gen_shrink_node(struct pglist_data *pgdat, struct scan_control *sc) +{ +} + #endif /* CONFIG_LRU_GEN */ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) @@ -5873,7 +6175,7 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) bool proportional_reclaim; struct blk_plug plug; - if (lru_gen_enabled()) { + if (lru_gen_enabled() && !global_reclaim(sc)) { lru_gen_shrink_lruvec(lruvec, sc); return; } @@ -6116,6 +6418,11 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) struct lruvec *target_lruvec; bool reclaimable = false; + if (lru_gen_enabled() && global_reclaim(sc)) { + lru_gen_shrink_node(pgdat, sc); + return; + } + target_lruvec = mem_cgroup_lruvec(sc->target_mem_cgroup, pgdat); again: From patchwork Wed Dec 21 00:12:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13078280 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 488EBC4332F for ; Wed, 21 Dec 2022 00:12:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0F60900003; Tue, 20 Dec 2022 19:12:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DBE6E900002; Tue, 20 Dec 2022 19:12:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B9BAD900003; Tue, 20 Dec 2022 19:12:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A2B3B900002 for ; Tue, 20 Dec 2022 19:12:47 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 67667140152 for ; Wed, 21 Dec 2022 00:12:47 +0000 (UTC) X-FDA: 80264387574.23.7D0A526 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf15.hostedemail.com (Postfix) with ESMTP id D2C63A0007 for ; Wed, 21 Dec 2022 00:12:45 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=BCLok8Tn; spf=pass (imf15.hostedemail.com: domain of 3fU-iYwYKCMgC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3fU-iYwYKCMgC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671581565; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gzZZ7Pl3axic12Be6+Sv18aJSfohVNrr9lclEYQlfdA=; b=QhGZrcPfIC/uTBXN0jhX/gkFNl4sxo+xXwsvs4In1M8FS8lXqkXd6OQgDpFF5PQAcO+ltm bT37dj32gG7C8cKdPBISU9667EN00T45n34DldNF7YRgUp9AOCLTl+7eh9DJQrSW1Vmflt a965nPY8BfkAttpHGzUdat4kd5RspM0= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=BCLok8Tn; spf=pass (imf15.hostedemail.com: domain of 3fU-iYwYKCMgC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3fU-iYwYKCMgC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671581565; a=rsa-sha256; cv=none; b=VzWWcixRm91ePmmEDGm30s8EFxuclMTUjlxX4tFa6WN6ejvu2PVtx0Z5rwO+bzwXnyNimf 85HZ1OtR4Xc9jwLCa3VnE3LSMOoCIDNvllDEjCM+Ek9XXCzQ4IlG9HzYWFIaQAL5D7PDFs 7VaZKpDDtgMoM7S00uMBMRt64MYTZe0= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-3ceb4c331faso158312607b3.2 for ; Tue, 20 Dec 2022 16:12:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gzZZ7Pl3axic12Be6+Sv18aJSfohVNrr9lclEYQlfdA=; b=BCLok8Tn1XTfQ1A8R6b7C5Ad/joLM/zJ52a9in4TvZQxFue8D9E+7yIFrIer98Eqj4 IT7c2Y7QptfbqvSrriBtpoLImc2rymZxl/aSRPL6iprTxYnoZ8GkLd6dpko1fT3qYpl3 j5aIasFOsO6exHTJNjrCfB/WGow8hUzKcSXoqt+CqvmuLFzcGhf9io/+i2C6Ss/e2mQV laXDGRPFMp9s74HpTyKqXCKzcyFji0RrSIXbOT0lve1VJZR/j3f6ef/UFnLZwT02BvE1 YIHTiq2hbzPBFT/mmXk6dbtrE2QduGkU5U0EBz06MoDqQSe1Flej1GG8CgMufMWwSm9d /umw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gzZZ7Pl3axic12Be6+Sv18aJSfohVNrr9lclEYQlfdA=; b=q5Ek5qEt6EZlC2vABFLUJTcrbqzz54qdQZ7poaJZI1Jnfx1wzgtoZd1xMWMwcjzwNg 24Lwi9N/NEqz/8VhB37rwU4DYAx0mGBTWBlQ3Wr7J0p685DDdSDzCdHIQo0JfX0Z6FAb mCy3P5EC2a7EtfhaliNMvJ7yZ08nfTK7m8KDchnHhK7krUcO81tTz2XUTjjEID+uNwmU nABUtl0SFDRAg4iGux4Ph8twsH859pLfffmG2BI4NExO8Hf0w3kpqk/q6vGTFLPQIQ3O 6xIVta9AFTB9xbnLuOVF2KVkKNDaJ8eLeJL1slAsuTib0QiZUG5tQ7YvE4uDgFpa59eM dANA== X-Gm-Message-State: ANoB5pmtZlYDH4Zv3kfwTFC+lgGu92Ec87UIbFIKJVX7z3TNl6hroQJH OkSrbMiLZ3OqWMC/mu1ZNvtvfMnn9Qs= X-Google-Smtp-Source: AA0mqf6rnjM1gkiEH2cSoCaqtQ9HV2F4qXN/ljYPU8+/Y/VmAHrOUkrolCEScMqy2/MGCZCUbXIT85hA/c0= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:a589:30f2:2daa:4ab7]) (user=yuzhao job=sendgmr) by 2002:a81:f90d:0:b0:404:7030:e1c0 with SMTP id x13-20020a81f90d000000b004047030e1c0mr6939777ywm.53.1671581565140; Tue, 20 Dec 2022 16:12:45 -0800 (PST) Date: Tue, 20 Dec 2022 17:12:07 -0700 In-Reply-To: <20221221001207.1376119-1-yuzhao@google.com> Message-Id: <20221221001207.1376119-8-yuzhao@google.com> Mime-Version: 1.0 References: <20221221001207.1376119-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Subject: [PATCH mm-unstable v2 7/8] mm: multi-gen LRU: clarify scan_control flags From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D2C63A0007 X-Rspam-User: X-Stat-Signature: c7hm75g4y5qoxw76grdegfnuj6mhny55 X-HE-Tag: 1671581565-332829 X-HE-Meta: U2FsdGVkX19X9PDv+UCdHls3nfMbHv0WTkNXUh7n+XQIsFpfJoqinNH6ZQmMKVOq2eTTBZtV4poIb28REQMZJwbmnYvtBytielGlF3hcnDEg4rblLpV25XDoMIEsr+Q7TxsZnsF50Y4zYimxcQ7UdEqR9Ojs9HRngUTQNr23oXkwb6csHa0/FyLkwVFOv1ICjy09M3US0GStnnylWk67KbH4rJ65h5u4c6SLRxBxKtRi1qJN7vShhkH7hOluwZHZKlOG4omhzx43iNtoT9kQaYEdN2u6UvJT9nDnCq5AhWOEb2PRGzdhK0KYqKrzM+8/jwUB6Tu+R0ILKZ4BP2t1rdq9RWpXryjbAAZ/EgV0Dc4T4WPXpgdj5HRt4wLBRrygbS5FeOn8TIgP6Sybz3L6DtmvWhdFlwIDn2pV01BMz+Epr02o5We3uYd/xmfOh99oNWXRuYuk99Ha6aziKDDPBGclCxzCA/R2ARaBGCrINO5z0+pj+W83bNTSv3NO00KwCd34FNgXwbQdkP8GmYPyF0rJz07yPcEq5SMQd0S3G0gmfd4e88XdnFYKVY7FCvkqyKNsORAa32cDyxLAAWUnfyohfalkmJOr5QyksBPJagxp16ZIrln801TNvfA5WOKyTiojTpPYl4pUNBe49lBgfVYAa6YnSjibQsiFa9KkkTOI2+7z7zOAzr9EHmczFDGWS0Fpg8NyqFBZEEOqZOY+RTTWU1AWFq/KVwP0+PdDmlvYSDrnvdGKdlP2AcVn4AR+1rfWR2SPptK+tKg7nEi7SGtkDYtzF6WcIpfAhqpGSrHUGBV2y7NdnCjMVKGPxgozaXMEvsjpsorpk37PM4qhk5fux7JlNzskWM+JgGXl8lCxwZIlCm28OSTwiONv5VqD8vB1fWhlMjb5e9X9OVbifM9QO4hwfgNNeVgnRpVXumLCPtLqSJOaWczvTx7pfUwgn4mixoyeV3mIoitbuuU h6SmqW4i wu5jYZb6YPqxJQZt1MFuc7v4esvrWm7mhk25FgkHSRldngcC9kSRux5Z7sbNWtBQSPVxJqxpbkVfbaWvlbWTnkiNZ/tV82tKPouH/0DQAwcKci7keRVuQELouqHFEL3UVxPj2BSHRN/EunX/xSfvUmGOWgeCnhbE7LJIUJqPYxkBx9FScaEfLQOAyIoEJ0AS95Kul/dPw07xOYa2NoQFvRpOvtriCkTd3/uZuFVq9EcdCMCuGX4yn/JcaAGjuDLePhIdzSIJDnguxNgRch2hgBI23QW6RAQoxrzhwddaPLVugAlffwT/eoSXt4Cz6iJnP1W1ruMsOkJWPL04QkagtYep559/Duv+OdHmuEcr13N0nkSbGsLXxw/vButfrKc3GyI1u+BrYK5a8mazepLU4qLaN9pxmieZ2wvGt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Among the flags in scan_control: 1. sc->may_swap, which indicates swap constraint due to memsw.max, is supported as usual. 2. sc->proactive, which indicates reclaim by memory.reclaim, may not opportunistically skip the aging path, since it is considered less latency sensitive. 3. !(sc->gfp_mask & __GFP_IO), which indicates IO constraint, lowers swappiness to prioritize file LRU, since clean file folios are more likely to exist. 4. sc->may_writepage and sc->may_unmap, which indicates opportunistic reclaim, are rejected, since unmapped clean folios are already prioritized. Scanning for more of them is likely futile and can cause high reclaim latency when there is a large number of memcgs. The rest are handled by the existing code. Signed-off-by: Yu Zhao Change-Id: Ic3b1a13ad1f88853427962b37669aa99942c9fb5 --- mm/vmscan.c | 56 ++++++++++++++++++++++++++--------------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index fd837a13617c..308c3c16d81d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3209,6 +3209,9 @@ static int get_swappiness(struct lruvec *lruvec, struct scan_control *sc) struct mem_cgroup *memcg = lruvec_memcg(lruvec); struct pglist_data *pgdat = lruvec_pgdat(lruvec); + if (!sc->may_swap) + return 0; + if (!can_demote(pgdat->node_id, sc) && mem_cgroup_get_nr_swap_pages(memcg) < MIN_LRU_BATCH) return 0; @@ -4238,7 +4241,7 @@ static void walk_mm(struct lruvec *lruvec, struct mm_struct *mm, struct lru_gen_ } while (err == -EAGAIN); } -static struct lru_gen_mm_walk *set_mm_walk(struct pglist_data *pgdat) +static struct lru_gen_mm_walk *set_mm_walk(struct pglist_data *pgdat, bool force_alloc) { struct lru_gen_mm_walk *walk = current->reclaim_state->mm_walk; @@ -4246,7 +4249,7 @@ static struct lru_gen_mm_walk *set_mm_walk(struct pglist_data *pgdat) VM_WARN_ON_ONCE(walk); walk = &pgdat->mm_walk; - } else if (!pgdat && !walk) { + } else if (!walk && force_alloc) { VM_WARN_ON_ONCE(current_is_kswapd()); walk = kzalloc(sizeof(*walk), __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN); @@ -4432,7 +4435,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, goto done; } - walk = set_mm_walk(NULL); + walk = set_mm_walk(NULL, true); if (!walk) { success = iterate_mm_list_nowalk(lruvec, max_seq); goto done; @@ -4501,8 +4504,6 @@ static bool lruvec_is_reclaimable(struct lruvec *lruvec, struct scan_control *sc struct mem_cgroup *memcg = lruvec_memcg(lruvec); DEFINE_MIN_SEQ(lruvec); - VM_WARN_ON_ONCE(sc->memcg_low_reclaim); - /* see the comment on lru_gen_folio */ gen = lru_gen_from_seq(min_seq[LRU_GEN_FILE]); birth = READ_ONCE(lruvec->lrugen.timestamps[gen]); @@ -4758,12 +4759,8 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca { bool success; - /* unmapping inhibited */ - if (!sc->may_unmap && folio_mapped(folio)) - return false; - /* swapping inhibited */ - if (!(sc->may_writepage && (sc->gfp_mask & __GFP_IO)) && + if (!(sc->gfp_mask & __GFP_IO) && (folio_test_dirty(folio) || (folio_test_anon(folio) && !folio_test_swapcache(folio)))) return false; @@ -4860,9 +4857,8 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, __count_vm_events(PGSCAN_ANON + type, isolated); /* - * There might not be eligible pages due to reclaim_idx, may_unmap and - * may_writepage. Check the remaining to prevent livelock if it's not - * making progress. + * There might not be eligible folios due to reclaim_idx. Check the + * remaining to prevent livelock if it's not making progress. */ return isolated || !remaining ? scanned : 0; } @@ -5122,9 +5118,7 @@ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, bool struct mem_cgroup *memcg = lruvec_memcg(lruvec); DEFINE_MAX_SEQ(lruvec); - if (mem_cgroup_below_min(sc->target_mem_cgroup, memcg) || - (mem_cgroup_below_low(sc->target_mem_cgroup, memcg) && - !sc->memcg_low_reclaim)) + if (mem_cgroup_below_min(sc->target_mem_cgroup, memcg)) return 0; if (!should_run_aging(lruvec, max_seq, sc, can_swap, &nr_to_scan)) @@ -5152,17 +5146,14 @@ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) long nr_to_scan; unsigned long scanned = 0; unsigned long nr_to_reclaim = get_nr_to_reclaim(sc); + int swappiness = get_swappiness(lruvec, sc); + + /* clean file folios are more likely to exist */ + if (swappiness && !(sc->gfp_mask & __GFP_IO)) + swappiness = 1; while (true) { int delta; - int swappiness; - - if (sc->may_swap) - swappiness = get_swappiness(lruvec, sc); - else if (!cgroup_reclaim(sc) && get_swappiness(lruvec, sc)) - swappiness = 1; - else - swappiness = 0; nr_to_scan = get_nr_to_scan(lruvec, sc, swappiness); if (nr_to_scan <= 0) @@ -5293,12 +5284,13 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc struct blk_plug plug; VM_WARN_ON_ONCE(global_reclaim(sc)); + VM_WARN_ON_ONCE(!sc->may_writepage || !sc->may_unmap); lru_add_drain(); blk_start_plug(&plug); - set_mm_walk(lruvec_pgdat(lruvec)); + set_mm_walk(NULL, sc->proactive); if (try_to_shrink_lruvec(lruvec, sc)) lru_gen_rotate_memcg(lruvec, MEMCG_LRU_YOUNG); @@ -5354,11 +5346,19 @@ static void lru_gen_shrink_node(struct pglist_data *pgdat, struct scan_control * VM_WARN_ON_ONCE(!global_reclaim(sc)); + /* + * Unmapped clean folios are already prioritized. Scanning for more of + * them is likely futile and can cause high reclaim latency when there + * is a large number of memcgs. + */ + if (!sc->may_writepage || !sc->may_unmap) + goto done; + lru_add_drain(); blk_start_plug(&plug); - set_mm_walk(pgdat); + set_mm_walk(pgdat, sc->proactive); set_initial_priority(pgdat, sc); @@ -5376,7 +5376,7 @@ static void lru_gen_shrink_node(struct pglist_data *pgdat, struct scan_control * clear_mm_walk(); blk_finish_plug(&plug); - +done: /* kswapd should never fail */ pgdat->kswapd_failures = 0; } @@ -5945,7 +5945,7 @@ static ssize_t lru_gen_seq_write(struct file *file, const char __user *src, set_task_reclaim_state(current, &sc.reclaim_state); flags = memalloc_noreclaim_save(); blk_start_plug(&plug); - if (!set_mm_walk(NULL)) { + if (!set_mm_walk(NULL, true)) { err = -ENOMEM; goto done; } From patchwork Wed Dec 21 00:12:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13078281 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9489DC41535 for ; Wed, 21 Dec 2022 00:12:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 275D5940007; Tue, 20 Dec 2022 19:12:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 22607900002; Tue, 20 Dec 2022 19:12:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 113E4940007; Tue, 20 Dec 2022 19:12:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id F2CC1900002 for ; Tue, 20 Dec 2022 19:12:48 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B1023C01CA for ; Wed, 21 Dec 2022 00:12:48 +0000 (UTC) X-FDA: 80264387616.08.6E2C913 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf10.hostedemail.com (Postfix) with ESMTP id 25FF9C0015 for ; Wed, 21 Dec 2022 00:12:46 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=K59wK9Xm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3fk-iYwYKCMkD9Ewp3v33v0t.r310x29C-11zAprz.36v@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3fk-iYwYKCMkD9Ewp3v33v0t.r310x29C-11zAprz.36v@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671581567; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XaREnfCRyQU3ad5Wd1iK/L1d7XHMzN1hNI71g9W9Aqc=; b=PMH2qO2VOJAfXa4+aBpX4bwtJm41IS8wn5YebnQxOFrX713ujd+24FbHFm0gkpMmnHpsP4 ROfQd2kFbs131ghC/lcz7IJeDp+ueOaRty/ee1Cl8QZf35hHlSvnuaDhuxKK4mnKg353ef 47XLpdF2Nbf2modVFZCR5q2dRmsXpZo= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=K59wK9Xm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3fk-iYwYKCMkD9Ewp3v33v0t.r310x29C-11zAprz.36v@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3fk-iYwYKCMkD9Ewp3v33v0t.r310x29C-11zAprz.36v@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671581567; a=rsa-sha256; cv=none; b=IN+S0HIccsbXsLBesY2+mNIMhhu979hayDLBH4Jk7jtWNwtUy+gQXOYMZJYFtuVeq0faIn g7bGBrkoivbsU3S3S0HdGlstJedM0hs1qs8vF13tqgZbc9cn0jNB3GwT8kNabYQrGEeci2 R4HmoFMjNUjOHbYnuzzQZNRinoci1M8= Received: by mail-yb1-f202.google.com with SMTP id k7-20020a256f07000000b006cbcc030bc8so15882793ybc.18 for ; Tue, 20 Dec 2022 16:12:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XaREnfCRyQU3ad5Wd1iK/L1d7XHMzN1hNI71g9W9Aqc=; b=K59wK9Xmrz3Bi5vvMVhN1G0pDO7CebJ60f+DjHwvBqntIUvRaRf11KCBF3IsW1uuCm LmdCEuyR6sl4QvqXqpqJlFmXBaasCbZ9nmXW6MTluImHgbKMWTiBxrs1SJxF+2QwuG8Y n3lT/WRDW8kZrenttBA6PiXNPTJQt4eM3DyDoOliigATORAsfTFpRCXSul0pGuAI9MlF xvpQD3dmjMzt8YndE3tmefxjvQ2G/DucE0e/w5OXs/Vyk/ThipPMV2jWTndUm75lJJeN Y5SOmz71wirINtVmmU6GITMwX4RwAhx/qeWMh267Z+OXa2GTQYBVO7aJK80W77QZ+2mh Zn+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XaREnfCRyQU3ad5Wd1iK/L1d7XHMzN1hNI71g9W9Aqc=; b=gD4Qf5GEbci/c8CcmtT3lJ85tzdrGgblTbpAQkkuW50PHnEVWojgS7UA8ROVJXfSvD HuxBSsQEan7rafodf6nQAt5vj3CMkn30uRXEJiLVVIhyI2HrKJpMmmKYLeCJjq2pMROC LCwV8MSJxvlx/mhNE/eVyjasOSEHGJH8KND+1wcMZLs5BcTSUDhpl1yDu4rv2C5GeHv7 lhkvm0rUCMAuKQXUD/IN7y8neUCvttFiCCFiGLDIyVwY8J0LHOP1U4JgnoKb467QerOY V8USqMk+c8cOB8xEK8bXpsEY3I30VA+smfZkqwttHXHw9oR56C3UQOkZXOn+mymIxc4d lynQ== X-Gm-Message-State: AFqh2ko+Vc8Oa/XClpHfXuWk+jsrO90Ivotn9BySOnMUYQpKARUZkPKK b4dBqXjZWuY1qjHZPDSJiVJVu1nwwJo= X-Google-Smtp-Source: AMrXdXur6oRo558EHTCVqmu2nmkUYNqNZaoP17m0xPtqeSBL8ltbKO3K5SoD9CmsnhBaHILyzV33vTOZCRs= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:a589:30f2:2daa:4ab7]) (user=yuzhao job=sendgmr) by 2002:a0d:cbd7:0:b0:388:7d2:575d with SMTP id n206-20020a0dcbd7000000b0038807d2575dmr768501ywd.21.1671581566428; Tue, 20 Dec 2022 16:12:46 -0800 (PST) Date: Tue, 20 Dec 2022 17:12:08 -0700 In-Reply-To: <20221221001207.1376119-1-yuzhao@google.com> Message-Id: <20221221001207.1376119-9-yuzhao@google.com> Mime-Version: 1.0 References: <20221221001207.1376119-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Subject: [PATCH mm-unstable v2 8/8] mm: multi-gen LRU: simplify arch_has_hw_pte_young() check From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Queue-Id: 25FF9C0015 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: hz75yrfwue1hrrrmqew6whhdf9kyo3q6 X-HE-Tag: 1671581566-996743 X-HE-Meta: U2FsdGVkX18k422k13UijIZpYYBCw9WdeUETJSF5WNlJ/MCxygx++bLgC4GogcRF6c9t3ZfsdfnTf2xW9sYeFC8+rQrWCdfG8hxSpaUp1snHrx8FdmE5GKSKO58QKGrz8nxvyF71YcnV7KPZbWrXaqBLMj7zFtvuKzIVZ4boq7DiAgzeWhWPnTY0MM4n6/etdVeNasq2za131mtALYFKHOKE6w0wfUbAPxys1x95mJy3tjJEc0Z+ChSd9ZGA6I0+gpm4DVTBHNQ+Oo4xcw+0B6QRhCGM7Ra6MyGKJH7fCRwJNxQAMsg8wVvzM2mWSKBFD26qPK59v0ZRkt2tIpaDZE416mU4dJX24X9IvbWIxVxCx33oyxVTJxBLiKqzhdxAmOc0uRtWOWxqKxqMZyhWcNogYVmHyetXQ+mB/CUVtzy+hXPJmxiBmitEI9IfUgNHdd2LI2DEldSyWmAP5DITwW059m7eEzOkKNPrKIPjHLV794839olwXCNpL1MnDdYHIY6o6Il7MofCYO7NDAhXm0PdnfQlsq/FjRzy5WLcs0SwTn0yRQColOZoMeaPuuWppD2X5+Ab/hGg3ubfqUbWcOK6cAgq6slgI3yskd8zADji9GN8hJtIJIY6LwWFU1CIAmOR2kcdJiIBE65baFBn9aWNW7AS6IN8fiOZnY/R/wJbZ95esOmCvA2p0xfNxq4Ji8X3mdn2JVN2dJOs0q9eSllr5pPn78ki31mhM27K1tN0sUvrVHkFRXVanbUD3MWO3gmfXIKRBIs03Nlbc6QM43fEXLdx4ZCkD+tpdYS36kpnl5MIIud5k+pocYhOxHkWE2Mr2FujSZFKHWy/cAk/h108CIRWqnGgfs+eZ3CeP7TBCXw0LRURHm4oWR2GI96EJkqNpVBN1eZyuYuWesskOVr7OWnYUr8cwGk/QhwDAIE/D44LMmK60Qssk65R6zpVPx+NbA4eRcs9Za5Kn/k 79nHnBcJ T7rVGQVN68zNOoK3+QoOVvLlQxdS/DjsU3aXAt7KPAzFtxpnm8xk+zyspN9u8ajz4VDLr0x49iPkus2s7wXQNe+Q+V48Z+3Xl02VYttKztNy5zJVNmkIq+TbVB2TABfCVqLzYqHbcH9ghqJ6memo4JnDSesnOrbo0gGxvBTAVY2PrRXv8+ibhTO/1LS8zBeFsDh35yccZS+zHGUMfBBcqmMK9PZkS+w2pqgwY4XkcAbKPUNBncrHNnRrmXNb7i1W7A4+J8z+k3p3Yo3CyKxNoLKstKsXqmJyl/bQyVwDyZZTzZ9wFwnu7dcxKhsAi2cQpCxzSBzbY0oJIpTqJYITb/34lv3QqSPhbGP9b/ePa4xxRll5BsbDJAbjCs1jG2ZSSVogdgKn3vC0K5PKf6d9FvfwPi1YYhUfq0yuH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Scanning page tables when hardware does not set the accessed bit has no real use cases. Signed-off-by: Yu Zhao Change-Id: I414310f1cf642d305a16045518517f09ce0eda23 --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 308c3c16d81d..88948f4c8d18 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4430,7 +4430,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, * handful of PTEs. Spreading the work out over a period of time usually * is less efficient, but it avoids bursty page faults. */ - if (!force_scan && !(arch_has_hw_pte_young() && get_cap(LRU_GEN_MM_WALK))) { + if (!arch_has_hw_pte_young() || !get_cap(LRU_GEN_MM_WALK)) { success = iterate_mm_list_nowalk(lruvec, max_seq); goto done; }