From patchwork Wed Dec 21 00:12:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13078275 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31D19C3DA6E for ; Wed, 21 Dec 2022 00:12:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3888D8E0001; Tue, 20 Dec 2022 19:12:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 336558E0007; Tue, 20 Dec 2022 19:12:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D7A58E0001; Tue, 20 Dec 2022 19:12:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E93748E0007 for ; Tue, 20 Dec 2022 19:12:40 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B706BA4108 for ; Wed, 21 Dec 2022 00:12:40 +0000 (UTC) X-FDA: 80264387280.11.C30BF85 Received: from mail-oo1-f73.google.com (mail-oo1-f73.google.com [209.85.161.73]) by imf02.hostedemail.com (Postfix) with ESMTP id 1FF4D80009 for ; Wed, 21 Dec 2022 00:12:38 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="COBa/Wmi"; spf=pass (imf02.hostedemail.com: domain of 3dk-iYwYKCME516ohvnvvnsl.jvtspu14-ttr2hjr.vyn@flex--yuzhao.bounces.google.com designates 209.85.161.73 as permitted sender) smtp.mailfrom=3dk-iYwYKCME516ohvnvvnsl.jvtspu14-ttr2hjr.vyn@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671581559; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gUYK6iVE8NyMy+ADEXjGNSzP24X8eYlgIjWsftpIbpg=; b=hLFjnElksCXiBXr4XASBAaJoKdJeeu3tXcC2PrausFB1pMUTE2aA8lttBWnsT62B3FaO9m ldTXSEr6ecnfPv+t/2uyTqGlPBwJ9cG+IzOOIxn/Ja0sjOjm6MMDVHu/8DH7Wg6SrSFkzM Md4/TxTI2JBvW6FMSELZfLI5ZePckI0= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="COBa/Wmi"; spf=pass (imf02.hostedemail.com: domain of 3dk-iYwYKCME516ohvnvvnsl.jvtspu14-ttr2hjr.vyn@flex--yuzhao.bounces.google.com designates 209.85.161.73 as permitted sender) smtp.mailfrom=3dk-iYwYKCME516ohvnvvnsl.jvtspu14-ttr2hjr.vyn@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671581559; a=rsa-sha256; cv=none; b=Nsn4ZPjiEJ8zuW+2VhqKfn73C61I4cBcd1BqhdDma4Zu+6W6vf+99x+CWgLLU0YGLSFOwp cfoJn2ii8I9pM3jw1PujYkrE07aKxHO9AGjUT16JNHwIS+9JaxKLmB/z/xd6oEdf7KPpSs K0ywnh3B66qiVkAZzR3YAQ/gVPJFy5o= Received: by mail-oo1-f73.google.com with SMTP id b6-20020a4a3406000000b004a5ffc77240so6251608ooa.17 for ; Tue, 20 Dec 2022 16:12:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gUYK6iVE8NyMy+ADEXjGNSzP24X8eYlgIjWsftpIbpg=; b=COBa/Wmi0UcxSY0BUzCvzZSg4FdxLNKz22ZgMbaxMfNqGEwpnSzK+Ad+FmIUryr6eD hrvb+id173KoljBaCO6ipPfL22rnaj9lF2DyB8tH/BSRRrNnR2DEJAUpA4gNsiZ6FlUc 9HZPbwK4HzsTPIPRO9gNO96gKLjojmVvKA7YaNzfORXUGx/AUbGwKTYiOdvcUazPysOC RVTg6rZoSWlPvMoi58ZR0BbdLKUE04gIzuDoW6fpxTCkbueXJZsu7T+TVfsMZwmTtFgV N+JSbUk0w5cKXD/XclOTxPcIVANu4eZylI/VRfKjcol9oGuLvaY+QWoSz4BSy92hlMpb IPuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gUYK6iVE8NyMy+ADEXjGNSzP24X8eYlgIjWsftpIbpg=; b=nEjPWWATxLDOGBK3pTV7KMHqCElOM3o83CQoQVYg/1eYkhB8QD+8xlacZZUZDDqBRG tMo7vWCz8Ds+MiASxNndgN75gHMlmStTiHf4SHKMVuDaOs8m7HkrRoSCSt9y+KQ4MNnu viGOVoDZp8vXXXXfthL8Ncl1sJw709lPW4L5AimTsNxtzn3LZuyd2aV0Dy0prqEYiAZu 4yGnXgqHSWSf2AInMbqyKWMhHoGyPEGIlgQ3PkzdAwhtjRFAer3UXn5xq/362WQhUPXi GjqW217N97BeYa8M1d/7GV2bYT9ZcLxLeMg4x0EdiDk0ybg7s6zCIfrLvdRsolRKV5Sr oBHQ== X-Gm-Message-State: AFqh2ko2KZ8fn3M6VBs2w0oPSeqpEn66cZcRctRh1WoXPEmKOJt7gAL+ kXcEi+ZrD6iD4T+t+GElqeTwrlgyVrE= X-Google-Smtp-Source: AA0mqf4/rwvL7nj+AYw5nim+vzF9pRGefJw2rUsq83pO0l/l8ORY4gc6wDzcNVgSfgwxkSSHUd1IdMkHbhA= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:a589:30f2:2daa:4ab7]) (user=yuzhao job=sendgmr) by 2002:a05:6870:30e:b0:144:74f1:173d with SMTP id m14-20020a056870030e00b0014474f1173dmr2117657oaf.241.1671581558131; Tue, 20 Dec 2022 16:12:38 -0800 (PST) Date: Tue, 20 Dec 2022 17:12:02 -0700 In-Reply-To: <20221221001207.1376119-1-yuzhao@google.com> Message-Id: <20221221001207.1376119-3-yuzhao@google.com> Mime-Version: 1.0 References: <20221221001207.1376119-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Subject: [PATCH mm-unstable v2 2/8] mm: multi-gen LRU: rename lrugen->lists[] to lrugen->folios[] From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 1FF4D80009 X-Stat-Signature: z5fhgddar8w64z94izjybskcjpjmrp6f X-Rspam-User: X-HE-Tag: 1671581558-498567 X-HE-Meta: U2FsdGVkX18K/MSnHO5P17FGwVwx742AUKnI2yqiMbpj/d/yONHUs07oc9yuGDAW7MGjsPW51ejHgrS7gtl/wnFUjC/OAHi6lKtuRyy2fSF67FksgASl1Z1r15Lw7xLTOCl1Rc8MgnWBVgODSNcOO1Gcq1xryUMRQmxX4lhNGeE9Gh8NQGnhNnu4PQ5EpgK3JC8RF8dWtugyTNPcGI27/wLH4cgIm47nnBA0GxDzAHB0Q1A2GpSrsBfNoNLVCay9g4eks9wsbKQ5vdqk3l8amx6DRxkSTzlieJpWM5o7+/3DjHiZv47o9WL++X2FJ5ZHKnQfdxXLrTtUJ0leX48Z+AsarZ3EEhtWT7Gz69PEfVJ1uB+ZzDRWv5srAT1aWiZIV3nzG/gC5BTATdJe5mn/icpLGjRhUAraLiptAsWT9FWnBmoS82E4Chy6Zhlcu6MTtNK0TMkQCrF/db2l7jbiuUtTXrzKrK71a8sBhdihqxTSZHWMN6M6y6ehnNrqdXjOWP1F//hWPI2iKvnt39O3kZRT5CY7+I1Gt7rGSdsO3kxd3U8rRsYg9I6sSDV3vGbKlOA6PEkZpAgYmObaiXB8dhes5x1q94zXlSEMnJzy83973qv8X9KVH9dQC51JQfeOnTJTtp+y+xFRN+uWDAe0lt4wGPxKlXgd8cyxtNqHRR7drhD5QF14Mk+z18di5Zn55d2CJyYLi8Q3ORBzGOsYrcn28aa9SuiIERNV0DB07EeXdhy97VIgykrA+niEzVQMkVPyL32rtQtkANLhBjjEMg95qSL7k9MwebGCqXs5wmy9/dp4pBIG7g7wWqIsrmUShPnj5+QBfa/yo5WproMBc0muzekrVDC76R2PCvKWBHhSI7u8EWkVvcwXJ7B/3CQCcEqna1fOB+MXZGJ+FRS4X2kA7Kd20lfowPqeN05OkQekW+Beuxq7f4b+CAr3Nggg6yYEKvSiC5TJJyEx+/W 4BdtDYrP 7PuNAdeCIc9NGJhjM04DWLCd2FOSmXsFYOWMkXIPq19uFctQzVRCe9UhBgaPxSt2O0OhK1LsNsWjTdErSXlVheQ6YLes8yZ1c2i0IbKqwcazbdkeqIOn+NkzU5+eZeCic4lqTGkVIfNx3NIi6sWaL72WP3Jp3g+4E05OfL4pLiMPbp19m9TxQr+yrxFa4sr05xlsSDB668X/5kloef2Y6nXhOp1xCLShYZmUoLB9rpmz8shuF2vSlOmE8WbSdiHftGF68Z3AkwbO2tMiXaqK/A8pgOCiWTHI9F//VBZk7Df6k8WjyynB7PO/SikrvW25nglZDRukCIKFp9uv5xmsL/h2fKJMnm7MRZHVrvGxnVRnfQno= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: lru_gen_folio will be chained into per-node lists by the coming lrugen->list. Signed-off-by: Yu Zhao Change-Id: Ib1bee101fd6b73da5409d32e67409fa6475d8cc6 --- Documentation/mm/multigen_lru.rst | 8 ++++---- include/linux/mm_inline.h | 4 ++-- include/linux/mmzone.h | 8 ++++---- mm/vmscan.c | 20 ++++++++++---------- 4 files changed, 20 insertions(+), 20 deletions(-) diff --git a/Documentation/mm/multigen_lru.rst b/Documentation/mm/multigen_lru.rst index d7062c6a8946..d8f721f98868 100644 --- a/Documentation/mm/multigen_lru.rst +++ b/Documentation/mm/multigen_lru.rst @@ -89,15 +89,15 @@ variables are monotonically increasing. Generation numbers are truncated into ``order_base_2(MAX_NR_GENS+1)`` bits in order to fit into the gen counter in ``folio->flags``. Each -truncated generation number is an index to ``lrugen->lists[]``. The +truncated generation number is an index to ``lrugen->folios[]``. The sliding window technique is used to track at least ``MIN_NR_GENS`` and at most ``MAX_NR_GENS`` generations. The gen counter stores a value within ``[1, MAX_NR_GENS]`` while a page is on one of -``lrugen->lists[]``; otherwise it stores zero. +``lrugen->folios[]``; otherwise it stores zero. Each generation is divided into multiple tiers. A page accessed ``N`` times through file descriptors is in tier ``order_base_2(N)``. Unlike -generations, tiers do not have dedicated ``lrugen->lists[]``. In +generations, tiers do not have dedicated ``lrugen->folios[]``. In contrast to moving across generations, which requires the LRU lock, moving across tiers only involves atomic operations on ``folio->flags`` and therefore has a negligible cost. A feedback loop @@ -127,7 +127,7 @@ page mapped by this PTE to ``(max_seq%MAX_NR_GENS)+1``. Eviction -------- The eviction consumes old generations. Given an ``lruvec``, it -increments ``min_seq`` when ``lrugen->lists[]`` indexed by +increments ``min_seq`` when ``lrugen->folios[]`` indexed by ``min_seq%MAX_NR_GENS`` becomes empty. To select a type and a tier to evict from, it first compares ``min_seq[]`` to select the older type. If both types are equally old, it selects the one whose first tier has diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index f63968bd7de5..da38e3d962e2 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -256,9 +256,9 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, lru_gen_update_size(lruvec, folio, -1, gen); /* for folio_rotate_reclaimable() */ if (reclaiming) - list_add_tail(&folio->lru, &lrugen->lists[gen][type][zone]); + list_add_tail(&folio->lru, &lrugen->folios[gen][type][zone]); else - list_add(&folio->lru, &lrugen->lists[gen][type][zone]); + list_add(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 1686fcc4ed01..6c96ee823dbd 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -312,7 +312,7 @@ enum lruvec_flags { * They form a sliding window of a variable size [MIN_NR_GENS, MAX_NR_GENS]. An * offset within MAX_NR_GENS, i.e., gen, indexes the LRU list of the * corresponding generation. The gen counter in folio->flags stores gen+1 while - * a page is on one of lrugen->lists[]. Otherwise it stores 0. + * a page is on one of lrugen->folios[]. Otherwise it stores 0. * * A page is added to the youngest generation on faulting. The aging needs to * check the accessed bit at least twice before handing this page over to the @@ -324,8 +324,8 @@ enum lruvec_flags { * rest of generations, if they exist, are considered inactive. See * lru_gen_is_active(). * - * PG_active is always cleared while a page is on one of lrugen->lists[] so that - * the aging needs not to worry about it. And it's set again when a page + * PG_active is always cleared while a page is on one of lrugen->folios[] so + * that the aging needs not to worry about it. And it's set again when a page * considered active is isolated for non-reclaiming purposes, e.g., migration. * See lru_gen_add_folio() and lru_gen_del_folio(). * @@ -412,7 +412,7 @@ struct lru_gen_folio { /* the birth time of each generation in jiffies */ unsigned long timestamps[MAX_NR_GENS]; /* the multi-gen LRU lists, lazily sorted on eviction */ - struct list_head lists[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; + struct list_head folios[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; /* the multi-gen LRU sizes, eventually consistent */ long nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; /* the exponential moving average of refaulted */ diff --git a/mm/vmscan.c b/mm/vmscan.c index c0c4b0cd2fe2..94477894b226 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4274,7 +4274,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) /* prevent cold/hot inversion if force_scan is true */ for (zone = 0; zone < MAX_NR_ZONES; zone++) { - struct list_head *head = &lrugen->lists[old_gen][type][zone]; + struct list_head *head = &lrugen->folios[old_gen][type][zone]; while (!list_empty(head)) { struct folio *folio = lru_to_folio(head); @@ -4285,7 +4285,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); new_gen = folio_inc_gen(lruvec, folio, false); - list_move_tail(&folio->lru, &lrugen->lists[new_gen][type][zone]); + list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); if (!--remaining) return false; @@ -4313,7 +4313,7 @@ static bool try_to_inc_min_seq(struct lruvec *lruvec, bool can_swap) gen = lru_gen_from_seq(min_seq[type]); for (zone = 0; zone < MAX_NR_ZONES; zone++) { - if (!list_empty(&lrugen->lists[gen][type][zone])) + if (!list_empty(&lrugen->folios[gen][type][zone])) goto next; } @@ -4778,7 +4778,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx) /* promoted */ if (gen != lru_gen_from_seq(lrugen->min_seq[type])) { - list_move(&folio->lru, &lrugen->lists[gen][type][zone]); + list_move(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } @@ -4787,7 +4787,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx) int hist = lru_hist_from_seq(lrugen->min_seq[type]); gen = folio_inc_gen(lruvec, folio, false); - list_move_tail(&folio->lru, &lrugen->lists[gen][type][zone]); + list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); WRITE_ONCE(lrugen->protected[hist][type][tier - 1], lrugen->protected[hist][type][tier - 1] + delta); @@ -4799,7 +4799,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx) if (folio_test_locked(folio) || folio_test_writeback(folio) || (type == LRU_GEN_FILE && folio_test_dirty(folio))) { gen = folio_inc_gen(lruvec, folio, true); - list_move(&folio->lru, &lrugen->lists[gen][type][zone]); + list_move(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } @@ -4866,7 +4866,7 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, for (zone = sc->reclaim_idx; zone >= 0; zone--) { LIST_HEAD(moved); int skipped = 0; - struct list_head *head = &lrugen->lists[gen][type][zone]; + struct list_head *head = &lrugen->folios[gen][type][zone]; while (!list_empty(head)) { struct folio *folio = lru_to_folio(head); @@ -5267,7 +5267,7 @@ static bool __maybe_unused state_is_valid(struct lruvec *lruvec) int gen, type, zone; for_each_gen_type_zone(gen, type, zone) { - if (!list_empty(&lrugen->lists[gen][type][zone])) + if (!list_empty(&lrugen->folios[gen][type][zone])) return false; } } @@ -5312,7 +5312,7 @@ static bool drain_evictable(struct lruvec *lruvec) int remaining = MAX_LRU_BATCH; for_each_gen_type_zone(gen, type, zone) { - struct list_head *head = &lruvec->lrugen.lists[gen][type][zone]; + struct list_head *head = &lruvec->lrugen.folios[gen][type][zone]; while (!list_empty(head)) { bool success; @@ -5846,7 +5846,7 @@ void lru_gen_init_lruvec(struct lruvec *lruvec) lrugen->timestamps[i] = jiffies; for_each_gen_type_zone(gen, type, zone) - INIT_LIST_HEAD(&lrugen->lists[gen][type][zone]); + INIT_LIST_HEAD(&lrugen->folios[gen][type][zone]); lruvec->mm_state.seq = MIN_NR_GENS; init_waitqueue_head(&lruvec->mm_state.wait);