From patchwork Fri Dec 9 17:29:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13070065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48E5AC04FDE for ; Fri, 9 Dec 2022 17:29:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D914C8E0003; Fri, 9 Dec 2022 12:29:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D41658E0001; Fri, 9 Dec 2022 12:29:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE1EB8E0003; Fri, 9 Dec 2022 12:29:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A9CF08E0001 for ; Fri, 9 Dec 2022 12:29:27 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 6B609A020F for ; Fri, 9 Dec 2022 17:29:27 +0000 (UTC) X-FDA: 80223454374.07.FD2320F Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf15.hostedemail.com (Postfix) with ESMTP id ACD9BA000B for ; Fri, 9 Dec 2022 17:29:25 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=HPuNxYxc; spf=pass (imf15.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670606965; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kuQDmXVI6AUKEcYPLNSszxFNnmXrkePaT3VtT+uEHZQ=; b=tFR1yr4ppys7XtraNkZPkskcoy96jRBJqhp4XucHu8F0rqnI9uxyuEhGE5+tmfMIIop6gi 3jrLR82nvxo1pUDElCbwvArEYZAnXb+ZM+y1grf80Y3Fre2opflidJfLBL0Rl1efewI8Ph 3iOx4T1EOJPM3No9mLkouPowLreTeYo= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=HPuNxYxc; spf=pass (imf15.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670606965; a=rsa-sha256; cv=none; b=GnFXZ7pmoBv0cfSlrYCpN/zlO9oQg6T9Mmymq9l5t8m5vB49GAior9XK+qxIe2+9AZYypp 7bh73afr7ZntByGZT344nTrZCxBKfHx1Eic60Iab7AlDM8y4a3Bx19tE5PWYeqVLMtUON2 SM89gBNmcjFJikebsgBV2yDEtV9J+28= Received: by mail-pl1-f178.google.com with SMTP id jl24so5578828plb.8 for ; Fri, 09 Dec 2022 09:29:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kuQDmXVI6AUKEcYPLNSszxFNnmXrkePaT3VtT+uEHZQ=; b=HPuNxYxcswp+YZr0FxtYoZgorvZcNvfgr727KgTVN1kVR9gXJWL1CqmBOHDPd3BTDh B1yXnSnkRJ9kmcK4hREN2GsL8ggUBH+lVsdc2CP69wO/ikZz2giVPPzXeiVtSKDpCBF0 kU8XEdMy5PmpOohYjejRZcSmOn8N2Ct4eUxpe2fLcSCAEBIr5MIZgcSFGrRb4LTrRL/q VJ4kNZLFuf+XWYW66bwQnWz4pfPhsOeTRCkznkS4kpLJ2qQgi6KOIzuPzTwHGOP7bY4q 6qhowfZmihUE/v+o/fDbj6lk8fCNaNkY/y84Cyd+pj57OEoR5d9Q2NChYxJJQD/DB/tm ySyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kuQDmXVI6AUKEcYPLNSszxFNnmXrkePaT3VtT+uEHZQ=; b=mIsLhaEa/RAT9etDBUbPygQmlPKu6iyCsn31ihQAmiI9EtByX0NIKOqxMzjJQZczgu LOP1t06d+R1jtOzr8HCeHXPL4++nGafvliy2UENR2vaYAWUX4ic9asR/6abkkgMbfRKK vIx3K5Ng7nr+yF8POlgC1YRGIpkG3CJzEL3EG0vBRJovsgBpNaood+VB1wBHV933cnuv 3AhPFVdKUfb7bUogN3o14du/72uYDwl+U33UyDnJ3QqXRKd3wyVKm1mRtd9mAhQ0+1os f6+D4kFb4DgsJgeDykaqpkAp4azR3ApjHr6j9VLTYd3xKqVcImCEILKDc8XD1j/p5MLZ /Pcw== X-Gm-Message-State: ANoB5pmOBW+0fRd971jg8kln/VLLbRQ0/AGGfsCgPApeAvlcVWr0+yI3 XiMQ+poZoGKduozUNbUs4F4= X-Google-Smtp-Source: AA0mqf75XoJ1pnnfx3MdPkMjxnv2Hj/GZA3os0MKBkwSpmBeVdS9RR5HtxMoEealyuWW6pVrLRQ1Mw== X-Received: by 2002:a17:902:c40f:b0:189:ea22:6d6a with SMTP id k15-20020a170902c40f00b00189ea226d6amr10405154plk.60.1670606964310; Fri, 09 Dec 2022 09:29:24 -0800 (PST) Received: from localhost (fwdproxy-prn-003.fbsv.net. [2a03:2880:ff:3::face:b00c]) by smtp.gmail.com with ESMTPSA id d8-20020a170902cec800b00189a50d2a38sm1613909plg.38.2022.12.09.09.29.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Dec 2022 09:29:23 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, bfoster@redhat.com, willy@infradead.org, kernel-team@meta.com Subject: [PATCH v3 2/4] workingset: refactor LRU refault to expose refault recency check Date: Fri, 9 Dec 2022 09:29:22 -0800 Message-Id: <20221209172922.3143160-1-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <202212091724.8wijxY41-lkp@intel.com> References: <202212091724.8wijxY41-lkp@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: ACD9BA000B X-Rspam-User: X-Stat-Signature: ce9myrcjixw46qt6ukgcndpxqyxeqx7k X-HE-Tag: 1670606965-861717 X-HE-Meta: U2FsdGVkX19kbfoPS+6d7q/TWEzAbzewTC8btERvNQbR5YuIcwPFSAQkjy7uCfZB+/LEvkjhCtuO+paqqGwsEdVlR1UwtwKuMv7DJU9GeQP4BgG0bJ1QwYRpcz6XIe5aWQ91khs3qa2JePUXdO2Zkw81i5B9xnbdZy3skSnVA0jv91n3K0NnZR4Jw9In5KEyS12mIIcJDh5Rjb1uwCfYZ+uUy+qkl0UpU9yCa5Xa0887TyaoTh8Ixmri5k/RgqLhbj/18sp5UcI7HRvVWmDXS1yXHvQlRZ4qw04rmpPgiyL/DGsN1GFfTfppOeaGIgs8ChhLHC/pMWlG6cAAXqB1rAwkxdccupCEnWmCuVgxke+iyA9R+dY0BM2p2eRbN5PDSSgccSzbCbd92c3oLkLtSJQ3VQDJRSyFk+1vhhWexMxavnI18SzHOqDnBpY3oeR2Gzs00etL3oHbsi4nxda2jEZ3qNe6uMuwfO9LMxIdCxn/AY+5eqiCYRT6CIgc7aNie5zg07uq5vQoxTP/a19O0pTUtO8zE3PuMarBFeQ++WzvmeiuxArlf5gQoUwrQrihgSvg6UEJc2LM7ozD9beoQIX5ba2y4V32R9D7WCuPmGA1IYFRUv/IYnCXZ/C07UhbVqFTvmkY3RhlE7t5weM08M4MaxwpJZaYdpLs0dzTP2u58x0UeVygDFaG+4VlO2IDip1rUZkVqGiCJXntOAH7vKrv34GOMQlnBBzwP2vrWH5v7pJoJ7wFiq9UTv6ohOVp3clNhvUiw/H5Rplu3E+hs1yI63MvzOVh0Qnj+fAbo9treVuAv4eq2zbml7CMi19m82o7dQiQzMMLmN/hrJdm7t+IKoM/s5JWfVKNaCXisvghwj4RxDgsKDzcCRlTlBH+mRyU9C8Hu0KWIpke3sW6pgEIujkCUcB+QlY075HDWJE5YOZkRM7Y6UQQJ6j69bbKk4GJ9sz0MFCJD42mNdR AqIOhU7M setKi60fWp95l/qwnLmCQzzOMx1EdYgGg3UOmyPPSr28xNf0ZR2l+S63q1fTwzVjhXDm51EmKAa03naD1/C/hLG6TbeDwk2ISVF28Nmq30l1DmKPBlZ1qRPAoBJkJFZn4VOxy X-Bogosity: Ham, tests=bogofilter, spamicity=0.000035, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for computing recently evicted pages in cachestat, refactor workingset_refault and lru_gen_refault to expose a helper function that would test if an evicted page is recently evicted. Signed-off-by: Nhat Pham --- include/linux/swap.h | 1 + mm/workingset.c | 129 ++++++++++++++++++++++++++++++------------- 2 files changed, 92 insertions(+), 38 deletions(-) Fix the issue reported by kernel test robot -- 2.30.2 diff --git a/include/linux/swap.h b/include/linux/swap.h index a18cf4b7c724..dae6f6f955eb 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -361,6 +361,7 @@ static inline void folio_set_swap_entry(struct folio *folio, swp_entry_t entry) } /* linux/mm/workingset.c */ +bool workingset_test_recent(void *shadow, bool file, bool *workingset); void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages); void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg); void workingset_refault(struct folio *folio, void *shadow); diff --git a/mm/workingset.c b/mm/workingset.c index 79585d55c45d..006482c4e0bd 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -244,6 +244,33 @@ static void *lru_gen_eviction(struct folio *folio) return pack_shadow(mem_cgroup_id(memcg), pgdat, token, refs); } +/* + * Test if the folio is recently evicted. + * + * As a side effect, also populates the references with + * values unpacked from the shadow of the evicted folio. + */ +static bool lru_gen_test_recent(void *shadow, bool file, bool *workingset) +{ + struct mem_cgroup *eviction_memcg; + struct lruvec *lruvec; + struct lru_gen_struct *lrugen; + unsigned long min_seq; + + int memcgid; + struct pglist_data *pgdat; + unsigned long token; + + unpack_shadow(shadow, &memcgid, &pgdat, &token, workingset); + eviction_memcg = mem_cgroup_from_id(memcgid); + + lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat); + lrugen = &lruvec->lrugen; + + min_seq = READ_ONCE(lrugen->min_seq[file]); + return !((token >> LRU_REFS_WIDTH) != (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH))); +} + static void lru_gen_refault(struct folio *folio, void *shadow) { int hist, tier, refs; @@ -306,6 +333,11 @@ static void *lru_gen_eviction(struct folio *folio) return NULL; } +static bool lru_gen_test_recent(void *shadow, bool file, bool *workingset) +{ + return true; +} + static void lru_gen_refault(struct folio *folio, void *shadow) { } @@ -373,40 +405,31 @@ void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg) folio_test_workingset(folio)); } -/** - * workingset_refault - Evaluate the refault of a previously evicted folio. - * @folio: The freshly allocated replacement folio. - * @shadow: Shadow entry of the evicted folio. +/* + * Test if the folio is recently evicted by checking if + * refault distance of shadow exceeds workingset size. * - * Calculates and evaluates the refault distance of the previously - * evicted folio in the context of the node and the memcg whose memory - * pressure caused the eviction. + * As a side effect, populate workingset with the value + * unpacked from shadow. */ -void workingset_refault(struct folio *folio, void *shadow) +bool workingset_test_recent(void *shadow, bool file, bool *workingset) { - bool file = folio_is_file_lru(folio); struct mem_cgroup *eviction_memcg; struct lruvec *eviction_lruvec; unsigned long refault_distance; unsigned long workingset_size; - struct pglist_data *pgdat; - struct mem_cgroup *memcg; - unsigned long eviction; - struct lruvec *lruvec; unsigned long refault; - bool workingset; + int memcgid; - long nr; + struct pglist_data *pgdat; + unsigned long eviction; - if (lru_gen_enabled()) { - lru_gen_refault(folio, shadow); - return; - } + if (lru_gen_enabled()) + return lru_gen_test_recent(shadow, file, workingset); - unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); + unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset); eviction <<= bucket_order; - rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might * have been deleted since the folio's eviction. @@ -425,7 +448,8 @@ void workingset_refault(struct folio *folio, void *shadow) */ eviction_memcg = mem_cgroup_from_id(memcgid); if (!mem_cgroup_disabled() && !eviction_memcg) - goto out; + return false; + eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat); refault = atomic_long_read(&eviction_lruvec->nonresident_age); @@ -447,21 +471,6 @@ void workingset_refault(struct folio *folio, void *shadow) */ refault_distance = (refault - eviction) & EVICTION_MASK; - /* - * The activation decision for this folio is made at the level - * where the eviction occurred, as that is where the LRU order - * during folio reclaim is being determined. - * - * However, the cgroup that will own the folio is the one that - * is actually experiencing the refault event. - */ - nr = folio_nr_pages(folio); - memcg = folio_memcg(folio); - pgdat = folio_pgdat(folio); - lruvec = mem_cgroup_lruvec(memcg, pgdat); - - mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - mem_cgroup_flush_stats_delayed(); /* * Compare the distance to the existing workingset size. We @@ -483,8 +492,51 @@ void workingset_refault(struct folio *folio, void *shadow) NR_INACTIVE_ANON); } } - if (refault_distance > workingset_size) + + return refault_distance <= workingset_size; +} + +/** + * workingset_refault - Evaluate the refault of a previously evicted folio. + * @folio: The freshly allocated replacement folio. + * @shadow: Shadow entry of the evicted folio. + * + * Calculates and evaluates the refault distance of the previously + * evicted folio in the context of the node and the memcg whose memory + * pressure caused the eviction. + */ +void workingset_refault(struct folio *folio, void *shadow) +{ + bool file = folio_is_file_lru(folio); + struct pglist_data *pgdat; + struct mem_cgroup *memcg; + struct lruvec *lruvec; + bool workingset; + long nr; + + if (lru_gen_enabled()) { + lru_gen_refault(folio, shadow); + return; + } + + rcu_read_lock(); + + nr = folio_nr_pages(folio); + memcg = folio_memcg(folio); + pgdat = folio_pgdat(folio); + lruvec = mem_cgroup_lruvec(memcg, pgdat); + + if (!workingset_test_recent(shadow, file, &workingset)) { + /* + * The activation decision for this folio is made at the level + * where the eviction occurred, as that is where the LRU order + * during folio reclaim is being determined. + * + * However, the cgroup that will own the folio is the one that + * is actually experiencing the refault event. + */ goto out; + } folio_set_active(folio); workingset_age_nonresident(lruvec, nr); @@ -498,6 +550,7 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr); } out: + mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); rcu_read_unlock(); }