From patchwork Wed Jan 15 12:06:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Donet Tom X-Patchwork-Id: 13940380 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8584C02180 for ; Wed, 15 Jan 2025 12:06:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5077B280004; Wed, 15 Jan 2025 07:06:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4B84B280002; Wed, 15 Jan 2025 07:06:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3584D280004; Wed, 15 Jan 2025 07:06:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 15A7F280002 for ; Wed, 15 Jan 2025 07:06:43 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 6F1F214124D for ; Wed, 15 Jan 2025 12:06:42 +0000 (UTC) X-FDA: 83009559444.16.997011F Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf04.hostedemail.com (Postfix) with ESMTP id 3DCCA40019 for ; Wed, 15 Jan 2025 12:06:40 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b="sYcRaUr/"; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf04.hostedemail.com: domain of donettom@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=donettom@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736942800; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=Q68++GGoXmKPwhjOkPGV8ZviSVNKRTXNxz2brydr0VQ=; b=2Cixv+bEMWdzQ8k3x653NckWF665AWcnBh3MmckgIJrp3fr8b3bRDgl0QmHK5R/fzgWzjE GUSf+Ota6r/bZ94PibEHZuXICUYJwmJ2kXgY6Ktjp6FO9+PGnTZ1yAelotZ0Pl+hBC4GmS iASGySU6nALOfw6pWP+gtSsjKr6Oobw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736942800; a=rsa-sha256; cv=none; b=4SUF1JstoFyUNl3A7hMMVdfUBiZkKnYya64G2XOU+91Qx+RDk0uajop5+VS9EDwMxCpc1r +HbfuUzaxqMiTYFDFUJvDiePXqAIfXKnpxucbvkb6A2M17g8pxJ38jM2X0xg753OiedEyZ So+6zxvbXBHN1Eyk+enlzAvKcFp+vLI= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b="sYcRaUr/"; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf04.hostedemail.com: domain of donettom@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=donettom@linux.ibm.com Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 50F1PkAF013432; Wed, 15 Jan 2025 12:06:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:message-id :mime-version:subject:to; s=pp1; bh=Q68++GGoXmKPwhjOkPGV8ZviSVNK RTXNxz2brydr0VQ=; b=sYcRaUr/+MjKgtzi8+8jvnBc92lu0ZBxgxbxwjq4Lsou SAuo/q4KIaU/rV0IUfHxoc6WGodWk9D+uYck1Nv3ZYUhVkWnSNm6aDULdFFm7xz7 XjvRoHcEhyrvw9Mj8q47lPvdVItZN7+ifRVcE36Di6GbXRmthBFCg2+P6mxyCwNx ld5rBF+9NAga7lecIdRstclzSOIBHX68FDqHYhZvPLtLwmSl4RKu0unT8j3KfUyf u73JLpRf/LPahgCoUWoM8lE0TEvdGggdtbI5jNZhYjuxtEmZxJ8mQ9yl8ayvQ+qk vlkQkq70sCeekebGhpLgBsnkoKm6+HDDsQSmVDbpkg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 445sd64vp1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 15 Jan 2025 12:06:31 +0000 (GMT) Received: from m0353725.ppops.net (m0353725.ppops.net [127.0.0.1]) by pps.reinject (8.18.0.8/8.18.0.8) with ESMTP id 50FC5Q6b028736; Wed, 15 Jan 2025 12:06:30 GMT Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 445sd64vny-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 15 Jan 2025 12:06:30 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 50FBr1ZL001110; Wed, 15 Jan 2025 12:06:30 GMT Received: from smtprelay03.fra02v.mail.ibm.com ([9.218.2.224]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 44456jyw9y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 15 Jan 2025 12:06:29 +0000 Received: from smtpav01.fra02v.mail.ibm.com (smtpav01.fra02v.mail.ibm.com [10.20.54.100]) by smtprelay03.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 50FC6Sd335520936 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 15 Jan 2025 12:06:28 GMT Received: from smtpav01.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 384C920043; Wed, 15 Jan 2025 12:06:28 +0000 (GMT) Received: from smtpav01.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 78F0B20040; Wed, 15 Jan 2025 12:06:26 +0000 (GMT) Received: from ltczz402-lp1.aus.stglabs.ibm.com (unknown [9.40.194.31]) by smtpav01.fra02v.mail.ibm.com (Postfix) with ESMTP; Wed, 15 Jan 2025 12:06:26 +0000 (GMT) From: Donet Tom To: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Gregory Price Cc: Yu Zhao , Ritesh Harjani , "Aneesh Kumar K . V" , Matthew Wilcox , David Hildenbrand , Huang Ying , Johannes Weiner Subject: [RFC] mm/swap.c: Enable promotion of unmapped MGLRU page cache pages Date: Wed, 15 Jan 2025 06:06:25 -0600 Message-ID: <20250115120625.3785-1-donettom@linux.ibm.com> X-Mailer: git-send-email 2.43.5 MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: LGMfVWDP16kUO1s0Ia1wCWkUkb321qRc X-Proofpoint-ORIG-GUID: YmM6d4CDL8wePF6S2SgvrI69I_a86sT1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-01-15_04,2025-01-15_02,2024-11-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 adultscore=0 clxscore=1015 phishscore=0 spamscore=0 suspectscore=0 priorityscore=1501 mlxlogscore=999 bulkscore=0 lowpriorityscore=0 malwarescore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2411120000 definitions=main-2501150091 X-Stat-Signature: huxysuhpo9kryc7fnjhz7rrmpwh1pinp X-Rspamd-Queue-Id: 3DCCA40019 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1736942800-619971 X-HE-Meta: U2FsdGVkX1+reId/+lq3Bsqz8TpagUuTjsaJ+i67YtgUrjVys4rXTdHzxSoI7918c+0Lojs90Jp8hKxR981Lbi4mnp7r2CD+KBxiD+QQ1xqkbphMBWpBzS9CTl9n6nV8mWNqQkW+YmQleujubRbWeev+DxvfrXb67xSirRVk1CvX8MqGUUuDj2oWrocqsutvd4EY5a8EzQAcg/Hwb4z0VvVJ0x3HSkmjkXynYJz1WtE5Z/7MSruS3bFxQRYvfZrVJOBokjjd8CGX/lhadKAvGD+kJQrlMP47v6HKAyvSWg6K6cCRlcDjZO+C04o7RERr/WRKnlwk2Z98wDj0AhfwABB4k4x60HZEgyBcL8lALaR6CgJ/QKKJhcedODCpB0rbYgJWJ2c+KJAIAW7nWn8dt7hChvmlh63X4x5Ud565GdgZ4B5J+gTf5K22x8ymEyP2XBN621hpDK4XE2mLWgl2Vl+JlKjNCyqYagAgj0USHJGFWjuZM3Ztd8XgO1qAPBESdpLayn6Eg/+vYFyWjHiB3fbdFarjroRASqAHbFpmLmJNDwxEF5YTicUpHynqbuOCTRSK86oynT6UJZ5aiK1siWl70UwbhNQkciLbOeyUnrLRx2mYQBHwRLeoJl5N2HKC81f9+3Y+qb8KYncIisNhTO7PUnTGbQX7uClmumddXpaUUJqcOKSNVCWVCb2SzZsDnUDV7IHf4JpG4/0xBQVAelPZJHk6Rtiv3/LcdL8apZEAvtzKXjfqFXw4THim5Z3BKaXpKhzc30NexXRzGi1gu5P9mEDV/ni8J3E22o9tuSMYuqS9JfD8bnKTsCq1ILZF0pVSwd6PFA1Tl+BHxrnBWADH+cEg5xD5uRcpPsPUR6Wm5RPK/1QMI369MyyR22VrU2jyUjVz0+ppx3+5NL9I86yuZYaMk4du87nUSsC1gLLUCVOQVcVj8zib7jEdjVgMhxAD5EqPwtDRCCYBDpE GGiN0pM/ TD5OeJp+LqbypxdF9oxofjMHmYMmXZSrrZ/ft2vpc6WuSB4o7BtUt2/eR3vqlYr+aWFpEnKby2+nLTNZA1P3z1vt5p+B+S89N7ujcmR6QoReIf3qK5HIbjRfY2FCFF/8CHm3NlUJWwYprRQu2lz3qdg91snnKZfRAh/XE5t113I0pnQv6whUYNtM4d1V/hGziTqJPccJ3z6q6SezwN+EOr37HczPxLv8mLX41Es5OzNt4nCMTWQCcfKOnhw0H/qv7a+CPqPnasvV1rshRW3wqcq/pMiNXso/Za3oah0dr8EZM/UE9hTPwcSp1bUNdMrnqYnwqbqFkbD/6eyOb/FN4Q8uMoKY14NcrjgCuxrBfc8Iz7gFh5BCWn3xaSwb4BaaDFqhwXbOD8Z+5XNUUjGmFAbeMtMDH8ucMTrD6nNeGDymV5ehY2/54cWvkqXtzjgzgJ7LUQIOHAizrkP1GxQWlwwhzeSPHZ5WtOmtLEYMMxMbbnoM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch is based on patch [1], which introduced support for promoting unmapped normal LRU page cache pages. Here, we extend the functionality to support promotion of MGLRU page cache pages. An MGLRU page cache page is eligible for promotion when: 1. Memory Tiering and pagecache_promotion_enabled are enabled 2. It resides in a lower memory tier. 3. It is referenced. 4. It is part of the working set. 5. It belongs to the active list. For MGLRU, the youngest generation and the youngest generation - 1 are treated as the active list. When a page is accessed through a file descriptor, folio_inc_refs() is invoked. In this function, we check whether the page is referenced, is part of the working set, and Belongs to the active list. If all these conditions are met, the page is added to the promotion list. The per-process task task_numa_promotion_work() take the pages from the promotion list and promotes them to a higher memory tier. Test process: We measured the read time in below scenarios for both LRU and MGLRU. Scenario 1: Pages are on Lower tier + promotion off Scenario 2: Pages are on Lower tier + promotion on Scenario 3: Pages are on higher tier Test Results MGLRU --------------------------------------------------------------- Pages on higher | Pages Lower tier | Pages on Lower Tier | Tier | promotion off | Promotion On | --------------------------------------------------------------- 1.5s | 3.2s |During Promotion - 6.6s | | |After Promotion - 1.6s | | | | --------------------------------------------------------------- Test Results LRU --------------------------------------------------------------- Pages on higher | Pages Lower tier | Pages on Lower Tier | Tier | promotion off | Promotion On | --------------------------------------------------------------- 1.5s | 3.2s |During Promotion - 4.2s | | | - 3.4s | | | - 5.6s | | |After Promotion - 1.6s | | | | --------------------------------------------------------------- In LRU, pages are initially added to the inactive list. When a page is referenced, it is moved to the active list, and promotion to a higher memory tier occurs from the active list. This process often requires multiple reads to trigger promotion. In contrast, MGLRU adds new pages to the youngest generation immediately. As a result, we observe that MGLRU promotes pages on the first read itself, unlike LRU which takes multiple reads to trigger promotion. This difference also impacts read latency: For MGLRU, the first read shows higher latency due to the combined overhead of accessing a lower tier and performing promotion. For LRU, the first 3–4 reads typically exhibit lower latency since promotion does not occur immediately. MGLRU and LRU are showing similar performance benefit. [1] https://lore.kernel.org/all/20250107000346.1338481-1-gourry@gourry.net/ Signed-off-by: Donet Tom --- mm/swap.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index b2341bc18452..121de1d7e938 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -386,6 +386,9 @@ static void __lru_cache_activate_folio(struct folio *folio) static void lru_gen_inc_refs(struct folio *folio) { + struct mem_cgroup *memcg; + struct lruvec *lruvec; + int gen; unsigned long new_flags, old_flags = READ_ONCE(folio->flags); if (folio_test_unevictable(folio)) @@ -399,13 +402,29 @@ static void lru_gen_inc_refs(struct folio *folio) do { if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) { - if (!folio_test_workingset(folio)) + if (!folio_test_workingset(folio)) { folio_set_workingset(folio); - return; + return; + } + goto promo_candid; } new_flags = old_flags + BIT(LRU_REFS_PGOFF); } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); + +promo_candid: + if (!folio_test_isolated(folio) && + (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && + numa_pagecache_promotion_enabled) { + memcg = folio_memcg(folio); + if (memcg) { + lruvec = mem_cgroup_lruvec(memcg, folio_pgdat(folio)); + gen = folio_lru_gen(folio); + + if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen)) + promotion_candidate(folio); + } + } } static bool lru_gen_clear_refs(struct folio *folio)