From patchwork Wed Jul 24 13:01:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13740916 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A46C1C3DA63 for ; Wed, 24 Jul 2024 13:01:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2AE086B0095; Wed, 24 Jul 2024 09:01:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 25F416B0096; Wed, 24 Jul 2024 09:01:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0894F6B0099; Wed, 24 Jul 2024 09:01:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id DBB796B0095 for ; Wed, 24 Jul 2024 09:01:49 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 86E21160938 for ; Wed, 24 Jul 2024 13:01:49 +0000 (UTC) X-FDA: 82374658338.19.9CA2AD0 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2066.outbound.protection.outlook.com [40.107.92.66]) by imf20.hostedemail.com (Postfix) with ESMTP id F093D1C0029 for ; Wed, 24 Jul 2024 13:01:42 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=qstCRrGE; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf20.hostedemail.com: domain of ziy@nvidia.com designates 40.107.92.66 as permitted sender) smtp.mailfrom=ziy@nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1721826064; a=rsa-sha256; cv=pass; b=ui60C+3663cyntk9ZZaNfkTT359lqDa/OYDDDPJk9FqjTPro3esVzJOLyXkc4r+rTHDzwe Ylrq2lXeX+ryVZZCVDIpquWR4IyNu34uHpiIVQsi6PyL4e+SI8ETZkPNgG6DSxOSOQ3Msi 6ENdfx/ToN87716JjZ5t5fVyKVu19Eg= ARC-Authentication-Results: i=2; imf20.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=qstCRrGE; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf20.hostedemail.com: domain of ziy@nvidia.com designates 40.107.92.66 as permitted sender) smtp.mailfrom=ziy@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721826064; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=de3HtV97/3HfU1Wwqz2WEK7UWyyj5aobEIaoBUZtd2I=; b=TglkMa8EyYlcR48XbMbDPkQrKNrEmZJXxn8ABUj/zvV1sUvocA2JNU+CGEoWSDxC4Hy4Rn flHB8SXL/lqMjATyc1hRPBfCGRfND7QeuewhJXjVxwisIofeNexiU8OfIopjaCtj7LzS+i ixE0REIOoO4V/WoiSFzLGMi0/490X0A= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=iQUX2ghHrM5OKpN6mlllpqbSjxg8AGuU1bv+7m0T7GZGWgOQwpe5419hI5fWhexQkCqYSUMc9Z4zwePSm7Z91Lf/Wij8D+mDONilKyZDnjmBSEauhyYRD4bScgQ8BSjWvzQya4tHKNa/UO5XnnbdsmubbnJV3hP0ne2ABNlK19L1boakZCA2iBOz45RhaoTAvYF7NEmWzIRzMtT48m+AYmLTPwLKa+DdnCYA2GdkJpv2zbYH8vcUE6NZ4RQYMz/9Tqs4sLOgZ/+807ODB2NROvxKBiQxmOLaTGBA5HVwTZbZd6EMGGU1LUeW3kj1b+s3y7r2gZtezGR5sEZECdaVyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=de3HtV97/3HfU1Wwqz2WEK7UWyyj5aobEIaoBUZtd2I=; b=kh9GUWVzbVpJHzbVLeQM6m7Q89u59cKJA1TKEkJDX9jEA1OPiYKK235KD0mjjfLeP1+prh1NjaLk5N1k4vdh6VmYBwBgTXz8YYxFKs477+AnNY5CQ6AjF00K2QZw1bRMGMC8lcUJJqhs5tuECF8QbiguZ6VQ1OgzE+3DblFLBTgQ31016Ky7cq/EdsGIMyaiCeaohYmvTn+F92Cyv1MiG0Aq0IZs2qCY1g9EIX39ReFYUzDZNci7vObDJzSvEucVelVeKIjw5dSpLRimo8Kylf+35ZGHkzJ6sjRumXzNXJ9Oxi0ogFzTsfYk1tM07Tftwf46Xs1a/BWO+HNbYyQK4w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=de3HtV97/3HfU1Wwqz2WEK7UWyyj5aobEIaoBUZtd2I=; b=qstCRrGE4Nx6o0ooMqjKn0DTR7PgDBleyJ8YHQUw+b94ShTkW+s7TR2GEPiDgrz7g5nc/7SgRkgt+mQrTlzQsO8IVwRnG9gyO74cJBFXNIxBjKcNWWEuszzyJwM9e4fP491l7R9RQkiEFZAzR5Gw5mcJggdTre4SPzVBvRLGs0599uG0xumkiKlUbePnEoxG+T/8qPXEKn+uC/1hKan+7GgmYs9+WixZj0zzYJxtckZmWrFVNqA7NKtK95uchfg4qypMtGUJJ6HohkMjn1Cu6TZDj7ZzYq09vM1JF/YIjzHupshyoC+s5rw3BOjY0cyJCmXay2+j6bQh1lmDpTTgqQ== Received: from DS7PR12MB5744.namprd12.prod.outlook.com (2603:10b6:8:73::18) by SJ0PR12MB6709.namprd12.prod.outlook.com (2603:10b6:a03:44a::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7762.28; Wed, 24 Jul 2024 13:01:21 +0000 Received: from DS7PR12MB5744.namprd12.prod.outlook.com ([fe80::f018:13a9:e165:6b7e]) by DS7PR12MB5744.namprd12.prod.outlook.com ([fe80::f018:13a9:e165:6b7e%4]) with mapi id 15.20.7784.017; Wed, 24 Jul 2024 13:01:21 +0000 From: Zi Yan To: Andrew Morton , linux-mm@kvack.org Cc: David Hildenbrand , "Huang, Ying" , Baolin Wang , Kefeng Wang , Lorenzo Stoakes , linux-kernel@vger.kernel.org, Zi Yan Subject: [PATCH v3 2/3] memory tiering: introduce folio_use_access_time() check Date: Wed, 24 Jul 2024 09:01:14 -0400 Message-ID: <20240724130115.793641-3-ziy@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240724130115.793641-1-ziy@nvidia.com> References: <20240724130115.793641-1-ziy@nvidia.com> X-ClientProxiedBy: CH0PR04CA0007.namprd04.prod.outlook.com (2603:10b6:610:76::12) To DS7PR12MB5744.namprd12.prod.outlook.com (2603:10b6:8:73::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB5744:EE_|SJ0PR12MB6709:EE_ X-MS-Office365-Filtering-Correlation-Id: 82296d42-9d23-40f2-3fe4-08dcabe0b6d5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014; X-Microsoft-Antispam-Message-Info: apb+s7tSbTXjL5nvvoIW4QtKIEZTFkZ57bc7CpbmYIo63CRLAKtkBcffFr0T3yhdMUPrtzbLXuoUx8fRXvX2OR4YKScJNujVU8N1PV4Y9cwENCQ0Br0Bx9YfkZRbzHNH31oqiVFcuq1TwBU3LWZ5YirmJ85/wYsXhP/8pctF/Z2V/zs1MEqRW5E38SxmC4QJtkgMLLwaSvtWzcV+6cXeNUkdThS/zPJ8mvsAO6qHudGdhjDJbD8zKDfAwJyNE5N5swQTPZHkjZL9BidbJJ7JPa7c9GvIMApMAH2bmH14fdh2Qry+KP1s+5/+i4mRXG24KBi4w/j6QNY43zR3jGYVqArxxT0H2iOQyaxPztgA2+yd9ooqHJuHlnMtnvqHxFPN5AXJiX7T5bs9Njvqvqm5vm48MtN/rwFstCmVR+hpqNOIDi5vuZrmXTHyIdAkb1vaB9E0uCCwMOujCsNbXCnQwaqKriOXAxt+8mNPemmWwQtm7XeAnnH8xkj/BuOZO235OPRr5doGBucoT5agrNIfcV4M92CumSmEtJaMD5DPfVygzRjlNNBqMnzeOKYUDEtQVyh+hRL2g+HcxEOYeRwUMur6PP3lUH2F4ybd0OFF6p2tbztVzgw1wGnX0mDFM1RCn+1mt5DHGe77byvKmaoILapWKV0HPd19jgde8nDVQxNcJAkJ3BEPIsuGATD+Jza8wl68rTvDziFduTOMYKap7uZHtu7mOUr+NUmwIcA+6fjpLYiplF6W5IoLuln6DbPOLuc9FCLPTruDq81I5PPbYkM8TnhuoET/zxt+mrbpPWCxqffO/ygIAe7aNzsmOiN00JJAYGF/iyRYl6KknEBWO0r7QFIktk5M0o5xCgbt5i8xyLfBIRVEG9j2C70qb76E5oYwe2bmsqZjnwhjDKeITbR5Ma6TxBDa3mYP7/KuPN1LraoCeEznzIGsm+TQmll6Cu9NJKNOJvIY9+oFpMEXM8Q6E3QqND9BiThOwHGlpBAjlrnZJlBTVJ/xMhrY0/kix2RXDowBnZhF4+1Be4idDNnN7VqpNCiMr67Yph9LNhGqO7zAuktol5pKRgQROYB72VBzTWvL7lGLpLAy5CQLpqSpZ20uEJNygEygR//qR1A08z0J6vAKxHI/gfCygtNlqSGwjp6FGWzucwR2JwObxeqMTA6NWfso3jZOkxLkgO3Qi4ZsGyvubY8RgP28YBfATRahY1pmDnwOpcEmEnMlaGpAlp1nXPML/Z51jH6p823TQm/JY72u62jbTyBK5Aedf0oG7lGnA4cOsBwP8OKvYuX6dYo/AmvuBOFLkO7N8goAEslJFYDU0Hc8s8K0YpHgHNzLM4VqMmELpe0UKyRMAw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB5744.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: M8E3r/N+6BRIya+hBsfajwiI0IXubOWiFRwJjr59foP3qFoYdEWtyIP2RSFOdE9eZH2CClhiVquNue0zG8oRtY/SSAxlzzgNGzjjK9KBy+LN2GqDNqAHC8hM+YYabIv+t04a3rCVomkNmqTMGKECmoCr2NVwYciukKrgz7hM251zZkiiUBEjaEGduxwgrz0wTVBMvWJul9zwBYVDyqYmphhDywye8odnR7tLluLyH4VAtabnora9r/U0/1eJp/jyIUzI5wwPJBbCO0f4eOKK38lfwl7T65meHdSjq1do19L/WpfpQyEIm2hmj+1Yz/2DD2hoZMy5Yzs5jCWFmtUxORmlbpKHI3V6u7LNWfEqyqH+ISluanX+njeFtwT0BlKkwAL7z4TBhnOE+k2kpVReVtGQk5Afw27MAmg3nLsHeQBcOT0G2PoleJ15sJNi/zUs51tohPQS8rmxb+dX/Ot7HNH+3TGKCq+l9Y3wMMu1exs/OdJzOLCj2KJMzRBaF32V6q42VfzLCWBecTldUHN6H4xWUKNT6FV1C4UpkDDU7/vixkABzwG3wEIdMEtnY9IFsjRSK0mqrqgytqFU89zok/BAv6f5YmxL5faLsxDciSYWY4FD3bQiA3AA6QzBBzpjY4BCZ/bK2t1R1l8+Xqfqo2PXfVYHYIaYom9Luve9CHREsgBL+C/kc6pVsWnFHCTz0jgmPJowg8vBClo2p7X56gwxwWmdIePxufXwZvFzBtNKreTkevqhhAk7gY6ukdTxghjCww8fBNfRLLXSp5QV34Mbv5ZH0BVrPYl8eWCjIrEq9kWE59TdYHU9MpEYAGsS7Lw7bLcDsx3YinXO2UuqQp5oyoQpvLdRyua84JVtnLETLiSmuY2ehdR0vmFx0AO3duUhamrhyOzd6hyaVzKhSBAUNLaOW9/PH6nB6g19c1eWTL/y9CVoPlHF8fffrZkTfveVL29HNIZr5Afa1SSsfSr4vhkGQ042iuDKuD3FNLhFqnj3VTNOLHHHT1g8Jx9ANa+zgQk8W+2Vpdj1HbTas7l2XmXV6cO9qf9HOnAnmZe4RfeTwZ2aXNTeHi46rh2lZiO7jB023QXVZmHzRLum4RvKv5mzvzj/PGHJhCe+VE7KPovFlxX/UrWBZbxmrjLHddiZViB9nJtzQarlbwXx5Q7NJJuJuWkN2qUHSMGKqCwoSY0yLgvd3GAoK5gZiR6DN7LgtM/TMIGv6N0IAj1QTCTyek9hlNEHKlyzTAaERhtJk13KKt+1PgRhO27wRKYv3LZfr7KX9/PKvAsoo3ad8enLFH0iI4XIkSXbL4edbtZwx28B9oGg684AR9fVpUbeM9wG+a2dfpEapquv0dk3NV+ya1L4Ap+E676eLuyswF6wg1rnpULw6nBx/326t48oLUrZX2EkDPni/nXX0+uOEJfgEo7JoDCMytlZmOvqaZ1xOYp50MqrzHnD6KrGyjBRo+ijm6/pne4pKeWt5/z1eghZC/xPZrLTu4ikXq7pP8DGADmpnV43A2/6Ev+b0d/iVwMrE5GdVcuM86VKkG4Hy1/+M8flo/Olq/QH19jCcqQ= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 82296d42-9d23-40f2-3fe4-08dcabe0b6d5 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB5744.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jul 2024 13:01:20.9717 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: TBpMoBRiLx/QDGcrqF6nwlt+WjghURwjXy7NFjFHORPQSiX2HCw5Oo5OYT8gv441 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB6709 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: F093D1C0029 X-Stat-Signature: bgabma7csamrwta3eykdhgpiu6z14h6j X-Rspam-User: X-HE-Tag: 1721826102-285914 X-HE-Meta: U2FsdGVkX19EA35An0bKiiaOfXwbZD0hPAQ80GCojOsfUGmn8QgcS0rcRa+U4O7u/msqBpXjzw9G0IjDf31G5HM9i//wibF8a7YIF5nru70H7+Y3ejlOW1ijIqEGNYJl/boKFGuN6LMec4RzsSfqbvBJW/l45Y47fZ3YjrSeC46IWI4FoPbPZCc9ZSgrjCqvPeQYuEtppzql7IP5AEM4TtEdUX1gcsvqRPU3/70pfW9fb6hs6gJRgV370gwbE4bedNOHHEDQwe6+nsaEBQlQ5Z5GIyqIHiz5sJ1q/8gGJeBvLAMdTx1zWmeXwIe4uN7RGph4EXRt4sWFSJK3u7QHH41GNCB7CLu97I60eu/Tpu73+Q1YQhlbGPWQ2oSZ+Tx1qHJsiKXQGSnTOiZKI95p47s2ZT658Lm55oWfTPBTPAdFSSLaTuBQwjOoif8jzk15g9ol4OVLnL1FhDPfQw1h5o3zrPK9ern4pjJd0AOGz8uGLZisMmyH6phnkYgazlB1r5OYn1JPovc17iyaEKcq01qGCPFqBa8EDykxNYuNJE8R8Xpx5KGDVGQ2DY/F4qMA2IpIr+TlvYTB8tzfAaskB/s8PQ13l1U9pqN6FZ282ybMgk85m8kVyaf2cB5wEGl8UC/JxhSqp0Jvj94SsN91iBGeB4cXdbXWxaSTSxkBsBwr8/eN9rSGYRdNaRefsbLGfjnuGC9z6n89CdloFj6GOdjBF6flUm336l6CGCsOEHSqQ+4WmBUfbdWMp+yFRS5K9zDkDctclCltchPSQx/XM9oeqy791bouTf4pG9qAbaudEayL78EzzFh4OVDT+qu9CqR1PfKDa4ZS6iODZ8i9rpHOQHxcSKl1c95GENihWftE/9Gtfk9VhRhiZ3sH7+Wokj7vCX4Zg7G2XesUjL8t65vFRyeQOdJlA81a6Qjp/vLlP8piU22cEGbopIigypwXD4Cx9z28JvfxqwWc39V vVPeS8nK 59fim756Hrga2T9eU+knRWF9aB2ncN2UkoAg59AF4kgkgxcqzD2Aok6quo4bNWk1bYoYUWJrxQyvlhugoF6ghWOt7jDJv1pylRQxNHwuexHLNvdGVGZEqFJZZemk0QG1oyiuupzZdWBFVPCeyBNT63x/g84FFEyVp5KIA235yTuLC3k8CNxriFtGgGPGcBeYRNP4+iUjY6/CaMF3HtZcSa6odc1CPyw8IHn9pyKt4Qt+Vkc8jXjQGSnj+rc0zzDPjpyJQKRypiS9fj7I43fANq0ZfaT+jLH3HMry9AxqkuTjkiLOvkMsemRrD/DNRMUH00lxaiqp7PN/OtyNJX4MxoKplQCqn67vcFR+2853YIQuRvSY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If memory tiering mode is on and a folio is not in the top tier memory, folio's cpupid field is repurposed to store page access time. Instead of an open coded check, use a function to encapsulate the check. Signed-off-by: Zi Yan Reviewed-by: "Huang, Ying" Acked-by: David Hildenbrand Reviewed-by: Kefeng Wang --- include/linux/mm.h | 6 ++++++ kernel/sched/fair.c | 3 +-- mm/huge_memory.c | 6 ++---- mm/memory-tiers.c | 19 +++++++++++++++++++ mm/memory.c | 3 +-- mm/mprotect.c | 3 +-- 6 files changed, 30 insertions(+), 10 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 17753a463e01..2c6ccf088c7b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1717,6 +1717,8 @@ static inline void vma_set_access_pid_bit(struct vm_area_struct *vma) __set_bit(pid_bit, &vma->numab_state->pids_active[1]); } } + +bool folio_use_access_time(struct folio *folio); #else /* !CONFIG_NUMA_BALANCING */ static inline int folio_xchg_last_cpupid(struct folio *folio, int cpupid) { @@ -1770,6 +1772,10 @@ static inline bool cpupid_match_pid(struct task_struct *task, int cpupid) static inline void vma_set_access_pid_bit(struct vm_area_struct *vma) { } +static inline bool folio_use_access_time(struct folio *folio) +{ + return false; +} #endif /* CONFIG_NUMA_BALANCING */ #if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 9057584ec06d..416e29b56cc4 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1840,8 +1840,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, * The pages in slow memory node should be migrated according * to hot/cold instead of private/shared. */ - if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && - !node_is_toptier(src_nid)) { + if (folio_use_access_time(folio)) { struct pglist_data *pgdat; unsigned long rate_limit; unsigned int latency, th, def_th; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 15234b2e252e..5c0a6a4e3a6e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1707,8 +1707,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) * For memory tiering mode, cpupid of slow memory page is used * to record page access time. So use default value. */ - if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || - node_is_toptier(nid)) + if (!folio_use_access_time(folio)) last_cpupid = folio_last_cpupid(folio); target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags); if (target_nid == NUMA_NO_NODE) @@ -2061,8 +2060,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, toptier) goto unlock; - if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && - !toptier) + if (folio_use_access_time(folio)) folio_xchg_access_time(folio, jiffies_to_msecs(jiffies)); } diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c index 4775b3a3dabe..2a642ea86cb2 100644 --- a/mm/memory-tiers.c +++ b/mm/memory-tiers.c @@ -6,6 +6,7 @@ #include #include #include +#include #include "internal.h" @@ -50,6 +51,24 @@ static const struct bus_type memory_tier_subsys = { .dev_name = "memory_tier", }; +#ifdef CONFIG_NUMA_BALANCING +/** + * folio_use_access_time - check if a folio reuses cpupid for page access time + * @folio: folio to check + * + * folio's _last_cpupid field is repurposed by memory tiering. In memory + * tiering mode, cpupid of slow memory folio (not toptier memory) is used to + * record page access time. + * + * Return: the folio _last_cpupid is used to record page access time + */ +bool folio_use_access_time(struct folio *folio) +{ + return (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && + !node_is_toptier(folio_nid(folio)); +} +#endif + #ifdef CONFIG_MIGRATION static int top_tier_adistance; /* diff --git a/mm/memory.c b/mm/memory.c index 802d0d8a40f9..833d2cad6eb2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5337,8 +5337,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) * For memory tiering mode, cpupid of slow memory page is used * to record page access time. So use default value. */ - if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && - !node_is_toptier(nid)) + if (folio_use_access_time(folio)) last_cpupid = (-1 & LAST_CPUPID_MASK); else last_cpupid = folio_last_cpupid(folio); diff --git a/mm/mprotect.c b/mm/mprotect.c index 222ab434da54..37cf8d249405 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -161,8 +161,7 @@ static long change_pte_range(struct mmu_gather *tlb, if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) && toptier) continue; - if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && - !toptier) + if (folio_use_access_time(folio)) folio_xchg_access_time(folio, jiffies_to_msecs(jiffies)); }