From patchwork Mon Jul 22 17:29:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13738981 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 313D6C3DA59 for ; Mon, 22 Jul 2024 17:31:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC6586B0089; Mon, 22 Jul 2024 13:30:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A76636B008A; Mon, 22 Jul 2024 13:30:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8EFEA6B008C; Mon, 22 Jul 2024 13:30:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 70FB46B0089 for ; Mon, 22 Jul 2024 13:30:59 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 19166161745 for ; Mon, 22 Jul 2024 17:30:59 +0000 (UTC) X-FDA: 82368079038.14.6795FD1 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2067.outbound.protection.outlook.com [40.107.102.67]) by imf05.hostedemail.com (Postfix) with ESMTP id 5293310001B for ; Mon, 22 Jul 2024 17:30:56 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=ksPGxcbn; dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf05.hostedemail.com: domain of ziy@nvidia.com designates 40.107.102.67 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721669405; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=77dpwOV1mf8CfX1TKVYlaP3iGgjjwq7Xmbx0eFVU8xk=; b=iE1HdHGluaTPDv58SMFJldGh7IisiXWQYeQu0Y3nnb7tpk2G2gRJA2RFH0xYwwiBehk5vK g0ulV1GG2lygo/qyA9b4+d+yEmxB+SkjoSNAXfFASoXfBTNggMhChF+7G5NIdz4EIlXsDZ 6vXJVjfIE5YKaNKnzdMrSRAZANGFG0U= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1721669405; a=rsa-sha256; cv=pass; b=HoyZXFqkgQaMMKSwltqY87P5MogKvCu5gNw3aPbwQo1wtWHJpQjXaINLEnqfra6lRGoc0d M18W9yfBx4i5fTSgM5ma0eOEBKAFJElIExF5RX50xikpwVw62NyfSHWxXwuuM3IPqWinrz ToZ1iWY0EDDjooOAH7wApPlskjvZ3c4= ARC-Authentication-Results: i=2; imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=ksPGxcbn; dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf05.hostedemail.com: domain of ziy@nvidia.com designates 40.107.102.67 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tbGLR68w2qxynFeG+2lTC8gCCmE9L1yWVJstD/dW4/Gq5NDzmC8JyRkWD84MDATQOYaYXEF1NaxplTs1fHtequg+ZG1Y6iBTUJ32O1AfJFC429Odft6kjpo77KOT8TZSpw7Nm0ewC/qap1lxOG3p9kpT/lMQ/tEsm+vVJ5+JrRJY0Y+80YYV1vHJ4VuoAYMaGqL+9JSaZPn19OkrNfmq0zlxKjgYcod7RC99QfG5H/VohhxfEcLfqpLt+YfpNV+6iuVBFxPMuXUYJ3FMgZk8TXXCc/97I4kQYaCFtROfYe4UbWS1Lwq4pvSRZl0H/C0ArjPAc11dlJvl/WF1TGFV4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=77dpwOV1mf8CfX1TKVYlaP3iGgjjwq7Xmbx0eFVU8xk=; b=vCvucg15FAmhNs5+K1KGyGL+rZexPLdj3zIn2/VfO4CKJG3dVosv40NeRJhBjAO2cKcT7vV9IYzbyNBARQZ+fPjldYRof+UloOiSYGkQ5C5kqltDt6I5RzJNcbf2bqCn0MJ3S26mqzlVD1IRFHcWBFZ3XuJ2qzeBwGT0Q27Djo3pmt50xHUyDDXtmviC6mMAGKBckH/33rTm16/sXEg6qZR3onIAEe5vH3h+z+qXN9f/VHHoyOLLgA2YvkbnU4LWslWo97TP3dL/APfDGytjwKKKHbKXn+ItjnPLYl+oCyx+Y9mLLu9m/12nmdJUpTMN3xzo9pjU3dY/yv+AHUusCA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=77dpwOV1mf8CfX1TKVYlaP3iGgjjwq7Xmbx0eFVU8xk=; b=ksPGxcbnz5maQoBUsWSPBvZA0nn+4dJESTSRH6LzEsUgYAFJUOu7JAI/RkMnzlDoer2ouYFYjCCha9PepgAV3r9G8RN91lOjhYndRuj8lRXnojV5yS3doAxfgOWvLWcYOWg2+ztwhC9Lj7w7wGCTw5Tv3H7GzVN0E/8SY0gxKbISyjaa12M2vkjgJJv2bHOVAbrtyB+lw3ViOFR/gSY4nyZYVNoZFyo3m4GarjIG3F/aQ8F0oJPBwsPXahSYD5Cu3iXuVufNPEH/lZR6cVfb82whXwzwDprbUG5uQTsTbDA53vAjgzkJBO1xg4CTqGFyEJlZekQZz3kn3c3P3d6aDg== Received: from DS7PR12MB5744.namprd12.prod.outlook.com (2603:10b6:8:73::18) by SJ2PR12MB7944.namprd12.prod.outlook.com (2603:10b6:a03:4c5::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7784.14; Mon, 22 Jul 2024 17:29:24 +0000 Received: from DS7PR12MB5744.namprd12.prod.outlook.com ([fe80::f018:13a9:e165:6b7e]) by DS7PR12MB5744.namprd12.prod.outlook.com ([fe80::f018:13a9:e165:6b7e%4]) with mapi id 15.20.7784.013; Mon, 22 Jul 2024 17:29:24 +0000 From: Zi Yan To: Andrew Morton , linux-mm@kvack.org Cc: David Hildenbrand , "Huang, Ying" , Baolin Wang , Kefeng Wang , linux-kernel@vger.kernel.org, Zi Yan Subject: [PATCH v2 2/3] memory tiering: introduce folio_has_cpupid() check Date: Mon, 22 Jul 2024 13:29:16 -0400 Message-ID: <20240722172917.503370-2-ziy@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240722172917.503370-1-ziy@nvidia.com> References: <20240722172917.503370-1-ziy@nvidia.com> X-ClientProxiedBy: BLAPR03CA0078.namprd03.prod.outlook.com (2603:10b6:208:329::23) To DS7PR12MB5744.namprd12.prod.outlook.com (2603:10b6:8:73::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB5744:EE_|SJ2PR12MB7944:EE_ X-MS-Office365-Filtering-Correlation-Id: 3ed615e1-d660-41b3-4ea2-08dcaa73d4b6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: thTG0HbTUUzV20Qf18YnacBVhWtbTuk94UKVeyYXQrrz/LmXzXpJJBKqE5RcWUJhYkIIU9YrzVERZDN+pzrLEMso/HsUPCPjnQET2JDwPd4EaUbF72QcEw6N+xOf0PoMvi5sfNOKpYIFEkPwm3QBZU/qCG9LMxWSfvwRLdN2iL/2eKV/VHRHwD87rx/1U5ZhIUJD2Ceh9/UwxGjii7/FOLvIcXg/HyJzonGyOAZvJRldCJRuqSmMJvMT/0LQya9QZZVzM1dU4N4kpRBjp1vU2IzKlyyWSTrR+ROENGjEsrZgnBMYuUyEGwuj0d9UyPsggPRavJ/xOO3RjiDuj/eioQSzVUCcb+q51yb9iYfAS25dN0cRMYX6PpabeuAJIytzxOY5H3LJfAXT+PIN5XUYZR3iL4zFstPhDYrswEf3EVwX3zinty1oqG6f6FK0NRNN8oCMvI8UdA9GfxjQI8worS6d8MOJKfiV6e7ssNalLBmauxRllKNK8vCfq2+kJDcEVFW7LIyq1MO+pWqYanaOI5ivv6gDHN4eYKRJhnOv7FfGaWlhHIzLs7gn0R4l2aKP7RYM3I2X1+G6N8zRRIJSriOrCmpYjiwPKf3TyWTwBwA065HlDhkIpdimXeJgq8OOqwZ32n8RMMLkeCQUZgRMz2n951oH63z7VfNhDKQNliDznr/8noVMbuFYh3ufU0wWNklBU5KLpMvJL1GeZVTI2ysEEegw6OPvVcLRTWMr2B91TQp7nC+Q0PfG7IltORpOgroBYpMBjHeA5GM/R0kE09tVqHbikI3HvTS6MipfQu4k2lVq5X8c9jF/MI6DImYK7WT6SVJjnNtFLaWgBmQYPF0j6JD66IrmsQqSs3Ioo+H5Hvrkjmyovl/xSXbr1gIXHxQQxlrsjgmUEGlr42Nk4wFhkKA/62/JojNpIDDhnX8xXhWDSzHvYaljx5ZwtsQeMondBRBawTnWp8kJa01rlPGOSn58bSa05qvuEcwLYKmh/MefxZxFDqzwNmNtliVlD1wjzqB/m29xa/nBV+cSqPlXckDjm7I+1RtGyvC7e0kKUddsuNm8ZzoTb/EKyDgYli7XgUx+jxHLWI9cmUQqfqXbOzzOqN8ezuUYfNbtfxis09j0byZpGjy6CnRoVx2M+vSUIBD1vqgo2w1JNsICcZU3VwVxbGQCr74NInnszUp/n2cV2gQYAlVhKjxpkADn1tZQLP0VA23mOC+A37RROt478PZdpVRcfOMAjGkZ6sxGreXYWk0dYVv1bRKrTND/ljS5UpgL5pUoMpNLsARZNw8lG9Crv9kC0R56E5OsbkxFPlGxglTHbajHojf+dw3w06qPVKxh1qYgYT1uoNpIEA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB5744.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +37r/LRMNW2pH/Ju/uEK96NOThkceTG0otMZgkgnEtSLStSsW/XvAlOn1/oz6M8/If0xKoJLgPWNCX9zf0zmTj9A6ls2V78JQEhVfvYAesXOCAwOLG/6anl7uvZM2hluSM16yfuD7Y/dC58CRtOfyf9zP5aWuWJPaHWVWSsxHtzY+E1Iad6Gz4YKZfZ/PvzW49MaHEtih53SI8LPkvaGaxYJpNvaXn8RTNWS8hV3V18QC4fE1DRtLN+0DyVaHSzKTziRh835ahJaOJw+RA/wq0D7nTAyNnRJ9tDccbuM7Rdmt7THNbMmFz9jdcYMgF+4It/mUlvPLSCBO6oa4aGn5eD5NWwZAJAvOZzt3x31WJrPQjskQRITskbUn4K+dYT5vkhNlZJZ5CGr8xHehJomlaeZdwWE5CnoJLLZ5oa9X6GbIujNSgqc3NJftE7svDkJbYQPqo6UTIUfL3ZeBzjBqsQIS8gF/mlRMsBpTrXQCBl0Zk4JVrKNK+GiTM0xpuktOuWugMb8rMuQEWXXqrfDfwTb6zukVVE8fFBXr4S+XMpvQL6lv9GVMQ6YaQQ0nIDw1Hn8DdCM2ahOJWv+w2bRwruq+M9JKPDAKf/CfDENu+j8IFWO59XOP4n49HS7wbZ9Uh1wDbc52k3bELGfWitkZ3ShHT2w15pVRTQZI/7S0eokJKrJPeTeyBlMeF6Q5Kgfg55co0FqISsYiXkE7kvaAph5DAb8uEz7VClD6pIcqAxNlb7ujtKvUjRdz/UVYdW2vJim0Eb0ONC4dASMDcDkCyQe7MI3vJBsEzKHmo09pKmPyIj3N0D5vU8K0ZOeOxPRoMgZYISYd1x6gubM9+rSJh+3XZvuSvgjBAn4yZenYTckdwgVP6j74e7zJgj6YLn7t4jTifgUDcM34YrBfpMyNMkqQyo1GIpLiQ3AsasLyK/Uta+2h4voNxusvTIDHQFWRfPhs1xiWxiLkf04dJRoeKEEMyrLadQcg2B3ZzOevo+bOevJ4YyffdsHx4QqFgQPbdvHmLbSLEcyIryPcf5c9RS2xDB2XUFQtLAU69JiO22LLm12nTDK3vIw2AMay8Xvm8hHzv5fA3u8exxKT4Hd8140BjNL/4W5iSfcERsIoT7QadSh2eLEHx90xKtQDAOZ7/EnbRgMJIck6r2O8s4PKvepDjOxUrZlZVk4yWs8cETExx1JAqiUSGgPzjMEiC3z99TOE6QWFpkvDb6k2xkjCNEsQyp9GLMlqnsCmmPgy/KK+JZ2a4bUgzCPnKLeGjYy8wdhVNiIoQ4zlUB9wwHDGpYm51qGjW1RNRJMIsFBbKf8RiuGymSOk99Zo+OhPuUHz5qgKy6xCxmiuyHiQsR7OY25164LcIja2y4mZ8hfYU4yI4aB8QLL94uhO+OPSxc1ydi7QWkl305GL/5IMKOBekneXWjsmGlehs03H18mxduMpXZcDwk4dP/Y/Dq9kgbDVEABpJDvlKpSH7BOAOnznSBphzW6mezjW52FWJW0Nnq+TJ5vVkBiuRsp/jeYWfDxRxDUfhLjnp+qK9X8rEl3gjBQVuD8xL4qT3ssSEAHh1E= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3ed615e1-d660-41b3-4ea2-08dcaa73d4b6 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB5744.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2024 17:29:24.7681 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: lNFAjMLwiZGC8QBP1OSwHuAebtZPTYic+bufBp3c6b3MH+2molyHBb3ibdXisQRZ X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB7944 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5293310001B X-Stat-Signature: qq3ya4pjjzcmqrnafpxhuq94x4wszcgj X-Rspam-User: X-HE-Tag: 1721669456-341580 X-HE-Meta: U2FsdGVkX19mnEbVj0v0wrFwGCHNv+5EHK9TYUriXOiXjGyToQ0NkOg+Q4tLw/HWX4s4AjAA9pKyKqpv0GtaGSWN7oHmC9J3Pna2qSmP5S3Yf0sFS6JZgyB/LjW73USE0fUtbXIkkJNkUeZVlACJZ8DqkSvYM67Wl06b/ZZJ2/dEHEYJtBZltf9mFXbyVHXv+8NVTX15ny3UA1uwIOHKAIQwCsHJy2zrLI+Q96cDYnzhOs8Rlhv65388hpxLL/7E1W9sTl29ZtRVSoIx+Sa+I8oLVGHad74IF31Q3aEnfpy6NrwDJQeHUc/6IXiockQyBrug9+3DfgyvYSHvgdYD9T53H08FkOFXKjxFTyZt0wUsZgzEyGU7H9yCF6Zlx8n5U5YQb8h7OcBB7yLfe7Ll6kxoUZm3qj+Ub8iN3a9cpkXVQWZYutvzBq9KxZ9IKLUPR7RzpPHR07CtZdYF7o7EWePlrPrz9e5v4wen7je9h5XIUN0Lorq+vw/V4VpRqwzeE4ITQYW8OnBgyaDX53Csj+qIxSPNjGJmxyTGgIWLfcJKUFLRmpBT58HQPqCCuK1ntVohkL/0ftTSKgREuV5er/XQpmD1eaNtUk4NnYCG4iOKur74ydyIdBcM9+3GsGsuBh6q5J9sAPYI9TJvTK75THunL4o3QDu8sxPhHzSf8+QFErUjb/7zsQMubcOUDPH979wvQsOSga647SvoCCfSI7apltfxq+Fm9y1ycMHhjTTlQAXbxse+/KRelwIBk55SAEZ8Z3ky168LyMQnUYOcKmtWa8oVxZgymfoHWwcp5sjEjk6btyTUWy/uoGpDxtaiukNQuV0ZR7lT/LH4zUn92eoOhdpYzg0nS7Ae7lfTtbQg8Eq3Oa9N9h9SdlOsYq2TrpOpbIZInkP1+jUw+PKYIgtD+SmpzbytZQ9pT+tP9a2+jXC50X+BRMVxATW64CxYRwFv82owIrlRQGKF3bL 3PRcmu3r ZvQd3/FEa3/TWzWGChkZ2i4CNbjaBQpgjsmK4BujyK+vV01c+M+AY+8aYClWs/4xYid1reyJGM4ezxCmzmh6d3rU/tCY5Imgtf6QtGgEoVxW2ovp/0qdYfNF1tYrJ8nSAkKvbPW/uNqsgH3ZP2ioRaGwdRgI1CfHeoukRVjJJlpPIb+TSuiTyIGs42A0+c2Oqigv1ifxf/6fd3bcp82oYhF13ghLw6SMTlNf/IcdsYozzwRscGwxJHu5S+Z3mIkjnjaTNEDEn561C9C/zI2Jnh3VLwTAacNYKdsACZQej0+2MNfbHLPHjcHpVUHh53KpFAEdVbsYFNTxku6p3mXYxfeLZfQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Instead of open coded check for if memory tiering mode is on and a folio is in the top tier memory, use a function to encapsulate the check. Signed-off-by: Zi Yan Reviewed-by: "Huang, Ying" Acked-by: David Hildenbrand --- include/linux/mm.h | 6 ++++++ kernel/sched/fair.c | 3 +-- mm/huge_memory.c | 6 ++---- mm/memory-tiers.c | 17 +++++++++++++++++ mm/memory.c | 3 +-- mm/mprotect.c | 3 +-- 6 files changed, 28 insertions(+), 10 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index c227f22ba810..048b2a56d8a3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1738,6 +1738,8 @@ static inline void vma_set_access_pid_bit(struct vm_area_struct *vma) __set_bit(pid_bit, &vma->numab_state->pids_active[1]); } } + +bool folio_has_cpupid(struct folio *folio); #else /* !CONFIG_NUMA_BALANCING */ static inline int folio_xchg_last_cpupid(struct folio *folio, int cpupid) { @@ -1791,6 +1793,10 @@ static inline bool cpupid_match_pid(struct task_struct *task, int cpupid) static inline void vma_set_access_pid_bit(struct vm_area_struct *vma) { } +static inline bool folio_has_cpupid(struct folio *folio) +{ + return true; +} #endif /* CONFIG_NUMA_BALANCING */ #if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 8a5b1ae0aa55..03de808cb3cc 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1840,8 +1840,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, * The pages in slow memory node should be migrated according * to hot/cold instead of private/shared. */ - if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && - !node_is_toptier(src_nid)) { + if (!folio_has_cpupid(folio)) { struct pglist_data *pgdat; unsigned long rate_limit; unsigned int latency, th, def_th; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 825317aee88e..d925a93bb9ed 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1712,8 +1712,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) * For memory tiering mode, cpupid of slow memory page is used * to record page access time. So use default value. */ - if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || - node_is_toptier(nid)) + if (folio_has_cpupid(folio)) last_cpupid = folio_last_cpupid(folio); target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags); if (target_nid == NUMA_NO_NODE) @@ -2066,8 +2065,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, toptier) goto unlock; - if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && - !toptier) + if (!folio_has_cpupid(folio)) folio_xchg_access_time(folio, jiffies_to_msecs(jiffies)); } diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c index 4775b3a3dabe..7f0360d4e3a0 100644 --- a/mm/memory-tiers.c +++ b/mm/memory-tiers.c @@ -6,6 +6,7 @@ #include #include #include +#include #include "internal.h" @@ -50,6 +51,22 @@ static const struct bus_type memory_tier_subsys = { .dev_name = "memory_tier", }; +/** + * folio_has_cpupid - check if a folio has cpupid information + * @folio: folio to check + * + * folio's _last_cpupid field is repurposed by memory tiering. In memory + * tiering mode, cpupid of slow memory folio (not toptier memory) is used to + * record page access time. + * + * Return: the folio _last_cpupid is used as cpupid + */ +bool folio_has_cpupid(struct folio *folio) +{ + return !(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || + node_is_toptier(folio_nid(folio)); +} + #ifdef CONFIG_MIGRATION static int top_tier_adistance; /* diff --git a/mm/memory.c b/mm/memory.c index 802d0d8a40f9..105e1a0157dd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5337,8 +5337,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) * For memory tiering mode, cpupid of slow memory page is used * to record page access time. So use default value. */ - if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && - !node_is_toptier(nid)) + if (!folio_has_cpupid(folio)) last_cpupid = (-1 & LAST_CPUPID_MASK); else last_cpupid = folio_last_cpupid(folio); diff --git a/mm/mprotect.c b/mm/mprotect.c index 222ab434da54..787c3c2bf1b6 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -161,8 +161,7 @@ static long change_pte_range(struct mmu_gather *tlb, if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) && toptier) continue; - if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && - !toptier) + if (!folio_has_cpupid(folio)) folio_xchg_access_time(folio, jiffies_to_msecs(jiffies)); }