From patchwork Fri Jul 19 14:43:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13737364 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 000EDC3DA5D for ; Fri, 19 Jul 2024 14:43:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84A616B0088; Fri, 19 Jul 2024 10:43:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FA436B0089; Fri, 19 Jul 2024 10:43:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 673406B008C; Fri, 19 Jul 2024 10:43:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 48FB36B0088 for ; Fri, 19 Jul 2024 10:43:18 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id BCF2980725 for ; Fri, 19 Jul 2024 14:43:17 +0000 (UTC) X-FDA: 82356770034.07.5B590AC Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2064.outbound.protection.outlook.com [40.107.96.64]) by imf22.hostedemail.com (Postfix) with ESMTP id CFC9BC0021 for ; Fri, 19 Jul 2024 14:43:14 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=rweOgWAC; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf22.hostedemail.com: domain of ziy@nvidia.com designates 40.107.96.64 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721400152; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I7ArFvuePSWI2SLkIisr3FxXyJh02NNwl14vvvrY/bw=; b=yWme5Hi1MYwfwooOKN0US76euxdYxIw0wcjo7PkIOauwuX7oe94d//pDyEhSfib/MaMyGC XKFiOrehZYTqBRTc9ZnB4Rb1sZon8RSMcJu5i9eZSzlgY2WdvBI8MbVi+iVt9JTgXK1QVO lod78+EnBMI6BQctHU+adtRAHxPcF+4= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1721400152; a=rsa-sha256; cv=pass; b=xRC4/K4GE9rUgEM1l5pVcJMH/RW/xUUEyoWDN6H/MrAFjFnzvpH3gTN2J6c6wx3k99J2GG 5GChpKq35ayywQkeV0PiF/tEy9+ovj0EdCPvWc2G0uC+hGeApwkWq2MgjHjaVVtMTBfB79 gjXqrXy1ITFhUb4QSrziHQe9FxhPUSM= ARC-Authentication-Results: i=2; imf22.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=rweOgWAC; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf22.hostedemail.com: domain of ziy@nvidia.com designates 40.107.96.64 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tqxLZQSjT9EX4VdpruIkiICEMs0t4bO7abTNiAllwnh3LfxdbLIUG47udIjCbSSbNXzejybnRHvk8yxJnfdxQZR60tsLeWmo/0sFFgNU2zloWDPUVRTBUUiJxaHG9VtXEIbLCL+e7LqA9i6zdnZH+qJNcFZrfK4Is/+VLI7T0ZsSPgrfTk71Q81P3qcCb2ROJViYgoGac3zc33C4z3ge3YaIH5Xa7vxGrdDguNI4eMW48loeIqdlXldISew3zpoiXiQKiKpmpX6d1phH7r7+hmBBAVRdlGhJzcDINMAKmTVeZ1zpsy61dbflVJwEMeUDhCZY3BQSjXXgrlFARbc1gA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=I7ArFvuePSWI2SLkIisr3FxXyJh02NNwl14vvvrY/bw=; b=GULG3zNCwYrH6VBbH44TZKq5v5LgE44S3cYDZ2DHBYFJmKmCYwRge01xK7b79tn9UPTsgSslU+ZBbF/0N3HsKLhPwO81htiRpKDqbn7aLGOW2b0vI/VqqA33vCkNxWppP+7DlL0vHkSsOd9hfONdTr7+c7dXsw36/7vpKQFhm20P+AyybsypmauecZerjgQWtIBlGPyXhfm0fUKOdnoUndB6/wy4d9DM2hQIjYXEiRiQgoNyWmfs5n56Bsvyx1aaxAp0FXIKyJ4F0a3Xe/lEan1bsx/XoARcV3cRGfFfw268Rbi5bwQgQtkniyhA373SmdGELr7YOrb/yjWDoF0EpQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=I7ArFvuePSWI2SLkIisr3FxXyJh02NNwl14vvvrY/bw=; b=rweOgWACw7miKnvht3vAmCN393VSYPNFD2ViA0qRn16sCbnEM//96/gBYjxG9ueFZf+K17AU2DahRTg5BetcJA18xTTPapZF5eytKLQeBarn9IyDNaA/0K4G/TDXQED60AwLEbtXxaIJ8eWshAF27PL/wo3Mq6PEQcVbKODwYZqI1nrxzYBW8J5ibg0GSdNvw3pltLV5mcRtNPilOmrOOIIYLJQOeAdTqsz8FDYnVr26BxrTnQgr8bY+SESZIUS3ynkLHXAVBeOlMg61KTq+vGVLN8yIVf6bt/WNea66Bd1scBzZOSDEklt7nXoMkwDLcqwyrjO/dzHn/apLEzOZSQ== Received: from DS7PR12MB5744.namprd12.prod.outlook.com (2603:10b6:8:73::18) by MN2PR12MB4079.namprd12.prod.outlook.com (2603:10b6:208:1d5::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7784.16; Fri, 19 Jul 2024 14:43:10 +0000 Received: from DS7PR12MB5744.namprd12.prod.outlook.com ([fe80::f018:13a9:e165:6b7e]) by DS7PR12MB5744.namprd12.prod.outlook.com ([fe80::f018:13a9:e165:6b7e%4]) with mapi id 15.20.7784.013; Fri, 19 Jul 2024 14:43:10 +0000 From: Zi Yan To: Andrew Morton , linux-mm@kvack.org Cc: David Hildenbrand , "Huang, Ying" , Baolin Wang , linux-kernel@vger.kernel.org, Zi Yan Subject: [PATCH 2/2] memory tiering: introduce folio_has_cpupid() check Date: Fri, 19 Jul 2024 10:43:06 -0400 Message-ID: <20240719144306.258018-2-ziy@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240719144306.258018-1-ziy@nvidia.com> References: <20240719144306.258018-1-ziy@nvidia.com> X-ClientProxiedBy: BL1PR13CA0001.namprd13.prod.outlook.com (2603:10b6:208:256::6) To DS7PR12MB5744.namprd12.prod.outlook.com (2603:10b6:8:73::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB5744:EE_|MN2PR12MB4079:EE_ X-MS-Office365-Filtering-Correlation-Id: d87e9ce8-72dc-4a2b-3113-08dca8011c6d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: ryJqqZP6Lab9I/2cgGFFWYR4yq10N79LEcRCmfi08pTgYMCVRPNsBdRP6bfveESEX28v9RKHgMb0sSFbIsviHqhITLDNNiYp01AcbpvXK7E7Hw3nYz1ALprrEnnKfDKGhrYcVFRLSJXvx7dLIhk0D6FTwwRuLzWuPtk1xa0miBpRVSE99eK8GFgeiR3Y7C5hiUEoMQYCMsEIQb5Tn8DZn6LjgtFPpgC47ycc0YwhyT/H/HlFN5Cl9WWPqcIaFAOy29oi0T9XioFtCvBOZ4a0y5dGyUkiUgbq3+ud22qqJlWMrw0QcpowF/NoeBev74F4p86F94nvEpyGLIDDY9KQzg1rfWJaBycca1fe/VdddLTVMxyJW3B5CjUtQxO67+cGjlMPRBVMSP7VfI1xMHG/dqKXSlbXBWi3oJzV8MLYHmvzhJhiadfes1PP4oEiEBIjn6pfx9k6QLlhewy5jT56cmVQxRARhT/PtL0ILweobAJtSsmBGuT1T3sxZyIIDTafoYEvbc2UVoaCS97jYN1m5IhQ6NUwhlifggBVNmyAauUqWsxDXbsT/gY/PDUtmoFgToIz3BAcAeAkC54j6pEBqYy8iLw8VLbFxrmt3lN/CpkTymzjiT0XDEoLHBndI3VcGeQJE+yKlERcS7jO+Xwel0V7TQmJJn7DXLx7OVrmoZXMAOi3zlD0CizosXH4BJsfMUb8bt9yzefGfkZSAcpBw05enoMROEIbZel53JfzzvenMecEryICxiTS7L+rxZZkcauRPtZ77JHnPgbvWXi0/c+ya4/cxO+juym/MPeq/Ww6mpGQ+kMjTUXs2lrLxNKiBvuKyiyKV+qwu6++iikjF7JGOYDSI/WElNDpH0uEzPkzhvsmA/MWQsKn4OWkLrUfDn5bB/DwxSQVTB28DkodS1POO9qM49oCtqgahEJzMqVB2GXjUZ4L2a449VGhMkTTY80LH0GvJX37UUY0QVS/Fzx4K+wF61CDQqVT2uZ9f/OWaVR1fQVY+dZCQzqhf3ZuzvxK/FQrhRjcEO2Bu/1Arge5nXwYVVItFaZehtlCNu/GagswVEViCXfT2zZATMpIQXAV034OinyyDnwJsYJhNcswxznIki1WHw4xYcGQ8vhBfuTFdWm+4eQWc3SKIyn/YgReaNgHAMhgCYfYVN747OFrOsQ3YINkEs0TusytwUEm2PlhGPdDW/VZqYUpgtV74hPg1Gbp+wj224hSJlFQs7rrGdVgIKdn35Ebs5T6KsW8jQwm4wXJ/hdAPNIRmJQqkGRcQ8WRrLSYCQSvFLvJb1fuzTNPWxCXcbBgTOpyUtyo1XedITsCHdLCO/R8OxF4F6h6yyu51cO0hbQ/NcN29A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB5744.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: JBVHmu/dkNfBDuqmJY7/Xv6PuOrl0wI6wcrJGKMqf1CQrNvuVSz1kDoHukKth+c092eqyGM4hpR/POKc5jaF3ACnEHkPVk7UswGDAoeEUyxdAkgp3vUSv6qsUX0mujFXsPk2xKMizrnwqMkCOoX5sYnpxVNZ1Ko1YvXTEFnoatM5pF9efDubafF0NbC6XK5TtKkaViGZirm1P3UuOEq+//ZPILCsDUBZLNBEo1u35KaET0BtYYAtYmTXPWD/lPXjG7TDLTcHAvQKDWz65S8/CK+/y/cUkKhfxPByr+CLai6CcL9P0CPoGl7kbVG3sxqkuwqYkIO0E8htA5d49xLR1rTczQyVeGTlrFNYY9FhpyiiBTkvka1/V3TeDZ7RPKmU8WdpGnjK5LcAfI4znTKopSK/wwPLljkEnLDP1ntcSwkdoywW3RuAIM8yYKVfFGCNZhuXmSqKnC8xukIuWh0050rRCixvG3Yqo+yka2a+k6ip4tunK4BihhYK98UASrcRm+D0SPzpPZ3clnZm8akZq6ImdmP06MPfS3MDJOQndF/VTk5sN4NEnVifXoowlnTf4RwUdNHhg1VQa8jK7pUVEBk1PBMheuZbuuELmqXCM16NfiFY5QNquQVwofn8qPO92PJVE1ssQk4lbKSqRfIT2/uVNMFcyBIo5PhJ4w1k/PwZYHoIG5spo/Rzd8XxXymjyRbDrBirpTdhGTvHZnCEb+y5CpyQYDUXwDk1m/S0eZ/ju64ECEsN4Km9X2yU6jw3/vGSPWeo26Anz+t82gQzZZfcNu4NIrS2/M4phv1QhTlzV5Y+xOTbY6MI7BUj/Rjm0ivTsHeeO8HbBU9PaG3bncJjZDi8C6kmJGrxhPRSJGAO0PM9zgt4ilk03cIIvilQdutlZ3s8xnrvmZFP6NC/dNr6jl3bxDwsGbuUIHQlZZrOlSkOUNymLgtQzT/51dj4owROgsO7cB8oX0F6PMK5aDGgPRsQ7d331UsuEligBFBbcMsBbaYmU/ebUDCHC6Vq6iKazsTnceaewEM11OOaSQdQCm/fdAU7+awzBoukWPuYwft5v9PtIzFSo4TcaJElX6X6nIuz5+L+1S7eQRk5Km0Wf3tFrPd9/6to1/CLcv6IRdb024Rxr7PI/rHVj+95I0cmxERY5RIrYArrzCulblaGiefx6C2idyzfeShoao+Px0EYFB9mdI21dgyn0c1LK7X2Blpztt1YhWH1bWszhO305mDbuZheuYd1Rsiju2rGBD5dl9N8rp2fWt9OKZ7whj/Awy5rccuvIoOS5P9OlgbnUOIe46y6tJfs4aVdNt43taRuwcNt/QvD3LPfvAc2HWaTzkT7h/IYM38XTvlAJjbAsiz2zHeDVXsGHL1JgS9/GUwx2GgrAVyIuyq/UojzsBE8S8YREPQqV9RH2Vj+/IxuHji90YJ2dsDFYJwv7gBskL/Xmft0LrnAY6/UGH6W7KFKQjuJ4pU+A1L6mzb+2ehblyIZra7DcKkXPS/Is4Z30EXjW5pWnf53CT9lQnIC7+hQYrpn42q4HVHwOSQZx4KfGVqUtnFvfg0sbkG7aL4= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: d87e9ce8-72dc-4a2b-3113-08dca8011c6d X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB5744.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2024 14:43:10.6293 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rfCvhpCFUX4GXx0czpa9OJZuNZAI3M+AOXSjRw9tP7vz9PBMxpwzmNDk9osuL7nC X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4079 X-Stat-Signature: 48tbaiuny55jx1zu7bguxeuxa61b9djg X-Rspamd-Queue-Id: CFC9BC0021 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1721400194-454454 X-HE-Meta: U2FsdGVkX199k4VxGM+s0qBbM6Z5KJ8yJHIS+DjWkYdS08vGnl0e/ZJaDTZNoIGIuiB7xN2mY7gpsIXWJ0OFtjIxntb6WH+Yt9/3DF3xeeoqObTk3MlqneEMPBHdPMRtk9WC9UZDdoQXWvzw9mUCgzoty5ukWYYx66iwvH0xw7QULCbTtAfQ96uM2eUqjFLw9/xvYLywwF4U9gLrKxmH4Hw5810LgVci+9XFwCnDgr4D0phNlGFCz5ju8AEMCr0d94bJ61r9FIbEhoJoVOrvaC2WFSuUVd44GpHu/j99ue0Rbek/RCVppuTvnZkqBrlrJ0jrGncCoBJMwGx9a2ewbzkEk7ptw1o/1EA/x7VEIQV4f6+kp33VGMAZdmPq7kokGlEu9dUiW14oa3SQfiyW8YmDZTka7u6f+FgLvNRFob39qFhIyqg29YQJkVQe6wBTQF5Gvljr263/ekTRSGAd7Fwo5WoKJSGBc+OwlXrilNPLX7eM/B1UjR2/kkAeP8MoGiF94+xNILq3f13HDuJXLx3YEaCVNmgjl972BXrKNCDfmxIPdCdhcOzKCJj6Tw82m5G5OrC74w24D8aL6QGYBrZe6axlQ1I0B1fXWc5ZX3l45Zm+/X5+P+2690VsKBXmTt5g01GB1DWXPYM8a5aefT6OAWth0g0EupRaA18uMklHcuX/qFGteXjQY4kyhzbN5gJZy5LCebNH/byFt6GL3uJiu9cP3nXm4FsRV0kaN5vdvfveeIOuCeyBE9vyauEv+kRe+ZqdLGTQ5ZrmGrkJMU57+/xwa1SwlImcTMH2WILsaiWFdDEAowDy5TIQlIixm+XWdq3H0loRGn36HtK9vtCfHTXKpcHKWKkW/HkDpcwZWxjmIyxrPkOfU7AQRAOk9Zfc6hN09inYp07f7E3mB1qTmHs+FZZ3BnXxP20Q3OwpuF2jfwaahEm2A5Okpr/6pJeNiCKkwf7NYdgZQCN d04RnC+d fMCSY4DfX1TTqMxBrY3jWzYd0W81KjaxI3kI3PGWTQJe6+Qh+nkAe5R2IN1lg1Tr0l6bvkY/6WtmLoaphX0S0f0gyIONh54hKr10BlLlyh4oicDMW9iY0f4v7uexsM5JUhC53iaD/DXN6ewBmCSyI72Y6OSRgLgNLIXGGOUw0EEceLvlHL14ZH9Ad+DsuyzHqj4iqwagPabAE/l7n6IIKO958eyAEuvEBdYXvdy3XvZS7PsqJ3Z6dsAKh9RuECvz4rxe53kjrocj8Sw8Z4ErV6eH6iHghn6E/yALeD89CWA8c1Oc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Instead of open coded check for if memory tiering mode is on and a folio is in the top tier memory, use a function to encapsulate the check. Signed-off-by: Zi Yan Reviewed-by: "Huang, Ying" --- include/linux/memory-tiers.h | 8 ++++++++ kernel/sched/fair.c | 3 +-- mm/huge_memory.c | 6 ++---- mm/memory-tiers.c | 17 +++++++++++++++++ mm/memory.c | 3 +-- mm/mprotect.c | 3 +-- 6 files changed, 30 insertions(+), 10 deletions(-) diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h index 0dc0cf2863e2..10c127d461c4 100644 --- a/include/linux/memory-tiers.h +++ b/include/linux/memory-tiers.h @@ -73,6 +73,10 @@ static inline bool node_is_toptier(int node) } #endif + +bool folio_has_cpupid(struct folio *folio); + + #else #define numa_demotion_enabled false @@ -151,5 +155,9 @@ static inline struct memory_dev_type *mt_find_alloc_memory_type(int adist, static inline void mt_put_memory_types(struct list_head *memory_types) { } +static inline bool folio_has_cpupid(struct folio *folio) +{ + return true; +} #endif /* CONFIG_NUMA */ #endif /* _LINUX_MEMORY_TIERS_H */ diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 8a5b1ae0aa55..03de808cb3cc 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1840,8 +1840,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, * The pages in slow memory node should be migrated according * to hot/cold instead of private/shared. */ - if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && - !node_is_toptier(src_nid)) { + if (!folio_has_cpupid(folio)) { struct pglist_data *pgdat; unsigned long rate_limit; unsigned int latency, th, def_th; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 825317aee88e..d925a93bb9ed 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1712,8 +1712,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) * For memory tiering mode, cpupid of slow memory page is used * to record page access time. So use default value. */ - if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || - node_is_toptier(nid)) + if (folio_has_cpupid(folio)) last_cpupid = folio_last_cpupid(folio); target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags); if (target_nid == NUMA_NO_NODE) @@ -2066,8 +2065,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, toptier) goto unlock; - if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && - !toptier) + if (!folio_has_cpupid(folio)) folio_xchg_access_time(folio, jiffies_to_msecs(jiffies)); } diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c index 4775b3a3dabe..7f0360d4e3a0 100644 --- a/mm/memory-tiers.c +++ b/mm/memory-tiers.c @@ -6,6 +6,7 @@ #include #include #include +#include #include "internal.h" @@ -50,6 +51,22 @@ static const struct bus_type memory_tier_subsys = { .dev_name = "memory_tier", }; +/** + * folio_has_cpupid - check if a folio has cpupid information + * @folio: folio to check + * + * folio's _last_cpupid field is repurposed by memory tiering. In memory + * tiering mode, cpupid of slow memory folio (not toptier memory) is used to + * record page access time. + * + * Return: the folio _last_cpupid is used as cpupid + */ +bool folio_has_cpupid(struct folio *folio) +{ + return !(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || + node_is_toptier(folio_nid(folio)); +} + #ifdef CONFIG_MIGRATION static int top_tier_adistance; /* diff --git a/mm/memory.c b/mm/memory.c index 802d0d8a40f9..105e1a0157dd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5337,8 +5337,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) * For memory tiering mode, cpupid of slow memory page is used * to record page access time. So use default value. */ - if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && - !node_is_toptier(nid)) + if (!folio_has_cpupid(folio)) last_cpupid = (-1 & LAST_CPUPID_MASK); else last_cpupid = folio_last_cpupid(folio); diff --git a/mm/mprotect.c b/mm/mprotect.c index 222ab434da54..787c3c2bf1b6 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -161,8 +161,7 @@ static long change_pte_range(struct mmu_gather *tlb, if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) && toptier) continue; - if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && - !toptier) + if (!folio_has_cpupid(folio)) folio_xchg_access_time(folio, jiffies_to_msecs(jiffies)); }