From patchwork Fri Aug 9 14:59:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13758867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DA39C52D7C for ; Fri, 9 Aug 2024 15:01:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B012A6B0089; Fri, 9 Aug 2024 11:01:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A8AD36B008A; Fri, 9 Aug 2024 11:01:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88F366B009E; Fri, 9 Aug 2024 11:01:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 602356B0089 for ; Fri, 9 Aug 2024 11:01:03 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 70B17C11C0 for ; Fri, 9 Aug 2024 15:01:02 +0000 (UTC) X-FDA: 82433019564.12.14A3B2D Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2073.outbound.protection.outlook.com [40.107.94.73]) by imf17.hostedemail.com (Postfix) with ESMTP id 0E62A4004E for ; Fri, 9 Aug 2024 15:00:58 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="msMk/chR"; spf=pass (imf17.hostedemail.com: domain of ziy@nvidia.com designates 40.107.94.73 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723215607; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Jp1ggtAq2TPL+tWQv2b9sD0zy1Lqc8F9tl7KovE4HMg=; b=xLnE6SOYfnE9otRZmEtFEU8YM9WyN1lQYzgQhYbRRFiBbgGbOY5TdZYqkbb95cNklabWes /r9Hqh1nHEuuxKqFNFz0zyq+BHE9Qd4INxDRnC5gXzsWScN51U03YyTRW9rh+RWcpS026P r8FdWQlVquj0yTcLuFQPcZAF469mFJE= ARC-Authentication-Results: i=2; imf17.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="msMk/chR"; spf=pass (imf17.hostedemail.com: domain of ziy@nvidia.com designates 40.107.94.73 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1723215607; a=rsa-sha256; cv=pass; b=El6MeX4qvnCq+9e9cwMRusLGrhDriuOg1vaKhIfaXtyBIRL2JNrFv6Bhblg0G0klwRbcs2 JQAlYsmgJa5KirhSlo1zcW1jbEspzle9rOuEapul/mbWRtqC5X10WaMV1W2VXlbzpAC835 diltSuwNi4zAkr3FeX9B/qbcep01ocU= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DbKuD2iwF4gFglTo+WrcYagwHK7VSz+mZGVDbqL1N4yhQxmgqyFkc1RhcOIWb4z48EgdbnMhSCKGcGzPu5oLu5KI8Mi/ID1jmbbLAPnNI4guKHnInZslbpxjjQ3/US9BUn748w1D5fSeYIqvIXgdCvJxfuQuB5RPVqmDJSAUw6UrrvCRhRAfXoCssU5GdAcv6IYxlleisu5BTwxFJTCxax6CjqWz9m2p/my7w0gG5KLJAdnByAScDtcUPzbTF7teJW5i+3RkD8ekSxJLomF6IQ7tdFPzhLLnsuGxmR8q2glzCSCTKBUkw2ZdIYHT+rYzc97ZUnp/xdz6G5ABSytihQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Jp1ggtAq2TPL+tWQv2b9sD0zy1Lqc8F9tl7KovE4HMg=; b=k0oJ1BcH0CK6azo2E/wqHYrp5oBnaQJDsjf3xZnJ0vemrNbFMgtN4Rvn2QxF578VT05IVz1wqBdcoVkxr9bIBgF/TJDUr2aLPdFXu8LQtO38tXhPk/kcDuavJ1I1q5LMdquP/ECE7SFWwq7TYusS93nBzLuL6GSKLZjZg1l574Z15MKVjrnVae3aVOaIeSFdkR63gntkBxsYpfT71/6UWEJplOr/yqV2ivT/MHa/4fO0fYOHF+UaR79/5/YFclYqecm18p8LGYu14hAIQfzwzH/lLAhcwjcCiA8liA69cEkz36a8JX2oKjycV0G+VCgfVV1S553nOY3vdmRx4z5+ow== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Jp1ggtAq2TPL+tWQv2b9sD0zy1Lqc8F9tl7KovE4HMg=; b=msMk/chRuSZUhIUp4Hy93Ik51puNiRdC0d3LILReIOE2R4V4l+LYQP0y/W2Qvagig+hW3V04Wwj86uugUIMAM4qZWesxgAVK7v2ZPcjfh2q03UafUypvQJ96EpehWCEkiOkrpqpUl6Y11ybMWYwJ4r2SiTbS9rUkbBanIkp5i9gZwe0PQK4IxnpxmU7AqVA+9AYeYxczL/MvLxu9A5M3tDiUNCwunv60XFf+S1zaTIaA6SlDxsAysU5fLqvUUplzvA2UYJ0zYKLHsdAhVM5+H55ntK/vrh3q4vO2EvD0eVY3qYnV7/bYzfjpR0X0KTghHyE3LKBt9yskxvD0MRiYrg== Received: from CYXPR12MB9320.namprd12.prod.outlook.com (2603:10b6:930:e6::9) by SA1PR12MB6823.namprd12.prod.outlook.com (2603:10b6:806:25e::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7849.14; Fri, 9 Aug 2024 14:59:23 +0000 Received: from CYXPR12MB9320.namprd12.prod.outlook.com ([fe80::9347:9720:e1df:bb5f]) by CYXPR12MB9320.namprd12.prod.outlook.com ([fe80::9347:9720:e1df:bb5f%3]) with mapi id 15.20.7849.014; Fri, 9 Aug 2024 14:59:23 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: Andrew Morton , David Hildenbrand , "Huang, Ying" , Baolin Wang , Kefeng Wang , Yang Shi , Mel Gorman , linux-kernel@vger.kernel.org, Zi Yan Subject: [PATCH v3 3/3] mm/migrate: move common code to numa_migrate_check (was numa_migrate_prep) Date: Fri, 9 Aug 2024 10:59:06 -0400 Message-ID: <20240809145906.1513458-4-ziy@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240809145906.1513458-1-ziy@nvidia.com> References: <20240809145906.1513458-1-ziy@nvidia.com> X-ClientProxiedBy: BLAPR03CA0088.namprd03.prod.outlook.com (2603:10b6:208:329::33) To CYXPR12MB9320.namprd12.prod.outlook.com (2603:10b6:930:e6::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CYXPR12MB9320:EE_|SA1PR12MB6823:EE_ X-MS-Office365-Filtering-Correlation-Id: fcc233f4-6fe5-404b-b093-08dcb883db08 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014; X-Microsoft-Antispam-Message-Info: ooUYHu3jv0TDp6L6f7V/oUy+zyp/KNRRBspA8U9Wyvh0DcWa11G6/Z22yMMOauztTJ0T5uWKXfclU0Ymr1mRwLLJFg/d4MzULAzyv822MoY3OZo9nmE/5tSuZiXp1xpXyIMkcsBzeotfzWG99NLegOarrCwJc8zzmXz87rLI6uJgdnBpVcXiK0iz/5+/i3DgaeOz8FPz9yZXF9SS+8lSp9rRCr3lfVEM3XnAuUs+wG5irV+W1LLuqrvAgPV9/K2Tpwc79K08m0kAFTbTuKZMsV6zyTXoT7tFXDl0u3P18Jpxg5JdKw/qbzcHtgTid8x/+UhKcxBziQmEHngyFCnVmyN1CH3UIO8QoG6gIHNr0ZCLu+9o682Tdf5MBQZNQwYgJvJD6XC7BvIRoC628RWbxcLGEFAIeWGfv08HT7/Bom4VhEt681iY5JagMOblk80R+/4yLQjoI+D1NBvt8nCpAD9mzNMCsXXU4e+PakHwDxp7AO0LdlO90TnfB6LoHgPw+qufOAe3g9N9S5HcafdLiiw8wM0xz/70dx1qHK6OGs/xY2YWJQ9nsMB+ivMnY92ATojeuP7xbelxEYEsRhupFNNPJpmpJc9KFFdKNUFkzJATCiitBQ7ylI4n4lk+pPSj4/OQEW4fdoYBjtAZA0FDxaORzbsYniXleO3Bj4MxEldnDZhiTbT4R/h9qYqoQZviJ5dkhdglDyfCye3W3NYg4IShwKdGzWY6il3BKu4wgGgKE0uk2tbYafCGRUoaRH3Fldto3q3g+VpICKNEYrtvL1EC1TyRbov+pmWlSuDKE5x5BMYi/BMP2GvGn8K58Mk4ftYvH1rO5zEfzwt7hCYNHC0p/JW68q2WX3jX3ntTWxpJ8/yktwuAW3sznAJhYSiixox/N3uUY7Hx9WJnukUri/fdc0Z8uPLB2gV4mHtBzQenH6GS5n5dgMpRkF+28hAkEuGw0pTaZdKGIXypx5ItEeTXYprFkCnFrwQtEJlCa2WCyS18DpEHnhxytlszQrFVodL0ONVsi4HrIW2wK/kKX0MSZOg6G+z0dmwlb0nogUCJmQ+9mz7JXMNffPYO43wFatyZAvCroCo5yagq35gpVjySDPg+He0mz8kd3FjwagKO+eAI5LZDtTyWbLaDnOBSFGqEKvi4OjaGUC4RhgpiCAQLreHM3oAO8aqhLr+Lu/L8HJtTC4xTxYpLcroVVBuy98TioGoyKAMcREmyuMZ8P50fN8Xy6mEuPpzZybV3EthsSB86kLyWLFvc7020XWucboInPl5pltX0OWd8MtU6ssDmz38jf81U5eJ4M3MF2o+R+sVQeDc+XzoSYMR3Wgj3q38MD0xTTaMGK29nhL/fdA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CYXPR12MB9320.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: EnIpbK5dj9/xYPibNM3XPy4mXGaS6lCzJzruuk5AREgyCupp5uLXB8662rzH37IXWGV6xC1VuGcrDOZntHNcv5NftEddccj4qdv7HHSYD0esqL4e8D4PGWJXsi7dZ0pnAs61NKtx09ApDMg86FLRutEDmEZbDGGyt3LoOM6N3c+ZvOWkjB2kNDUpK0TbCrWSd9e1zUrXrNsFrSONHu9u6d4K5EE7Q0CWjrdSOhpJ4xdIF39Kf/5j1uSYGnmIEgi/pu2sTNBO9Ah0egRLGbqkKRz4eAIvA7MXHz9HeIwkkFgt1D08JmkXawj1Ganinj/BRe4FONgtS0Qvo8gXjiypE8NarCI2ArqjMJqwpEC5ogNvDa4OLPYG2EHskOEhP5q9Ca2Xie58bcnvx+RF8f9IjuPKMiRvPYrORDFgaK9V+xRg2hN+fKkhKPbKZa63rDWhdePgmGbSm+VPOwQbWoIMGQp8Xvz3SrcO5A2V+87wEv5PTWxu2k2RxYHWKZ8EUZfclbv1epzqhOlFApfFqoiat6cKPMonFbe3T03saQ1zirqY1k8ia0szB1yiNvlYnxoIGyF5FmlW919byysPhQn6TnAjNVd8Mtw9f17h+EIqbSedao3VwEOzoQEJAda8Q5GWjFdIq1Ju82/0J9veXp1i1oPeDh8SSGHICXxJKqRnkT+flPPrTLBWPVQlEwDtbcDCtjPkzxKzvPKBbg+EmCJxRaJFmD1hPcjNyXbfI7paOHjAXWTNbbrQtInwUCWBKw05OHzgpLGkrvhHoSoeXl6EN5qdA6ZqFVtfuCkPv1Iu1GJdmLXpxNtGfIwyo3+35lTBn3NX5Sw9TYl1NRpDyLjR0Pw6bHcGBQPlSTSJR0FfAS5EFWKKCBwIJA2peEtDVjv3v34HNFJimcXO0qUH1LL2i4OzuwmC3oE9NYpZM3NOGNv7wX5KCpoT0RcR6QJ6VX6C1I00AxybhGHqgHuq7KVKgF2BDt7Qvc2apN8QJ5kdIBXcJ7tjQy6P3wd5Uhz18p7XrefQpXYleoni//I19RDJtgAzgu02a3HeR6KV184mxEvHyOKVr5sE7u0qXuy0glHfgHUTsFFfDTh2b1rKjSLss4fPPQlbIFosjGIwgD5JOBccd4UqbdBpQKrs1o6e5sKs7rOLh0d2MpyxgvNePEE9LbgMPAGOOOwL10VOMvkctEU10KuwVutQ29ZsZgsG/IPL4xzF47irdn5ws9nLlLk5x74RZ180FqfqUN/xxWdjncPngpuMG8bMBQ2COFTWhAsS/rK2/Q9keJYtRGZj5vBArkrv73eqRI8QFvXj0P3ApCCN9+jIaEjHyIbar4g6dxQbfewVMJXShX18kWKBB3fVTPorb27cLAz8CKQ11/Vc+0DrSiiT4KnteFTeW/hlo7rLar48AxA2VRk9VAIJrBUNDgUoKEk2MAWlLw0HbGLu3xN3An5t9p5e15cbkZ2v8c2+ssQ+GJ0MAOpK8mmpY6srtuJS3JUvdde9SJePX6rDRCZepa0VzWKntEAep8dLSq9eLsbqSDIb//cPsFbNKafB5a7WB2M7XLnZpqkdNTjYUGvOP6znYSEpBeMdsDWrFuoE X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: fcc233f4-6fe5-404b-b093-08dcb883db08 X-MS-Exchange-CrossTenant-AuthSource: CYXPR12MB9320.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Aug 2024 14:59:23.5882 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Df0qtp2IIRRfrJzjzaQPUeVR0ovLobssFrabwTHHkUoiRx1d22byt8FwOVnVpBlw X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6823 X-Stat-Signature: dhtq6s36oczdobyz6nkjbkjgg6mb3io3 X-Rspam-User: X-Rspamd-Queue-Id: 0E62A4004E X-Rspamd-Server: rspam02 X-HE-Tag: 1723215658-433123 X-HE-Meta: U2FsdGVkX19O4UemuhDEjHOb1qFT7NcW++QbkmQm2o0wFP3XuRlVFh2SnsSzdAHyza6VGD0Z8MoE7G53Nb+5WNwY43j2ggT13ELXFhWZPk+pK8k6wQ/R62323hH6gHGtGQhtG2r0ryJ2XzYojR736oWcuLHAeV1H5JmQUJk9YWKeTr0sIL0WDgcXO2KyH2/6CNjTND6xJ8Ghp7Goc23wB5+/KNoksXxSrksm0dxkouARB0fIOiFXEQhYySf4zP9uWC4P0WBW0huPtPJxqFAOQUyJSWHKVJ0O/tvb/ompM9bD/HqXV3b47x6w6afX0VKOtceWzTOCjrB2Gnh84PDF1Sz1srZyhrUAsA6Nt3qkclYlo8WJZq4A2b0gBgJRvS6rovy3ZR1AlygZUZtQT+Sn16VflKAZPPyjy6zCDKoKJKdI4hlTJdPT1MPl8XEYNN83rUkOGNq0v3Vch0kgWei651X3oRsnMNnljuwsEHf62aIP69mhF31YyjENpWreD8dq5S/t+DIdqgunfyv8e3A/1oJ+/5Qmeq60luPBvA8CtdLiI2xyJ8/YndjYHQ+a2GSz8Lo+7wtxT9oR6iAJpoVSs526BGr31OB0Xhfo0w3DvOgo6rOMeBK/f7CMszLifCrTe2zPA6+MB295JJxFOZVZfsIvkuo3l2n+n7W7FbcybI6PHZvuCZAhYjoGW/tbh3d4JPohsrNsP+s6Vpfg3oSLwqovSAmKEVJm2ltFnDQpl7tmzAB6QG8icYA2pS2QpFMOGVILdFrHw63JwBw263yvkC7YDhJCoDWcZicAFFzECCCnkxLOqsmkjWbK+aeg9K2fP7CfbAykTjBuo/2JTgn9IpxCymjTsOQba4vmtk0HrVU1FK/L4lv99NbVlfnyvXnqq/LLxfa8vqz3ewAPk2HKHhYjYoXn+/LTAgJko83pUVIl/c6Jmfb3hA1XEEEQjJB3mfGc+UfBDpvm10xnbJF CS6JdzLM rDvg1YXkGBS1ddaUnUHs/AhhF5uUbCDqf4UhTb6yLv6mtHhWHz75D4idjNqbLw9syy7ILxGw6/ZfI6EbGOBbvzZsEao9fvu1AfFmu/X+HKTBDEMU9txi4q5yZ4d6Lm0cd5xdbLOg7C7QN9YOqPOwX1mo4qnpEaYvI0hx6wkDtCfB77PWiyWhGiYEbtG4J3DNTSVt9sZjof1ZhyjclNKLBWHckMfvaob055QmZlGDVTPE2Y5aW1DHMYi1vgxD14atJuMs1pUHjh0GxSJCJ4WYf2KBC1vLke7dR3RS3IpHBCs/ITV9r5qEOn6x19HCwWXIDtZJeR1CLSXpvnEYKz/H9v5HPjZvz0xMHwl08YAc5LGVpjpmXYlA8oWUkf95fZOENI4Ma X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: do_numa_page() and do_huge_pmd_numa_page() share a lot of common code. To reduce redundancy, move common code to numa_migrate_prep() and rename the function to numa_migrate_check() to reflect its functionality. Now do_huge_pmd_numa_page() also checks shared folios to set TNF_SHARED flag. Suggested-by: David Hildenbrand Signed-off-by: Zi Yan Reviewed-by: "Huang, Ying" Reviewed-by: Baolin Wang Acked-by: David Hildenbrand --- mm/huge_memory.c | 29 +++++++++------------- mm/internal.h | 5 ++-- mm/memory.c | 63 +++++++++++++++++++++++++----------------------- 3 files changed, 47 insertions(+), 50 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 666fa675e5b6..f2fd3aabb67b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1669,22 +1669,23 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma, vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; - pmd_t oldpmd = vmf->orig_pmd; - pmd_t pmd; struct folio *folio; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; int nid = NUMA_NO_NODE; - int target_nid, last_cpupid = (-1 & LAST_CPUPID_MASK); + int target_nid, last_cpupid; + pmd_t pmd, old_pmd; bool writable = false; int flags = 0; vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); - if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { + old_pmd = pmdp_get(vmf->pmd); + + if (unlikely(!pmd_same(old_pmd, vmf->orig_pmd))) { spin_unlock(vmf->ptl); return 0; } - pmd = pmd_modify(oldpmd, vma->vm_page_prot); + pmd = pmd_modify(old_pmd, vma->vm_page_prot); /* * Detect now whether the PMD could be writable; this information @@ -1699,18 +1700,10 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) if (!folio) goto out_map; - /* See similar comment in do_numa_page for explanation */ - if (!writable) - flags |= TNF_NO_GROUP; - nid = folio_nid(folio); - /* - * For memory tiering mode, cpupid of slow memory page is used - * to record page access time. So use default value. - */ - if (!folio_use_access_time(folio)) - last_cpupid = folio_last_cpupid(folio); - target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags); + + target_nid = numa_migrate_check(folio, vmf, haddr, &flags, writable, + &last_cpupid); if (target_nid == NUMA_NO_NODE) goto out_map; if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) { @@ -1730,13 +1723,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) flags |= TNF_MIGRATE_FAIL; vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); - if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { + if (unlikely(!pmd_same(pmdp_get(vmf->pmd), vmf->orig_pmd))) { spin_unlock(vmf->ptl); return 0; } out_map: /* Restore the PMD */ - pmd = pmd_modify(oldpmd, vma->vm_page_prot); + pmd = pmd_modify(pmdp_get(vmf->pmd), vma->vm_page_prot); pmd = pmd_mkyoung(pmd); if (writable) pmd = pmd_mkwrite(pmd, vma); diff --git a/mm/internal.h b/mm/internal.h index 52f7fc4e8ac3..fb16e18c9761 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1191,8 +1191,9 @@ void vunmap_range_noflush(unsigned long start, unsigned long end); void __vunmap_range_noflush(unsigned long start, unsigned long end); -int numa_migrate_prep(struct folio *folio, struct vm_fault *vmf, - unsigned long addr, int page_nid, int *flags); +int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, + unsigned long addr, int *flags, bool writable, + int *last_cpupid); void free_zone_device_folio(struct folio *folio); int migrate_device_coherent_page(struct page *page); diff --git a/mm/memory.c b/mm/memory.c index bf791da57cab..e4f27c0696cb 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5368,16 +5368,43 @@ static vm_fault_t do_fault(struct vm_fault *vmf) return ret; } -int numa_migrate_prep(struct folio *folio, struct vm_fault *vmf, - unsigned long addr, int page_nid, int *flags) +int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, + unsigned long addr, int *flags, + bool writable, int *last_cpupid) { struct vm_area_struct *vma = vmf->vma; + /* + * Avoid grouping on RO pages in general. RO pages shouldn't hurt as + * much anyway since they can be in shared cache state. This misses + * the case where a mapping is writable but the process never writes + * to it but pte_write gets cleared during protection updates and + * pte_dirty has unpredictable behaviour between PTE scan updates, + * background writeback, dirty balancing and application behaviour. + */ + if (!writable) + *flags |= TNF_NO_GROUP; + + /* + * Flag if the folio is shared between multiple address spaces. This + * is later used when determining whether to group tasks together + */ + if (folio_likely_mapped_shared(folio) && (vma->vm_flags & VM_SHARED)) + *flags |= TNF_SHARED; + /* + * For memory tiering mode, cpupid of slow memory page is used + * to record page access time. So use default value. + */ + if (folio_use_access_time(folio)) + *last_cpupid = (-1 & LAST_CPUPID_MASK); + else + *last_cpupid = folio_last_cpupid(folio); + /* Record the current PID acceesing VMA */ vma_set_access_pid_bit(vma); count_vm_numa_event(NUMA_HINT_FAULTS); - if (page_nid == numa_node_id()) { + if (folio_nid(folio) == numa_node_id()) { count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL); *flags |= TNF_FAULT_LOCAL; } @@ -5479,35 +5506,11 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) if (!folio || folio_is_zone_device(folio)) goto out_map; - /* - * Avoid grouping on RO pages in general. RO pages shouldn't hurt as - * much anyway since they can be in shared cache state. This misses - * the case where a mapping is writable but the process never writes - * to it but pte_write gets cleared during protection updates and - * pte_dirty has unpredictable behaviour between PTE scan updates, - * background writeback, dirty balancing and application behaviour. - */ - if (!writable) - flags |= TNF_NO_GROUP; - - /* - * Flag if the folio is shared between multiple address spaces. This - * is later used when determining whether to group tasks together - */ - if (folio_likely_mapped_shared(folio) && (vma->vm_flags & VM_SHARED)) - flags |= TNF_SHARED; - nid = folio_nid(folio); nr_pages = folio_nr_pages(folio); - /* - * For memory tiering mode, cpupid of slow memory page is used - * to record page access time. So use default value. - */ - if (folio_use_access_time(folio)) - last_cpupid = (-1 & LAST_CPUPID_MASK); - else - last_cpupid = folio_last_cpupid(folio); - target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags); + + target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags, + writable, &last_cpupid); if (target_nid == NUMA_NO_NODE) goto out_map; if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {