From patchwork Fri Aug 9 14:59:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13758864 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87DBEC52D7C for ; Fri, 9 Aug 2024 14:59:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F1F666B0089; Fri, 9 Aug 2024 10:59:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EA9E96B008A; Fri, 9 Aug 2024 10:59:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFA7D6B008C; Fri, 9 Aug 2024 10:59:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id AE82A6B0089 for ; Fri, 9 Aug 2024 10:59:28 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5577F810A3 for ; Fri, 9 Aug 2024 14:59:28 +0000 (UTC) X-FDA: 82433015616.14.B76E1B5 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2071.outbound.protection.outlook.com [40.107.93.71]) by imf06.hostedemail.com (Postfix) with ESMTP id 6E91C18002D for ; Fri, 9 Aug 2024 14:59:24 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=ifdlw5nZ; spf=pass (imf06.hostedemail.com: domain of ziy@nvidia.com designates 40.107.93.71 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723215532; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=edoJuHT7gWila84psYDlaeDgJJza53cSwF1dPd/agJ8=; b=ghjtVOyr65rlTJEpRXapYTO8EP2b1rgdFDbLojwm+sQKblWx3WQR8bCwo0LqjgE2nSV5ZE NE8Ut3APobH5nbhvNM9tB/WqFuS5IlXn9ZzeSS2KNNyCmA6K12sYKAnk3y3e9b/2ahym1w fFfc5wQLdlfD50AL7hJAl0Qugytb6Yk= ARC-Authentication-Results: i=2; imf06.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=ifdlw5nZ; spf=pass (imf06.hostedemail.com: domain of ziy@nvidia.com designates 40.107.93.71 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1723215532; a=rsa-sha256; cv=pass; b=GQ4cj++MSS6aJbPXj7KOtLf7MLnmvGpokz84MItVngs9i2maCgi+Y2C+1GbuszPtSPR3k7 SehVtevBPSNwf+Hyi/gsJS9yVa6LK9mLaIVXnV05J+xkLu5uXRQzXcABSB67F/Qft0Kn8I 3hizjnl8I4CdTsH/qC18YHqHlZ5BuYc= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kNqaHLv5gNzECjF1Dp4j++PobWqNnLKnNSRXSHIZkUArNW80x7/oBMX4/c59b6UBuXW9wUNgynLKd7Xuc5sHsF7YW1YHorj9s25RjYIsJDGPb6Faph1JU9isLaVBDtVj/Bd5ihAgWGvsGuPVKQH/IePzVe8mdZFE5Eglekykzuo19K++nSShhCvyFJ4O06h8rrtrAffhHoDf9W6bQ6rvh+ojh2g9XwUOiuSkCkqfrvOMf+vF6mw7gHk4Q04eneB+gRxm2Ois41xaC7C+mpp5AYifV6O9omZW4OA6p3iOtG+TR488AIDb6GGYz4MMGFWcQ8LaKOmHpZY44gRPUoIjdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=edoJuHT7gWila84psYDlaeDgJJza53cSwF1dPd/agJ8=; b=ACzyjykpcGCbcANyyQMmfBo6rAt7Z/xz1VVeC59DPGJ0nasTJQP4t90ihJKD9VbQ8OnWq2UciNMWpq3F5tP9d+RM3bn158j5H6A9bAwyKWjYKwowrGR46SubQhUNRKj6V28tB7sxrTmrhLQh3eJizbqMo2mEDUukoBbKE+HPggchdxOH24WftXKvCyak6ckcaUB0BvNyeekpo23hUG/rnadhBuvK1u8drMQ6sYuhResTbFqRjprqpUx6NY/Y8Mp2TEi9S+yeCD4gqWXst+pnRrnMfj/LaeX3oZaL/ojvnarJEXZk36XxC1Iqul+ZWNnU7tPOhRdRQavu16jveaC2FQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=edoJuHT7gWila84psYDlaeDgJJza53cSwF1dPd/agJ8=; b=ifdlw5nZj/Vp5u8Qoo43/xhTWmg8eRtrt+IvcffviFotbTFHBhpyzdyqnc9E1TqpL+iQCl888itNL7juNb8oFLArUGqP7QHXir0jVmV9ppgDG0lMLSfr7y/tCtxpODJzX/gDqTxmgD12rWhxBYF7StrUzn8Y2LlDuVHpYyJ9Xts9feUcHDETt+yDx1cLDaVL3j3O8TJDyrX4uF1EVG/HJrikJKaQmZx2weLwOgB40zlXq45SSK+W0bzVdc6iiW3vaKFYpqUH70odIXUwEqgz7VQyKAtVfSsmdUWLzpAuUkGdot3fFB92SSCjY/aS2x//7RPHhfN9wakT381eCxjkdA== Received: from CYXPR12MB9320.namprd12.prod.outlook.com (2603:10b6:930:e6::9) by SA1PR12MB6823.namprd12.prod.outlook.com (2603:10b6:806:25e::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7849.14; Fri, 9 Aug 2024 14:59:19 +0000 Received: from CYXPR12MB9320.namprd12.prod.outlook.com ([fe80::9347:9720:e1df:bb5f]) by CYXPR12MB9320.namprd12.prod.outlook.com ([fe80::9347:9720:e1df:bb5f%3]) with mapi id 15.20.7849.014; Fri, 9 Aug 2024 14:59:19 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: Andrew Morton , David Hildenbrand , "Huang, Ying" , Baolin Wang , Kefeng Wang , Yang Shi , Mel Gorman , linux-kernel@vger.kernel.org, Zi Yan , stable@vger.kernel.org Subject: [PATCH v3 1/3] mm/numa: no task_numa_fault() call if PTE is changed Date: Fri, 9 Aug 2024 10:59:04 -0400 Message-ID: <20240809145906.1513458-2-ziy@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240809145906.1513458-1-ziy@nvidia.com> References: <20240809145906.1513458-1-ziy@nvidia.com> X-ClientProxiedBy: BLAPR03CA0084.namprd03.prod.outlook.com (2603:10b6:208:329::29) To CYXPR12MB9320.namprd12.prod.outlook.com (2603:10b6:930:e6::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CYXPR12MB9320:EE_|SA1PR12MB6823:EE_ X-MS-Office365-Filtering-Correlation-Id: c2ddc1f3-7a4a-449c-31a5-08dcb883d8d7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014|7416014; X-Microsoft-Antispam-Message-Info: BcV98Ixm6bFTO0Y8qjsi3CJsQoguHCTzfeDNaKgbTVwaK3X6NSgedOUOgZvhHfbsGxY7z1ktAizjQbL1LaBGW6vYxUjvLvHtvEGeI1j1t+dS4NApCvaTuIW/pBM92nQ/QcynwQ3z5p2iR6d5Z7SIDAa0EqH3rav7yHvbzuIn8GQceTf+1jZaIOn1Ti+dAgK9x3u3U2QPIHbq1BsbsjmhgYseO8Yiu5d42AvX4gks8UhtALpKmiacYgeTVkOInR8ssb7FZUFcg0HYDHO/cSWMqUcGbKXuQQr+Y5rfHI27yfqyrpt/9A8iizGUCUl6F6UEQG5dKGM6ucA/yuwyFNk6Cmbyk0r7qUFFgzncdA8cRj5AeomZKmjz7w7v8BbunuyUTRxIzj1q9N+FFr98PuOSYtByzs/RCRK8rxORX1TxoDT4dpJs8VFpqyyFS/gXb6r7pJOEC4DSSJhw0gjsQfwkiWo6eGsFSqZcZMOcykyQNkwJMhf1hjgjqPb/CJqsE2NRkSbYdBY3sITOBsCMVeYA075yhRMjmYtPlZctsuRYH00cgznAyyW87T/gWV8sBNjuf136ar+UMZ/NO3PRIN9yBeoWIy50Ry+hkZvlfMEOpDBsvv+6dPmqlpQz8kCneX8Qv9BIFc4mKJPCW1mfWNkxU52Fdk621aWFT28rsj5yx0wjsmOv4FBqvt4VkQ8amND7smBRdsVzb0ZeT6vEj25VSmZ2yHt/juq/+Y0nyx/ZgwHmP4+wv5SfHSjDHc4w0Q/HrBxysI9ImSnYzBm/bY6NivAsUh3cs0cpvNiarFeEj75S60fvZX+aOr6E0ISyzRk2+kkKIseL7a7WuKur31G2EfcZaODGkJ6kSGI1c/ma0rxoMD6Q/e30CPVJmtGxX+LoT2tEFLKIOLqPLHcU6m6UQIY+5xJbxMcz7D6lS+yomGCmZkUSAYjPnDz1hE7rMTavRg0+CCcbjdm2QHmqSfuTAT/W1gu+fcHXDUq6aYCpgcmXdUcNtL3l4VfTSkjyZUmF7TiIRzOnycl6Zue12gisIx+7C23BzMMKfWyee5bf/hF+aYQbduyIzuA49A2ItYfG2L+ek9W8JCj+Z58UmbfWOjo0ljmLSdeixVTmkIC2wKnDEtpM50dyTtdUN1VSYAuemSvfLnqMDVlGAyntvbFDPtXmirNxJqQWpn74DTjIpgJnAWHC0oJsYjA9z0P14D+IYsggBUbnx8HPUbjfkogc7OvEY9XYugg4tt+TyTT8saEcUWJTdDKD9+v2IFR3rBXM7CHP2Zq8XzLwUFTsdUp2P2L8rcPMWdQnzpcKp8LzjJDvd+SNWgFeWi7SQbjvco3FkJAzP0I1suELMKreEXu8dw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CYXPR12MB9320.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014)(7416014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: k9Wio8SjiAzNpz3xTlFaNsXwl0I3SXEqtJVuL4ymN2CbmrzpGmjxvf7RbDrOOLyMPafd38dL9U9n6Kzurc02Y6r2TjqvV6CbNK7ehNLIDBeBMyyaQpSFZlqUUfrTG7xaDilVXvHCxmyuorbnED3K+tk+JRRliLs4JWpN9oANHvwgSCeTWO/GSVZXuSykacaSQskMvh3OPELJz8UGyqgz+m4ia7D1vO38DJ4tRSF+F9lGL/8fbzik3mdcoQlxgGUUyqEQEl0G9IB6PHMGH1WROqjXpk8h2IPQoPTPU36g6I0wDrlM8AujO4lgYGekSjOGIWa0Cd28LyytgPdujpWK9lNKzf0imUCsPyDeabcCNw1o7BPYvFV9waAA6J2LL46+0wVMd36Z8mh0QMc5xaIUqfhiye/Sv07VLtCy4fugxLeJQKrkGZc+6mFf1thVv2o54mL/5NQqPo8lfKdfZShscD6asBYjefKJKb0vI2RgJ22//MuD0NhcldsnsRZPCGUJ7jA8gg6Q8u1BskSsGlUyH6N5bhC54/v4VOj6EC+dmgQAsgTtwTtbfWzX3r3Y7vXLc+YzsGK6dqS9lhmh9jRZN2DIVfi6943in/A89wpuJEccAvwYWcsfvXTpbhSEbTXLYBD1DVILgLyOiHXbQP4VX9uhA/Pj7KnW+HOMBft01pBwJlXLoEBWNai4U3ZevuVZiEe3olUSvrb8Lk8m2cg+1Z8VTt5j7fserfX4JiLep9xHqedtKJNPKT9le4mtohGqO6LMba3OkpfrhpAnY4WZZ6HQu55D+S4dn7COTb0wNstTn0+yKE74WmRn5R081u/A1om0MQQe9teH5vLL2fyGcbHyeCg54SjIYt4KftJGaivS+4gphus103WIPD3Ueh9QuCcf2s6d701e5lF+7S+iIs2yYks9SDzDhVY724vevFnqzQ79O+JWoEtQXGOnsHz6nBNo5/E4RCMVJeUvFelD/zMqBaiLdyeBsQKGBcCCUQBF1s1IIe/P9KIzJ0NZfs+fvlhUfUEeHU+JXA53em3GSVrJA+c57o1dnkGeO6CPxJWUihrcsnEzdSp5l7PMa6d114TxwjN4Vuuujjv/ERzUF37eK8A3BupvpPDT8Jqkhc0VqjiJ8UzPErpKzsjcpfddcD/elBDA4UlG8hj5slOO2kdeEbGZFcHxoYTTgaydRLX6hueFsFLY2wBd+AWAcMlFEf4G/LTNbbjolP/F+1y4nCo5th+p47KZNxbrfw50uYKnUz+AQjbrMnzXWmfQtHZ8ugKYmK3XkfqpcNtx0DF4ulXW5S6BPcUSHFCOBHBwltjtInRyRym2XXhLPpatxSEr8svxVaBSgstF/mFJ9/Tk70vER4hzIWBsKBNyIemTNEyHdwOep8KeFm1SxUpVin2AuC8lR7CuBLCmSeoD2KSqXexZwKK3B99sWZ1LMuuhpByU/XxQu0leu+GfzkdkFqGeDVGXtIn6nm3HR4Y4+KndtqxwIqw/WyG3x+HI/Gv6VwHUPAEh0hpFhvMdfCBFW3CBTKjVj5wmI8tI5Qp4mmZuZjEjizAeikIk0DhHiwIAkBJNy1FdKjIvzgBXlUOKD41W X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: c2ddc1f3-7a4a-449c-31a5-08dcb883d8d7 X-MS-Exchange-CrossTenant-AuthSource: CYXPR12MB9320.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Aug 2024 14:59:19.9173 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cNfpbJyV7zMWue2Usz7z2XcTFv/g8gJpnoNNEdg5+ojbST0y2+BcUR8jTgYERrP6 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6823 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 6E91C18002D X-Stat-Signature: nezrdfdpn5dkrcgmaypbkpsepodsm3ry X-HE-Tag: 1723215564-338520 X-HE-Meta: U2FsdGVkX1/Y6Ji/vkhLkFzjiPktGm7vFTAeYCtfVxe0maAJD9qui2DV21tKSEn8znb+Ph4xNmCYQbArREcjUDtPsDkU/q1QWSC9xBRfBA41N07LBjYPheHklDIsVGMjx7acZ2dU9CbmrCupQlk373Dal3kayaxsCh645HjwR/pK9T/coFjQq1LYvmuAXehZe924yL0jMJwST0YTNU5b4vmlefbQBwDJ9dSAY/L3QttmUNp4e98eSyP0AFnVghdfVRyAX1oT5ot0FbqWA4MiLRjTERn+cBL3M2QTQ8lmsoEt/xKatI4/8wV7dg3TCA7iXkXXZ/jlMvEiP9TS3YdOB80rtQtgBeb3B21XIXg4oBTduNLiq3UBusgfEM5QtpopCYbDvGOe3MKOEo2XeQcGgEpTz1uO/tBgKYG9tmTy2if5wTzBdrLyl7FFNvoeR5imfg7ZwNnoIfeN6AfWW2TDsLHT97m/Pr4pb7nDNxxti6+iEQU3fg/CTXHymhsmbsbQ0oj9xvGbp0NKEO+EqL2zf/6MEX92bSyCM8V6HPOCCHp7jJ0J9xRGWmwjlizfnNNisZii+TjOIavwi8/AfuSNCHGksRh/RQlHUmsZvS03WgL8UsfVzFMUbXmxqyjxUR2YWKHJFol+bhMe3yZBCYDAabTbqcRrz+RSYRqkFetLpTt9XLvOSJ8f6LPG9IB9Mfoz4q36jBZTlmq9ailvZw5b5ybRtlhsqG+kUe7JW5WaUnHpZfPU3HPcsEKu5qBnX3hLyjypA8/8wcqmhe0vh6LL7vNUKo++nxCa1+ba9e0Tqf2We+SX185RR2pbyJg0ptR/tQbR17AZS2/i1kRLILazzacuemS4QVeQ9pctYJHzmoIDabPyJRBkrnNDEYBhHvKSXJW43bOjT2R5NxFYZSz8nxT19S0sLWDg9Jk7di32p8Vyf1L3h+p3jRD28OOGwGiKaSB3QZSc6YPMEn4F2a5 0uuvIWZ8 N1djZwzY5CZLmUzCndL28FAarZ7XBuktseDdW1D9TbqzKAd2xzOIkpor7ssmvkm180cCcVoXWVDXbqLIhHK3jxknrnVthhA5gU4/UyVBZgpd/1/WlBtI1PzAcSZbtNKWYzmttUmyJBwMsUB6HiUZqYKIfOYN8ba1bgzGfryPPgExvcNMe1LhYxFo1LpHw5Xhw2IBmtBLwT9wi6+NONkNpQ/kgCLQX7PpUwLWlZ0JOErYmK8GEVG1EYE8cHSPsrFELF02lBRED0fOp6zSNGILJABnehB9ckS89I+NM/sTRBYUkln94KWNA+SoHns1qAmtOIL/w77c7pyIMCPHQvB0KYpzrw4pFBo269HkYbk8V21t171IZ5pHnQnjpcpG6q0LSbh/ZEq719MVAdm4C6YCpWvLPgw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When handling a numa page fault, task_numa_fault() should be called by a process that restores the page table of the faulted folio to avoid duplicated stats counting. Commit b99a342d4f11 ("NUMA balancing: reduce TLB flush via delaying mapping on hint page fault") restructured do_numa_page() and did not avoid task_numa_fault() call in the second page table check after a numa migration failure. Fix it by making all !pte_same() return immediately. This issue can cause task_numa_fault() being called more than necessary and lead to unexpected numa balancing results (It is hard to tell whether the issue will cause positive or negative performance impact due to duplicated numa fault counting). Reported-by: "Huang, Ying" Closes: https://lore.kernel.org/linux-mm/87zfqfw0yw.fsf@yhuang6-desk2.ccr.corp.intel.com/ Fixes: b99a342d4f11 ("NUMA balancing: reduce TLB flush via delaying mapping on hint page fault") Cc: Signed-off-by: Zi Yan Acked-by: David Hildenbrand --- mm/memory.c | 33 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 17 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 67496dc5064f..bf791da57cab 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5461,7 +5461,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) if (unlikely(!pte_same(old_pte, vmf->orig_pte))) { pte_unmap_unlock(vmf->pte, vmf->ptl); - goto out; + return 0; } pte = pte_modify(old_pte, vma->vm_page_prot); @@ -5523,23 +5523,19 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) if (!migrate_misplaced_folio(folio, vma, target_nid)) { nid = target_nid; flags |= TNF_MIGRATED; - } else { - flags |= TNF_MIGRATE_FAIL; - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); - if (unlikely(!vmf->pte)) - goto out; - if (unlikely(!pte_same(ptep_get(vmf->pte), vmf->orig_pte))) { - pte_unmap_unlock(vmf->pte, vmf->ptl); - goto out; - } - goto out_map; + task_numa_fault(last_cpupid, nid, nr_pages, flags); + return 0; } -out: - if (nid != NUMA_NO_NODE) - task_numa_fault(last_cpupid, nid, nr_pages, flags); - return 0; + flags |= TNF_MIGRATE_FAIL; + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + if (unlikely(!vmf->pte)) + return 0; + if (unlikely(!pte_same(ptep_get(vmf->pte), vmf->orig_pte))) { + pte_unmap_unlock(vmf->pte, vmf->ptl); + return 0; + } out_map: /* * Make it present again, depending on how arch implements @@ -5552,7 +5548,10 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) numa_rebuild_single_mapping(vmf, vma, vmf->address, vmf->pte, writable); pte_unmap_unlock(vmf->pte, vmf->ptl); - goto out; + + if (nid != NUMA_NO_NODE) + task_numa_fault(last_cpupid, nid, nr_pages, flags); + return 0; } static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf) From patchwork Fri Aug 9 14:59:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13758865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 967C0C52D7C for ; Fri, 9 Aug 2024 14:59:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1641E6B0088; Fri, 9 Aug 2024 10:59:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0EDC76B008C; Fri, 9 Aug 2024 10:59:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E0D106B0092; Fri, 9 Aug 2024 10:59:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BBE126B0088 for ; Fri, 9 Aug 2024 10:59:32 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4B1091C2212 for ; Fri, 9 Aug 2024 14:59:32 +0000 (UTC) X-FDA: 82433015784.07.A292F21 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2046.outbound.protection.outlook.com [40.107.92.46]) by imf10.hostedemail.com (Postfix) with ESMTP id 834ACC002F for ; Fri, 9 Aug 2024 14:59:29 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=tJmVXXx3; spf=pass (imf10.hostedemail.com: domain of ziy@nvidia.com designates 40.107.92.46 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1723215536; a=rsa-sha256; cv=pass; b=gbDIYoW+/PyN3Xcy+2IJNmU4s/ENPDP9CnnmNkJdpe/yjL5cWU88TpRuTHjt3bTiEHip1Q qpGJnEaW/Xx6JoOMLL8/J8D064ExiO9v3hmz1Zj95OfNj06mygjMx02UrCsjIpI4OK8ivt cgr0k+y9D9vbt/QgBqdHPbNLp5y5Kog= ARC-Authentication-Results: i=2; imf10.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=tJmVXXx3; spf=pass (imf10.hostedemail.com: domain of ziy@nvidia.com designates 40.107.92.46 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723215536; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1XYiZT7rhmKPSP0vkQcCuc/N49agUoRG3Dy29+8nYjw=; b=e4nTboMRO9+C4nYUv++SJbL67etDg0ytLrGsjYXClZWy8hFKQzIE8Q5CVwjxdZ2BOaNtMN G+tdvORAGcWKQDva+FWWUznTzbjHmlVUCjn40cSzqJcHDEj79ASi6H42fdoD1zCY213LHR wy/kvkhSYm/P7zKc7aWJZj0WdbAxTH0= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lJrUNknsdqer/ahxrBmkom08NyFnAxDLZoVTYRdkEfRdqj0xELHFfImwzbnnMrT29sdcIH1VbLU4mrZhjAceyAS0dLi13LtjFd9Emz6sjIMMbh5GiN22IVYVy1/5+QigY91gtHmcy3NLXcjcBTb+TIhD1k5gv+9DvWWhW2yS7JIflw99Vmcwj8rutB2SqE9YzIX72mKXMUoGLjNuIVINyK/BYv3yLtDP6iwt3pUz0dsj/mN8w9Np//U70BTsfr+8B9jgraDxrykCW6jCqv5nTRQpzpIbp0kGGnH/g6x4LLK4xiGPPDXvGBAYTQklUsLt5MVu132i6ef01n3SUHIOtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1XYiZT7rhmKPSP0vkQcCuc/N49agUoRG3Dy29+8nYjw=; b=pV0vlrlKIrYbcGP5gWNKDZyJX3AqSiJO4ZaeHNBGfCUwLGhg1WuPyD/fmpl41l6D7EmXSXxEnXfCnNKR2AL3CO6Wth13rhk1VFFViVJYR3MuaqNgx0UUjebABXJyRnGFQD59UJ1ovVPnZmaWDBuPIZnUw2PqMIoW3RozlWWyj/lRP3Ceyqoi7e/6rKs3e2Bb8qwXdLmAjFkVdcGSDD9PZKY1nGhZF1slPd8I+z0X9fb2kfShd4JR8DLpq1CshTWQmwq2Yo66dmnpvqRNwR9gpiQTNdLzpMGrX4LSvFoYu+wS1n3p4ahcOQAtACxcki2wabBE59Cu9J7tsXdwW1gr/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1XYiZT7rhmKPSP0vkQcCuc/N49agUoRG3Dy29+8nYjw=; b=tJmVXXx3MSWTndsVbGYnccDdTDn6ExcqgngxFID5vz3Gn5Eft+rfhnOLIXdLpPR8xuYMrxOljBH6VRwUb9EFjt6ggZGpaJ30RkDy4Sbc2Q0PRmVRXPgHnh1acopDIMgayVZhPKldzb3KJnYP8x6jT42p3+5c88AOHbOb8IMALjH0mes257vYUrKeML0eRxaErnZE2RePoaswwq1W68o8InSAluQcXMDseROp0CR/ZD7C1nJeBpSiNP4jnyw93IgCJgIu69IblsT0PllLitse6cNAL7O1C93mSUpdl+AZJ711pisdlDAT5HPqRHaf5ppP5gXQUKbdF6hYvRkz5FEXpA== Received: from CYXPR12MB9320.namprd12.prod.outlook.com (2603:10b6:930:e6::9) by SA1PR12MB6823.namprd12.prod.outlook.com (2603:10b6:806:25e::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7849.14; Fri, 9 Aug 2024 14:59:21 +0000 Received: from CYXPR12MB9320.namprd12.prod.outlook.com ([fe80::9347:9720:e1df:bb5f]) by CYXPR12MB9320.namprd12.prod.outlook.com ([fe80::9347:9720:e1df:bb5f%3]) with mapi id 15.20.7849.014; Fri, 9 Aug 2024 14:59:21 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: Andrew Morton , David Hildenbrand , "Huang, Ying" , Baolin Wang , Kefeng Wang , Yang Shi , Mel Gorman , linux-kernel@vger.kernel.org, Zi Yan , stable@vger.kernel.org Subject: [PATCH v3 2/3] mm/numa: no task_numa_fault() call if PMD is changed Date: Fri, 9 Aug 2024 10:59:05 -0400 Message-ID: <20240809145906.1513458-3-ziy@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240809145906.1513458-1-ziy@nvidia.com> References: <20240809145906.1513458-1-ziy@nvidia.com> X-ClientProxiedBy: BLAPR03CA0077.namprd03.prod.outlook.com (2603:10b6:208:329::22) To CYXPR12MB9320.namprd12.prod.outlook.com (2603:10b6:930:e6::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CYXPR12MB9320:EE_|SA1PR12MB6823:EE_ X-MS-Office365-Filtering-Correlation-Id: 033aea5b-0b48-4cf7-4272-08dcb883d9df X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014|7416014; X-Microsoft-Antispam-Message-Info: kF8MiWo6w8j10A2B2YK8g5fFiggSqMhA9C6wS2D9hbbbgCW5lcX3tyURZDdGc2Cho4TBGP5wbKO+inaCdtQ37VuR0irJakQG9mQchJIUPv6Z/XNwJ3xQYqZWRFxYZc8Zp9S7kSqxhXKvrGe60LNh4FKF5DK5rndj8oM6efLQweGp0y2vZF9d70RrSP+Xzb+tGoPsPs9D2QywvZO8/HXBgbLZ9hfW6XA7zhMDdgL16hnsBklCO5kdpMqHyGPiUZn5yQ5l/lwwf1fU73AQxSv0KpG2ejPHbJtvWdBuMFBJ411NDatOzyqHl7BuOvfeAmpp091VK5RblvmWZqSm3SLs2DhVTJDT9cwxkUw4f1+RPYZVNkEXqvNgfWrEDWnioedgO/1m+VuVjyVfvNOOZ33N3Gev1flo8YVM14v6RZQ8E4ElqtSG5eDmZCr3qwT7PBBfJbdb0RrHKK9ksPF8tP2cPuVJDWffYCiodYrPFwqdfaSNC730nKJEWmeGKaRDQ4Of9q3R86Lx8EOLNKU3l5X5YH5yt+UcBgXjqtCf3k5xmI2YjR1OS2V0W5glf97IG0sD78joYtMTE4gOR3cRYbVGy4LzUDOxLRom8MtIWKdTeZCMqC0BlmXjwIQPNMXKmvdGwFTt8bRosu/FKCr7PzH7F7qA7Vg1FllAJJeliAUImbtNzTqjO9fDY4jIssI/Y0LbMfz5uRvXflFk+6bvsw+3XVFLTjI7F48oqgRt+CdK8phDwf5R2hc2/vUGaGcDS8cV+BM+iqhrmWDPcrx3ePh9V6n+BqIalUTZquoxEaOQclDaWfzu1OWdj7JVfeAAGo3dt8mGcNqtKlUkXfVDknJeSfjWnIxyPw2YBoqsAFiGsunfKdH3LO5mwszMU2wvkIIhCV7JCrrCbER0TDplCCowA7Dvr0cd/ldGxNW/yMVYd10q0VGGvi6b4PIF7E0i4mB8w6Geq9tt+WOLg8l2rpslyAKDoWwAZm6Gwc80TqkfTR0tYKtF4KF8xfxNaZm/KHNstDgZsatXkoIB3jg4LR5zNecm91y3hGvL16BRwTBRX9Xvf5M89V7uqyhPLWNNKm0/zOCLfV5WRATL1Oqn6O9O2AyoVHE06D504ACrKzE5E51w1V1SdM0uAF8ZzOroYqPEDrGo1m1s8QIcDZJ0sRere4KomNinM/ANYgtC9f+tzRe+Ta0rTleIjsxldHi1ShYh5cM9D+Hose8Yi/2s8p5Pb8w/OfZj2g+8dFEENXdck7eBHT4uuzMBCwl0K4lgf+d5G62/Z/rkx0cdM+PcUsFRowGqn+1unI68J74zphZsRfhgjBPp8N4qieM0gQU+cIzB4o3ulK4Q/H5sV6jpXzsKZujdPJX/00+UkW4k6FIr20M= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CYXPR12MB9320.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014)(7416014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: URpX3NK/YKDy0tNgoA4oxpJP1Y1xr0YYBSMaGZvS0o9NJTlywQwylPF/03tpQ4RwNfX+LTsQK5T7Z3ml12100/LmO558upw1fvYtFMTa75yuwJpigx5uQNNHU1qsAoqUG65PnOBQDRd2EKdeUjF35Wxe9kzmHZIN+PixJkNG0LW7sCxOwtBSP5/MFgkUs+2bYOW2CSNKsANWEFxeFWDXzVrHJ5M54yFyw4TSgTljeQqX8hiVf9AkyY7Ta6u1MFlAZrNkdVaE2bxQVshdhfa5C6xgy1UqGw814+2LPRNRfMSoo/4hUF51QUPEwrpgezyxSYELOftp78+YZ8sNHWryZU7V/+7MxvC3TsWIMMbVEbC2TI2nqnyGPQmcYB7LS3LbuxP8H1fGVI8w88CNE2EjFa61AwjGTkahObKi+JZdJshfAipdBUMmnd8j6I6BFprcf4XJJwAlSMFWIZ2lXg0uHQ7msbKlY0lYR4GMV9QjPfmXCTatQYWoMDpRC4FGbVOEjpXwSb0ST7gV5GOU3mufYcxs7seNt9mIAVqPqJ4Vu6Z7ZCJgoQzpym5bMhoblBRAcPZm3xVjnsHhJ5KEiUeEwJMH2SyiKHXTj0pjm92sZBIZB3oWNrrEtkgohNQvCd0kKLd7xiNkaq6wbj6EXPX+i40ghAT1SAZAbNW9SJ3cF25OcJnkWp1M2TwGDhcHugj3Ynh/ouy6jPjbYq4hh8jJGwhG8zIGt53IzOdOlgQnjhc8v3JGb9EvdY72vgtIoqVbG6x/liimxmFhK8tMhw6BRVqQiBADYwamTI6uH1M6DlEJQGs/dKbmUeILB5jBGDHnjwd0nAImuWaE5qR8R1Otv5IM0H19b2JMZhtj/KUWlNIKolQgZdEhGh51TDZw2aMY8Bou/BFPxluuK98/jOfhqSTT6d/VVnk2hGBdoyOwn2Zdh8wwddVJ2iGtevc6DKGWWqZYDO7p5Ud/q+zPDNn5+Bn66cFK7rbZOro7xhjCIz3lGrTNtdVjaNNmJRtGZlIVbRN/Wy8qrhuLJ123G/VvnyY/6tuYsv0Sovd152EAchapGg8NKIOPzww7yPningfp+bWRKlxoZvgb4n/ldcC0XkPaxZusBfNmiFnWbXlluXFa9/GTzVd2WlGHqgOJlbhhqFtFNXql60xue49KCyPhVnU1vd4QLEGmIwsayIAcVl5VP+bStO93Sebt3qCOYAuUDKGLI5rxkyhoo3DyuAsi7pBnsWLXRp41kHQNUc+pY6soUaUCmzpWfR5qaj8kN5/ZHxM7EvhBQ+F5DhSaKiZz4zNCRpvJoVpxECUtkEeTcXXQCvIcL7q2W6xBr/yz6rkEFZsJJowrrUHryIQJpBo7RBV3W2q478O5hTAgelmZRkQJ3Kjn6AmBiHAQpajjinmd6E/1TobdCiLeqIgI0olQTPiZkZA/km5lmwDTaBPmfwnq5cEbI/YVKhJdAtStzDkjYSGrJfdV0QAY/RnUKee46y/IcRA9O1tmYR/jIB6ufX7ikCZ3P/cZcUvJHRODWTGwA+9Np8qgtobD9NXllSD7J1v4i50K6y6zgCThUjh+SUziUJCMv2YqINL0M9poWHu7 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 033aea5b-0b48-4cf7-4272-08dcb883d9df X-MS-Exchange-CrossTenant-AuthSource: CYXPR12MB9320.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Aug 2024 14:59:21.6054 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: gu1i5YaaeqcuWRxGgDyxwsN9BGpuvjQIskos8790AZrQe4J/Zb+Uh/Fr2pp8sZeA X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6823 X-Stat-Signature: d6pqazgkmnbjasfimyiah8n98x5gxiop X-Rspamd-Queue-Id: 834ACC002F X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1723215569-296739 X-HE-Meta: U2FsdGVkX1+0UPpMIxVhqNm2D+BgiNB51lvcEDGa7xyI/sVvvy/cBkKiGNKd0Nb1taoG66UwOOPsklNmf60mPwWA61kYsvm1HXmmB6dirAGV5OtBjRRynIlJHhQhggcMNhNO2i6Gyac0OupKH76rGtk+XBj9mxDS4rR3M3szU0IdMe6eVIMIpzMRXloxy1SBuT3L8UU+cwfQRWbJtc921THRK16awydGLyswSzoZZLIzjgIt9bjWmBdSaoRf6MnmlAcaCaa+j1HpXMmY7kBKf5rnTR4rtFn1nnayMi+5MBudIza2jAxLdqWFA5KxdK+hxj//dO3EGRglccltYCYjsNqPUmotrt/FuapaVBBL5ujojliJASAXeEIeYBXLuDeWTPbt28w4/opTUxDBr+oLzue2M5zSZzFjfeJavLKqJyZGYuCjEdz/JPvlALxWy/zymZuU6Yw+992foIsauLO7DEFK1J0OA5zXWsD+KjSq2Y+rAboZvJYSkk1iiSAKAniovLAmMA/XqEo2/pixhpRFParMlMj2oFTUcCbIaMDNEcVU+atP8sbXQW80nDn/J77xicAdMj4SVhS9N9K3A94fuvXqZkj28mLh8Yx1WKrBG5C8V2ijneFMauKOTYyLL2+zOgNR7Ju3Y+2+NxntgTJ4CYubpHC5BMrBp9NZH7MK39LcGKix3bh4Gc+llSfVd76KzJOORLWBaAhw7rlJ4LHoskJa9TSTqkqp+tHGkDk4SvL61oWIuF0un/f6K62N/ezB4X+eg71r7esypCjBLmoK/69OipTSUY3PHSoU2Fuu1CmauMfsykkZrHChxI6zpH60chbgS9MQSFCQbfMz2fZx6poKTesn74uDdGsxwkpVIqiDv9ESDJ3mbvdcnNbyHDl2c7eWnaWFvHrOVo+LMrxrtiv7+NOtsRla/+9WNqIgWTpdafPaTyKqZV/YfD7bn3DkNxJtrxf5Yi32oy5sbLB zf/FnDVM d5ShA/NLiUg/aZQIc3efPiFBUDPsA2ytw76d9guzI+D0iiOlvGHfsIUOh9DCAuLKLopBOGBMXbzMdofgTW50yjTkUaNx8vr8ysGaDoQ+RhGCF1tStfGM3DUz4CDRA/JvkEShBWDB7aupsW11jcHCDrEKFHD/mP9ZrOO11EqT28p0ZqOINFz8SsRv9WPNPSKHY127t8GZO9UxX4j/dL5MRA7YQ5M3JuyFoKBSu/COPXT1ujkymTe0b0GWBxPOv9t1KKEpRiiv4sJTv+uYxeLBpPTRLH34FpgXz/3nHJobyNJhqcrJ/72CWiRujhqawY7ONcrwRmrd+cbAYM+ZyNcdz0Cg1G+K21iN2QhvAR2e8njfrTn8mBoiqeTG1QcAPpxwj8qEtKfZgAkMzV9h3wOzstISqOQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When handling a numa page fault, task_numa_fault() should be called by a process that restores the page table of the faulted folio to avoid duplicated stats counting. Commit c5b5a3dd2c1f ("mm: thp: refactor NUMA fault handling") restructured do_huge_pmd_numa_page() and did not avoid task_numa_fault() call in the second page table check after a numa migration failure. Fix it by making all !pmd_same() return immediately. This issue can cause task_numa_fault() being called more than necessary and lead to unexpected numa balancing results (It is hard to tell whether the issue will cause positive or negative performance impact due to duplicated numa fault counting). Reported-by: "Huang, Ying" Closes: https://lore.kernel.org/linux-mm/87zfqfw0yw.fsf@yhuang6-desk2.ccr.corp.intel.com/ Fixes: c5b5a3dd2c1f ("mm: thp: refactor NUMA fault handling") Cc: Signed-off-by: Zi Yan Acked-by: David Hildenbrand --- mm/huge_memory.c | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0024266dea0a..666fa675e5b6 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1681,7 +1681,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { spin_unlock(vmf->ptl); - goto out; + return 0; } pmd = pmd_modify(oldpmd, vma->vm_page_prot); @@ -1724,22 +1724,16 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) if (!migrate_misplaced_folio(folio, vma, target_nid)) { flags |= TNF_MIGRATED; nid = target_nid; - } else { - flags |= TNF_MIGRATE_FAIL; - vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); - if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { - spin_unlock(vmf->ptl); - goto out; - } - goto out_map; - } - -out: - if (nid != NUMA_NO_NODE) task_numa_fault(last_cpupid, nid, HPAGE_PMD_NR, flags); + return 0; + } - return 0; - + flags |= TNF_MIGRATE_FAIL; + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); + if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { + spin_unlock(vmf->ptl); + return 0; + } out_map: /* Restore the PMD */ pmd = pmd_modify(oldpmd, vma->vm_page_prot); @@ -1749,7 +1743,10 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) set_pmd_at(vma->vm_mm, haddr, vmf->pmd, pmd); update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); spin_unlock(vmf->ptl); - goto out; + + if (nid != NUMA_NO_NODE) + task_numa_fault(last_cpupid, nid, HPAGE_PMD_NR, flags); + return 0; } /* From patchwork Fri Aug 9 14:59:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13758867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DA39C52D7C for ; Fri, 9 Aug 2024 15:01:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B012A6B0089; Fri, 9 Aug 2024 11:01:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A8AD36B008A; Fri, 9 Aug 2024 11:01:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88F366B009E; Fri, 9 Aug 2024 11:01:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 602356B0089 for ; Fri, 9 Aug 2024 11:01:03 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 70B17C11C0 for ; Fri, 9 Aug 2024 15:01:02 +0000 (UTC) X-FDA: 82433019564.12.14A3B2D Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2073.outbound.protection.outlook.com [40.107.94.73]) by imf17.hostedemail.com (Postfix) with ESMTP id 0E62A4004E for ; Fri, 9 Aug 2024 15:00:58 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="msMk/chR"; spf=pass (imf17.hostedemail.com: domain of ziy@nvidia.com designates 40.107.94.73 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723215607; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Jp1ggtAq2TPL+tWQv2b9sD0zy1Lqc8F9tl7KovE4HMg=; b=xLnE6SOYfnE9otRZmEtFEU8YM9WyN1lQYzgQhYbRRFiBbgGbOY5TdZYqkbb95cNklabWes /r9Hqh1nHEuuxKqFNFz0zyq+BHE9Qd4INxDRnC5gXzsWScN51U03YyTRW9rh+RWcpS026P r8FdWQlVquj0yTcLuFQPcZAF469mFJE= ARC-Authentication-Results: i=2; imf17.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="msMk/chR"; spf=pass (imf17.hostedemail.com: domain of ziy@nvidia.com designates 40.107.94.73 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1723215607; a=rsa-sha256; cv=pass; b=El6MeX4qvnCq+9e9cwMRusLGrhDriuOg1vaKhIfaXtyBIRL2JNrFv6Bhblg0G0klwRbcs2 JQAlYsmgJa5KirhSlo1zcW1jbEspzle9rOuEapul/mbWRtqC5X10WaMV1W2VXlbzpAC835 diltSuwNi4zAkr3FeX9B/qbcep01ocU= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DbKuD2iwF4gFglTo+WrcYagwHK7VSz+mZGVDbqL1N4yhQxmgqyFkc1RhcOIWb4z48EgdbnMhSCKGcGzPu5oLu5KI8Mi/ID1jmbbLAPnNI4guKHnInZslbpxjjQ3/US9BUn748w1D5fSeYIqvIXgdCvJxfuQuB5RPVqmDJSAUw6UrrvCRhRAfXoCssU5GdAcv6IYxlleisu5BTwxFJTCxax6CjqWz9m2p/my7w0gG5KLJAdnByAScDtcUPzbTF7teJW5i+3RkD8ekSxJLomF6IQ7tdFPzhLLnsuGxmR8q2glzCSCTKBUkw2ZdIYHT+rYzc97ZUnp/xdz6G5ABSytihQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Jp1ggtAq2TPL+tWQv2b9sD0zy1Lqc8F9tl7KovE4HMg=; b=k0oJ1BcH0CK6azo2E/wqHYrp5oBnaQJDsjf3xZnJ0vemrNbFMgtN4Rvn2QxF578VT05IVz1wqBdcoVkxr9bIBgF/TJDUr2aLPdFXu8LQtO38tXhPk/kcDuavJ1I1q5LMdquP/ECE7SFWwq7TYusS93nBzLuL6GSKLZjZg1l574Z15MKVjrnVae3aVOaIeSFdkR63gntkBxsYpfT71/6UWEJplOr/yqV2ivT/MHa/4fO0fYOHF+UaR79/5/YFclYqecm18p8LGYu14hAIQfzwzH/lLAhcwjcCiA8liA69cEkz36a8JX2oKjycV0G+VCgfVV1S553nOY3vdmRx4z5+ow== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Jp1ggtAq2TPL+tWQv2b9sD0zy1Lqc8F9tl7KovE4HMg=; b=msMk/chRuSZUhIUp4Hy93Ik51puNiRdC0d3LILReIOE2R4V4l+LYQP0y/W2Qvagig+hW3V04Wwj86uugUIMAM4qZWesxgAVK7v2ZPcjfh2q03UafUypvQJ96EpehWCEkiOkrpqpUl6Y11ybMWYwJ4r2SiTbS9rUkbBanIkp5i9gZwe0PQK4IxnpxmU7AqVA+9AYeYxczL/MvLxu9A5M3tDiUNCwunv60XFf+S1zaTIaA6SlDxsAysU5fLqvUUplzvA2UYJ0zYKLHsdAhVM5+H55ntK/vrh3q4vO2EvD0eVY3qYnV7/bYzfjpR0X0KTghHyE3LKBt9yskxvD0MRiYrg== Received: from CYXPR12MB9320.namprd12.prod.outlook.com (2603:10b6:930:e6::9) by SA1PR12MB6823.namprd12.prod.outlook.com (2603:10b6:806:25e::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7849.14; Fri, 9 Aug 2024 14:59:23 +0000 Received: from CYXPR12MB9320.namprd12.prod.outlook.com ([fe80::9347:9720:e1df:bb5f]) by CYXPR12MB9320.namprd12.prod.outlook.com ([fe80::9347:9720:e1df:bb5f%3]) with mapi id 15.20.7849.014; Fri, 9 Aug 2024 14:59:23 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: Andrew Morton , David Hildenbrand , "Huang, Ying" , Baolin Wang , Kefeng Wang , Yang Shi , Mel Gorman , linux-kernel@vger.kernel.org, Zi Yan Subject: [PATCH v3 3/3] mm/migrate: move common code to numa_migrate_check (was numa_migrate_prep) Date: Fri, 9 Aug 2024 10:59:06 -0400 Message-ID: <20240809145906.1513458-4-ziy@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240809145906.1513458-1-ziy@nvidia.com> References: <20240809145906.1513458-1-ziy@nvidia.com> X-ClientProxiedBy: BLAPR03CA0088.namprd03.prod.outlook.com (2603:10b6:208:329::33) To CYXPR12MB9320.namprd12.prod.outlook.com (2603:10b6:930:e6::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CYXPR12MB9320:EE_|SA1PR12MB6823:EE_ X-MS-Office365-Filtering-Correlation-Id: fcc233f4-6fe5-404b-b093-08dcb883db08 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014; X-Microsoft-Antispam-Message-Info: ooUYHu3jv0TDp6L6f7V/oUy+zyp/KNRRBspA8U9Wyvh0DcWa11G6/Z22yMMOauztTJ0T5uWKXfclU0Ymr1mRwLLJFg/d4MzULAzyv822MoY3OZo9nmE/5tSuZiXp1xpXyIMkcsBzeotfzWG99NLegOarrCwJc8zzmXz87rLI6uJgdnBpVcXiK0iz/5+/i3DgaeOz8FPz9yZXF9SS+8lSp9rRCr3lfVEM3XnAuUs+wG5irV+W1LLuqrvAgPV9/K2Tpwc79K08m0kAFTbTuKZMsV6zyTXoT7tFXDl0u3P18Jpxg5JdKw/qbzcHtgTid8x/+UhKcxBziQmEHngyFCnVmyN1CH3UIO8QoG6gIHNr0ZCLu+9o682Tdf5MBQZNQwYgJvJD6XC7BvIRoC628RWbxcLGEFAIeWGfv08HT7/Bom4VhEt681iY5JagMOblk80R+/4yLQjoI+D1NBvt8nCpAD9mzNMCsXXU4e+PakHwDxp7AO0LdlO90TnfB6LoHgPw+qufOAe3g9N9S5HcafdLiiw8wM0xz/70dx1qHK6OGs/xY2YWJQ9nsMB+ivMnY92ATojeuP7xbelxEYEsRhupFNNPJpmpJc9KFFdKNUFkzJATCiitBQ7ylI4n4lk+pPSj4/OQEW4fdoYBjtAZA0FDxaORzbsYniXleO3Bj4MxEldnDZhiTbT4R/h9qYqoQZviJ5dkhdglDyfCye3W3NYg4IShwKdGzWY6il3BKu4wgGgKE0uk2tbYafCGRUoaRH3Fldto3q3g+VpICKNEYrtvL1EC1TyRbov+pmWlSuDKE5x5BMYi/BMP2GvGn8K58Mk4ftYvH1rO5zEfzwt7hCYNHC0p/JW68q2WX3jX3ntTWxpJ8/yktwuAW3sznAJhYSiixox/N3uUY7Hx9WJnukUri/fdc0Z8uPLB2gV4mHtBzQenH6GS5n5dgMpRkF+28hAkEuGw0pTaZdKGIXypx5ItEeTXYprFkCnFrwQtEJlCa2WCyS18DpEHnhxytlszQrFVodL0ONVsi4HrIW2wK/kKX0MSZOg6G+z0dmwlb0nogUCJmQ+9mz7JXMNffPYO43wFatyZAvCroCo5yagq35gpVjySDPg+He0mz8kd3FjwagKO+eAI5LZDtTyWbLaDnOBSFGqEKvi4OjaGUC4RhgpiCAQLreHM3oAO8aqhLr+Lu/L8HJtTC4xTxYpLcroVVBuy98TioGoyKAMcREmyuMZ8P50fN8Xy6mEuPpzZybV3EthsSB86kLyWLFvc7020XWucboInPl5pltX0OWd8MtU6ssDmz38jf81U5eJ4M3MF2o+R+sVQeDc+XzoSYMR3Wgj3q38MD0xTTaMGK29nhL/fdA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CYXPR12MB9320.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: EnIpbK5dj9/xYPibNM3XPy4mXGaS6lCzJzruuk5AREgyCupp5uLXB8662rzH37IXWGV6xC1VuGcrDOZntHNcv5NftEddccj4qdv7HHSYD0esqL4e8D4PGWJXsi7dZ0pnAs61NKtx09ApDMg86FLRutEDmEZbDGGyt3LoOM6N3c+ZvOWkjB2kNDUpK0TbCrWSd9e1zUrXrNsFrSONHu9u6d4K5EE7Q0CWjrdSOhpJ4xdIF39Kf/5j1uSYGnmIEgi/pu2sTNBO9Ah0egRLGbqkKRz4eAIvA7MXHz9HeIwkkFgt1D08JmkXawj1Ganinj/BRe4FONgtS0Qvo8gXjiypE8NarCI2ArqjMJqwpEC5ogNvDa4OLPYG2EHskOEhP5q9Ca2Xie58bcnvx+RF8f9IjuPKMiRvPYrORDFgaK9V+xRg2hN+fKkhKPbKZa63rDWhdePgmGbSm+VPOwQbWoIMGQp8Xvz3SrcO5A2V+87wEv5PTWxu2k2RxYHWKZ8EUZfclbv1epzqhOlFApfFqoiat6cKPMonFbe3T03saQ1zirqY1k8ia0szB1yiNvlYnxoIGyF5FmlW919byysPhQn6TnAjNVd8Mtw9f17h+EIqbSedao3VwEOzoQEJAda8Q5GWjFdIq1Ju82/0J9veXp1i1oPeDh8SSGHICXxJKqRnkT+flPPrTLBWPVQlEwDtbcDCtjPkzxKzvPKBbg+EmCJxRaJFmD1hPcjNyXbfI7paOHjAXWTNbbrQtInwUCWBKw05OHzgpLGkrvhHoSoeXl6EN5qdA6ZqFVtfuCkPv1Iu1GJdmLXpxNtGfIwyo3+35lTBn3NX5Sw9TYl1NRpDyLjR0Pw6bHcGBQPlSTSJR0FfAS5EFWKKCBwIJA2peEtDVjv3v34HNFJimcXO0qUH1LL2i4OzuwmC3oE9NYpZM3NOGNv7wX5KCpoT0RcR6QJ6VX6C1I00AxybhGHqgHuq7KVKgF2BDt7Qvc2apN8QJ5kdIBXcJ7tjQy6P3wd5Uhz18p7XrefQpXYleoni//I19RDJtgAzgu02a3HeR6KV184mxEvHyOKVr5sE7u0qXuy0glHfgHUTsFFfDTh2b1rKjSLss4fPPQlbIFosjGIwgD5JOBccd4UqbdBpQKrs1o6e5sKs7rOLh0d2MpyxgvNePEE9LbgMPAGOOOwL10VOMvkctEU10KuwVutQ29ZsZgsG/IPL4xzF47irdn5ws9nLlLk5x74RZ180FqfqUN/xxWdjncPngpuMG8bMBQ2COFTWhAsS/rK2/Q9keJYtRGZj5vBArkrv73eqRI8QFvXj0P3ApCCN9+jIaEjHyIbar4g6dxQbfewVMJXShX18kWKBB3fVTPorb27cLAz8CKQ11/Vc+0DrSiiT4KnteFTeW/hlo7rLar48AxA2VRk9VAIJrBUNDgUoKEk2MAWlLw0HbGLu3xN3An5t9p5e15cbkZ2v8c2+ssQ+GJ0MAOpK8mmpY6srtuJS3JUvdde9SJePX6rDRCZepa0VzWKntEAep8dLSq9eLsbqSDIb//cPsFbNKafB5a7WB2M7XLnZpqkdNTjYUGvOP6znYSEpBeMdsDWrFuoE X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: fcc233f4-6fe5-404b-b093-08dcb883db08 X-MS-Exchange-CrossTenant-AuthSource: CYXPR12MB9320.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Aug 2024 14:59:23.5882 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Df0qtp2IIRRfrJzjzaQPUeVR0ovLobssFrabwTHHkUoiRx1d22byt8FwOVnVpBlw X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6823 X-Stat-Signature: dhtq6s36oczdobyz6nkjbkjgg6mb3io3 X-Rspam-User: X-Rspamd-Queue-Id: 0E62A4004E X-Rspamd-Server: rspam02 X-HE-Tag: 1723215658-433123 X-HE-Meta: U2FsdGVkX19O4UemuhDEjHOb1qFT7NcW++QbkmQm2o0wFP3XuRlVFh2SnsSzdAHyza6VGD0Z8MoE7G53Nb+5WNwY43j2ggT13ELXFhWZPk+pK8k6wQ/R62323hH6gHGtGQhtG2r0ryJ2XzYojR736oWcuLHAeV1H5JmQUJk9YWKeTr0sIL0WDgcXO2KyH2/6CNjTND6xJ8Ghp7Goc23wB5+/KNoksXxSrksm0dxkouARB0fIOiFXEQhYySf4zP9uWC4P0WBW0huPtPJxqFAOQUyJSWHKVJ0O/tvb/ompM9bD/HqXV3b47x6w6afX0VKOtceWzTOCjrB2Gnh84PDF1Sz1srZyhrUAsA6Nt3qkclYlo8WJZq4A2b0gBgJRvS6rovy3ZR1AlygZUZtQT+Sn16VflKAZPPyjy6zCDKoKJKdI4hlTJdPT1MPl8XEYNN83rUkOGNq0v3Vch0kgWei651X3oRsnMNnljuwsEHf62aIP69mhF31YyjENpWreD8dq5S/t+DIdqgunfyv8e3A/1oJ+/5Qmeq60luPBvA8CtdLiI2xyJ8/YndjYHQ+a2GSz8Lo+7wtxT9oR6iAJpoVSs526BGr31OB0Xhfo0w3DvOgo6rOMeBK/f7CMszLifCrTe2zPA6+MB295JJxFOZVZfsIvkuo3l2n+n7W7FbcybI6PHZvuCZAhYjoGW/tbh3d4JPohsrNsP+s6Vpfg3oSLwqovSAmKEVJm2ltFnDQpl7tmzAB6QG8icYA2pS2QpFMOGVILdFrHw63JwBw263yvkC7YDhJCoDWcZicAFFzECCCnkxLOqsmkjWbK+aeg9K2fP7CfbAykTjBuo/2JTgn9IpxCymjTsOQba4vmtk0HrVU1FK/L4lv99NbVlfnyvXnqq/LLxfa8vqz3ewAPk2HKHhYjYoXn+/LTAgJko83pUVIl/c6Jmfb3hA1XEEEQjJB3mfGc+UfBDpvm10xnbJF CS6JdzLM rDvg1YXkGBS1ddaUnUHs/AhhF5uUbCDqf4UhTb6yLv6mtHhWHz75D4idjNqbLw9syy7ILxGw6/ZfI6EbGOBbvzZsEao9fvu1AfFmu/X+HKTBDEMU9txi4q5yZ4d6Lm0cd5xdbLOg7C7QN9YOqPOwX1mo4qnpEaYvI0hx6wkDtCfB77PWiyWhGiYEbtG4J3DNTSVt9sZjof1ZhyjclNKLBWHckMfvaob055QmZlGDVTPE2Y5aW1DHMYi1vgxD14atJuMs1pUHjh0GxSJCJ4WYf2KBC1vLke7dR3RS3IpHBCs/ITV9r5qEOn6x19HCwWXIDtZJeR1CLSXpvnEYKz/H9v5HPjZvz0xMHwl08YAc5LGVpjpmXYlA8oWUkf95fZOENI4Ma X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: do_numa_page() and do_huge_pmd_numa_page() share a lot of common code. To reduce redundancy, move common code to numa_migrate_prep() and rename the function to numa_migrate_check() to reflect its functionality. Now do_huge_pmd_numa_page() also checks shared folios to set TNF_SHARED flag. Suggested-by: David Hildenbrand Signed-off-by: Zi Yan Reviewed-by: "Huang, Ying" Reviewed-by: Baolin Wang Acked-by: David Hildenbrand --- mm/huge_memory.c | 29 +++++++++------------- mm/internal.h | 5 ++-- mm/memory.c | 63 +++++++++++++++++++++++++----------------------- 3 files changed, 47 insertions(+), 50 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 666fa675e5b6..f2fd3aabb67b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1669,22 +1669,23 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma, vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; - pmd_t oldpmd = vmf->orig_pmd; - pmd_t pmd; struct folio *folio; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; int nid = NUMA_NO_NODE; - int target_nid, last_cpupid = (-1 & LAST_CPUPID_MASK); + int target_nid, last_cpupid; + pmd_t pmd, old_pmd; bool writable = false; int flags = 0; vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); - if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { + old_pmd = pmdp_get(vmf->pmd); + + if (unlikely(!pmd_same(old_pmd, vmf->orig_pmd))) { spin_unlock(vmf->ptl); return 0; } - pmd = pmd_modify(oldpmd, vma->vm_page_prot); + pmd = pmd_modify(old_pmd, vma->vm_page_prot); /* * Detect now whether the PMD could be writable; this information @@ -1699,18 +1700,10 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) if (!folio) goto out_map; - /* See similar comment in do_numa_page for explanation */ - if (!writable) - flags |= TNF_NO_GROUP; - nid = folio_nid(folio); - /* - * For memory tiering mode, cpupid of slow memory page is used - * to record page access time. So use default value. - */ - if (!folio_use_access_time(folio)) - last_cpupid = folio_last_cpupid(folio); - target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags); + + target_nid = numa_migrate_check(folio, vmf, haddr, &flags, writable, + &last_cpupid); if (target_nid == NUMA_NO_NODE) goto out_map; if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) { @@ -1730,13 +1723,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) flags |= TNF_MIGRATE_FAIL; vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); - if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { + if (unlikely(!pmd_same(pmdp_get(vmf->pmd), vmf->orig_pmd))) { spin_unlock(vmf->ptl); return 0; } out_map: /* Restore the PMD */ - pmd = pmd_modify(oldpmd, vma->vm_page_prot); + pmd = pmd_modify(pmdp_get(vmf->pmd), vma->vm_page_prot); pmd = pmd_mkyoung(pmd); if (writable) pmd = pmd_mkwrite(pmd, vma); diff --git a/mm/internal.h b/mm/internal.h index 52f7fc4e8ac3..fb16e18c9761 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1191,8 +1191,9 @@ void vunmap_range_noflush(unsigned long start, unsigned long end); void __vunmap_range_noflush(unsigned long start, unsigned long end); -int numa_migrate_prep(struct folio *folio, struct vm_fault *vmf, - unsigned long addr, int page_nid, int *flags); +int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, + unsigned long addr, int *flags, bool writable, + int *last_cpupid); void free_zone_device_folio(struct folio *folio); int migrate_device_coherent_page(struct page *page); diff --git a/mm/memory.c b/mm/memory.c index bf791da57cab..e4f27c0696cb 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5368,16 +5368,43 @@ static vm_fault_t do_fault(struct vm_fault *vmf) return ret; } -int numa_migrate_prep(struct folio *folio, struct vm_fault *vmf, - unsigned long addr, int page_nid, int *flags) +int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, + unsigned long addr, int *flags, + bool writable, int *last_cpupid) { struct vm_area_struct *vma = vmf->vma; + /* + * Avoid grouping on RO pages in general. RO pages shouldn't hurt as + * much anyway since they can be in shared cache state. This misses + * the case where a mapping is writable but the process never writes + * to it but pte_write gets cleared during protection updates and + * pte_dirty has unpredictable behaviour between PTE scan updates, + * background writeback, dirty balancing and application behaviour. + */ + if (!writable) + *flags |= TNF_NO_GROUP; + + /* + * Flag if the folio is shared between multiple address spaces. This + * is later used when determining whether to group tasks together + */ + if (folio_likely_mapped_shared(folio) && (vma->vm_flags & VM_SHARED)) + *flags |= TNF_SHARED; + /* + * For memory tiering mode, cpupid of slow memory page is used + * to record page access time. So use default value. + */ + if (folio_use_access_time(folio)) + *last_cpupid = (-1 & LAST_CPUPID_MASK); + else + *last_cpupid = folio_last_cpupid(folio); + /* Record the current PID acceesing VMA */ vma_set_access_pid_bit(vma); count_vm_numa_event(NUMA_HINT_FAULTS); - if (page_nid == numa_node_id()) { + if (folio_nid(folio) == numa_node_id()) { count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL); *flags |= TNF_FAULT_LOCAL; } @@ -5479,35 +5506,11 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) if (!folio || folio_is_zone_device(folio)) goto out_map; - /* - * Avoid grouping on RO pages in general. RO pages shouldn't hurt as - * much anyway since they can be in shared cache state. This misses - * the case where a mapping is writable but the process never writes - * to it but pte_write gets cleared during protection updates and - * pte_dirty has unpredictable behaviour between PTE scan updates, - * background writeback, dirty balancing and application behaviour. - */ - if (!writable) - flags |= TNF_NO_GROUP; - - /* - * Flag if the folio is shared between multiple address spaces. This - * is later used when determining whether to group tasks together - */ - if (folio_likely_mapped_shared(folio) && (vma->vm_flags & VM_SHARED)) - flags |= TNF_SHARED; - nid = folio_nid(folio); nr_pages = folio_nr_pages(folio); - /* - * For memory tiering mode, cpupid of slow memory page is used - * to record page access time. So use default value. - */ - if (folio_use_access_time(folio)) - last_cpupid = (-1 & LAST_CPUPID_MASK); - else - last_cpupid = folio_last_cpupid(folio); - target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags); + + target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags, + writable, &last_cpupid); if (target_nid == NUMA_NO_NODE) goto out_map; if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {