From patchwork Thu Aug 8 23:37:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13758252 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF44EC3DA4A for ; Thu, 8 Aug 2024 23:37:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73ED76B0083; Thu, 8 Aug 2024 19:37:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6EF406B008A; Thu, 8 Aug 2024 19:37:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 568DC6B008C; Thu, 8 Aug 2024 19:37:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 37E866B0083 for ; Thu, 8 Aug 2024 19:37:37 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8ED6114039B for ; Thu, 8 Aug 2024 23:37:36 +0000 (UTC) X-FDA: 82430692512.23.2FF93DB Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2054.outbound.protection.outlook.com [40.107.92.54]) by imf21.hostedemail.com (Postfix) with ESMTP id C6B0E1C000A for ; Thu, 8 Aug 2024 23:37:33 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="nQPQ/rnW"; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf21.hostedemail.com: domain of ziy@nvidia.com designates 40.107.92.54 as permitted sender) smtp.mailfrom=ziy@nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1723160200; a=rsa-sha256; cv=pass; b=s+XPM/EnnQ6dnRwbRmbq+lmzw8MZAGDXXXaRf1gHRE6IAXX57TRKVLu1Oilb/z17hH0qxV M3bckoPLAg2WFDWxxqTA/ikPRGVkF1NS6+ajyFVVPctT0lmKpnaxr8HY5SUo+WLVRZucPZ 1M6eOyX0PO5/4EiGN+x/S/XIBx3qbLU= ARC-Authentication-Results: i=2; imf21.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="nQPQ/rnW"; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf21.hostedemail.com: domain of ziy@nvidia.com designates 40.107.92.54 as permitted sender) smtp.mailfrom=ziy@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723160200; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=7SBpyOCq2nBVCUIqsAnd9mjHJhRIz8EP39UOPaCGWQ4=; b=EweHsZQ4F1CwcvIZF7BP+9LVcm828TCr/0G40kWqShGQ+gG9SZZGYTRg/FwviCryoGpFaH BpOHxnEQzMNPM8bHkRSP9qxZxnx8js0kpou1OSuy8r/teQ5RfhWWnJutV8eeG49ZTaayLB /ray2HQRuv1VI81Z9touIXu7YK5jdYI= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=p7qnqj5sBcQRATgdLQCzG4doUSvajEV8xswTebcwjV9KaAhv+8sUA5JQfaS2EvxR4EwTfF9zcE3OV+vIwRUq8ZV3WLG+olHgFPUUpyIUkTR8d7yKnfTBHjPCQTvXqRRrpij9V64L34GCLS/lvgtu3M4A6kDFkq0nfJhG24zR4QhF9QPhslnyQ7WxDr3tY8cJjhjrMoxrEB6ISTI6ZYBwtDdqJ+L5ezHyZ4YAzw7tVpfZ0+4OIP5f1M41BaoY7A8Z8y9hZwgrF7diauvI8LfgxBi9Gy5AcSUPsIpnIVRfTHE8rpBIw+3CB0PksQy/ASfmm1AxLM3F+xD7aRKtig5ckQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7SBpyOCq2nBVCUIqsAnd9mjHJhRIz8EP39UOPaCGWQ4=; b=vAbhCssbtnyFmGznilImT5YqoyMR1bDyNbT08Z932pYOrWVJusGuSR2jjV2I2fIqxtWcVtXIGIkWymIan8/PU2mYRE6w98mPIKMrFjWbOgx0loaUpbuwS5mNXv/kgux6chxPRhqTPmsgc/xkJ+veap7bO75mQe28NpmZsePvDbavnQZRDo765JNyP4Cq1AQuA/VQh+YjPjtUjsMVkMLrOIAEq7bm3p7b9In39kwqpm9lKKH0GmiEzlDk+2YvjNwT4vltMzfgrN+NZzHte69rTvM0vPcMxVSokoZ7eoBpNoxP6/wNAb/zPQ+uEvAfrp0KnlFuKoFia1l0JA15fa0x4w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7SBpyOCq2nBVCUIqsAnd9mjHJhRIz8EP39UOPaCGWQ4=; b=nQPQ/rnWs83F6tBg3DDpaJyGWKqa62duUMKvv4nayfxg5EU7bdqSERH7LVr8MPxn+LfQytmQwTWFI6p5eWd2htgntK55eJ1LFsDwSRs7/CLfDCkF10eiYbwo1NeCWLrGnJIO8gzvbTnyaKP+XKtnRRb676hwD4q9KFV9nWoRrRfyWBC8Y+0DfmukZZGWzJ4A4dWPtO8d5dcBsmQ4W+g3p97i1cCTX0MO0XIRvYtiCub3pI/rHtCOUDS9mgNledZqg64BT9j2mF66y8kKUGp36E+V4JvyV5H3IrL4xARbupX1sfFkCY/zaNSAmJUgU9PgOi0EnEP60uI9Cm3fpuqbRw== Received: from CYXPR12MB9320.namprd12.prod.outlook.com (2603:10b6:930:e6::9) by PH7PR12MB8056.namprd12.prod.outlook.com (2603:10b6:510:269::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7828.23; Thu, 8 Aug 2024 23:37:30 +0000 Received: from CYXPR12MB9320.namprd12.prod.outlook.com ([fe80::9347:9720:e1df:bb5f]) by CYXPR12MB9320.namprd12.prod.outlook.com ([fe80::9347:9720:e1df:bb5f%3]) with mapi id 15.20.7828.023; Thu, 8 Aug 2024 23:37:30 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: Andrew Morton , David Hildenbrand , "Huang, Ying" , Baolin Wang , Kefeng Wang , linux-kernel@vger.kernel.org, Zi Yan Subject: [PATCH v2] mm/migrate: move common code to numa_migrate_check (was numa_migrate_prep) Date: Thu, 8 Aug 2024 19:37:28 -0400 Message-ID: <20240808233728.1477034-1-ziy@nvidia.com> X-Mailer: git-send-email 2.43.0 X-ClientProxiedBy: MN2PR19CA0010.namprd19.prod.outlook.com (2603:10b6:208:178::23) To CYXPR12MB9320.namprd12.prod.outlook.com (2603:10b6:930:e6::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CYXPR12MB9320:EE_|PH7PR12MB8056:EE_ X-MS-Office365-Filtering-Correlation-Id: 16a3eeaa-c63e-4dc7-b708-08dcb80311f9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016; X-Microsoft-Antispam-Message-Info: OO/tc+AHGV0H2YeEIQHBbMGSUbiNKhppi5agyB0SBDxGo+1HrJ1qXwke3XDVDk2Ze8tVt71kGLioCdV8X89/5rzOgxL7+RO1XZuVAsZ6oGbMtD15xXMNdkbnBHuhvsqcjywEcsjrkx7Ofi/yg76tmdybbdhV3XNrZIryLlmg0TgZ3v1rCiih5XO/16nl4DLVmiF+45KOgPPMDTywQB6Lw5oUiaq49t3OJfX4HH0etl2BYC0m7QgYrz1QTGhfd92CJmuo6ZmxGGL8ECkBjiDEy5MHutLbXZISCsyazpn/1ThR6G9Y7gNWBQcrrw/R68D3e1O9ZElYmR0z45y7k/94uYJXGzMdRraJ5QNBsU8JUvpyYS8/ESQOcviN4Oe45rUea694z9iFMNh1dvT8VWnjtOimHWmC4zzeJfxcHos+mXZrZbBBG1ag3KZsT3D07DaR7GPxwWt+AOAw1sxx3F8Y25nTb9yB8/PVXUyo2q6/TWd6agODwybwuufN9pvRSwumBfZ58i3gQ9Ixk1DaEnn/Y2QMQ1w7A3LonETp+n7FqZY1njkGdiyt0qEAV0VRpAnbqHPFvrWAfnlKeeYGw2U9D2GCpcC/6suHgedDKhepYQM3Bh8wxvPPcT8+fARTwBoC7rzzy45M9eXeW5F1HvtT0pNbOXL55q6lJlJgjDD35VeOzHyBwTxqw5NIQMN4CfO7NXYs4MRnLRnoKScmoNjGTDs9kgcB54KMV+xZXdX5fmmRCkqMDqf138bnR6wbZEn+kzRbKeMrsNvw2V5YgeE+Fhlu24iV28b+sY8hvOBxBvGvU18Qb8mgcwkuiKPy+yyyXHgG7eUaH6FyOyz21pGsHg17Mg+5Ylz90rXoZ1TMwiPpz4ymaH2+P8btdEHAEMYHOcpF+MkLRghw3D2i5uM9ba/FcdDrc3unL2I3UnkHlO09BKFV3qeKaix5dU1VyxZTDVRQMLB5fmJkALGZLsIhsCaOUOEBAfcZELx9V/0vj/vRtlHz80evBKht5TONdTnl//xDmrNcQZZYrLxZF6dgLBLQSZziqGLfWdNRdQwTve1U4DpMEb9iqmJpM+aOG2nxwVerVBbPbycmrwCdFQ9VHAY6K1cTlYckgsTxRtBncyAxsphJrrFHRtpWJJNxTqlUU7cQ47IzBVOXJWRgv5267R6fWaSxbR1aoGOFz0pI6b2fKrOQeuJ5j8UHjFtT7/PzaBY0OtuR3IoAss7gFW6ihyqruo8sqxRaUZSrMai18DTqYqaKRgN/PCiJCvb+OYAtL+zsJHx9mr/MzlRA8MjbTS0LZHK8ZEIt6ziR2XmT/ATYWoVhRkkifE6lWd0R/SPksmKevL6hYCd4K4Vyiz+h4A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CYXPR12MB9320.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: /EVWsqO3kZI+CsP4jQxwVlUuB+KK9658IF7JdronmbNlyuwQ4t8bIiw9FK7rIb1rMclduHjHOAs/g2FlnZAUShfBpHWOU0p3ZJfvcoNG4F+vBtNHYR92q7NPf2DGzdiyUn+4dAY2FBio2ZP5qEPRvkrZdrHmMBEMYN+BzQUpv36YS6ziLHqKQXyn/rk/wVm8ou2r9WjxlsyX1rhuFaPd1NhV2q8t+GqUc6lcKtahXyUwjQb08bR+RGu2/qsi4cdHbTsVa3Yid2SWSkPl5noD2Qqj6hNjdZzi5RIUAYEYuAVm+U+3kdGBZNZ5Z0bYeJnfmiav+H0iVJSo7f0g9yf/zkqLHAxIOjuq2lTpuFmO4dUco6P79++IVoR6tqUsSq4GfvTRQvK0VAMIgSEU4AxJCuQPUMBy5DVcdCZtlY4riGXQs8+2yVIClpqaZt0hxMEvyNRQH8/5WAlRf0weAsDepwt9mmzgsB83H22yPGdZcV4Fi6DF3cMdUInLak3bLARml2uAnnWnrNWEv1WCkti491gCBxP/BmXFvNfDHrl0B+AE+hKwOnDNF2Uv1jxYDZj98tNN8wXyOOH0rPm1jLNPvn35RtAYXG40bHQHbkhTNsbAZMyAfv+GxPO7pTGznlT2Gd6poND58XEdlpCSibjLwBQbcXXVfGKeIh3PmQ7sxKj9m05HWLm0PjYc4xdJ4mEp7b4QqQQeTOU88O4yoqgysvAC3IVUbRnk5Mr0ydb8AR+MVdflgvKAFgzRineXHssfvvBYOgdyVEj6HohC7r83SAdXvCA2hTmjX+jwkgiNMbK2Y9zM6XJICj2+OAxgv/3pqUAh1bSV9UCYtvn7g0u9uYnVSF8rcYy3ImOSoj0Gl1wWBT+mHaNT72XLVcr8SQWtWyv4pkzfw2M4eP9KzlBIIRezZ66tAf+GusDm1cpjp78e0CNNwvVsUx7CmIkdwaHa9hkCuugTCc178YAy0+OoCP/DN84kMlfyFKy33YzArMN7SDhYebt0rgcpoHLrd5VWvLP6yUyp4/ePS2l7caTuTHEweiwKjeL8ioY3GT/DfLSZTovXj2fmMTSZqFxMEaOvMZlPhmK902XOKXhD6OCKmjxwsLcoi0vnt3PYthloN1CDqQeL3PW0IH4Tbn+uOS8r2ZS4Sw2AQdtxlXHEuhpV7I0tk6XWof/w/h2Qk8jaugW0/aX3cZ5EmTIHbEt9UoUYc0l57DVnKG/0rSZZQY7PLyn8mpMIZ/11tbUTmQUOwu3Hp3oozg61G06GRu6ZU/eHkAvz7tNip10WdADiYhfCHF6AkYW37dRLHfLyr1NiiPFVBDcAK47B5bBP6a5FtKj5/fKyLWOeSX535MJ2EGCPhy36ykRci0MLGnXTuxeI8td55NJzkG/N3onrwjnoLVbMr9IAdOptfUt7VWGd9fAW3Y0F1d08kGwXt4l5eXKihvII99+sFoOxrv1rnrxSalE124u5FYXlr0htCm3h5krl3k/5aWCbGgt3NenS3ZMSFZepl/bg7ZTB0yaOWTy0Xj82mNqTEW2eyDNA64RcPV4WblyW3l2wRK2GCJ07qB8XsBk= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 16a3eeaa-c63e-4dc7-b708-08dcb80311f9 X-MS-Exchange-CrossTenant-AuthSource: CYXPR12MB9320.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Aug 2024 23:37:30.7489 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: N0qa1zOQZxiGjFqUyGvnsigjSudsi2is6Baax1xLYoD+ja41Bv41zCEOCy5jznEI X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB8056 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: C6B0E1C000A X-Stat-Signature: bu7n43suqbcqcembx6xem8hzk19gbt79 X-Rspam-User: X-HE-Tag: 1723160253-778396 X-HE-Meta: U2FsdGVkX1/d+FUwZZIxhZd2Yw+KdwyB38CrCisOsZtB92yIl7bliNgYD6tOuxDKn+zhD5rb55a9sh+PLsMTHf29+FTucVrlMSumcTQnCmjoYzN6ks/KRDBfEQIVZm8abXlYd0SNiRaAHYSO84Xym20AMlfeK/RG9ht4IpQTd+L+phVBWnHtITmG3Cqn4VDlsqxq6xtP0RYmisxkkOACrsNzI5+zOi+K04n1cjbwAUF8KBvgDxXuO2KdYPi/zoNqETVKEtxn47VXtU/4xsh+qCNXMNeLqZ+NeBh+gRsv97NKoyGwreHOgmgtU8rmjixlUTfWRGJxlaExWQZn15hxwbEKiME2zwf0kNpAOrDYwq4NWfR4QOAgS7tuQ+pl8YBCIxE4OLaorJXsPhHF7QstIKKfpf2p7AR2+kpQ7wUCATk+R1ltFTi+vKVhPKjBT/lxMaTn0OBMn1EUj08UQZ2MJlWtZp1BpG7JgOmcGIB2U+NiD/g7Cful+hoB57q4cSYdojIhWL6OVnrWB+oKAn4QiTqmJXadtK2UiSeSXJVF9oTiXKPpSAXvS0OXEZKwR/cGJHKtYga0PUMbfkQAQMDNeXEuzAJd5ZV04xzb61yTX8VdmKezrq41uocgk32WLh6Vz2SQZEf37yVsByB8DMOosb7E478+kvDtueSFxkzJSPW+b+ip5aWXkVw/9u+h5suBIuBvkas8Fv3RCMTdL47wjoY8KwntqXtJL9l3OWcbRklgu/2L2IETzbXrF0M/LmLlJlsr8J0vl6t3dy+Aw91X45QdrFu4KoCWWgsLI3jefBHfVeorsGKc4910jj/I/bIsV9ejKQ253VMRO8GIT3tNapys8fwlD8u63AAJIsg155CsWcSup+LFClJRtzimIpHwC9Hn6GDqnOAcMxy2BnHJl6paHhiTnR6b6PuhMtiirYlVLW8hZViJvnywrcnxIaZUsvHxNikgorqvrAdUzAf wugm00lw EEVTPqFxij0D81uQB3+1gNX65KKKp13mSEXoqW/+2+Zw81USJGg7DnnBjl9K/cHxzpuVwmPyB63h2jKg0gHxAHNkR1teo1PYcO/UUbAve81cOTIdGVSCpPfXue5JAr1frZmMfxHKWLBnB6k7LbiNSiLO8IrKGLVGKmyURscBmiIYGFqsaKTXc68rrBbGeYEWqpja0A5NSH5AIbm74IkW/SKqnppDlmqXizWFCWT101c0NcpQDSTUtVEWXLahR4M1AL7P2eV0mxGWSQ8Sn8WMSC51Jo4bQLIlk+lhf7xdrAO7Dl1xHxNjNZS+Vc/10zaN+8Hd1ez/gspVaJNXYUmhKSFBWZg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: do_numa_page() and do_huge_pmd_numa_page() share a lot of common code. To reduce redundancy, move common code to numa_migrate_prep() and rename the function to numa_migrate_check() to reflect its functionality. Now do_huge_pmd_numa_page() also checks shared folios to set TNF_SHARED flag. Suggested-by: David Hildenbrand Signed-off-by: Zi Yan Reviewed-by: "Huang, Ying" Reviewed-by: Baolin Wang --- mm/huge_memory.c | 29 +++++++++------------- mm/internal.h | 5 ++-- mm/memory.c | 63 +++++++++++++++++++++++++----------------------- 3 files changed, 47 insertions(+), 50 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 4e4364a17e6d..96a52e71d167 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1669,22 +1669,23 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma, vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; - pmd_t oldpmd = vmf->orig_pmd; - pmd_t pmd; struct folio *folio; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; int nid = NUMA_NO_NODE; - int target_nid, last_cpupid = (-1 & LAST_CPUPID_MASK); + int target_nid, last_cpupid; + pmd_t pmd, old_pmd; bool writable = false; int flags = 0; vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); - if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { + old_pmd = pmdp_get(vmf->pmd); + + if (unlikely(!pmd_same(old_pmd, vmf->orig_pmd))) { spin_unlock(vmf->ptl); return 0; } - pmd = pmd_modify(oldpmd, vma->vm_page_prot); + pmd = pmd_modify(old_pmd, vma->vm_page_prot); /* * Detect now whether the PMD could be writable; this information @@ -1699,18 +1700,10 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) if (!folio) goto out_map; - /* See similar comment in do_numa_page for explanation */ - if (!writable) - flags |= TNF_NO_GROUP; - nid = folio_nid(folio); - /* - * For memory tiering mode, cpupid of slow memory page is used - * to record page access time. So use default value. - */ - if (!folio_use_access_time(folio)) - last_cpupid = folio_last_cpupid(folio); - target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags); + + target_nid = numa_migrate_check(folio, vmf, haddr, &flags, writable, + &last_cpupid); if (target_nid == NUMA_NO_NODE) goto out_map; if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) { @@ -1728,7 +1721,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) } else { flags |= TNF_MIGRATE_FAIL; vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); - if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { + if (unlikely(!pmd_same(pmdp_get(vmf->pmd), vmf->orig_pmd))) { spin_unlock(vmf->ptl); return 0; } @@ -1736,7 +1729,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) out_map: /* Restore the PMD */ - pmd = pmd_modify(oldpmd, vma->vm_page_prot); + pmd = pmd_modify(pmdp_get(vmf->pmd), vma->vm_page_prot); pmd = pmd_mkyoung(pmd); if (writable) pmd = pmd_mkwrite(pmd, vma); diff --git a/mm/internal.h b/mm/internal.h index 52f7fc4e8ac3..fb16e18c9761 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1191,8 +1191,9 @@ void vunmap_range_noflush(unsigned long start, unsigned long end); void __vunmap_range_noflush(unsigned long start, unsigned long end); -int numa_migrate_prep(struct folio *folio, struct vm_fault *vmf, - unsigned long addr, int page_nid, int *flags); +int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, + unsigned long addr, int *flags, bool writable, + int *last_cpupid); void free_zone_device_folio(struct folio *folio); int migrate_device_coherent_page(struct page *page); diff --git a/mm/memory.c b/mm/memory.c index d9b1dff9dc57..3441f60d54ef 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5368,16 +5368,43 @@ static vm_fault_t do_fault(struct vm_fault *vmf) return ret; } -int numa_migrate_prep(struct folio *folio, struct vm_fault *vmf, - unsigned long addr, int page_nid, int *flags) +int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, + unsigned long addr, int *flags, + bool writable, int *last_cpupid) { struct vm_area_struct *vma = vmf->vma; + /* + * Avoid grouping on RO pages in general. RO pages shouldn't hurt as + * much anyway since they can be in shared cache state. This misses + * the case where a mapping is writable but the process never writes + * to it but pte_write gets cleared during protection updates and + * pte_dirty has unpredictable behaviour between PTE scan updates, + * background writeback, dirty balancing and application behaviour. + */ + if (!writable) + *flags |= TNF_NO_GROUP; + + /* + * Flag if the folio is shared between multiple address spaces. This + * is later used when determining whether to group tasks together + */ + if (folio_likely_mapped_shared(folio) && (vma->vm_flags & VM_SHARED)) + *flags |= TNF_SHARED; + /* + * For memory tiering mode, cpupid of slow memory page is used + * to record page access time. So use default value. + */ + if (folio_use_access_time(folio)) + *last_cpupid = (-1 & LAST_CPUPID_MASK); + else + *last_cpupid = folio_last_cpupid(folio); + /* Record the current PID acceesing VMA */ vma_set_access_pid_bit(vma); count_vm_numa_event(NUMA_HINT_FAULTS); - if (page_nid == numa_node_id()) { + if (folio_nid(folio) == numa_node_id()) { count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL); *flags |= TNF_FAULT_LOCAL; } @@ -5479,35 +5506,11 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) if (!folio || folio_is_zone_device(folio)) goto out_map; - /* - * Avoid grouping on RO pages in general. RO pages shouldn't hurt as - * much anyway since they can be in shared cache state. This misses - * the case where a mapping is writable but the process never writes - * to it but pte_write gets cleared during protection updates and - * pte_dirty has unpredictable behaviour between PTE scan updates, - * background writeback, dirty balancing and application behaviour. - */ - if (!writable) - flags |= TNF_NO_GROUP; - - /* - * Flag if the folio is shared between multiple address spaces. This - * is later used when determining whether to group tasks together - */ - if (folio_likely_mapped_shared(folio) && (vma->vm_flags & VM_SHARED)) - flags |= TNF_SHARED; - nid = folio_nid(folio); nr_pages = folio_nr_pages(folio); - /* - * For memory tiering mode, cpupid of slow memory page is used - * to record page access time. So use default value. - */ - if (folio_use_access_time(folio)) - last_cpupid = (-1 & LAST_CPUPID_MASK); - else - last_cpupid = folio_last_cpupid(folio); - target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags); + + target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags, + writable, &last_cpupid); if (target_nid == NUMA_NO_NODE) goto out_map; if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {