From patchwork Fri Jan 3 17:24:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13925733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94F9CE77188 for ; Fri, 3 Jan 2025 17:27:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 336746B0098; Fri, 3 Jan 2025 12:27:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BF606B0099; Fri, 3 Jan 2025 12:27:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0EAF66B009A; Fri, 3 Jan 2025 12:27:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DF4EB6B0098 for ; Fri, 3 Jan 2025 12:27:48 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7E11E807AC for ; Fri, 3 Jan 2025 17:27:48 +0000 (UTC) X-FDA: 82966820412.01.8ADBF23 Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2051.outbound.protection.outlook.com [40.107.101.51]) by imf01.hostedemail.com (Postfix) with ESMTP id 3C5A940016 for ; Fri, 3 Jan 2025 17:27:06 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=ZUm3JNYu; spf=pass (imf01.hostedemail.com: domain of ziy@nvidia.com designates 40.107.101.51 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735925226; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pkrR6pMPjoPUyRjyi/crq5jutxBvrS/X6+z25YveCkI=; b=7+X3Vwpo893ONwUtqNg9iyZ/V/AOsAlnh750B7WMHYVeni6Uhi0NyM4+K5nj4f5dh8vBir MR/kIQDQnfzhgXHNT5+CzDvZFaE7zgIhLPL64fcrVc1M2vnxhGMuVK0lTJXS5la3QBKZZX OhhEHcdDVibm4G2YKXSbfNIo/0dj4dI= ARC-Authentication-Results: i=2; imf01.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=ZUm3JNYu; spf=pass (imf01.hostedemail.com: domain of ziy@nvidia.com designates 40.107.101.51 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1735925226; a=rsa-sha256; cv=pass; b=VwnjrwCw5Orn0jguzJLun2OyT0XbRGCWGumbKxdet0SnBgaxSuY0jPBtqXSQbBzvymxjpd F1s+f4RTZo5Pt1GkerCQhIdT42laqhc04mrRSSRXz1ZEnEitcAm7guV/i78gcPnacpxgqA 1S8qyfKbfR58kehr6U/sa7Zvej7KSU8= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PyEOL7Mq+daLVE+woSiEnU6l+7zecW146HG88Cf/PHb6oa0h/irCF3pnNsbqT+508ZGzpjbW2P4ngvKXYs0Rbe99u5fA35A5xhmHRp3FhqNG4XxPtI8yfp0F72OFwiZYFa0AQE8FuJ2muAaKhaLNGdFt0dUDhBGa9gUwho3phaZ+WpYG5rMni4Sk7upTlaI31cDCFwNGvzKoHpZPx53p+3Aok0OnJydXCSVPWIu+8lh2ZLqS5WPBEqYjtDKBU5KeD2f0+YbGQunC16NCOE5uBdWRx8PCGfjMFoUyGYFVVTBqavQq+jC3aKkruU+cQZs+RQvVVxkU54iTPNP48AWU/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pkrR6pMPjoPUyRjyi/crq5jutxBvrS/X6+z25YveCkI=; b=GfS90n7WeS+XGai41l30ArSykwx8b/RzGsJV+PILyqhSLioQQOxRHNadWWtW19Ruqb4O+RttBxnaEgTG1c7CuWXKFe+Y/lrF0jOcrQY3q2nUt+/a99vRB0eG1aVBG9Che2/ssWh/J/53UM5LQGdSaIh2adW3sZog6zfdUqi2rT5oPcX9gvBVp3equV3MgMORfo++5jr9X/2vmIVgZMS7awsBr0DbNF/eUwiQelrMm0eJJhe98UMytVRYAcQ21pcxlyNemO9M8kfP39apy6kYdlAAUkhDBfBQUkMiIspSRlQqLy0/ViJ9h/S1XCZmJJFpNWW3IFi9eFQ9FNBifARLyA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pkrR6pMPjoPUyRjyi/crq5jutxBvrS/X6+z25YveCkI=; b=ZUm3JNYupyRAG5V7LDQVR3T7WKX1lZAXXNn5rvF4E3CusAEJ8joVKMqTeezGBckXuJav7frH2jRZ8jIIBw/umGIWGAlGLVZmO2qsxYvPQdSK6jjRtz6ZJZK4B3nsP82Xxa80JuZM8wXhXPv9/h+RvZSZtRyg5lQsE1J+JCt4B5gspa5a9/NegiBoXPRiuLbLEWutNFmJ4svnmcC3W/9VtqUZhbd5Okw7lo6lLaeMt48J7IWkcs5Dhl+tL7OUeveAGplIpnhvOVA28jle3L+/fqa0ncCHZJDZqsL5kFL5HOhjHfXdqwxeQ4czluZ79B2sAk2mX32r3/pOE7PsZfbPtQ== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY8PR12MB8066.namprd12.prod.outlook.com (2603:10b6:930:70::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.11; Fri, 3 Jan 2025 17:24:31 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%3]) with mapi id 15.20.8314.013; Fri, 3 Jan 2025 17:24:31 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: David Rientjes , Shivank Garg , Aneesh Kumar , David Hildenbrand , John Hubbard , Kirill Shutemov , Matthew Wilcox , Mel Gorman , "Rao, Bharata Bhasker" , Rik van Riel , RaghavendraKT , Wei Xu , Suyeon Lee , Lei Chen , "Shukla, Santosh" , "Grimm, Jon" , sj@kernel.org, shy828301@gmail.com, Liam Howlett , Gregory Price , "Huang, Ying" , Byungchul Park Subject: [RFC PATCH 1/5] mm: separate move/undo doing on folio list from migrate_pages_batch() Date: Fri, 3 Jan 2025 12:24:15 -0500 Message-ID: <20250103172419.4148674-2-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250103172419.4148674-1-ziy@nvidia.com> References: <20250103172419.4148674-1-ziy@nvidia.com> X-ClientProxiedBy: BL1PR13CA0198.namprd13.prod.outlook.com (2603:10b6:208:2be::23) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY8PR12MB8066:EE_ X-MS-Office365-Filtering-Correlation-Id: ef0ec1b5-013f-4c3d-eefc-08dd2c1b7c2b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: NjlRwJUvYdEK2xcKV5aPKMLbHv98gJMVCK1UUfPD1/QZA9WbahVsJnSLNrsNFWbWzGL+wN8AzxEiFjJM9CmzNRMf4JGz9jybtA48zo4tt/cCFUkmKqcQ4hHtk/G3OqwqwiJuWHsy8p3ILMTTgt0blRjbQJvXTXItxwMvTcD//CZfb8AX/UZdDMLzb72568YpPpLsa9ZWVwutEYVLOcaSFPP8Q2dR4dZspXaaCCjp0Fva7GvqYiMVQV9oB4HNDri39QsGuWan9/xJcyeRRmxp7dDbV5uUnbO8HxxOEIeoxufGyAn/M3ePhqb2OSIp/2qvqW3pMou2mHShwPsYa5fMVYT3Sg95UA8augWzzuLKjE4NyUlbVAvp89n6prp3TtK6Ib7/mOceMqD5O0v91tv6I9bEBioAN57xF+1lfDA8ZLDUVk0IxiWWLQXbPXzbFEgejt0Gexx2rGPnEcJxtozOpQc8zkRxapU1UVlkkQnkyvAWg6us8owKmsIht8ECUjCOfNmOdJI2vvubgxmE7F1rbd5FFXo0SNvh7hAgSjtsQ76Giqoi1dHL0WRR25+D0EMlU7w9+WNRciKMzHDcyukKHtXpXsO9xb6osAaSqTHTWhb/Rhj0ibhDuxp+eusA9EWFSNHNYcc0P0TTxy8BzqZBND62DHn/HNGk2jjUCTG0wnghylVFfVwzWmoiv/kLkZKqVUhib5yQ01ZeFoiSZ8pj7xdfbXl0aChgbMDMLuCEyE8zKQocEjwd//mwhXmmicrG3UkW2o5kY54x+CyJAIowribW1PIQc1BXB0wDAt+c5eRmXBCcPW+lFoWDe37NyenJHX4AzpHPWSADl0a8ythIQbwChPGosYBAdRai87cDp0aoW31ASXBEvF2hhSP/sqNlaukJ04+JAlUugbPiAtOAsgDE2uvSjR/1kbr1rmxtDn38qVpU1XxXxmy6tzFchbH2jC1P9PfDG+88w4fXYoV4Qx1zGDupFd5ATV7E2LWmAcY5jPVuQyjrjDj4biSaOWbns4r9hr0XCNGA3kId3qB9rmsxhAqaHNyFYfHsI77mfaU+OK1dioiJTKjLYg6XnEbNDFsV1nhU486h+C1lV1g8h928lqGD1pSO5E6fnb/B6BUFymOt0lfd/MPoGZI5tpbdxnp5Y8g/IKJ9b+eXIxtKFH53u+sK64GiqqD0+nQRYZkriIqz1xSOFOBZufRXEXbozMG1ZgRGJUB9svhgtqsfeBm5uBtJiKS2+0DsIrmB1GnNVMsrPqzAgIJ6U56K2o8PJD4nZDWiqROduL+GmvZdzamY2OCY2jQVoZOKYeMHf1K5u1i49ZPFM/MhK0ComJD4ZFTOzN8Yy9LkrsaACchclDDbgb24TdzBoPKCQmE+u8uhkNtOJK6YHbxQWy7hX5B/ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: C4o6Mww8vDCuscw+/E8qZV+3z4cbmevKK6xnRKxokzE/PvdgUijCbYy5c7y/JJUmZ4YvyM9hDfgw82HVskrGLR0cvUBXAsKvuRJfs1WYYaNm776DtHja70eSrk7sPUtfciH52W/cl6S7uaAGu9+aLloiBBl75wcYk+IPHr1mvnBdTXN/LzLgsXmM1xjcyX9xX/t48avKxuATloMtMBQMlV02Kwg4jUZZGAS/NJOvxy12PUfCVEy+JicuZhBCnId5OzhWKzwCaFMcTmNz5ORs1ewE43gWde7r/gCGilwFQtuxQIDmTZYWyCSz/qUSE4K4qCQO7pNjHwJ6Q6YXXVeOldycOTivRDHlHc+7HowWoc23QHGvcImc2ECHMuJkOjTlCs8FG6Fbre1v3t5fR9qfcqHgSJEnnJXYJ+1hX9xN9OGVOCLq/6nEsoLI0+myweVoBvJHxUmF7LzYMZE1IpT5cEc27pEHno1sgwYDdccSAdzKLaseUB/OnAS2xyVSswfRO4p3Q+lZOqC4NzD0803WQS/Cmb8wqWBfGQAVV4ipH5XJcKxiwcmnETHgxrrD8ovWSpyBHwgAJ1ELw3+UScVY0l4g9UC/djqnVXKqtIV4B+O+svZvfV/OhzunfHTBtv750xSyAdblE2MzGdMC+v8WDwUwf4r8gRttOM6Gya0+g99ENAvVr/P33DAmEh/QSnu+F3tNO1T6k1LwaOZnElRRl0/GFfawOpRTQs6kkl9I8ft/Fb5jl+j8uEgEnIqbSPABUX9rzaOIGKMNa6cULLd1uDBPTjpeoxm7D0EWvWkH9pj0S10WjwvMPO2hOmTfhRnr+J2G/4+fGeWt4Gxu78KjRStpVxoedRsnHfL+8CsDhyfsumi2R1QsW159FEt/57Bvt2RDi/w4Pndr4h/SESLacBz0+7Buob+N8B+uJxAZmnpOtcBp9gxxe6Miif2RXJ21ZWXh1rqMhz1mFIOxE5SuPWxzlBf1QuKXoOPTl2gN1kdpHSV204tNrOYw9rKapUTZE1L4tg+3B5f6uCZtZp8usKd8g3jqpqnNGIhDySdrvjL1wL7ymhY67DuB1Q8+JHzJ8tAaeHrVFaEKxa9Qsiwuzzm3HPyAKJ4P6tJdBxPw3jziGmQ0gqgorzI60U33U7z2QWv0/TwBcLTgjs7e2FF1U94S4+pQfKgPF+xvnx/EzK1bsaw5P+VVIQnZ9eVbVoO23nW+0o5N8Ifb0SZwVTz+aQ8scyNjPkFfKy7x7jz7QquU3v61FlpLRYQezcto/wnvUGxCEGN1dIW+VHxpNd0cKP4oX4xflQSvgdSALKQYTs0tOHJouOWuEFCZWLJ7kxAJRwUCfXUIno5XvWpC6TlVm5XYGcerWtXOektw/86xJC/6V/SSrDYAyKBfr0aOPofSUQUhyQXRJjP2Kr24sd3PTqxtrDz9EyvzQzzUQzdbGHqqYYG/PvOW0BQaIj2CCEVzlRVFhDTqIGDmklVQgG3jpZzb9LiHHwsx2SjfB6BCGxt6vD1lQtwB5ZiI+nB8SM8aKXJF5M/bXQFi+Jb0WTCn4UeWzvbD64EYZihgARf8pPvrv0W7t+KVNK7SxFKEI2Fx X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: ef0ec1b5-013f-4c3d-eefc-08dd2c1b7c2b X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2025 17:24:31.6742 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: I6xYX+Lz1Vmthy+Jim/RlUfHpXds5oltnukYHBjWOyjz+c//UQBxdiPjuHZzkh9M X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8066 X-Rspamd-Queue-Id: 3C5A940016 X-Rspamd-Server: rspam12 X-Stat-Signature: fe8565uxuymna7eot5to5w6eiwhd43xh X-Rspam-User: X-HE-Tag: 1735925226-76549 X-HE-Meta: U2FsdGVkX1/7JlmGsp1F/kmm/1K7DyZpAzaRv/Q3Re17JBeWDjYXWkxme7qcCkXHhifmN3Gk4HtFPjUV7k5TpNRZGGuPHdysEGUdfunrAXQTDUMooL5Zgm5FJs4eFxqSmy4nSQnXv2QvIfRRMHDdvLhYAyH4JkxEBzyYZJCfVKflMCLH/G763ED7uXreXDSBTYtYoy1SIj4YKAinq/Lgkjvj7Rximea6COMG91Krxjpsdlny3jqODwwuS/jsVjEX/rrI5pSzR4L46fHdBsKzeGg2QQuBaVxP/pGaTzceZBwGmxprtT/7DKeCqUR8ZKuxZrbPRzt4SLZXtHdMWBSY7I8aHuKn7Wns0ElcYNwXhWrgm6beJVMj58sCnoPbGG49hBfoVvpQGc3V3bfKIapixg9xc4E4U9tTutsp+6b2k5XnnDA7nx/D5MaDj5vQ5q5ygQ1CWX3IdIMwxcMfBS153XrIHvpT/YA6C7GLAe9BQKSVKbqt43pxy7fUoN7CfXHMEF75+hpcq23KR5+S6MHH/h9pz9U4M93umSJeUGWukFZ9jd7MPYoKM5TGZqjE0z1Omy1IZl+lIJbA8EAjR0d4Fmqe6IxqAsPI9zpZWmmicQY6Ub0hPEAOLTHKMw0Zj6w4ecpRIaOzmJaZzYpUGZEu8uaAfyBM0p0O+00mTpV7j5YPYfz+t1rndOiaQzHNqamHUKvrja4cotOLKhfcqEdhO0q55U2Zba09W4he4yl3nkFtDg+4M5sv85D6yvG6bpEFLl7NPKhbYMVyeSNPZAlgleixD1wd0PopWpmvFQaUcfX/BTRJymhzm1bTCi+Dre5r4fbJAVkRUX4ksw1MYQhqm7zeuyzUFmtkrNDcJS0daXyxMAqAQs3L15Lvi67reLd1Bu6dYFOn0WntaeLJev6Jj+cox44z0Z6X64Sp54yyQ/DlkS2FVAF0QuBjvLaaSBjWx6kWfQLFQVSG8hWIWit Jn/srE/l wXiY/k8CnFg8H9tmztvON5DC4HCxDBYOtWoNsIFd+m2DLJai+3xStgEXUriBvgUbVqtN9J6sPTK8xTofx9amtmuobwaneWu+0+RwrI9T4r6O96r97/BYCHhd9H4/6rj/n9/7OIZsFtpIUSHmnHMB1C4tjWGWipTSDU/5uUUGUfgPAbu46eRb/6hYO3mME+rA1jAdoaTj5XqD8gXRRltnny0u8Ilh24p0eP9+0ChHxbxl0Zmv4I1WvRvQbN6jcO6s7C3AlKWHFGTLCcBlefoLiFkh8svNGBPG8HPKBGi9fa/EG9SBq+MgsSp58NaARC6DeuobY7B9cAFHJ9xbYkq7cIPuXW4O7EUZN2b1P+z8dj5mQ6g3lWs4G5dLs1cV/3doGVdd/DciTtL5Vh+O/fGQ8k/a2p4cNW72EfgbJ1oRs0QaV3LK5BRnF9ZFHZX2/Uz2E7jgqsK6X5Zcwbo1Pqd8J+WfCaAw7ktzyCWgeKY3x4TOug3k+mGZ4NFfVorfvn0YizlWZ1LiSU+aXf8U= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Byungchul Park Functionally, no change. This is a preparatory patch picked from luf (lazy unmap flush) patch series. This patch improve code organization and readability for steps involving migrate_folio_move(). Refactored migrate_pages_batch() and separated move and undo parts operating on folio list, from migrate_pages_batch(). Signed-off-by: Byungchul Park Signed-off-by: Shivank Garg --- mm/migrate.c | 134 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 83 insertions(+), 51 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index caadbe393aa2..df1b615c8114 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1690,6 +1690,81 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio, return nr_failed; } +static void migrate_folios_move(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + enum migrate_mode mode, int reason, + struct list_head *ret_folios, + struct migrate_pages_stats *stats, + int *retry, int *thp_retry, int *nr_failed, + int *nr_retry_pages) +{ + struct folio *folio, *folio2, *dst, *dst2; + bool is_thp; + int nr_pages; + int rc; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = migrate_folio_move(put_new_folio, private, + folio, dst, mode, + reason, ret_folios); + /* + * The rules are: + * Success: folio will be freed + * -EAGAIN: stay on the unmap_folios list + * Other errno: put on ret_folios list + */ + switch (rc) { + case -EAGAIN: + *retry += 1; + *thp_retry += is_thp; + *nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + break; + default: + *nr_failed += 1; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; + break; + } + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +} + +static void migrate_folios_undo(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + struct list_head *ret_folios) +{ + struct folio *folio, *folio2, *dst, *dst2; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + int old_page_state = 0; + struct anon_vma *anon_vma = NULL; + + __migrate_folio_extract(dst, &old_page_state, &anon_vma); + migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_folio, private); + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +} + /* * migrate_pages_batch() first unmaps folios in the from list as many as * possible, then move the unmapped folios. @@ -1712,7 +1787,7 @@ static int migrate_pages_batch(struct list_head *from, int pass = 0; bool is_thp = false; bool is_large = false; - struct folio *folio, *folio2, *dst = NULL, *dst2; + struct folio *folio, *folio2, *dst = NULL; int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); @@ -1883,42 +1958,11 @@ static int migrate_pages_batch(struct list_head *from, thp_retry = 0; nr_retry_pages = 0; - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 = list_next_entry(dst, lru); - list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { - is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); - nr_pages = folio_nr_pages(folio); - - cond_resched(); - - rc = migrate_folio_move(put_new_folio, private, - folio, dst, mode, - reason, ret_folios); - /* - * The rules are: - * Success: folio will be freed - * -EAGAIN: stay on the unmap_folios list - * Other errno: put on ret_folios list - */ - switch(rc) { - case -EAGAIN: - retry++; - thp_retry += is_thp; - nr_retry_pages += nr_pages; - break; - case MIGRATEPAGE_SUCCESS: - stats->nr_succeeded += nr_pages; - stats->nr_thp_succeeded += is_thp; - break; - default: - nr_failed++; - stats->nr_thp_failed += is_thp; - stats->nr_failed_pages += nr_pages; - break; - } - dst = dst2; - dst2 = list_next_entry(dst, lru); - } + /* Move the unmapped folios */ + migrate_folios_move(&unmap_folios, &dst_folios, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1927,20 +1971,8 @@ static int migrate_pages_batch(struct list_head *from, rc = rc_saved ? : nr_failed; out: /* Cleanup remaining folios */ - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 = list_next_entry(dst, lru); - list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { - int old_page_state = 0; - struct anon_vma *anon_vma = NULL; - - __migrate_folio_extract(dst, &old_page_state, &anon_vma); - migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, - anon_vma, true, ret_folios); - list_del(&dst->lru); - migrate_folio_undo_dst(dst, true, put_new_folio, private); - dst = dst2; - dst2 = list_next_entry(dst, lru); - } + migrate_folios_undo(&unmap_folios, &dst_folios, + put_new_folio, private, ret_folios); return rc; } From patchwork Fri Jan 3 17:24:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13925729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B9FCE7718F for ; Fri, 3 Jan 2025 17:24:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E7E766B0088; Fri, 3 Jan 2025 12:24:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E2F246B0089; Fri, 3 Jan 2025 12:24:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA7E56B008A; Fri, 3 Jan 2025 12:24:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id AD0646B0088 for ; Fri, 3 Jan 2025 12:24:53 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 3456E1C52F1 for ; Fri, 3 Jan 2025 17:24:53 +0000 (UTC) X-FDA: 82966814070.10.7D4E19D Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2043.outbound.protection.outlook.com [40.107.212.43]) by imf12.hostedemail.com (Postfix) with ESMTP id A095D40018 for ; Fri, 3 Jan 2025 17:24:27 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="m+ivf/XK"; spf=pass (imf12.hostedemail.com: domain of ziy@nvidia.com designates 40.107.212.43 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735925066; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uwRxhfDx0PPdt2UKRR7o8d87pn7X56g+pIfPcK/e5J0=; b=6CiM7uL8LuYQnNN1xb4tiBQLcSGMyr8L9A8CKXXPxy65bMxYrD+h8oD9svwPOlDZbgOblA 0I8IidYwhNa9P27SzYdTIXY5pGOLU4qw8mwaWZWzmrvRetvWOb2Mwxp2MqTvrZtHvNUPsm VWQ5i33vf7eyy2I/il8MSLqKl9Jh7RI= ARC-Authentication-Results: i=2; imf12.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="m+ivf/XK"; spf=pass (imf12.hostedemail.com: domain of ziy@nvidia.com designates 40.107.212.43 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1735925066; a=rsa-sha256; cv=pass; b=IryrzIz6svz4jfN6XiakbmddMYKhpogbQdtcnF2rBGhJ72zrSKzJNpFuLVCWYl09nTzmsa HAQKfIVvljVV8w4tbCaYvyTZV8Vv1g/GqqGi/lXPJ5sHPDzy3Ipl2pwww4DjIAjCpQLGM4 vxlvcbVQfo3zROHZpoEnXKltZDppr3E= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=G08pdsDn6bOlabAryqgvUPXKJolk0z0xxSytV4fGa3+/C/gCFHpFotGdNGOg+a7VW30OtQPUpwzwdieDLImCZz9PQebrTHkghNHOxtYOnI5oAu5hHBnK/gamO9h86x6td5gjJHFkQ32PDQ4KyPtUxZM+sclUrqwPgHFA+I38jKCsJT4OubSH8O6J7gOcezy8iAcAVuOnsnmNkHgf0fbDeRkGuRcufItiKZ96gOOetGF5vZiceL7XtLRDvr02XhK3+isLizC2Z49seiYgsN8GWEJ7HcQZmmOTFLN2pE2lMsQAum3ZfGrBsLpoQQG9eb5QhNPfnG4EpHLrZzJIge7Y0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=uwRxhfDx0PPdt2UKRR7o8d87pn7X56g+pIfPcK/e5J0=; b=ybdCywDmMdAhGOQB67mEbvnTqpZFU4B6RP2xbLgcbEtSTZNMVxqCNV6hmuUf+24ja5tASY640Hgr20N+4vkdUrl54EIP5AZnsI2p6Bim/yARnIq/cYLqBf5vaPaWrbXJ/9NGGvSS0kRvKZD4zsx57zeY/Ds83KNEYIpPPlZm2ArwnARmood4duiCg4uID4oPjxOCPWOOAOvYBNCW+hkH/WeSPodnu47r55ywhggdN5EWZHdDW/EpZZngn18RbZgLJOa0VnnBQDVy+wA99CewMS/kkurkNUGAkVDygkv8f0kJzifPT727KTSXuxRD+09w9BgBQPc+osJy1yHZoeyNag== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uwRxhfDx0PPdt2UKRR7o8d87pn7X56g+pIfPcK/e5J0=; b=m+ivf/XKxRBnVRQrr/Rn8p+Ejp/lo0uWjDjcd1G8Ez5zkKK7pkh3TPhn+kTuW6qgR1yoT4RnbaZGqgSivgPs8PMFXzUaF/CF1fveYfWIblBJ26RZ6s7HhVPAfdMszyFWoQ2Ym7SnqV3MehoMEUbIChYbKbi1379QwhMGJNtpYko/HUjHzZM9jajQYTSkCAl4cUliM2g90cGFM5Cg+FruHywc/x4OQHPqi9faqNDtRFP+27TzDJR8oLC+QKSfkHI1t3YEQgK6IR3TCw0hJORivdH9bhtnTAKf9y1jf5N2pePIcac5VIYitteRVaGmg9bcjf2i8uQlsjSHlxOrvDXdzA== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY8PR12MB8066.namprd12.prod.outlook.com (2603:10b6:930:70::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.11; Fri, 3 Jan 2025 17:24:33 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%3]) with mapi id 15.20.8314.013; Fri, 3 Jan 2025 17:24:33 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: David Rientjes , Shivank Garg , Aneesh Kumar , David Hildenbrand , John Hubbard , Kirill Shutemov , Matthew Wilcox , Mel Gorman , "Rao, Bharata Bhasker" , Rik van Riel , RaghavendraKT , Wei Xu , Suyeon Lee , Lei Chen , "Shukla, Santosh" , "Grimm, Jon" , sj@kernel.org, shy828301@gmail.com, Liam Howlett , Gregory Price , "Huang, Ying" , Zi Yan Subject: [RFC PATCH 2/5] mm/migrate: factor out code in move_to_new_folio() and migrate_folio_move() Date: Fri, 3 Jan 2025 12:24:16 -0500 Message-ID: <20250103172419.4148674-3-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250103172419.4148674-1-ziy@nvidia.com> References: <20250103172419.4148674-1-ziy@nvidia.com> X-ClientProxiedBy: BL1PR13CA0205.namprd13.prod.outlook.com (2603:10b6:208:2be::30) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY8PR12MB8066:EE_ X-MS-Office365-Filtering-Correlation-Id: 5baec480-c932-4d6b-1e02-08dd2c1b7d24 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: K6k8+fFIByCriKA/qWkDv7j9vZknOJ6Xf8B9hwEzSVH2d1bDKw/uqbcaequ8/o9Y98hOseDu0vW+U9V7c78KG3JanA63Lw51RG2CSgXIZihW8TSReKNSSKloxmpKbGgKk3TYjpKK6fcjanfR0DUB7QiK7fO1raJQv5TdzALmQ3UpYOU1oQhJDa0BRLlDxeo7Zj8/gp7Y3p/QBghbyubSoDLZvez6lMcCrwwwXwstpNnCtZtceO/QdH/BmgoHDod4oiTyIdBqz+UBetI6eAGSYzxtHWfKjbaFkEb6wmSPlW51vZcHl6U19QWHzkuwED88RNEoirvadnTkCZ0YfTlKWSzj91OKRlcTqwwKsRimCByu4620bXC7n6XDCkbPXfPbtFBp/kyZVE93wY4bYUhMgudk/TbLMGrHbQivd0usOlv7WQY/fSp3vMPSqLPA9yLSHYU22ucZnLEwqlj9fohx3G39a5l7KFj6WeaWXpVMx543LCpp+j5nILgS9Fxbxmn0ofieyNmRbmTTPHC5A0gCtS+N41yGzaZoySHH34KlbYPcUM2+vFPiHpnsDKP4hcGv0O7l6BDZjsiKtQ1tJsNt0bVrtkI28QjIV3bB/wbop23RTm7czEyAmMqXWId/6Ck1ixLNdsCjNVcIemtuKYNIxgqXvx44noriPkBxO8iPYHi2vJ+6aJSfVnoOaT/FpK+7oU1GuaHEE9cWGitl46n0c2YgY2xe873egyX9Lh6VSbRusR5ixBpetp+hUzXBFcG3VL+Sp0CSo7xNtxCgRy7AsQzHqClVli+Q4EWtDraYkX4mt3IKp6Yi9fe0r+E0G2vAC441Vrq+3fCg4EQQPMrMEcAXg+mQKBRt848CZ++8Wg22vmDztUG+QhkFMv9kf5nCQxcg2VE4qU3J2OJGW/u21HMbOFkU/1knhZl3txNerYlmrsSE9L76QL5Z3BgkvUIfvc9X+BaKkZ8YId8TTqoxuP5ndpKERerDjPCIaOaNkBJeb6nYLcYRIcY/R1+HdQOu+izClA10l6NCPcYhSGQAar2zXRnowa9Gjt9D3S7vly+Y1vmXSYxdtullo4glbpuCg4b+dQayI3A2CBi5J8RGoNYkYMpOlgVLQJHuHiM88/XXV/51FeEaOW+GtgAYEWqY3py3d+3WxcGPGE8IpxC32cRQZmZguI7hJPjKBcI1dUBjLlZPhWIzni9TosxO1Wm9E0oXS730V0u/Nb7VDejbibnrHZj6Z6LUVU+kWFzqdnQrDrEkXRBnkQC6W8vf6TzvMqKbdDFpS8igEYRRKJW3xzBA8gg04nz1SWLEq1v1/d9Z/GUBBEarFTs0nRe/J1KmY9azfwe3eg9Ds9aT1VgGx+hqeJiDThVlqTwbVFRF7lOGX329JmgwfOd+nPtYY6E5 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: l+6eK/ORncnf5MRbZWjPsEz7ZxIxwYMkIpYUlz8Nsp0KjOFUeEp3DdjKLPrFSqH5/fcQyfW6Hq49HRPE5d9qJvqKt/2JjR/BR1WkZuikxzdOrBkmW0ExfNJ2PpwlQgWsl6NX07Tu1GPrtNKsoNmQC75zyQy6dIwbK92naZrBjJxTk/B1nssteKOS+CJXLLJRbJbW6URBvttUAeqICUDCrreO4+zlnDL0lUVWlmw3Yq6v7rK/Z59cCYLhkM3J2oTvDCyjj0TxedzI1gkwHAP4+3Yz1+h7QbVvayyE81NhIWJhd44gcOABb84+aZ3jqEQG/aa6fnHFYShj2ZkjOuUkbT4PJzXAG1ArtBTYU2zknR/szh53MxEOZKSYTUbKGBQZKI9em6T5Q2BEPvcZ3uT+mAO3CsBlxqN4XOkTUTOQDjSgGIGi2oQRHDfFh7ownC2zNO1U7JMsILu4NDG/eJpthBzaWZmz2APfZTK87PjHKQgUa0plvTcfAZ7axcPrtk93Gx36pUckilvSrAx01oiKzxJ9S7JJO5wP5jihI+DOVMBF0hlM8EaYznI4FZRmCY/747+pdkWA/yYUFyAezurzwzN4UxAJCE3+qmTWMNs/xNheZ8HXa9po6/qUHFD1axOvkWLMd8obFVOmXS+7gel7x/oP9QvVeQiTPSvm9BBybC4BTAFsryltIpxPcjV4T3dxk57iPb5X5duIEhZBrponZPDeMul5widbs7gTxrHFOpXiQ/3E/MvjstXM99f8A4VHYKVDcOssXOWEWACk9geGX8uUuejmx9Js0GiTQvfURH46PKRocnYXNAPltGpy59yMjvNXMm0NbnPw5+dIiisRMjslGE485aft9AkFL2nTZNOhylRVWC7zU6BwuQQDnBkULNk6hDUNVQRGO7FgXVOBVlTPgkP20mQO0sWWek7e8n1Tr3aJyqvbXMXQsHWpy+WkqsTAHx40jW+wDCGkUbZINDKFHWAvigP7ovMKuAO6OdjphV7+44zuK7TXaP2Hjt3UR8UHRZeul3K4G/6W+6ssTPd+YGgCoeX7dZhbskQFT8QxbzLERqOixhqc4WAQ55WTFKl2pT2z2wkuyqnwGvdbQ3CKo6ku1s2J1o9gWNY2D4RUHiNmyCKC88P9W/m8MycB1zxgEUhx0pyjTOULCOHUaETSNOzvpobYA+Q3wR6Z2T1dxsNggSnCRCTmiUer9rSWHw41BP6YaQk1atruk1JVVhMUEnmGBOBZprO1IonO6GPKFvBRULOgTABqjtswunjKZ040GRlMqxBI6XwMisQsOz44hMk2ydFzrAMInlfoOA9S72g1G5ZTCugm7MHRvMOEKwysVwB229fcQQI2loAdXUCsTdBCKz0XAO7euDGn5cYZWNlA7cWKmHrwhfSSystFoc3yrMqs2n0h6Ypcy4aMUtyymGkuoj3kDz49XMI9PFmDA5PW+b3VKr757nX8sq88nZExuVrgGs/Ybxhk+ctX/Duc1TWrhnr/EItu/ChZZD3BkkrDnQ8Mj0aJVCEZ+P73zspedbHee8OO5offyJmLzsfVylZaCdMsX2zgE1GhbVk= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5baec480-c932-4d6b-1e02-08dd2c1b7d24 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2025 17:24:33.2817 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yXPeTobEfuzjRGlFy1Tzzy1jzmzDETE+Hk3s7lGCeNAd9COywFwDJIGnkA4vhS0E X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8066 X-Rspamd-Queue-Id: A095D40018 X-Stat-Signature: oc9a41qnfipzkmc7z9op3ji5hudxgyrh X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1735925067-104110 X-HE-Meta: U2FsdGVkX1/KYECRhHltUnz27kgNbSG8w5BwAYTgYrjR3qSNeIqAH5WKUb+LQshqyrEF1ncRvW+FM0g29CAA8lQ2GQlpvoZGDtRon5/pMbDkSbW5wu4iUEPFwcvxWoDNTTUuUS6bCu3r41kvBQUcK1JLm4t7NWpn+QriPUWqfoW7PNhDcvK7Yt7SyLUK4PfaQRbymwc02RLVwkasFu8Rmg3q77lKYnX1la1Cff6zi1/gtKf07lxOHDae7rqZwFcOosueuXAwY4lViNZL5Ezig+a/tWmZXu+Ldlk+9pPRCOzcPVixYkH9SigbeIZW+VoWa9wUlQw+sCfRzJA2TGESbEPWBqXAiC2OFLNFyaUU/36USSx2yHRd8e0DXyn7HdwFIEg/7M1eSXFPOaH95zUlNRbwx7MTGgJosJpaesK1HErD44EmHGSmrkBeIFElv0ExPcTp6Zg43AkA7ZS0+TuHkRN6CHmimLnoAPMAMupxpVjStFxCN6DXQftl3x23MZNa04V2vUbsqI3c02D6CGZlMfRcf5nnBJp/fzLawVRXBwwzq/JfaCokLAk2eOq6kzwlI1ZdZlVMrFngkI9w02tQrkdRIION0K41gNsX2vROyD9CsRHs4zefBX3/c5T4T7EjL+Bd/23PBwhAz2GnlDp8bTRkC/EjB0vFju7PmFTgXThX3SMoH68h+Q7iw6zLVD06VM53wyMiRnREZrwQMWiBweyBjw2zMwdYWQtCc03KRlhSROdxSUfWo0CRac5GMshFTb99JSJN+NuV0B+Id1vJDAvMOe0ijx7+cCK++4Hdq/RLz7QYMvArTaV0Kq7MFADFLPypW2G/Ey4h8sPgPyth12xyRdnELhjTk+N/BpXY70MEXdM2idr25k7qJ6Ys6niQUpOdcWtB7OnbvxEWGFUFoXVgWJR7musoxjEr8xrgvdpYPQy3k85OtYhmPYLWaKUe1ROlXiMMq47gf8TWZfQ z3DQe2ck UdW86YXP45jtmCq6ZJXNfWEnD0mH3X+J3/MtyJYZ2afxrWrZOC4lYfaSpCG/pOlKH5al+mpZXEMg2cYvTCcjj7MTBH2BlXA1thVeuGva588mSv+L4oTuZMocNZVpJjdI67QriXqjDLB7KLLncieDRdxuGULWj6IemtgT+Qq6xQyPHhZlF0ZqGzFCr5pWK7JWGVK/whfr7x70PZ+ruiTCw14JCNHVuZ1tgCVCz7Sfx1s0Ch6MnY627kCPqYZTxdRS+EW5TtqXjQk+oVkciNngoRFVxcbP8WjDwRL+y6NZQMu4YCy82cQ7cdFJAXiHalal+2i6AKscihiSVYLEQZ6O8fBHczGExktUZztOeoF+JEB1lPjdNiPaVXdfcmXDSc+1tUlelIxjXjSkjdaTGf89gUVDOuYSuk8iR56X8affqSmftjFzQfjkf4yyQpO/32VVKpMpbY0rtsK1ta/wXLJ6jBKOvaQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: No function change is intended. The factored out code will be reused in an upcoming batched folio move function. Signed-off-by: Zi Yan --- mm/migrate.c | 101 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 65 insertions(+), 36 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index df1b615c8114..a83508f94c57 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1014,18 +1014,7 @@ static int fallback_migrate_folio(struct address_space *mapping, return migrate_folio(mapping, dst, src, mode); } -/* - * Move a page to a newly allocated page - * The page is locked and all ptes have been successfully removed. - * - * The new page will have replaced the old page if this function - * is successful. - * - * Return value: - * < 0 - error code - * MIGRATEPAGE_SUCCESS - success - */ -static int move_to_new_folio(struct folio *dst, struct folio *src, +static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, enum migrate_mode mode) { int rc = -EAGAIN; @@ -1072,7 +1061,13 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, WARN_ON_ONCE(rc == MIGRATEPAGE_SUCCESS && !folio_test_isolated(src)); } +out: + return rc; +} +static void _move_to_new_folio_finalize(struct folio *dst, struct folio *src, + int rc) +{ /* * When successful, old pagecache src->mapping must be cleared before * src is freed; but stats require that PageAnon be left as PageAnon. @@ -1099,7 +1094,29 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, if (likely(!folio_is_zone_device(dst))) flush_dcache_folio(dst); } -out: +} + + +/* + * Move a page to a newly allocated page + * The page is locked and all ptes have been successfully removed. + * + * The new page will have replaced the old page if this function + * is successful. + * + * Return value: + * < 0 - error code + * MIGRATEPAGE_SUCCESS - success + */ +static int move_to_new_folio(struct folio *dst, struct folio *src, + enum migrate_mode mode) +{ + int rc; + + rc = _move_to_new_folio_prep(dst, src, mode); + + _move_to_new_folio_finalize(dst, src, rc); + return rc; } @@ -1344,29 +1361,9 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, return rc; } -/* Migrate the folio to the newly allocated folio in dst. */ -static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, - struct folio *src, struct folio *dst, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) +static void _migrate_folio_move_finalize1(struct folio *src, struct folio *dst, + int old_page_state) { - int rc; - int old_page_state = 0; - struct anon_vma *anon_vma = NULL; - bool is_lru = !__folio_test_movable(src); - struct list_head *prev; - - __migrate_folio_extract(dst, &old_page_state, &anon_vma); - prev = dst->lru.prev; - list_del(&dst->lru); - - rc = move_to_new_folio(dst, src, mode); - if (rc) - goto out; - - if (unlikely(!is_lru)) - goto out_unlock_both; - /* * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will @@ -1382,8 +1379,12 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, if (old_page_state & PAGE_WAS_MAPPED) remove_migration_ptes(src, dst, 0); +} -out_unlock_both: +static void _migrate_folio_move_finalize2(struct folio *src, struct folio *dst, + enum migrate_reason reason, + struct anon_vma *anon_vma) +{ folio_unlock(dst); set_page_owner_migrate_reason(&dst->page, reason); /* @@ -1403,6 +1404,34 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, put_anon_vma(anon_vma); folio_unlock(src); migrate_folio_done(src, reason); +} + +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, + struct folio *src, struct folio *dst, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) +{ + int rc; + int old_page_state = 0; + struct anon_vma *anon_vma = NULL; + bool is_lru = !__folio_test_movable(src); + struct list_head *prev; + + __migrate_folio_extract(dst, &old_page_state, &anon_vma); + prev = dst->lru.prev; + list_del(&dst->lru); + + rc = move_to_new_folio(dst, src, mode); + if (rc) + goto out; + + if (unlikely(!is_lru)) + goto out_unlock_both; + + _migrate_folio_move_finalize1(src, dst, old_page_state); +out_unlock_both: + _migrate_folio_move_finalize2(src, dst, reason, anon_vma); return rc; out: From patchwork Fri Jan 3 17:24:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13925731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F126AE7718F for ; Fri, 3 Jan 2025 17:26:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 88D936B0092; Fri, 3 Jan 2025 12:26:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 83B736B0093; Fri, 3 Jan 2025 12:26:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B59A6B0095; Fri, 3 Jan 2025 12:26:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 470566B0092 for ; Fri, 3 Jan 2025 12:26:21 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 090848097F for ; Fri, 3 Jan 2025 17:26:21 +0000 (UTC) X-FDA: 82966817346.07.26C0F9A Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2080.outbound.protection.outlook.com [40.107.101.80]) by imf25.hostedemail.com (Postfix) with ESMTP id DC775A0008 for ; Fri, 3 Jan 2025 17:25:40 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=DbxOdeVi; spf=pass (imf25.hostedemail.com: domain of ziy@nvidia.com designates 40.107.101.80 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735925142; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QuwWf93KZrL6mtGmG8Qf0lVvYao/70tVo+I+7ylzu1w=; b=z/T25OLwNbfsEVFoqNEN3T2daESv8D/bk1TOEhNTLLJBLTFoB9lGspxtB5bv7HA/ZH2Wvp qFg2BmAVZe6ZH0RGKZY3XobZLIwzw0vbqbpsvxwcmVqCZv5+gx0D2G9u9xz9cuQAZ3VN1j WPmymSFl+vsu89+QGdPvFyCdUCUDrlw= ARC-Authentication-Results: i=2; imf25.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=DbxOdeVi; spf=pass (imf25.hostedemail.com: domain of ziy@nvidia.com designates 40.107.101.80 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1735925142; a=rsa-sha256; cv=pass; b=GEj8o6oVpl0LBxLUVH4FJe6trNavKRU7BsoeBvnwn0pUiHypMkWFFlt/jljAyUBFspkmeJ yjlqs2sdZfC7DW+N0ixut4My+TMPZIck8YdK1P+XsUq0LufZxrlGNqGE5hFDYPfrNEgXYn muFTYUuAbB/XhVxC2CymfgC2ikK4ptg= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=fyTjPaIyRr4xGoKRlInmu5u8ds2jZ7Ri/ggF/bb15As09QVPXE9d99GlMkvZ+EYlZmL2Wg0xJRbIwAeciJNrYN8MeHTd5PgXCIqgr1G++YpekTn/uZauT3IkJhDRlrTklYV0pc8YjKicXPiiv9mCgzcgyQ5Xk8RG3dNEDHkfnjZ2/PTSRa5lPPBCxpxFmYo7DMCemoa8zplXOcDtRQjrFNtUjYhX78Frau2SI77/jdGbZ+YElnqHszepBWU6pAexYmB4rlq3LzDttwmPV3wGmB0D+N35KxSayxvramgo/0qs9gedEd3gDwm4z5alyfRTK9a62dYEFLX/L2J99/QlHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QuwWf93KZrL6mtGmG8Qf0lVvYao/70tVo+I+7ylzu1w=; b=cMgsRXBlnhOBqhIq3y3Ad0xBTtxymfvsTmtamj9V9QZKwEgWcWr+KTY/1g6omxiT2komeULobyOsZauk2WZAj2vTZl6KOfCKHPmwG0Ng3GrMb2+5kF7sKVmW6fcFlRK/2r6pPXFWKiFnt2uTFXzK6Du4gHAQTH+NQBiivOwfbYTitRz7uII5R3DtiZjEJpXX2m01LOaf9zMFvOHADGuHzZnq4r/brVVQtCOoOAQjotXgj455Q1s7Lro6o2QBy8Y8HmFB/cL8Dn3rInpzoARgfS6/7Ry6n5/tIN+yCgDOo4q3jWOhqmKqxhOAWrmCL+nSbHgmHowLzho2QbEBtA7N9Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QuwWf93KZrL6mtGmG8Qf0lVvYao/70tVo+I+7ylzu1w=; b=DbxOdeVitYwFpoqI6sSOo3yj/cocRVG5R1UBzV5LTwYj1w8wREAQiIgCX8RZqdkf4H776208QowFdb/iSceeV1FgzQXDFDNqgIQoX3y3EyXK1vy9e53F5IfVNvejOmzIzF6YJfn9ltoChZM0VoCrv/U2VVYq5AVnNdIH3fyWf+yfS8x/5CBw77worx9Fy+VH7mMEkI8YFocA8Oc79+nu7EEvC+RvupLhBU94igZyLQVfBHi9Nesb2FCpbr2gvIzvA5b4U2KyDuV6e140JstXgzCHtMZzXWwaEbpw/1SEjECRjCLflv8YMUDhXVzRdqrMYY5BCCHakboSZ0PjvhVf8A== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY8PR12MB8066.namprd12.prod.outlook.com (2603:10b6:930:70::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.11; Fri, 3 Jan 2025 17:24:34 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%3]) with mapi id 15.20.8314.013; Fri, 3 Jan 2025 17:24:34 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: David Rientjes , Shivank Garg , Aneesh Kumar , David Hildenbrand , John Hubbard , Kirill Shutemov , Matthew Wilcox , Mel Gorman , "Rao, Bharata Bhasker" , Rik van Riel , RaghavendraKT , Wei Xu , Suyeon Lee , Lei Chen , "Shukla, Santosh" , "Grimm, Jon" , sj@kernel.org, shy828301@gmail.com, Liam Howlett , Gregory Price , "Huang, Ying" , Zi Yan Subject: [RFC PATCH 3/5] mm/migrate: add migrate_folios_batch_move to batch the folio move operations Date: Fri, 3 Jan 2025 12:24:17 -0500 Message-ID: <20250103172419.4148674-4-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250103172419.4148674-1-ziy@nvidia.com> References: <20250103172419.4148674-1-ziy@nvidia.com> X-ClientProxiedBy: BL1PR13CA0183.namprd13.prod.outlook.com (2603:10b6:208:2be::8) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY8PR12MB8066:EE_ X-MS-Office365-Filtering-Correlation-Id: 39cb1bff-3ba3-4cb9-98f6-08dd2c1b7df0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: QBawTc3z5iu3lVNttX7q2/k7kThMU3foimtOlqsBcjy/5wtOoj335uxyfcyAEjAajfrtFajdJLdN4cLJZAEWsEXGCAbrmH1gZKKlC7F7rA5vwRws3T9mJCoy+SSmX1y1Oc3NSIP97T/HKvUkeCPSSagRbGzjVX+mOu+xe5vPZHV0FYEfV8UqxA2C32S+pQtADtrlzvzJe490NNn506Jlg42Y8HeKYNmbte6qrpBBp6g8wgmlV9WSwj07WYeNnRWPisNDPrpFH1PqYFRjJyzI+Tb0uNAmdwvDp/VxXCv9WXtHoA6GkAj17b/kucZT+q2ITQUDM0aawaza0W+KgAN+I8o0s9n8XgetSwZ5Sg9kN6Wo5D/6nWikcN+k4wHqNEE+Hzh/9lp1LsEGgG4qxkZvpx2KrPTJM4ACJdQUUk481+cBG9DMcMSQZwcWPuIXniVFu5MoIF1KSeYzEqr/NwMVcnIvn4HGUwAt9zl5Kp0hyZfrvjG32PRaAzmvnFKO3tn+6ZWZCZNIb37CQzVQIDuv5vbF5kdt+M1/MJx4heFBKj3tz+8aAE4tryOg7NYnDCYvvv9S+58gwJQcBUwmz6xyldxL2de9P0jdaWSECLWs9LKYrjNYKbFs1z3keokYW5+LF7weDbWqrwoUe4V9i02wECqXrSXUthMn6E8xmuO41qVHATEWPHlAr9zq3XZt7yrKzjCkn4Q22r8pBXZBEhz2YSNBR/WX7Uur27Az3e63ByeD3aH4q/eSvqxxzmcsHikoMcVYI51k8M6wxlLMC4RvZLk8P6qJ1Cx/xJip32ix2l8tk0bx/y76gzEY+gKGy0Hz7B86Sj844s2PKSQBK2eZvb06RPTngSbyyn7t6Elq4XSLRCRLJb6iEL5Q3KIkvvbojNcILYEF9/olE8mPFUdP/Y9sLYts5fVCz7o6fZMvOWeJCWoBsxbXTeu1aN34BZ9GwZPrt+BjVFZDujdWmuuPtDKPl4csCLz33vNWumP5X1OXT1xytSaO/kQ9yh/Ly7D+bBk6FaVYUG9nQkarRPyA9uc+EFovQ0BgyubcHf925op9/438JlAdzPWieciCfxJs2B6m8lQEZVeOEUR8FTr0KnDIyBcxFkquI21XTzaKvpYhRg/wLZjTLoijQ3FFzQPKI8dt3Y7r5nRTpg8vpa8HjVkVhdfT3y13av5pXkD4wfRxmed4/9cFnpU11hKBpfWdy/gLrO1ChU83vEF8q9YG06TH6bIwqZxOklvqJeSaO4+Qw/gPFYqin7r9M4kqW7cXzE/3VPNsvYPPCksebqGhYTNVhbGPqbGveLwVFzp+RxQb2jY9aKnhs5btRhHTyBCZ7bk5tNNsLXkS+ltyIi7/8fq24UySI1M2+w5Q5qmOT18IesESitWL8Qr2g16KiuMi X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: w4aHQC6JPkAWgk37MakXupaqjjwMcN9OujJ4z4vAjtQRz0lkp0/0fRJIf3wp8VBjadFpytorn2TLg6yd9s2FJMpt5mvzu7SKhbZNpk6AobazWzSaNQDhtGDCUJf8ARJrbOjXEcFOTZ8iUXRl9w0NYMd5Q2t2fGlOxXG7bjHV7iKBUiS4vO7YcdEp/4KMWMyAPoRyKt56MqJwOTUX0pFjv54Ik77yz001dtuyZDJzOUfrzuoJvOA01Lo+luf3vVdqZtmjnGNlsh5ZBhyQVw5frtk5ulEW0CNd97Hem2TM3ilwLeqQ8Tx24Gp0KYSpoRGmv38rEteuG5cgS3DzKQ98469mjmrQiP9ZBtqqRh8hHk3u5Hw/d5QOJtBJsi5d5n+vo8iHkvqKiPptA8Dxpae3IBVPurUG5J9v5PRAsJL/NVwsdG3LLvhVGHc6eq9+3JAlWS8R3pXgg7pTnPquKAjNWxwiaOeW/4jOABf37NK9aVcf0SXUMuQf6dy0g1NetWSnssmhq5TPXfciBo019fxV6wXOSDYJb2SlvquGXlJfEP1ftttGknl2Yd2L1CZb3kqzsVGnPLlD4KVi9gsKzR0CJUFzHlHs33u9G9UK7bglUdmOfA+lIEpsTeJiB7tByzAUP7hyZcrlb6uMhvk1rq8SLhC+C3ObIgPUPgco3V8ox99CyXA+bv6NBzRJctImX0SMm7hg5kokmuXL9kX1zrzwAHP8PpniBexf+AuaB5bwOyjf8XdEwNtLbcuY/9zktzsy1lYNf3lMMbpGGJInqHsoh0PxphH7R7oTfsWgDapJhQMPXpdi5XePXVA5cpA8dAhDjw2l2sjM3OBF5dPfdFk1UkEhLZ/M7g2NosIy8P4UvugwKK5N0dCrvtuXOOAgSTObT2eRKnj5BwCAd3jCMzLU/eYo09yt63oHvCoLkHwFVR78oq7Tl5ZouciuatBIxUOK257SnuHblyUY/LT/HIKYlU6Pb0vsvuo0mdqWAqmk1X9CTemqp5QqUtii1cQM69miSgK1PfoIe9+elATryMg8ki2wTZvXynbAtI8+9VSJVwLhY9oMMZPeb6JNt+p2aewx9e78C6H8Z5wQ6XbMQrb/RLZQKmWM6Ccww8/0QOQjReunhP04bZQSEaX0qiJsV9Mv4+zKlUF/QOmphRfMOjEgDnzWVqZgZUV5jhyvwQeUSjk2QIr0Enp74EMTVj1P4KEiChmn16hgnwcACLH3YSdAP5YF22FdPpJnVor2/BbQlmfucJ4r/1UzkQVZu5wCK08ScJrQ2CDMFs5ob5wf3IygS7FEZdftEu8GAdbUOVSImYeoQqmvO8uxpwPY6v8uYQMZsOMTkdLdyEBUcnxsreDVtrkV05li2DFzf+rUD3D/fZWp0FOSGoApNZ8UVM3nzMVPDdzn/4TiokfGgIhWBsOQa//sdrdH9VRZojipa/1ppvhNGs5LPAvB9h90pr9LtU7fjZMpOOUPRFxBENx+b8FD9grwc7mRZt+tSrzdVKS5yBm/WlKmYsLMzg+SxIOWFNXWbcK7Sbu8iEF2PKiHMYvJC13UNrp2u9Cd95QFjJ1KQGI= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 39cb1bff-3ba3-4cb9-98f6-08dd2c1b7df0 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2025 17:24:34.6361 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: AvuaQzy9chQAOvbB8XdYWftPqM20QrxCDGIjv1uO5y/795S2QpbAx6Msc9m7t7gD X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8066 X-Rspamd-Server: rspam05 X-Stat-Signature: q3dedmoxerd7y8j9cxwjfoybkqjs81ka X-Rspamd-Queue-Id: DC775A0008 X-Rspam-User: X-HE-Tag: 1735925140-976617 X-HE-Meta: U2FsdGVkX1+4O2nPba+Df5KDJY6PSYox7BIp6GgO/jdozLWKfYZB6SC7Tpgu+ZC5ivGnlRRB5I2f7A6CIg5qzKXguV1oXhilU4fEE5dWPOdWTO71Hjtu/y3/HXsbBpGXxBkt2p0D4ai5rw189ZN+gVyhMv/f08lXOGkuL3vI97BG5YuA17MTBL9rNMeWgznxGRLrDZqF8tDQfmQM+LBwoXuUfOfdYjI2YFmc3s+9l8tFX+bsakN24Om01A69yCkLlCN4AJgved8pF/tK42dQ7tWU7eAhVuDUoMgeyv9tGpzFI7aVuM65IuT4ImwTlAXGCBe4+frwGMmUR8CJmRCJC5lJDT2KbRklF0nMKxdFx4O+mDM7cHIpNFuDnzfx30wssZeMG1qRn7SJmUddjV+GHFmCoRHEH0XwaFtnjMpuCz0qTAEGQYiJ12NVCmDoaHHp9t3ngFzlI2jwy4yj6Qqmi3sBiAxKL/K33Sxj3MNhGwsIWPQbknR9zp/ORnvvS7xHabuc5aMdMVSmHHwFWeLvF0cHxO5w20kTrUl9SJVk5Zozb0I36hbeJQmbYy2WBrfzJlMuEA7T6u52TlugUPJze/p2VdW+sQYZ+8NKNGMGuTA/68P69hPJn7beZxQOS9e3aZTZFxeQvj7GYgCe1QvVuhyC91vm92sXq/uFMXO1Vrrnz6j+3SNnaLibEeziPXBBuU1BqcRjnBLQlOg19EEHLi6sBVlDw3Yy61Yxz+vn0S7tQD51kVhJkSflUR3cPcvQmvTiTZAtzewlbTD96WJ+S1006p8jCZfxl1x/ApEHnjLiRbnQwmI1FqdG1/do1NR2s3lUZceh6UZhdBknhgVbD8yRfkk4GfVh7xFkYnkcQ4oP0Zll98Uvnvn/BpeZwNmHhY320F2KcxUM1rr1+iSUQo4kvO1WZhNO79Xpl7V8W3Os7rRhDLDGKoPmdPylJvzzB7XKG7DycZz1ciuuwYL j/2mZDEr Ti4VoYSAvqP7u3VLauTp7JaHpfmos+ui4WYBVVHxt9iSVTwqhCardZn10gEggxIKFW5RMewdPuW30ziwUyaQ2v6TQ2DQWHj+zKyJZabuP6pOdOilDS43DsmEeCV4l+ZegoXeXmp8SFjnnwxTJa1bwZz09tjnxvG1eRFbj4mAO39xtup/7g4dO559QQHztfV1NHAJdqWahlONfXX4UAxAuix1XW7pG4AvCVWp9Q/2TLHBEYYxKOA+4zO6zuCpQnph7sXSRj7ypuUoZNHsrWkMTz0gC7MMTdiKjY7Nd+uf1m0yIcaD3czC2AZhnYmBWZZ6c3VwYoAzeTuFdKnELuKiUlWKeFV8nCTPtmjCVy6BhPcASPR+eO1G0QY1ThKHpa4oXY/8Gw7RbJtVJIghxjtQt+b/AovDZNy6VjY8I90xuIsLIFlZs+olEjDdCB80mS7GhQdx0jXcOES+DmB8aKDZJC3eZbEv6PUvjrbSCQUBqko3WraI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparatory patch that enables batch copying for folios undergoing migration. By enabling batch copying the folio content, we can efficiently utilize the capabilities of DMA hardware or multi-threaded folio copy. It also adds MIGRATE_NO_COPY back to migrate_mode, so that folio copy will be skipped during metadata copy process and performed in a batch later. Currently, the folio move operation is performed individually for each folio in sequential manner: for_each_folio() { Copy folio metadata like flags and mappings Copy the folio content from src to dst Update page tables with dst folio } With this patch, we transition to a batch processing approach as shown below: for_each_folio() { Copy folio metadata like flags and mappings } Batch copy all src folios to dst for_each_folio() { Update page tables with dst folios } dst->private is used to store page states and possible anon_vma value, thus needs to be cleared during metadata copy process. To avoid additional memory allocation to store the data during batch copy process, src->private is used to store the data after metadata copy process, since src is no longer used. Originally-by: Shivank Garg Signed-off-by: Zi Yan --- include/linux/migrate_mode.h | 2 + mm/migrate.c | 207 +++++++++++++++++++++++++++++++++-- 2 files changed, 201 insertions(+), 8 deletions(-) diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h index 265c4328b36a..9af6c949a057 100644 --- a/include/linux/migrate_mode.h +++ b/include/linux/migrate_mode.h @@ -7,11 +7,13 @@ * on most operations but not ->writepage as the potential stall time * is too significant * MIGRATE_SYNC will block when migrating pages + * MIGRATE_NO_COPY will not copy page content */ enum migrate_mode { MIGRATE_ASYNC, MIGRATE_SYNC_LIGHT, MIGRATE_SYNC, + MIGRATE_NO_COPY, }; enum migrate_reason { diff --git a/mm/migrate.c b/mm/migrate.c index a83508f94c57..95c4cc4a7823 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -51,6 +51,7 @@ #include "internal.h" + bool isolate_movable_page(struct page *page, isolate_mode_t mode) { struct folio *folio = folio_get_nontail_page(page); @@ -752,14 +753,19 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst, enum migrate_mode mode) { int rc, expected_count = folio_expected_refs(mapping, src); + unsigned long dst_private = (unsigned long)dst->private; /* Check whether src does not have extra refs before we do more work */ if (folio_ref_count(src) != expected_count) return -EAGAIN; - rc = folio_mc_copy(dst, src); - if (unlikely(rc)) - return rc; + if (mode == MIGRATE_NO_COPY) + dst->private = NULL; + else { + rc = folio_mc_copy(dst, src); + if (unlikely(rc)) + return rc; + } rc = __folio_migrate_mapping(mapping, dst, src, expected_count); if (rc != MIGRATEPAGE_SUCCESS) @@ -769,6 +775,10 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst, folio_attach_private(dst, folio_detach_private(src)); folio_migrate_flags(dst, src); + + if (mode == MIGRATE_NO_COPY) + src->private = (void *)dst_private; + return MIGRATEPAGE_SUCCESS; } @@ -1042,7 +1052,7 @@ static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, mode); else rc = fallback_migrate_folio(mapping, dst, src, mode); - } else { + } else if (mode != MIGRATE_NO_COPY) { const struct movable_operations *mops; /* @@ -1060,7 +1070,8 @@ static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, rc = mops->migrate_page(&dst->page, &src->page, mode); WARN_ON_ONCE(rc == MIGRATEPAGE_SUCCESS && !folio_test_isolated(src)); - } + } else + rc = -EAGAIN; out: return rc; } @@ -1138,7 +1149,7 @@ static void __migrate_folio_record(struct folio *dst, dst->private = (void *)anon_vma + old_page_state; } -static void __migrate_folio_extract(struct folio *dst, +static void __migrate_folio_read(struct folio *dst, int *old_page_state, struct anon_vma **anon_vmap) { @@ -1146,6 +1157,13 @@ static void __migrate_folio_extract(struct folio *dst, *anon_vmap = (struct anon_vma *)(private & ~PAGE_OLD_STATES); *old_page_state = private & PAGE_OLD_STATES; +} + +static void __migrate_folio_extract(struct folio *dst, + int *old_page_state, + struct anon_vma **anon_vmap) +{ + __migrate_folio_read(dst, old_page_state, anon_vmap); dst->private = NULL; } @@ -1771,6 +1789,174 @@ static void migrate_folios_move(struct list_head *src_folios, } } +static void migrate_folios_batch_move(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + enum migrate_mode mode, int reason, + struct list_head *ret_folios, + struct migrate_pages_stats *stats, + int *retry, int *thp_retry, int *nr_failed, + int *nr_retry_pages) +{ + struct folio *folio, *folio2, *dst, *dst2; + int rc, nr_pages = 0, nr_mig_folios = 0; + int old_page_state = 0; + struct anon_vma *anon_vma = NULL; + bool is_lru; + int is_thp = 0; + LIST_HEAD(err_src); + LIST_HEAD(err_dst); + + if (mode != MIGRATE_ASYNC) { + *retry += 1; + return; + } + + /* + * Iterate over the list of locked src/dst folios to copy the metadata + */ + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + is_lru = !__folio_test_movable(folio); + + /* + * dst->private is not cleared here. It is cleared and moved to + * src->private in __migrate_folio(). + */ + __migrate_folio_read(dst, &old_page_state, &anon_vma); + + /* + * Use MIGRATE_NO_COPY mode in migrate_folio family functions + * to copy the flags, mapping and some other ancillary information. + * This does everything except the page copy. The actual page copy + * is handled later in a batch manner. + */ + rc = _move_to_new_folio_prep(dst, folio, MIGRATE_NO_COPY); + + /* + * -EAGAIN: Move src/dst folios to tmp lists for retry + * Other Errno: Put src folio on ret_folios list, remove the dst folio + * Success: Copy the folio bytes, restoring working pte, unlock and + * decrement refcounter + */ + if (rc == -EAGAIN) { + *retry += 1; + *thp_retry += is_thp; + *nr_retry_pages += nr_pages; + + list_move_tail(&folio->lru, &err_src); + list_move_tail(&dst->lru, &err_dst); + __migrate_folio_record(dst, old_page_state, anon_vma); + } else if (rc != MIGRATEPAGE_SUCCESS) { + *nr_failed += 1; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; + + list_del(&dst->lru); + migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + migrate_folio_undo_dst(dst, true, put_new_folio, private); + } else /* MIGRATEPAGE_SUCCESS */ + nr_mig_folios++; + + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + + /* Exit if folio list for batch migration is empty */ + if (!nr_mig_folios) + goto out; + + /* Batch copy the folios */ + { + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && + folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + rc = folio_mc_copy(dst, folio); + + if (rc) { + int old_page_state = 0; + struct anon_vma *anon_vma = NULL; + + /* + * dst->private is moved to src->private in + * __migrate_folio(), so page state and anon_vma + * values can be extracted from (src) folio. + */ + __migrate_folio_extract(folio, &old_page_state, + &anon_vma); + migrate_folio_undo_src(folio, + old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_folio, + private); + } + + switch (rc) { + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + break; + default: + *nr_failed += 1; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; + break; + } + + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + } + + /* + * Iterate the folio lists to remove migration pte and restore them + * as working pte. Unlock the folios, add/remove them to LRU lists (if + * applicable) and release the src folios. + */ + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + /* + * dst->private is moved to src->private in __migrate_folio(), + * so page state and anon_vma values can be extracted from + * (src) folio. + */ + __migrate_folio_extract(folio, &old_page_state, &anon_vma); + list_del(&dst->lru); + + _move_to_new_folio_finalize(dst, folio, MIGRATEPAGE_SUCCESS); + + /* + * Below few steps are only applicable for lru pages which is + * ensured as we have removed the non-lru pages from our list. + */ + _migrate_folio_move_finalize1(folio, dst, old_page_state); + + _migrate_folio_move_finalize2(folio, dst, reason, anon_vma); + + /* Page migration successful, increase stat counter */ + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +out: + /* Add tmp folios back to the list to let CPU re-attempt migration. */ + list_splice(&err_src, src_folios); + list_splice(&err_dst, dst_folios); +} + static void migrate_folios_undo(struct list_head *src_folios, struct list_head *dst_folios, free_folio_t put_new_folio, unsigned long private, @@ -1981,13 +2167,18 @@ static int migrate_pages_batch(struct list_head *from, /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); - retry = 1; + retry = 0; + /* Batch move the unmapped folios */ + migrate_folios_batch_move(&unmap_folios, &dst_folios, put_new_folio, + private, mode, reason, ret_folios, stats, &retry, + &thp_retry, &nr_failed, &nr_retry_pages); + for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; thp_retry = 0; nr_retry_pages = 0; - /* Move the unmapped folios */ + /* Move the remaining unmapped folios */ migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry, From patchwork Fri Jan 3 17:24:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13925732 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAD89E77188 for ; Fri, 3 Jan 2025 17:27:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 558766B0095; Fri, 3 Jan 2025 12:27:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E18F6B0096; Fri, 3 Jan 2025 12:27:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E66E6B0098; Fri, 3 Jan 2025 12:27:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 067246B0095 for ; Fri, 3 Jan 2025 12:27:35 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B088CC07E4 for ; Fri, 3 Jan 2025 17:27:34 +0000 (UTC) X-FDA: 82966820034.19.BDEF943 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2079.outbound.protection.outlook.com [40.107.100.79]) by imf15.hostedemail.com (Postfix) with ESMTP id 12FB3A000F for ; Fri, 3 Jan 2025 17:25:55 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="Im4T0/jG"; spf=pass (imf15.hostedemail.com: domain of ziy@nvidia.com designates 40.107.100.79 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1735925203; a=rsa-sha256; cv=pass; b=tDq99JV9UTnmH6ConNJ1fTnKJTJxQ4mkmjqlijpGjV0BkIi3EDAe+cYEoAOYxhQ2oK8K5e eCCeqDmbASMw/4UegeSy+DtNAyNyp+G1yHesWF88QZ7LZW1zXPizxkcLnJywCc1C6c+WrL ziHNnHCSX1wSHb217PiLrojqXRupI3o= ARC-Authentication-Results: i=2; imf15.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="Im4T0/jG"; spf=pass (imf15.hostedemail.com: domain of ziy@nvidia.com designates 40.107.100.79 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735925203; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ej3UokVF/XKxua5H3iLTNC9XIyooIX5KbLGnOa5F/Ww=; b=rcFJVrlc043fvqqn6Gomnba9aWgQOfHR5LARPvvpBCscaAHk6tK6huSoSlPtHYck2+b7Q7 t9KfZl8MMaRE8uRwIJYh3EizIKmCLbmRgqPf6C5JluS5Y1bp032FQmKNfMfgm7MUrcsci8 EV27i8TdZSzejFYa0ejGxEDbdFLekbY= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cNWtCDrldY8B92XhGZzCst8DkHwD/N1eYBZi9ASGebWZfl4U8TE+YdWt0rB7KwCoGe3Daz34UKr52+jrwduqhDV+XtbgrxbTlNp76Mda8IeFBUo7+x4YC9+5E8MaFHCIgW6KO1jP7avW1vO7pmOf77qqg6NwzMJ1+VYGXQNmh/fMUnsvBNy7g2iAEnSoyndUXA5YFm1I8atvS/QoLdMprii71jOTkCtIuKKc03tsOOOsK9wIrD9yG/sCPRIzvJ3rI2aa4Gt1SrcBYDgns/vRjQ7d7Z2L//z+QJtraFm3/pvHSgJSE969IufFllmiP79/ick1yrEaEhFV7hLmDE2rnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ej3UokVF/XKxua5H3iLTNC9XIyooIX5KbLGnOa5F/Ww=; b=bI+WfXOKZXemscJlKHxc1IG5HA2xu3C7cHoJXi7fI0cdtj9kk85kpNv331bQ5uZ0YoqaEGxyOkb22Su4GIr+/VseWE5w9diMRcgPB84YFR3qzMZFql+g8stwmA8B7ZEp+Lcs0XUwajE1P1K2v+FzBKAceT0VjIXM/zVRMehNvJV4QakS2k43FgAHCrs5GNI+tUIkkYK13PF60v5N9+gCRHkqzP9DBWlCrlzj6YmPKYQ69jV3TxCam5+2sAL/dmWovSaIxkyXJmUXB1inGsweQztozbUY25URmCJ69ZHSVB5e7lDp7TCAMA009/X0YBZM8rMfsIpGK0r6n7Qlg2Zxyw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ej3UokVF/XKxua5H3iLTNC9XIyooIX5KbLGnOa5F/Ww=; b=Im4T0/jGRQuRLshw5sgs0tuvjadVbez0A+vFN38sraPilVNN321Cwz+IHogS5PeCubIoLMWJf9HctNDpxbKam4Mn18+Y3uRVh9xeYi4LnXvAW42ke3QhYQQ4wEualkj8MOhj5eaLHl58Mbo8aC0oLQlecW++hb1vwvQ6OGAGkm1RCFUoWf0wz+OZMhNAUUwli6+3S2ir86gmfZmeD/B6ag5cMeZGfuYkWDME5nxs+Bj2Hrru3KVqAMFtzECYaGw79l8sd3M3qaQLdphJyCV9nUZv7BoOeTJwvKF6IqMMnhtQ8G0uHfovXqNCRvIlFP5zubIVEm3ZqeuyJ05mE26Lng== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY8PR12MB8066.namprd12.prod.outlook.com (2603:10b6:930:70::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.11; Fri, 3 Jan 2025 17:24:36 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%3]) with mapi id 15.20.8314.013; Fri, 3 Jan 2025 17:24:36 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: David Rientjes , Shivank Garg , Aneesh Kumar , David Hildenbrand , John Hubbard , Kirill Shutemov , Matthew Wilcox , Mel Gorman , "Rao, Bharata Bhasker" , Rik van Riel , RaghavendraKT , Wei Xu , Suyeon Lee , Lei Chen , "Shukla, Santosh" , "Grimm, Jon" , sj@kernel.org, shy828301@gmail.com, Liam Howlett , Gregory Price , "Huang, Ying" , Zi Yan Subject: [RFC PATCH 4/5] mm/migrate: introduce multi-threaded page copy routine Date: Fri, 3 Jan 2025 12:24:18 -0500 Message-ID: <20250103172419.4148674-5-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250103172419.4148674-1-ziy@nvidia.com> References: <20250103172419.4148674-1-ziy@nvidia.com> X-ClientProxiedBy: BL1PR13CA0189.namprd13.prod.outlook.com (2603:10b6:208:2be::14) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY8PR12MB8066:EE_ X-MS-Office365-Filtering-Correlation-Id: cd39796c-9537-46ee-196e-08dd2c1b7eb7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: uTtOAOG77Ry9B2b5HUjMKJ6nHqsgcnsnBTT5Cc4F3lS3DS+FBRVaguDUk4X3yZ0og7ULsG+Hc/3Ptr9NoLdRV31mDAwmXxiziUBUVIPe/m94a/0+yIAkCE8TIUYRMQcT7x3RElWXcugAo0arRcPi6DRWXPZQOTI7iXJ/dutZNm7U3jmfZcsu1PmNzrdaT4OmY1iEkCHca5kvS8DS18BSnh/Mpls96+zMF4qF0PexynF/6qpJCY9/aeB07wdqlYCRjZ124NRJrSdIDVUC6v23Nozp9/zXGZsGccd7WcMkRVThd9C5LPLDCDMoDhFX+M02JAX4u9KBYKyFZX1mDxA+kl6yZ2ykGK8tpPPW01kSgL9b/U6SqkTpm3TlDLCpZnZURy5OHxKCtrxLP+HIbTInicBJqSuZWZcqoY+eqJCwlB2l7d6RNjR3mFerth+e5Q8hKY3tUiLki4sT8/oW0O7IR5xPbOV7qOTjHgMv/HDBGC/YICMR/OtZoLzxpjsPar5XiTiKqlHZXP96PgLgM55Fh7dhPJXRSynQwtoHkpbueUHgymplYWAENIUqOzrJOA+twD+viyxNodB9SK2gu1i/q+NasRLXjGRmCbMRT7f4k78d1hO5RjIr4ZjcpvvR2qohGkOJl4vNtW/LpjqdF3k2Xa97tGeO29/87xE0O0U5Q3V322D7uAy4IrcdvIPdBQfCl7Zal/OvNC3u6LJnjP6pP0sexB+lRKw8BPaMBl/yKIkIwe8Lm3MJoki4h4HlnlItXtWkNQvhlb+yr1Z3OVoyQ9KJt+kN/GZVLxEnrIjoVxKZX2dnwcxHasgdTqrLF2ugANS0ST+2G4g4DhP/peAQVj5aA0+7OiqiZ+v2dnKGEKy0XbBxTn2ihz7rdQzLuEoUHglAjkICxK4ckIWoF6w2mqrei+nX2VgyDcxxdLb8u6TrrnVHQGtSrEeiA4RLPyJR5K5lDLgi6Rl2IYxn/rjwdpLtc391KtGL0CsLL3qTimZ2evytJFhGZcKWRTictmt2Dbm3ygJM2VH3xeLMU0U1V5V9PezjMBos2XwX0F1nXIguSfZKCH0ynO9KF8T0AY+RU14HJtrB4QHuneR7rTrykumU7cU10qlCZKFzkMD5/RatqUZs0ez6fv0pcMeEYfyAQ8V1+ADSTneNdbMRdxQTQGj5OJWag8c+e3eHETpJekwHWO2FdnZVs7ZW4hFAZWCAn8yhrfJKowENW6Z2SCGYPGhktyh0FogItsLM141ztkVLuAisvLbZXzdsYAigC8lYK7FgGPtn7H4X1N3QS0B9hYH3TnzrX6dKnEuvFuKS0mvD1h5Ni54BildRVr1OO08t+tbjWyZ1YkjrZ1Md+Q151hO0BjfMFZtjaX+LMa30C8qU2SuIdJizY8pUiK+S71O8 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 09ygDoo4FCdmLBq91l9X7z+TxlrXK62OxfMfdZXTWIlxC+Z7geJ/o602poj88Hfh9zPw2I2qg0OsQGn1gHfO8s4gxGCAk0F4outSUD9w/F0eI+s9ViQtDCkKNB26QNVbG0vwL8ubG6APOm/lO+EMRqAy+ZOzvg4GxYFj+J9VLgdHnKzjn80lMVWk6OPQWtIedhOnp1hoC9e+TLWGUgiSksieKJnMF7H5lTpBvqOFGa0LC6im6OCXaKuf0tb0iEHld2ZDXNDaER+fSs5Zm2Khi4Eha332K3yJUmzTWGrUEnxAL1oonPOZIU7FZi0j+ZDPUyYnJ205H/0AzpuRnMYxXl0Uh3bEEjS3sYqIR0kPS+xdcvyaf52VOCOfcq+TLxpNr24JOVahSMqejNwB/Z66wEVe7K6YhZmSEEfiJbjOqVIo6jTAOhtjgq6sZZWFY48i/uUuO/IsKLJvTqzLSyHs8sCUwLW6o4vuNNrqFOyJ4m6s47elCiPdjzRQg+5Gn8sVHBqlwl8PMQzFIvspQITLLqxPQT2LzlmbcjBtINns4DtCMfi00dpnJf+8u2+lG3OKuL6cBSKK1ejW2Lb2lR5fcNgH2ZqKXoEgqxzn0s0aYPN2uEs09PTnXmpHVI7MBTnR/SgNZ6D02cI+KDTpOBHhzqNHxlTleQCVCtNY5pv9ZkZbArqeQajnUMmfr4idWu42SrXXA8KjoQD8G5FqhpfFIRUMSQz1ySkBFaQhwCdxlLAmfncVG/GR8sH8yGGVPeucUQxuRftRpE708DpRh8iJDfvYohN00JPVhGuxbh61h9N/ChGcOmWi0jtoLd5PFbhWd7ICbjh1uK59QW/pw/e8VaUsc8nzuTZkRRJ886hRqE+i/1plKn4xm9+ZCtSjddgPaOmgYVV3g70TZklNPenkMJcM0bbhuLAOb+G0/YLNEq8QKxocGNQ5cXtbvKyARt4wpmdLkSU7Ru5aEVYvbqJBYDMne6kMjJ8ZDxSQFU4tS5oI89GlQVU5/jJrbsTmqtPTjA1wQBUNF8GQsygMBObrUsKlX5MndkeeXdjoX0yBOjfRfNqxGnxyR34RFWHMyzqCTg/SHJKvZ5CCXLwDSRfIqtEnLU3PptUp28LkmMVzhsF0XbIG82ApjrwApWFi48/PPM2CkSAoLocPV7hvQHORIsOBdY+bl1vRBbTess/817/oXFdOdmWHwsqO5bKOvnoQqIF93XPxs6Qd9Itao/BGoe/2+sl3/I5VWXPoNWwstErTcUu7asmcAq3CocVaDm6acmkzGg7wuNTeuFSEFquzDbQdHbQtt0MMeoT4y9QvTIeqj5b7VsJv85Uj2sl4sLuuR9/iqaPlxwkZ+KzEZ9cvxEuVfK3dFUfHGOQQ1nnOjIEKHuysjDSzPT/SjEQxDJFevzfqw+9C3laoO/+ZFExwzXKJQzq9+sCtuuRr12sSOJ9tHx2E4QCvkZ1B7QZpk9odOAWHHvTGofA9dGwDQdrJoGEsBEizOjo9xPOfrQ/RKUg/b1N7fRCKao8hP0u1d6nv6kKrGUKv/4GrSqr0jEH6uFNI1OkQE81inR4MFyTOZRQ= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: cd39796c-9537-46ee-196e-08dd2c1b7eb7 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2025 17:24:35.9423 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +5tiddOO9fhidpFg4cbWJ1X1MhrzIED0xV9ZMl9dZ713CI2uyICsgoCvf0DHB3KG X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8066 X-Rspamd-Queue-Id: 12FB3A000F X-Stat-Signature: da1dh67pwge4ie59ezrjpwr7qfrej9pr X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1735925155-790194 X-HE-Meta: U2FsdGVkX186mDNhAmnW6I91pZNKSvgKinNIdpT7SMGS+pqi5fPf14ji+r/3thFLTFSLjZwvaSjyFOeQbimu+Aw4G/xr7a8U+wHeQLtYzuFRZ2jExuNUV5EEN3j8qsIvDDZqPZ6NGxgbXOEZL7+a1iGR9IowKB5Cn86z4BKNVsunet9GMtVw1bE1YHc0YHS96DUFXmrg05BHMV/LJfM2yfPiBFZOHs1p6LzAbfOEtVCrwnZCRHArbgRMe5oa4G2rw41bD1k6LDoQwaen2Xthh+ymdp8og8lKkYt+rLYpLgBS2rOtqYkHHtRTn1/fj6krAvSPBo7o7QIyDF+1n6kpbLKqSiywtQR7rVTSjfbp/0r7huHKl4l5MlTt4cbwN34DSzEzOkxBEHGSmTzTIfteFv68mTINXpPzXLrwTwmWQKnQbamiJLZctae5QUwrBubp09X4Pu1OFx5NAGo6cReJSLmzZ/Vptg5wHbYGJAm2t78oLn+roFX6efsQeznh7k7wLhwbh5X7uB8BFV0PTwtGPgL/nq7vHMvJTtmixZOjD8Ro4+bifHFBBHgA7kbVRYJSuJuX4Plexqq4N9bviSdtUy5Bm+x6kaMpPwhrkAxekXuvUC0HqhXx1yEd2+3pVVV8KNypjnSHpdbH58P1NN98J3tqvaIVabdZZLvbkniYMbFwwp483oytozBEra680FYUXPTseoparuB+TztFq8IDFDt4xr0Wcr0cfxDzKPOmePVw9Rbyn8kqtQWUmz1gi2gNXGNe6Ia5gS1TynvHf0SNJh2kmusBjhZCGFjkKZpflgRA651Fj9k5wVUj4brOA3PGG1ljpW4Sg2KJlSiDLTFOdn8kkclRWGBV0h/+VVnU4osJPLHRnS1cTfClH25mKpMSzB+O97+QprkQj2pIRpCBOX+s+b4yw2soKcqObNuDnqZ8HHcbpOsucUrjvlP443AokakpOfdT8cEd3cdPykE 1qRHCMg0 y8P/GTgttD9uB69/CXgrTsA5WnXBNdumo1hr36ltSDw3Cd+So3kPG7nREgPaGRSdAjHFs3VVfNsWzkww56KOIAOmSmoGapNXSrhzP/8IjA+2Iupe0iXJuy5WZblkOEdbqHY+AGmcAzSIGNqsAfogmMQ83HOd7SmnxkDugnt0O4rYEn037frUEiiRKLDFrKw765fkjeE4JKioWzXmPq4E2QUAd9s3MZr2Ev89RmiKbvWdfFoDggjVudBgEyDj5ztvGaua6H9KhZB9KJHADpNCknsoJFWxTM7yazWBRV03/yS0eBU+jgb5x9TwGsHpzbqialQqemcJedICWoXiBSE0ZO4pD0CZWatrSybq+9LBEf/q67Vj3k82Lwqp7YHxVvYMjI9wxoVbw7BBkJD14OHe1MDqRum0Uzn9YAWfh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now page copies are batched, multi-threaded page copy can be used to increase page copy throughput. Add copy_page_lists_mt() to copy pages in multi-threaded manners. Empirical data show more than 32 base pages are needed to show the benefit of using multi-threaded page copy, so use 32 as the threshold. Signed-off-by: Zi Yan --- include/linux/migrate.h | 3 + mm/Makefile | 2 +- mm/copy_pages.c | 186 ++++++++++++++++++++++++++++++++++++++++ mm/migrate.c | 19 ++-- 4 files changed, 199 insertions(+), 11 deletions(-) create mode 100644 mm/copy_pages.c diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 29919faea2f1..a0124f4893b0 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -80,6 +80,9 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio); int folio_migrate_mapping(struct address_space *mapping, struct folio *newfolio, struct folio *folio, int extra_count); +int copy_page_lists_mt(struct list_head *dst_folios, + struct list_head *src_folios, int nr_items); + #else static inline void putback_movable_pages(struct list_head *l) {} diff --git a/mm/Makefile b/mm/Makefile index 850386a67b3e..f8c7f6b4cebb 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -92,7 +92,7 @@ obj-$(CONFIG_KMSAN) += kmsan/ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o obj-$(CONFIG_MEMTEST) += memtest.o -obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_MIGRATION) += migrate.o copy_pages.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o diff --git a/mm/copy_pages.c b/mm/copy_pages.c new file mode 100644 index 000000000000..0e2231199f66 --- /dev/null +++ b/mm/copy_pages.c @@ -0,0 +1,186 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Parallel page copy routine. + */ + +#include +#include +#include +#include +#include + + +unsigned int limit_mt_num = 4; + +struct copy_item { + char *to; + char *from; + unsigned long chunk_size; +}; + +struct copy_page_info { + struct work_struct copy_page_work; + unsigned long num_items; + struct copy_item item_list[]; +}; + +static void copy_page_routine(char *vto, char *vfrom, + unsigned long chunk_size) +{ + memcpy(vto, vfrom, chunk_size); +} + +static void copy_page_work_queue_thread(struct work_struct *work) +{ + struct copy_page_info *my_work = (struct copy_page_info *)work; + int i; + + for (i = 0; i < my_work->num_items; ++i) + copy_page_routine(my_work->item_list[i].to, + my_work->item_list[i].from, + my_work->item_list[i].chunk_size); +} + +int copy_page_lists_mt(struct list_head *dst_folios, + struct list_head *src_folios, int nr_items) +{ + int err = 0; + unsigned int total_mt_num = limit_mt_num; + int to_node = folio_nid(list_first_entry(dst_folios, struct folio, lru)); + int i; + struct copy_page_info *work_items[32] = {0}; + const struct cpumask *per_node_cpumask = cpumask_of_node(to_node); + int cpu_id_list[32] = {0}; + int cpu; + int max_items_per_thread; + int item_idx; + struct folio *src, *src2, *dst, *dst2; + + total_mt_num = min_t(unsigned int, total_mt_num, + cpumask_weight(per_node_cpumask)); + + if (total_mt_num > 32) + total_mt_num = 32; + + /* Each threads get part of each page, if nr_items < totla_mt_num */ + if (nr_items < total_mt_num) + max_items_per_thread = nr_items; + else + max_items_per_thread = (nr_items / total_mt_num) + + ((nr_items % total_mt_num) ? 1 : 0); + + + for (cpu = 0; cpu < total_mt_num; ++cpu) { + work_items[cpu] = kzalloc(sizeof(struct copy_page_info) + + sizeof(struct copy_item) * max_items_per_thread, + GFP_NOWAIT); + if (!work_items[cpu]) { + err = -ENOMEM; + goto free_work_items; + } + } + + i = 0; + /* TODO: need a better cpu selection method */ + for_each_cpu(cpu, per_node_cpumask) { + if (i >= total_mt_num) + break; + cpu_id_list[i] = cpu; + ++i; + } + + if (nr_items < total_mt_num) { + for (cpu = 0; cpu < total_mt_num; ++cpu) { + INIT_WORK((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + work_items[cpu]->num_items = max_items_per_thread; + } + + item_idx = 0; + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(src, src2, src_folios, lru) { + unsigned long chunk_size = PAGE_SIZE * folio_nr_pages(src) / total_mt_num; + /* XXX: not working in HIGHMEM */ + char *vfrom = page_address(&src->page); + char *vto = page_address(&dst->page); + + VM_WARN_ON(PAGE_SIZE * folio_nr_pages(src) % total_mt_num); + VM_WARN_ON(folio_nr_pages(dst) != folio_nr_pages(src)); + + for (cpu = 0; cpu < total_mt_num; ++cpu) { + work_items[cpu]->item_list[item_idx].to = + vto + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].from = + vfrom + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].chunk_size = + chunk_size; + } + + item_idx++; + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + + for (cpu = 0; cpu < total_mt_num; ++cpu) + queue_work_on(cpu_id_list[cpu], + system_unbound_wq, + (struct work_struct *)work_items[cpu]); + } else { + int num_xfer_per_thread = nr_items / total_mt_num; + int per_cpu_item_idx; + + + for (cpu = 0; cpu < total_mt_num; ++cpu) { + INIT_WORK((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + + work_items[cpu]->num_items = num_xfer_per_thread + + (cpu < (nr_items % total_mt_num)); + } + + cpu = 0; + per_cpu_item_idx = 0; + item_idx = 0; + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(src, src2, src_folios, lru) { + /* XXX: not working in HIGHMEM */ + work_items[cpu]->item_list[per_cpu_item_idx].to = + page_address(&dst->page); + work_items[cpu]->item_list[per_cpu_item_idx].from = + page_address(&src->page); + work_items[cpu]->item_list[per_cpu_item_idx].chunk_size = + PAGE_SIZE * folio_nr_pages(src); + + VM_WARN_ON(folio_nr_pages(dst) != + folio_nr_pages(src)); + + per_cpu_item_idx++; + item_idx++; + dst = dst2; + dst2 = list_next_entry(dst, lru); + + if (per_cpu_item_idx == work_items[cpu]->num_items) { + queue_work_on(cpu_id_list[cpu], + system_unbound_wq, + (struct work_struct *)work_items[cpu]); + per_cpu_item_idx = 0; + cpu++; + } + } + if (item_idx != nr_items) + pr_warn("%s: only %d out of %d pages are transferred\n", + __func__, item_idx - 1, nr_items); + } + + /* Wait until it finishes */ + for (i = 0; i < total_mt_num; ++i) + flush_work((struct work_struct *)work_items[i]); + +free_work_items: + for (cpu = 0; cpu < total_mt_num; ++cpu) + kfree(work_items[cpu]); + + return err; +} diff --git a/mm/migrate.c b/mm/migrate.c index 95c4cc4a7823..18440180d747 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1799,7 +1799,7 @@ static void migrate_folios_batch_move(struct list_head *src_folios, int *nr_retry_pages) { struct folio *folio, *folio2, *dst, *dst2; - int rc, nr_pages = 0, nr_mig_folios = 0; + int rc, nr_pages = 0, total_nr_pages = 0, total_nr_folios = 0; int old_page_state = 0; struct anon_vma *anon_vma = NULL; bool is_lru; @@ -1807,11 +1807,6 @@ static void migrate_folios_batch_move(struct list_head *src_folios, LIST_HEAD(err_src); LIST_HEAD(err_dst); - if (mode != MIGRATE_ASYNC) { - *retry += 1; - return; - } - /* * Iterate over the list of locked src/dst folios to copy the metadata */ @@ -1859,19 +1854,23 @@ static void migrate_folios_batch_move(struct list_head *src_folios, migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, anon_vma, true, ret_folios); migrate_folio_undo_dst(dst, true, put_new_folio, private); - } else /* MIGRATEPAGE_SUCCESS */ - nr_mig_folios++; + } else { /* MIGRATEPAGE_SUCCESS */ + total_nr_pages += nr_pages; + total_nr_folios++; + } dst = dst2; dst2 = list_next_entry(dst, lru); } /* Exit if folio list for batch migration is empty */ - if (!nr_mig_folios) + if (!total_nr_pages) goto out; /* Batch copy the folios */ - { + if (total_nr_pages > 32) { + copy_page_lists_mt(dst_folios, src_folios, total_nr_folios); + } else { dst = list_first_entry(dst_folios, struct folio, lru); dst2 = list_next_entry(dst, lru); list_for_each_entry_safe(folio, folio2, src_folios, lru) { From patchwork Fri Jan 3 17:24:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13925730 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C054E77188 for ; Fri, 3 Jan 2025 17:26:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E7ED86B008A; Fri, 3 Jan 2025 12:26:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E2F896B008C; Fri, 3 Jan 2025 12:26:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA94C6B0092; Fri, 3 Jan 2025 12:26:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AB5EC6B008A for ; Fri, 3 Jan 2025 12:26:07 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 291CE4418C for ; Fri, 3 Jan 2025 17:26:07 +0000 (UTC) X-FDA: 82966817724.16.2188E8C Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2055.outbound.protection.outlook.com [40.107.212.55]) by imf24.hostedemail.com (Postfix) with ESMTP id 56951180014 for ; Fri, 3 Jan 2025 17:25:57 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=DzjP4foA; spf=pass (imf24.hostedemail.com: domain of ziy@nvidia.com designates 40.107.212.55 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1735925115; a=rsa-sha256; cv=pass; b=qTE8ZiRIPXhw/9uS9xrXkpoC8yLbdJXjUkRXpwG27SkTnXl8Ia9FiT14GpOFxgZKlAWREI cJRO8JEYl91orU/O6nJOt0AFHZ5ljMgQWFjYcwNOFXLYIhJvuiQBqZLOXse8KZH0zyL1A7 nQNS9K7mG8aDlrzm5c0r/eSPCNFnJd0= ARC-Authentication-Results: i=2; imf24.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=DzjP4foA; spf=pass (imf24.hostedemail.com: domain of ziy@nvidia.com designates 40.107.212.55 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735925115; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vGh53MmAsD9e3UWlUB2RQAs903jeDpCr5tqEtYEh+TY=; b=K1w20PHmRr6rAZ/3OliVw7j7zeYYDQKjvsPBo//t9tlRf0c/bVZxAWXDo2TJ7nCPtljIEw st2OH6tHGyEATmydPz+XH6Mm2ivZH5sS0DW7LK7bhXb6/ucGIuIoQF/6X4HCAnCCb9ZGa6 PPUXmCICEMAiXRj3Gg6YGBul2021rFo= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=fo2eRtsUAzYhjQpNrQPewosTszzUg0xUssm3q5hnf+8wOgG75NDOqzVGsUy/U13TeQyF8cN+RIws1vQOOc6CU+fF8sopSnbjGQQAMVUtOxE5P2+jY4cja/KYOAAvQBanldrjsXnhIvxBSWMYMMuIULfAmPrM5omi1qgms8IEOWD6iE9U1+Y2ebNhkbu+YdybbxJMvDgZtUv9quNs9kNknqeV4hWTA+qJHrNUmdP0LEZSweVz5POnGY1+MIJsCk/Mgk/va+msSkceWMWQwKV5YqUdW3Pt7fDnRixBIn9V2WFFs7PlKsUmKjd8J/D0EjVnryS1/ObVr1uIpXwLSeqkTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vGh53MmAsD9e3UWlUB2RQAs903jeDpCr5tqEtYEh+TY=; b=FQr+ZA6mgMOK/eHPBbijpb3+yfo04P+5nO+qTEHFh+ENLcbnKFV647/m3lc6EYEq7pvsNvjZnWaLRLqX3ob4mA0v/WJ/ZLX7GAdePS8ZoqpeBqxUMMbNprqk5i3V121I+GSMkMmzEqhOvXlJUUA4J0pTBqsNzD9aCLoOBUZqulx80dd4PIu/a2bZlNyqY7GsgbqX9YMTeq2i99xRKZ0GHuFwt+jsx/Z+TL2xaY0WJkiPzwrz18TNaXZ0WcF6O+JjJNesdIJGXh9cINVvXg35+wekPCUZ9eKV7gAQexKw4+9kC2kMsfpTHh1WxXvV8rscPZCSw3DUBQYzb01ItQHp9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vGh53MmAsD9e3UWlUB2RQAs903jeDpCr5tqEtYEh+TY=; b=DzjP4foAOhmkux48pPlpOXsCIVcbjWE5XW7m4ZkqlDY+Pkc1c7HnNH1B9M7KSuKcWeM1LmUP/hW5VJKrELCCo7nFn8IcbPX6OhrkTyUFi79Ck6nCNglnN+LFWwC//KMEEFhI0u+G0n7M7E+/4fw0UWb+XrkurZ7LJ1/weeCM7DCknZjeT6Nxd77rJhi/RTlDcKbfU/OlI6xLwXaRDISJldEgc0fTAUfZpTabBSU+RAxTnutH9HEdeDj7xW+EbIgWX2SxUXea80HGw80bgoARC5f43eQXRGCNGzTpcYqXXTUnkUn+sAlZxQnt5UbT6s+eVlH5LDgO68+tX3us2uTcRg== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY8PR12MB8066.namprd12.prod.outlook.com (2603:10b6:930:70::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.11; Fri, 3 Jan 2025 17:24:37 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%3]) with mapi id 15.20.8314.013; Fri, 3 Jan 2025 17:24:37 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: David Rientjes , Shivank Garg , Aneesh Kumar , David Hildenbrand , John Hubbard , Kirill Shutemov , Matthew Wilcox , Mel Gorman , "Rao, Bharata Bhasker" , Rik van Riel , RaghavendraKT , Wei Xu , Suyeon Lee , Lei Chen , "Shukla, Santosh" , "Grimm, Jon" , sj@kernel.org, shy828301@gmail.com, Liam Howlett , Gregory Price , "Huang, Ying" , Zi Yan Subject: [RFC PATCH 5/5] test: add sysctl for folio copy tests and adjust NR_MAX_BATCHED_MIGRATION Date: Fri, 3 Jan 2025 12:24:19 -0500 Message-ID: <20250103172419.4148674-6-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250103172419.4148674-1-ziy@nvidia.com> References: <20250103172419.4148674-1-ziy@nvidia.com> X-ClientProxiedBy: BL1PR13CA0187.namprd13.prod.outlook.com (2603:10b6:208:2be::12) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY8PR12MB8066:EE_ X-MS-Office365-Filtering-Correlation-Id: aa4d26f3-5c87-47e1-37b2-08dd2c1b7fb0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: skQen7OtqsoNThIXNUrzMgLpqxB9dVfdItRrK9FXt/TS+IcibzEzZShLL4dnxGzZtYk+zpWEjlJyARkFysuLYmGOITq8KnD31kA3yQQwJ7kDdoZdNZSwEaAsQJD/DQiSz3Vga+4XRrKhwcSo6KY9xlH6GSMPvzHtkbRaofWDy96TzkztdID9fIQ92Ayy2hzZj8HZ08ywGarC/4Da13sI7pHf89fyZHZtObuULIQdCF1aNnPBAvZKRmImLaTUI402RBy92uJSZKPxWXF1e73V+I3kH8mTqv8AQ5SNMWxdQtqOCJTzsNLCocYhXwDo/Kicy+ds1B9myCuxhz9c6F8NI+/ISC+KbBecaROmni6KMpNN36+LExKHKginwuMhI4fnBgvnuKAs09IDLa+FBq1rGFNbmvKxvJm42GdABpEhPHWVpyhZ3zbeSJUHIz/1YpifWHo9PuQaNAbuCz7rbclh3iHw/WRU1Xql8AuNgh16qs6pA63dnDb+6pywnx332dPErKeRJtCX7oTZe+O9OqTA0iQcURKYfoPt9o2x8KHUxLlrng9p+RqOQJMUyQS1H8SBvaVsH2ayMpdtSxFsFzmMFW27SIRWKn+SNaBFQ/OfaOQWIftSZgdFSxfwZxdH1kGvuvQFqRAnB1dufmh/f3DvrRqWpFQ962HYXY74C0hsYVwijD+su3/km7MujlqBQwKubevt9GFqoQKjdrpgD0h2yTnJ/07qD6QKlsDYFEcIDy0rYlhBqa/nmzAluWhKKNomfZujICqYRd9GcbXshhl7R9IaKZtDJyWcb+ZsETcFG7X1JOvf0g+5/ryjNqtL9farfiCDCTsV59Dxq838jzH8sZtpKnW+OToynnX/3kh/rKS9ataDY8FWDHxjnh/oRavaON0y/P9G5PEKT2ELYZm69uA/9t0lO0ipWyuc1cpaOHGlKn62Z5J19u3OphWDGaRCn45guReJ4mgfUCcQaC+ugMxCW8T4NUFokTFkKA1E86PmxTlJQf5TZmlVPRvn4z3aVdRNPoxahD1mt2da7RxaoIs+HW08XUrztmdzauXJJBL1DGIJFIc/FgaNnrsNsQtbz7LQAzP8Su2jYRT3JsnzAS3DEVhs7yr3xrrYUHWDhq7h9nh2CMVZiuauLhTm8fd/qbgWqBV9zkiEHraAHNq7zYIw1mOp7q6YCdT3oqud1lLmUeDV0BX8E4IUD60OwNoP6qTJR/wr0AK+1/hK6vOCTf1SSTm8+4liuC7BLrX16rp7P8E6wJHEam+LS/bGUIJlNPMlLW52vYAInTRtfTPIiAEi9x0mTYk5DOetvOAY7FBGuWBTJPyPtYLSmhUsh3gu0OmP5L6yqEWVsFwPnjk428AewblVsnASwpi8nK7SgHyeMer0sIaGsy2mrO0RmjIz X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +25rtkqRGgj0/l9/eIrgJRvdGKpIWukW7IGp2jYCJWP29x80IbpoEDD9ogj048J8q93XcmOwOJCyFTPvY7s3+bk/5oidz3224tl5OeqJDxhS32nqXUXo6rbpQY2I3wCBOvzPuKLimWkZ2hYe/o2QYCG1i84Kh2mCjxuY+KLCwHoZlCS/1gLCi7ph96nkHfq8mN4ITGG38bmeIch2mjJqRRZRolDPiX9eG8jJcw46dPZXYMjxyBXGVME99vuwCXzZpVPaaUBgZn2Blfw6eU1palq5nEY1jJQX/FEbaIsLM5uO59d/tX3eTJEoVsf6IcH1i00sic5rwiWvXBen2EBjhTeQGblMvgt67E39sal7Al7jpLZ7FN0ncDWoS96Z8OdPq1Gn8NtOFdlQTrH7AyUMxRy3Tfph2Y+QPKxY02uFFnzZxo4mOqP9MCWQh+D9+bKN95s5ryZjKAtu+7Im4kyTau4+7fr2V9sqFCcNYsjImuu7VMyo1BAkXrbQ3xznOrvRjdoUEUYWff3q9syiUzm4T+Wn67Tw8PaWDvq0WociKfhWSCw6X6ZI1N9o2A4KYmft3uol2M/GWg92efqb1jcAmyafpUrnNcu9a5s5QfDDMaJCOS1wnOhz/IVPau4eeuDRYic5CiNDbfZEkYiQxfsMmn4PYqv9Pd1Z1iAcYLe/a771okrs/CWFi+ZcNLovmo6XmJB1ijCoqXLrpYNgGyzX6uIdLzDUiF7J1TM1sfivXHt/RsdodkJ58rScb0tdYCH4OvIT1rT3KSZEjOqqRTXx8pJivPuPJ3hu4/pFAjIU4RqpKqmm1jDZHAURqQdrAL731PDqbZAxcbYRQh0uEPllMwVaIWwcGGb8zH/eT10StHVhqI6HGIv6f31qYaH3CeZm2FRluyXcyP+cMnvtIgw548O/GUQtR8CFs3QAmSDYKvuUTYQ3X+RZLNkNz09vFKl6gTvi2bAcjcNgTmitfR54pMVj1TusqLI7it8RWPZVWlpID0k1MJB+7xfVQ4TWBmaM0CzpOTSyD2Jcpej7v8QUIzhB2VJcH2JDfGY4qY02BnAxTc/XDJfB0Q8dNXs12In5fI9oQasNqDWfJGOb2TYmfX9fDJBP7zs97v39aGcMUerG9ZxqiGGnPuiLQKtscw/Pq83bnHm9BuXRTZJnuOsi+LCP0izoszL8Hg/PC+1fK3JBPNIb6+/ayLIwXHQ6KGYpXo0SFoyGd6B7IRtt44wwaH6gf+Qd4fAsyVQUqMi0Wkze/Ulouq+fGrui4OL86te6hp8lhmUKKHCBvJtedAFGs0V9dNAPGBmqwJSn/r9MZCqh+BUbTTLIaQ+96f+c7R/Q1EFmHuG2mHJ8fMTofoDgHWpSWgCIHrk9VoXowVcpJMw80uf++8HiYwKHzqF9D49ifBaHLuKHVwI6jWguH7AT0iGn4wvIzsZ22lywdVibwMZbkdF8wuzdD8/emLAuKvfpDD5WADwfSVK0euE6+WE9wy0Ane1b/xdYhTcOk4a4Re0dvkU8/pg9eChupb89pr1RR68+Yd6Siym5K5hAHQcT2KXpfXWiLe0I4d/Ht0ujt8g= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: aa4d26f3-5c87-47e1-37b2-08dd2c1b7fb0 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2025 17:24:37.5846 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mIhyGzW5QmDJWHBBA/BoHCfJEl9X3jAbzMlJsjEQwbE9l5eif5BPew19Ibs0pY3v X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8066 X-Rspamd-Queue-Id: 56951180014 X-Stat-Signature: j7gmmxyijt4pw3ce9y4ij83sxxi3ayno X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1735925157-850574 X-HE-Meta: U2FsdGVkX193bZe2vDyR7Z720GK13Ja+11Jw5TwFlL9hbJirq4FO8dSiToMmvWPe4KsxKuP3SF8mDmylzvqIDgMnueigYphsvjML2beI6w8bk1yirh+ISElAKWbwWuCHfMf8+Ff3AECj0LXBMAL7cJ4dOaF3kIeSRMMCBHgdetHYb3cSD+r4V/jTekoNftLFHiTfofMH4AEardDCQtvGLQvDjDIKpj0OejMlFDK1pw2UmWxFAAqtviD89IE8p0gmP0SQq/s9te46y4aLCil8uwkaa6gHFASvSYug/WY9qg+WU8OrJqXF+7E5rDIwcGzfS5SGPOa7cK3YGXeqSZnKheEBiW7s2OrcTt4RkKzgSP5HEJMDQnb+pRR3GfDXwY2jkwSSxbSHAPN/qZXNXpsSpnc3Vk16ienTRjnE879vRTcyZdj/wPonngBPoTe9gTfRtAhYU+MnvkT708rdOQwQPG9b6A0jSdb9ON3CiDdXEecLHR0oiUeGjfhNnXQQN8HnGwUaOCzZjwq9CrTWsidCYJLFG7claqnrCP6U/XReUObbaVcCDRPmdYpkJwlje5r4YpJPFYzLuPBbX3y8HQJGitCesJx9H6J+mGApLpcrUt7dqrUNYRymqx9GBlCvF0BBeRrgeUKz6QGiufzTbwdDSPOyIx5uE257uxLZ4MiTlAn+Hr9cnOsPmJZm3qCqNU3BtlD1zMMgEU3ZeQYP+0LhhExnF88lLOa386HpJDcmT1FC63DeVh3eAoW+vIFQdsWbN68BKtdgRrgpxRD/wPsjIpZQNpgnl8FGlEpapmWj+j47h/bUavywSGFgWvkKmxtv+UDD5BwVY8t0Z2lJcTbVtIhRaKDK8XDUy7CXo4R7A8hCGccrg0Yh2FQuPmGy/Wn7uViOnDNAlFv2iQ56SB3Xk8QZdbf0+PXDn3MBjWZuTeC4qWN3/MvsMI1rbiaycu6ovQbVNU4uv5kTxdRs5RT gQoopC82 2PXz3yAIj4wVDs7f/kqgqRf537UVBF5C5g11MpKtG8ha3PE/O8ZRVgNmDYHtY/zSwhIWiQQmGac9OS81U/4YBeP5SMkxje9QmBYde+38bA2Vor15tgtxEQ+EXLyPLEplAB4ZP7VcHMEEdtJ3pMKOiqhUW0Qi/fFDRWjs6LkfghOBBWrd1vfqLD1Gmk5SIPd59qBMhGFL8eqgziOtagH10oegOrHyMp75adPTMY0d6Fxww+NxPliwx1DB5HZ/gIUWxYjv3SVI4sC6NNgLsMZMwo52FhAL7T7sVvw7bKto1IxODLvlvMaegY3plMQ/s30BJUO/vPxv/TFV6OHqxRKou9WZWeuIPvGZUFHVZy8rI6YUF6EI+dDNlSmJexN5xorJrq/ZEwjStawL5ze7gghvwZrMLIEdlgae+OOPMKD87eI1mSdJUrBChm2VYHz/eKonw2hXLqNTN8vMRmixZLwAk6v+IqQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: 1. enable multi-threaded copy 2. specify how many CPU threads to use 3. push from local CPUs or pull from remote CPUs 4. change NR_MAX_BATCHED_MIGRATION to HPAGE_PUD_NR to allow batching THP copies. These are for testing purpose only. Signed-off-by: Zi Yan --- include/linux/mm.h | 4 ++++ include/linux/sysctl.h | 1 + kernel/sysctl.c | 29 ++++++++++++++++++++++++++++- mm/copy_pages.c | 10 +++++++--- mm/migrate.c | 6 ++++-- 5 files changed, 44 insertions(+), 6 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1a11f9df5c2d..277b12b9ef0d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -42,6 +42,10 @@ struct pt_regs; struct folio_batch; extern int sysctl_page_lock_unfairness; +extern int sysctl_use_mt_copy; +extern unsigned int sysctl_limit_mt_num; +extern unsigned int sysctl_push_0_pull_1; + void mm_core_init(void); void init_mm_internals(void); diff --git a/include/linux/sysctl.h b/include/linux/sysctl.h index 40a6ac6c9713..f33dafea2533 100644 --- a/include/linux/sysctl.h +++ b/include/linux/sysctl.h @@ -52,6 +52,7 @@ struct ctl_dir; /* this is needed for the proc_dointvec_minmax for [fs_]overflow UID and GID */ #define SYSCTL_MAXOLDUID ((void *)&sysctl_vals[10]) #define SYSCTL_NEG_ONE ((void *)&sysctl_vals[11]) +#define SYSCTL_32 ((void *)&sysctl_vals[12]) extern const int sysctl_vals[]; diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 5c9202cb8f59..f9ba48cd6e09 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -82,7 +82,7 @@ #endif /* shared constants to be used in various sysctls */ -const int sysctl_vals[] = { 0, 1, 2, 3, 4, 100, 200, 1000, 3000, INT_MAX, 65535, -1 }; +const int sysctl_vals[] = { 0, 1, 2, 3, 4, 100, 200, 1000, 3000, INT_MAX, 65535, -1, 32 }; EXPORT_SYMBOL(sysctl_vals); const unsigned long sysctl_long_vals[] = { 0, 1, LONG_MAX }; @@ -2091,6 +2091,33 @@ static struct ctl_table vm_table[] = { .extra2 = SYSCTL_ONE, }, #endif + { + .procname = "use_mt_copy", + .data = &use_mt_copy, + .maxlen = sizeof(use_mt_copy), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, + { + .procname = "limit_mt_num", + .data = &limit_mt_num, + .maxlen = sizeof(limit_mt_num), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ONE, + .extra2 = SYSCTL_32, + }, + { + .procname = "push_0_pull_1", + .data = &push_0_pull_1, + .maxlen = sizeof(push_0_pull_1), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, { .procname = "drop_caches", .data = &sysctl_drop_caches, diff --git a/mm/copy_pages.c b/mm/copy_pages.c index 0e2231199f66..257034550c86 100644 --- a/mm/copy_pages.c +++ b/mm/copy_pages.c @@ -10,7 +10,9 @@ #include -unsigned int limit_mt_num = 4; +unsigned int sysctl_limit_mt_num = 4; +/* push by default */ +unsigned int sysctl_push_0_pull_1; struct copy_item { char *to; @@ -45,11 +47,13 @@ int copy_page_lists_mt(struct list_head *dst_folios, struct list_head *src_folios, int nr_items) { int err = 0; - unsigned int total_mt_num = limit_mt_num; + unsigned int total_mt_num = sysctl_limit_mt_num; int to_node = folio_nid(list_first_entry(dst_folios, struct folio, lru)); + int from_node = folio_nid(list_first_entry(src_folios, struct folio, lru)); int i; struct copy_page_info *work_items[32] = {0}; - const struct cpumask *per_node_cpumask = cpumask_of_node(to_node); + const struct cpumask *per_node_cpumask = + cpumask_of_node(sysctl_push_0_pull_1 ? to_node : from_node); int cpu_id_list[32] = {0}; int cpu; int max_items_per_thread; diff --git a/mm/migrate.c b/mm/migrate.c index 18440180d747..0f7a4b09acda 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -51,6 +51,7 @@ #include "internal.h" +int sysctl_use_mt_copy; bool isolate_movable_page(struct page *page, isolate_mode_t mode) { @@ -1621,7 +1622,7 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f } #ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define NR_MAX_BATCHED_MIGRATION HPAGE_PMD_NR +#define NR_MAX_BATCHED_MIGRATION HPAGE_PUD_NR #else #define NR_MAX_BATCHED_MIGRATION 512 #endif @@ -1868,7 +1869,8 @@ static void migrate_folios_batch_move(struct list_head *src_folios, goto out; /* Batch copy the folios */ - if (total_nr_pages > 32) { + /* if (total_nr_pages > 32) { */ + if (sysctl_use_mt_copy) { copy_page_lists_mt(dst_folios, src_folios, total_nr_folios); } else { dst = list_first_entry(dst_folios, struct folio, lru);