From patchwork Fri Jan 3 17:24:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13925733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94F9CE77188 for ; Fri, 3 Jan 2025 17:27:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 336746B0098; Fri, 3 Jan 2025 12:27:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BF606B0099; Fri, 3 Jan 2025 12:27:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0EAF66B009A; Fri, 3 Jan 2025 12:27:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DF4EB6B0098 for ; Fri, 3 Jan 2025 12:27:48 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7E11E807AC for ; Fri, 3 Jan 2025 17:27:48 +0000 (UTC) X-FDA: 82966820412.01.8ADBF23 Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2051.outbound.protection.outlook.com [40.107.101.51]) by imf01.hostedemail.com (Postfix) with ESMTP id 3C5A940016 for ; Fri, 3 Jan 2025 17:27:06 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=ZUm3JNYu; spf=pass (imf01.hostedemail.com: domain of ziy@nvidia.com designates 40.107.101.51 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735925226; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pkrR6pMPjoPUyRjyi/crq5jutxBvrS/X6+z25YveCkI=; b=7+X3Vwpo893ONwUtqNg9iyZ/V/AOsAlnh750B7WMHYVeni6Uhi0NyM4+K5nj4f5dh8vBir MR/kIQDQnfzhgXHNT5+CzDvZFaE7zgIhLPL64fcrVc1M2vnxhGMuVK0lTJXS5la3QBKZZX OhhEHcdDVibm4G2YKXSbfNIo/0dj4dI= ARC-Authentication-Results: i=2; imf01.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=ZUm3JNYu; spf=pass (imf01.hostedemail.com: domain of ziy@nvidia.com designates 40.107.101.51 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1735925226; a=rsa-sha256; cv=pass; b=VwnjrwCw5Orn0jguzJLun2OyT0XbRGCWGumbKxdet0SnBgaxSuY0jPBtqXSQbBzvymxjpd F1s+f4RTZo5Pt1GkerCQhIdT42laqhc04mrRSSRXz1ZEnEitcAm7guV/i78gcPnacpxgqA 1S8qyfKbfR58kehr6U/sa7Zvej7KSU8= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PyEOL7Mq+daLVE+woSiEnU6l+7zecW146HG88Cf/PHb6oa0h/irCF3pnNsbqT+508ZGzpjbW2P4ngvKXYs0Rbe99u5fA35A5xhmHRp3FhqNG4XxPtI8yfp0F72OFwiZYFa0AQE8FuJ2muAaKhaLNGdFt0dUDhBGa9gUwho3phaZ+WpYG5rMni4Sk7upTlaI31cDCFwNGvzKoHpZPx53p+3Aok0OnJydXCSVPWIu+8lh2ZLqS5WPBEqYjtDKBU5KeD2f0+YbGQunC16NCOE5uBdWRx8PCGfjMFoUyGYFVVTBqavQq+jC3aKkruU+cQZs+RQvVVxkU54iTPNP48AWU/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pkrR6pMPjoPUyRjyi/crq5jutxBvrS/X6+z25YveCkI=; b=GfS90n7WeS+XGai41l30ArSykwx8b/RzGsJV+PILyqhSLioQQOxRHNadWWtW19Ruqb4O+RttBxnaEgTG1c7CuWXKFe+Y/lrF0jOcrQY3q2nUt+/a99vRB0eG1aVBG9Che2/ssWh/J/53UM5LQGdSaIh2adW3sZog6zfdUqi2rT5oPcX9gvBVp3equV3MgMORfo++5jr9X/2vmIVgZMS7awsBr0DbNF/eUwiQelrMm0eJJhe98UMytVRYAcQ21pcxlyNemO9M8kfP39apy6kYdlAAUkhDBfBQUkMiIspSRlQqLy0/ViJ9h/S1XCZmJJFpNWW3IFi9eFQ9FNBifARLyA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pkrR6pMPjoPUyRjyi/crq5jutxBvrS/X6+z25YveCkI=; b=ZUm3JNYupyRAG5V7LDQVR3T7WKX1lZAXXNn5rvF4E3CusAEJ8joVKMqTeezGBckXuJav7frH2jRZ8jIIBw/umGIWGAlGLVZmO2qsxYvPQdSK6jjRtz6ZJZK4B3nsP82Xxa80JuZM8wXhXPv9/h+RvZSZtRyg5lQsE1J+JCt4B5gspa5a9/NegiBoXPRiuLbLEWutNFmJ4svnmcC3W/9VtqUZhbd5Okw7lo6lLaeMt48J7IWkcs5Dhl+tL7OUeveAGplIpnhvOVA28jle3L+/fqa0ncCHZJDZqsL5kFL5HOhjHfXdqwxeQ4czluZ79B2sAk2mX32r3/pOE7PsZfbPtQ== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY8PR12MB8066.namprd12.prod.outlook.com (2603:10b6:930:70::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.11; Fri, 3 Jan 2025 17:24:31 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%3]) with mapi id 15.20.8314.013; Fri, 3 Jan 2025 17:24:31 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: David Rientjes , Shivank Garg , Aneesh Kumar , David Hildenbrand , John Hubbard , Kirill Shutemov , Matthew Wilcox , Mel Gorman , "Rao, Bharata Bhasker" , Rik van Riel , RaghavendraKT , Wei Xu , Suyeon Lee , Lei Chen , "Shukla, Santosh" , "Grimm, Jon" , sj@kernel.org, shy828301@gmail.com, Liam Howlett , Gregory Price , "Huang, Ying" , Byungchul Park Subject: [RFC PATCH 1/5] mm: separate move/undo doing on folio list from migrate_pages_batch() Date: Fri, 3 Jan 2025 12:24:15 -0500 Message-ID: <20250103172419.4148674-2-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250103172419.4148674-1-ziy@nvidia.com> References: <20250103172419.4148674-1-ziy@nvidia.com> X-ClientProxiedBy: BL1PR13CA0198.namprd13.prod.outlook.com (2603:10b6:208:2be::23) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY8PR12MB8066:EE_ X-MS-Office365-Filtering-Correlation-Id: ef0ec1b5-013f-4c3d-eefc-08dd2c1b7c2b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: NjlRwJUvYdEK2xcKV5aPKMLbHv98gJMVCK1UUfPD1/QZA9WbahVsJnSLNrsNFWbWzGL+wN8AzxEiFjJM9CmzNRMf4JGz9jybtA48zo4tt/cCFUkmKqcQ4hHtk/G3OqwqwiJuWHsy8p3ILMTTgt0blRjbQJvXTXItxwMvTcD//CZfb8AX/UZdDMLzb72568YpPpLsa9ZWVwutEYVLOcaSFPP8Q2dR4dZspXaaCCjp0Fva7GvqYiMVQV9oB4HNDri39QsGuWan9/xJcyeRRmxp7dDbV5uUnbO8HxxOEIeoxufGyAn/M3ePhqb2OSIp/2qvqW3pMou2mHShwPsYa5fMVYT3Sg95UA8augWzzuLKjE4NyUlbVAvp89n6prp3TtK6Ib7/mOceMqD5O0v91tv6I9bEBioAN57xF+1lfDA8ZLDUVk0IxiWWLQXbPXzbFEgejt0Gexx2rGPnEcJxtozOpQc8zkRxapU1UVlkkQnkyvAWg6us8owKmsIht8ECUjCOfNmOdJI2vvubgxmE7F1rbd5FFXo0SNvh7hAgSjtsQ76Giqoi1dHL0WRR25+D0EMlU7w9+WNRciKMzHDcyukKHtXpXsO9xb6osAaSqTHTWhb/Rhj0ibhDuxp+eusA9EWFSNHNYcc0P0TTxy8BzqZBND62DHn/HNGk2jjUCTG0wnghylVFfVwzWmoiv/kLkZKqVUhib5yQ01ZeFoiSZ8pj7xdfbXl0aChgbMDMLuCEyE8zKQocEjwd//mwhXmmicrG3UkW2o5kY54x+CyJAIowribW1PIQc1BXB0wDAt+c5eRmXBCcPW+lFoWDe37NyenJHX4AzpHPWSADl0a8ythIQbwChPGosYBAdRai87cDp0aoW31ASXBEvF2hhSP/sqNlaukJ04+JAlUugbPiAtOAsgDE2uvSjR/1kbr1rmxtDn38qVpU1XxXxmy6tzFchbH2jC1P9PfDG+88w4fXYoV4Qx1zGDupFd5ATV7E2LWmAcY5jPVuQyjrjDj4biSaOWbns4r9hr0XCNGA3kId3qB9rmsxhAqaHNyFYfHsI77mfaU+OK1dioiJTKjLYg6XnEbNDFsV1nhU486h+C1lV1g8h928lqGD1pSO5E6fnb/B6BUFymOt0lfd/MPoGZI5tpbdxnp5Y8g/IKJ9b+eXIxtKFH53u+sK64GiqqD0+nQRYZkriIqz1xSOFOBZufRXEXbozMG1ZgRGJUB9svhgtqsfeBm5uBtJiKS2+0DsIrmB1GnNVMsrPqzAgIJ6U56K2o8PJD4nZDWiqROduL+GmvZdzamY2OCY2jQVoZOKYeMHf1K5u1i49ZPFM/MhK0ComJD4ZFTOzN8Yy9LkrsaACchclDDbgb24TdzBoPKCQmE+u8uhkNtOJK6YHbxQWy7hX5B/ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: C4o6Mww8vDCuscw+/E8qZV+3z4cbmevKK6xnRKxokzE/PvdgUijCbYy5c7y/JJUmZ4YvyM9hDfgw82HVskrGLR0cvUBXAsKvuRJfs1WYYaNm776DtHja70eSrk7sPUtfciH52W/cl6S7uaAGu9+aLloiBBl75wcYk+IPHr1mvnBdTXN/LzLgsXmM1xjcyX9xX/t48avKxuATloMtMBQMlV02Kwg4jUZZGAS/NJOvxy12PUfCVEy+JicuZhBCnId5OzhWKzwCaFMcTmNz5ORs1ewE43gWde7r/gCGilwFQtuxQIDmTZYWyCSz/qUSE4K4qCQO7pNjHwJ6Q6YXXVeOldycOTivRDHlHc+7HowWoc23QHGvcImc2ECHMuJkOjTlCs8FG6Fbre1v3t5fR9qfcqHgSJEnnJXYJ+1hX9xN9OGVOCLq/6nEsoLI0+myweVoBvJHxUmF7LzYMZE1IpT5cEc27pEHno1sgwYDdccSAdzKLaseUB/OnAS2xyVSswfRO4p3Q+lZOqC4NzD0803WQS/Cmb8wqWBfGQAVV4ipH5XJcKxiwcmnETHgxrrD8ovWSpyBHwgAJ1ELw3+UScVY0l4g9UC/djqnVXKqtIV4B+O+svZvfV/OhzunfHTBtv750xSyAdblE2MzGdMC+v8WDwUwf4r8gRttOM6Gya0+g99ENAvVr/P33DAmEh/QSnu+F3tNO1T6k1LwaOZnElRRl0/GFfawOpRTQs6kkl9I8ft/Fb5jl+j8uEgEnIqbSPABUX9rzaOIGKMNa6cULLd1uDBPTjpeoxm7D0EWvWkH9pj0S10WjwvMPO2hOmTfhRnr+J2G/4+fGeWt4Gxu78KjRStpVxoedRsnHfL+8CsDhyfsumi2R1QsW159FEt/57Bvt2RDi/w4Pndr4h/SESLacBz0+7Buob+N8B+uJxAZmnpOtcBp9gxxe6Miif2RXJ21ZWXh1rqMhz1mFIOxE5SuPWxzlBf1QuKXoOPTl2gN1kdpHSV204tNrOYw9rKapUTZE1L4tg+3B5f6uCZtZp8usKd8g3jqpqnNGIhDySdrvjL1wL7ymhY67DuB1Q8+JHzJ8tAaeHrVFaEKxa9Qsiwuzzm3HPyAKJ4P6tJdBxPw3jziGmQ0gqgorzI60U33U7z2QWv0/TwBcLTgjs7e2FF1U94S4+pQfKgPF+xvnx/EzK1bsaw5P+VVIQnZ9eVbVoO23nW+0o5N8Ifb0SZwVTz+aQ8scyNjPkFfKy7x7jz7QquU3v61FlpLRYQezcto/wnvUGxCEGN1dIW+VHxpNd0cKP4oX4xflQSvgdSALKQYTs0tOHJouOWuEFCZWLJ7kxAJRwUCfXUIno5XvWpC6TlVm5XYGcerWtXOektw/86xJC/6V/SSrDYAyKBfr0aOPofSUQUhyQXRJjP2Kr24sd3PTqxtrDz9EyvzQzzUQzdbGHqqYYG/PvOW0BQaIj2CCEVzlRVFhDTqIGDmklVQgG3jpZzb9LiHHwsx2SjfB6BCGxt6vD1lQtwB5ZiI+nB8SM8aKXJF5M/bXQFi+Jb0WTCn4UeWzvbD64EYZihgARf8pPvrv0W7t+KVNK7SxFKEI2Fx X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: ef0ec1b5-013f-4c3d-eefc-08dd2c1b7c2b X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2025 17:24:31.6742 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: I6xYX+Lz1Vmthy+Jim/RlUfHpXds5oltnukYHBjWOyjz+c//UQBxdiPjuHZzkh9M X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8066 X-Rspamd-Queue-Id: 3C5A940016 X-Rspamd-Server: rspam12 X-Stat-Signature: fe8565uxuymna7eot5to5w6eiwhd43xh X-Rspam-User: X-HE-Tag: 1735925226-76549 X-HE-Meta: U2FsdGVkX1/7JlmGsp1F/kmm/1K7DyZpAzaRv/Q3Re17JBeWDjYXWkxme7qcCkXHhifmN3Gk4HtFPjUV7k5TpNRZGGuPHdysEGUdfunrAXQTDUMooL5Zgm5FJs4eFxqSmy4nSQnXv2QvIfRRMHDdvLhYAyH4JkxEBzyYZJCfVKflMCLH/G763ED7uXreXDSBTYtYoy1SIj4YKAinq/Lgkjvj7Rximea6COMG91Krxjpsdlny3jqODwwuS/jsVjEX/rrI5pSzR4L46fHdBsKzeGg2QQuBaVxP/pGaTzceZBwGmxprtT/7DKeCqUR8ZKuxZrbPRzt4SLZXtHdMWBSY7I8aHuKn7Wns0ElcYNwXhWrgm6beJVMj58sCnoPbGG49hBfoVvpQGc3V3bfKIapixg9xc4E4U9tTutsp+6b2k5XnnDA7nx/D5MaDj5vQ5q5ygQ1CWX3IdIMwxcMfBS153XrIHvpT/YA6C7GLAe9BQKSVKbqt43pxy7fUoN7CfXHMEF75+hpcq23KR5+S6MHH/h9pz9U4M93umSJeUGWukFZ9jd7MPYoKM5TGZqjE0z1Omy1IZl+lIJbA8EAjR0d4Fmqe6IxqAsPI9zpZWmmicQY6Ub0hPEAOLTHKMw0Zj6w4ecpRIaOzmJaZzYpUGZEu8uaAfyBM0p0O+00mTpV7j5YPYfz+t1rndOiaQzHNqamHUKvrja4cotOLKhfcqEdhO0q55U2Zba09W4he4yl3nkFtDg+4M5sv85D6yvG6bpEFLl7NPKhbYMVyeSNPZAlgleixD1wd0PopWpmvFQaUcfX/BTRJymhzm1bTCi+Dre5r4fbJAVkRUX4ksw1MYQhqm7zeuyzUFmtkrNDcJS0daXyxMAqAQs3L15Lvi67reLd1Bu6dYFOn0WntaeLJev6Jj+cox44z0Z6X64Sp54yyQ/DlkS2FVAF0QuBjvLaaSBjWx6kWfQLFQVSG8hWIWit Jn/srE/l wXiY/k8CnFg8H9tmztvON5DC4HCxDBYOtWoNsIFd+m2DLJai+3xStgEXUriBvgUbVqtN9J6sPTK8xTofx9amtmuobwaneWu+0+RwrI9T4r6O96r97/BYCHhd9H4/6rj/n9/7OIZsFtpIUSHmnHMB1C4tjWGWipTSDU/5uUUGUfgPAbu46eRb/6hYO3mME+rA1jAdoaTj5XqD8gXRRltnny0u8Ilh24p0eP9+0ChHxbxl0Zmv4I1WvRvQbN6jcO6s7C3AlKWHFGTLCcBlefoLiFkh8svNGBPG8HPKBGi9fa/EG9SBq+MgsSp58NaARC6DeuobY7B9cAFHJ9xbYkq7cIPuXW4O7EUZN2b1P+z8dj5mQ6g3lWs4G5dLs1cV/3doGVdd/DciTtL5Vh+O/fGQ8k/a2p4cNW72EfgbJ1oRs0QaV3LK5BRnF9ZFHZX2/Uz2E7jgqsK6X5Zcwbo1Pqd8J+WfCaAw7ktzyCWgeKY3x4TOug3k+mGZ4NFfVorfvn0YizlWZ1LiSU+aXf8U= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Byungchul Park Functionally, no change. This is a preparatory patch picked from luf (lazy unmap flush) patch series. This patch improve code organization and readability for steps involving migrate_folio_move(). Refactored migrate_pages_batch() and separated move and undo parts operating on folio list, from migrate_pages_batch(). Signed-off-by: Byungchul Park Signed-off-by: Shivank Garg --- mm/migrate.c | 134 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 83 insertions(+), 51 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index caadbe393aa2..df1b615c8114 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1690,6 +1690,81 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio, return nr_failed; } +static void migrate_folios_move(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + enum migrate_mode mode, int reason, + struct list_head *ret_folios, + struct migrate_pages_stats *stats, + int *retry, int *thp_retry, int *nr_failed, + int *nr_retry_pages) +{ + struct folio *folio, *folio2, *dst, *dst2; + bool is_thp; + int nr_pages; + int rc; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = migrate_folio_move(put_new_folio, private, + folio, dst, mode, + reason, ret_folios); + /* + * The rules are: + * Success: folio will be freed + * -EAGAIN: stay on the unmap_folios list + * Other errno: put on ret_folios list + */ + switch (rc) { + case -EAGAIN: + *retry += 1; + *thp_retry += is_thp; + *nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + break; + default: + *nr_failed += 1; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; + break; + } + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +} + +static void migrate_folios_undo(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + struct list_head *ret_folios) +{ + struct folio *folio, *folio2, *dst, *dst2; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + int old_page_state = 0; + struct anon_vma *anon_vma = NULL; + + __migrate_folio_extract(dst, &old_page_state, &anon_vma); + migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_folio, private); + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +} + /* * migrate_pages_batch() first unmaps folios in the from list as many as * possible, then move the unmapped folios. @@ -1712,7 +1787,7 @@ static int migrate_pages_batch(struct list_head *from, int pass = 0; bool is_thp = false; bool is_large = false; - struct folio *folio, *folio2, *dst = NULL, *dst2; + struct folio *folio, *folio2, *dst = NULL; int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); @@ -1883,42 +1958,11 @@ static int migrate_pages_batch(struct list_head *from, thp_retry = 0; nr_retry_pages = 0; - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 = list_next_entry(dst, lru); - list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { - is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); - nr_pages = folio_nr_pages(folio); - - cond_resched(); - - rc = migrate_folio_move(put_new_folio, private, - folio, dst, mode, - reason, ret_folios); - /* - * The rules are: - * Success: folio will be freed - * -EAGAIN: stay on the unmap_folios list - * Other errno: put on ret_folios list - */ - switch(rc) { - case -EAGAIN: - retry++; - thp_retry += is_thp; - nr_retry_pages += nr_pages; - break; - case MIGRATEPAGE_SUCCESS: - stats->nr_succeeded += nr_pages; - stats->nr_thp_succeeded += is_thp; - break; - default: - nr_failed++; - stats->nr_thp_failed += is_thp; - stats->nr_failed_pages += nr_pages; - break; - } - dst = dst2; - dst2 = list_next_entry(dst, lru); - } + /* Move the unmapped folios */ + migrate_folios_move(&unmap_folios, &dst_folios, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1927,20 +1971,8 @@ static int migrate_pages_batch(struct list_head *from, rc = rc_saved ? : nr_failed; out: /* Cleanup remaining folios */ - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 = list_next_entry(dst, lru); - list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { - int old_page_state = 0; - struct anon_vma *anon_vma = NULL; - - __migrate_folio_extract(dst, &old_page_state, &anon_vma); - migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, - anon_vma, true, ret_folios); - list_del(&dst->lru); - migrate_folio_undo_dst(dst, true, put_new_folio, private); - dst = dst2; - dst2 = list_next_entry(dst, lru); - } + migrate_folios_undo(&unmap_folios, &dst_folios, + put_new_folio, private, ret_folios); return rc; }