From patchwork Fri Jan 3 17:24:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13925729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B9FCE7718F for ; Fri, 3 Jan 2025 17:24:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E7E766B0088; Fri, 3 Jan 2025 12:24:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E2F246B0089; Fri, 3 Jan 2025 12:24:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA7E56B008A; Fri, 3 Jan 2025 12:24:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id AD0646B0088 for ; Fri, 3 Jan 2025 12:24:53 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 3456E1C52F1 for ; Fri, 3 Jan 2025 17:24:53 +0000 (UTC) X-FDA: 82966814070.10.7D4E19D Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2043.outbound.protection.outlook.com [40.107.212.43]) by imf12.hostedemail.com (Postfix) with ESMTP id A095D40018 for ; Fri, 3 Jan 2025 17:24:27 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="m+ivf/XK"; spf=pass (imf12.hostedemail.com: domain of ziy@nvidia.com designates 40.107.212.43 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735925066; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uwRxhfDx0PPdt2UKRR7o8d87pn7X56g+pIfPcK/e5J0=; b=6CiM7uL8LuYQnNN1xb4tiBQLcSGMyr8L9A8CKXXPxy65bMxYrD+h8oD9svwPOlDZbgOblA 0I8IidYwhNa9P27SzYdTIXY5pGOLU4qw8mwaWZWzmrvRetvWOb2Mwxp2MqTvrZtHvNUPsm VWQ5i33vf7eyy2I/il8MSLqKl9Jh7RI= ARC-Authentication-Results: i=2; imf12.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="m+ivf/XK"; spf=pass (imf12.hostedemail.com: domain of ziy@nvidia.com designates 40.107.212.43 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1735925066; a=rsa-sha256; cv=pass; b=IryrzIz6svz4jfN6XiakbmddMYKhpogbQdtcnF2rBGhJ72zrSKzJNpFuLVCWYl09nTzmsa HAQKfIVvljVV8w4tbCaYvyTZV8Vv1g/GqqGi/lXPJ5sHPDzy3Ipl2pwww4DjIAjCpQLGM4 vxlvcbVQfo3zROHZpoEnXKltZDppr3E= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=G08pdsDn6bOlabAryqgvUPXKJolk0z0xxSytV4fGa3+/C/gCFHpFotGdNGOg+a7VW30OtQPUpwzwdieDLImCZz9PQebrTHkghNHOxtYOnI5oAu5hHBnK/gamO9h86x6td5gjJHFkQ32PDQ4KyPtUxZM+sclUrqwPgHFA+I38jKCsJT4OubSH8O6J7gOcezy8iAcAVuOnsnmNkHgf0fbDeRkGuRcufItiKZ96gOOetGF5vZiceL7XtLRDvr02XhK3+isLizC2Z49seiYgsN8GWEJ7HcQZmmOTFLN2pE2lMsQAum3ZfGrBsLpoQQG9eb5QhNPfnG4EpHLrZzJIge7Y0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=uwRxhfDx0PPdt2UKRR7o8d87pn7X56g+pIfPcK/e5J0=; b=ybdCywDmMdAhGOQB67mEbvnTqpZFU4B6RP2xbLgcbEtSTZNMVxqCNV6hmuUf+24ja5tASY640Hgr20N+4vkdUrl54EIP5AZnsI2p6Bim/yARnIq/cYLqBf5vaPaWrbXJ/9NGGvSS0kRvKZD4zsx57zeY/Ds83KNEYIpPPlZm2ArwnARmood4duiCg4uID4oPjxOCPWOOAOvYBNCW+hkH/WeSPodnu47r55ywhggdN5EWZHdDW/EpZZngn18RbZgLJOa0VnnBQDVy+wA99CewMS/kkurkNUGAkVDygkv8f0kJzifPT727KTSXuxRD+09w9BgBQPc+osJy1yHZoeyNag== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uwRxhfDx0PPdt2UKRR7o8d87pn7X56g+pIfPcK/e5J0=; b=m+ivf/XKxRBnVRQrr/Rn8p+Ejp/lo0uWjDjcd1G8Ez5zkKK7pkh3TPhn+kTuW6qgR1yoT4RnbaZGqgSivgPs8PMFXzUaF/CF1fveYfWIblBJ26RZ6s7HhVPAfdMszyFWoQ2Ym7SnqV3MehoMEUbIChYbKbi1379QwhMGJNtpYko/HUjHzZM9jajQYTSkCAl4cUliM2g90cGFM5Cg+FruHywc/x4OQHPqi9faqNDtRFP+27TzDJR8oLC+QKSfkHI1t3YEQgK6IR3TCw0hJORivdH9bhtnTAKf9y1jf5N2pePIcac5VIYitteRVaGmg9bcjf2i8uQlsjSHlxOrvDXdzA== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY8PR12MB8066.namprd12.prod.outlook.com (2603:10b6:930:70::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.11; Fri, 3 Jan 2025 17:24:33 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%3]) with mapi id 15.20.8314.013; Fri, 3 Jan 2025 17:24:33 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: David Rientjes , Shivank Garg , Aneesh Kumar , David Hildenbrand , John Hubbard , Kirill Shutemov , Matthew Wilcox , Mel Gorman , "Rao, Bharata Bhasker" , Rik van Riel , RaghavendraKT , Wei Xu , Suyeon Lee , Lei Chen , "Shukla, Santosh" , "Grimm, Jon" , sj@kernel.org, shy828301@gmail.com, Liam Howlett , Gregory Price , "Huang, Ying" , Zi Yan Subject: [RFC PATCH 2/5] mm/migrate: factor out code in move_to_new_folio() and migrate_folio_move() Date: Fri, 3 Jan 2025 12:24:16 -0500 Message-ID: <20250103172419.4148674-3-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250103172419.4148674-1-ziy@nvidia.com> References: <20250103172419.4148674-1-ziy@nvidia.com> X-ClientProxiedBy: BL1PR13CA0205.namprd13.prod.outlook.com (2603:10b6:208:2be::30) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY8PR12MB8066:EE_ X-MS-Office365-Filtering-Correlation-Id: 5baec480-c932-4d6b-1e02-08dd2c1b7d24 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: K6k8+fFIByCriKA/qWkDv7j9vZknOJ6Xf8B9hwEzSVH2d1bDKw/uqbcaequ8/o9Y98hOseDu0vW+U9V7c78KG3JanA63Lw51RG2CSgXIZihW8TSReKNSSKloxmpKbGgKk3TYjpKK6fcjanfR0DUB7QiK7fO1raJQv5TdzALmQ3UpYOU1oQhJDa0BRLlDxeo7Zj8/gp7Y3p/QBghbyubSoDLZvez6lMcCrwwwXwstpNnCtZtceO/QdH/BmgoHDod4oiTyIdBqz+UBetI6eAGSYzxtHWfKjbaFkEb6wmSPlW51vZcHl6U19QWHzkuwED88RNEoirvadnTkCZ0YfTlKWSzj91OKRlcTqwwKsRimCByu4620bXC7n6XDCkbPXfPbtFBp/kyZVE93wY4bYUhMgudk/TbLMGrHbQivd0usOlv7WQY/fSp3vMPSqLPA9yLSHYU22ucZnLEwqlj9fohx3G39a5l7KFj6WeaWXpVMx543LCpp+j5nILgS9Fxbxmn0ofieyNmRbmTTPHC5A0gCtS+N41yGzaZoySHH34KlbYPcUM2+vFPiHpnsDKP4hcGv0O7l6BDZjsiKtQ1tJsNt0bVrtkI28QjIV3bB/wbop23RTm7czEyAmMqXWId/6Ck1ixLNdsCjNVcIemtuKYNIxgqXvx44noriPkBxO8iPYHi2vJ+6aJSfVnoOaT/FpK+7oU1GuaHEE9cWGitl46n0c2YgY2xe873egyX9Lh6VSbRusR5ixBpetp+hUzXBFcG3VL+Sp0CSo7xNtxCgRy7AsQzHqClVli+Q4EWtDraYkX4mt3IKp6Yi9fe0r+E0G2vAC441Vrq+3fCg4EQQPMrMEcAXg+mQKBRt848CZ++8Wg22vmDztUG+QhkFMv9kf5nCQxcg2VE4qU3J2OJGW/u21HMbOFkU/1knhZl3txNerYlmrsSE9L76QL5Z3BgkvUIfvc9X+BaKkZ8YId8TTqoxuP5ndpKERerDjPCIaOaNkBJeb6nYLcYRIcY/R1+HdQOu+izClA10l6NCPcYhSGQAar2zXRnowa9Gjt9D3S7vly+Y1vmXSYxdtullo4glbpuCg4b+dQayI3A2CBi5J8RGoNYkYMpOlgVLQJHuHiM88/XXV/51FeEaOW+GtgAYEWqY3py3d+3WxcGPGE8IpxC32cRQZmZguI7hJPjKBcI1dUBjLlZPhWIzni9TosxO1Wm9E0oXS730V0u/Nb7VDejbibnrHZj6Z6LUVU+kWFzqdnQrDrEkXRBnkQC6W8vf6TzvMqKbdDFpS8igEYRRKJW3xzBA8gg04nz1SWLEq1v1/d9Z/GUBBEarFTs0nRe/J1KmY9azfwe3eg9Ds9aT1VgGx+hqeJiDThVlqTwbVFRF7lOGX329JmgwfOd+nPtYY6E5 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: l+6eK/ORncnf5MRbZWjPsEz7ZxIxwYMkIpYUlz8Nsp0KjOFUeEp3DdjKLPrFSqH5/fcQyfW6Hq49HRPE5d9qJvqKt/2JjR/BR1WkZuikxzdOrBkmW0ExfNJ2PpwlQgWsl6NX07Tu1GPrtNKsoNmQC75zyQy6dIwbK92naZrBjJxTk/B1nssteKOS+CJXLLJRbJbW6URBvttUAeqICUDCrreO4+zlnDL0lUVWlmw3Yq6v7rK/Z59cCYLhkM3J2oTvDCyjj0TxedzI1gkwHAP4+3Yz1+h7QbVvayyE81NhIWJhd44gcOABb84+aZ3jqEQG/aa6fnHFYShj2ZkjOuUkbT4PJzXAG1ArtBTYU2zknR/szh53MxEOZKSYTUbKGBQZKI9em6T5Q2BEPvcZ3uT+mAO3CsBlxqN4XOkTUTOQDjSgGIGi2oQRHDfFh7ownC2zNO1U7JMsILu4NDG/eJpthBzaWZmz2APfZTK87PjHKQgUa0plvTcfAZ7axcPrtk93Gx36pUckilvSrAx01oiKzxJ9S7JJO5wP5jihI+DOVMBF0hlM8EaYznI4FZRmCY/747+pdkWA/yYUFyAezurzwzN4UxAJCE3+qmTWMNs/xNheZ8HXa9po6/qUHFD1axOvkWLMd8obFVOmXS+7gel7x/oP9QvVeQiTPSvm9BBybC4BTAFsryltIpxPcjV4T3dxk57iPb5X5duIEhZBrponZPDeMul5widbs7gTxrHFOpXiQ/3E/MvjstXM99f8A4VHYKVDcOssXOWEWACk9geGX8uUuejmx9Js0GiTQvfURH46PKRocnYXNAPltGpy59yMjvNXMm0NbnPw5+dIiisRMjslGE485aft9AkFL2nTZNOhylRVWC7zU6BwuQQDnBkULNk6hDUNVQRGO7FgXVOBVlTPgkP20mQO0sWWek7e8n1Tr3aJyqvbXMXQsHWpy+WkqsTAHx40jW+wDCGkUbZINDKFHWAvigP7ovMKuAO6OdjphV7+44zuK7TXaP2Hjt3UR8UHRZeul3K4G/6W+6ssTPd+YGgCoeX7dZhbskQFT8QxbzLERqOixhqc4WAQ55WTFKl2pT2z2wkuyqnwGvdbQ3CKo6ku1s2J1o9gWNY2D4RUHiNmyCKC88P9W/m8MycB1zxgEUhx0pyjTOULCOHUaETSNOzvpobYA+Q3wR6Z2T1dxsNggSnCRCTmiUer9rSWHw41BP6YaQk1atruk1JVVhMUEnmGBOBZprO1IonO6GPKFvBRULOgTABqjtswunjKZ040GRlMqxBI6XwMisQsOz44hMk2ydFzrAMInlfoOA9S72g1G5ZTCugm7MHRvMOEKwysVwB229fcQQI2loAdXUCsTdBCKz0XAO7euDGn5cYZWNlA7cWKmHrwhfSSystFoc3yrMqs2n0h6Ypcy4aMUtyymGkuoj3kDz49XMI9PFmDA5PW+b3VKr757nX8sq88nZExuVrgGs/Ybxhk+ctX/Duc1TWrhnr/EItu/ChZZD3BkkrDnQ8Mj0aJVCEZ+P73zspedbHee8OO5offyJmLzsfVylZaCdMsX2zgE1GhbVk= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5baec480-c932-4d6b-1e02-08dd2c1b7d24 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2025 17:24:33.2817 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yXPeTobEfuzjRGlFy1Tzzy1jzmzDETE+Hk3s7lGCeNAd9COywFwDJIGnkA4vhS0E X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8066 X-Rspamd-Queue-Id: A095D40018 X-Stat-Signature: oc9a41qnfipzkmc7z9op3ji5hudxgyrh X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1735925067-104110 X-HE-Meta: U2FsdGVkX1/KYECRhHltUnz27kgNbSG8w5BwAYTgYrjR3qSNeIqAH5WKUb+LQshqyrEF1ncRvW+FM0g29CAA8lQ2GQlpvoZGDtRon5/pMbDkSbW5wu4iUEPFwcvxWoDNTTUuUS6bCu3r41kvBQUcK1JLm4t7NWpn+QriPUWqfoW7PNhDcvK7Yt7SyLUK4PfaQRbymwc02RLVwkasFu8Rmg3q77lKYnX1la1Cff6zi1/gtKf07lxOHDae7rqZwFcOosueuXAwY4lViNZL5Ezig+a/tWmZXu+Ldlk+9pPRCOzcPVixYkH9SigbeIZW+VoWa9wUlQw+sCfRzJA2TGESbEPWBqXAiC2OFLNFyaUU/36USSx2yHRd8e0DXyn7HdwFIEg/7M1eSXFPOaH95zUlNRbwx7MTGgJosJpaesK1HErD44EmHGSmrkBeIFElv0ExPcTp6Zg43AkA7ZS0+TuHkRN6CHmimLnoAPMAMupxpVjStFxCN6DXQftl3x23MZNa04V2vUbsqI3c02D6CGZlMfRcf5nnBJp/fzLawVRXBwwzq/JfaCokLAk2eOq6kzwlI1ZdZlVMrFngkI9w02tQrkdRIION0K41gNsX2vROyD9CsRHs4zefBX3/c5T4T7EjL+Bd/23PBwhAz2GnlDp8bTRkC/EjB0vFju7PmFTgXThX3SMoH68h+Q7iw6zLVD06VM53wyMiRnREZrwQMWiBweyBjw2zMwdYWQtCc03KRlhSROdxSUfWo0CRac5GMshFTb99JSJN+NuV0B+Id1vJDAvMOe0ijx7+cCK++4Hdq/RLz7QYMvArTaV0Kq7MFADFLPypW2G/Ey4h8sPgPyth12xyRdnELhjTk+N/BpXY70MEXdM2idr25k7qJ6Ys6niQUpOdcWtB7OnbvxEWGFUFoXVgWJR7musoxjEr8xrgvdpYPQy3k85OtYhmPYLWaKUe1ROlXiMMq47gf8TWZfQ z3DQe2ck UdW86YXP45jtmCq6ZJXNfWEnD0mH3X+J3/MtyJYZ2afxrWrZOC4lYfaSpCG/pOlKH5al+mpZXEMg2cYvTCcjj7MTBH2BlXA1thVeuGva588mSv+L4oTuZMocNZVpJjdI67QriXqjDLB7KLLncieDRdxuGULWj6IemtgT+Qq6xQyPHhZlF0ZqGzFCr5pWK7JWGVK/whfr7x70PZ+ruiTCw14JCNHVuZ1tgCVCz7Sfx1s0Ch6MnY627kCPqYZTxdRS+EW5TtqXjQk+oVkciNngoRFVxcbP8WjDwRL+y6NZQMu4YCy82cQ7cdFJAXiHalal+2i6AKscihiSVYLEQZ6O8fBHczGExktUZztOeoF+JEB1lPjdNiPaVXdfcmXDSc+1tUlelIxjXjSkjdaTGf89gUVDOuYSuk8iR56X8affqSmftjFzQfjkf4yyQpO/32VVKpMpbY0rtsK1ta/wXLJ6jBKOvaQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: No function change is intended. The factored out code will be reused in an upcoming batched folio move function. Signed-off-by: Zi Yan --- mm/migrate.c | 101 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 65 insertions(+), 36 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index df1b615c8114..a83508f94c57 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1014,18 +1014,7 @@ static int fallback_migrate_folio(struct address_space *mapping, return migrate_folio(mapping, dst, src, mode); } -/* - * Move a page to a newly allocated page - * The page is locked and all ptes have been successfully removed. - * - * The new page will have replaced the old page if this function - * is successful. - * - * Return value: - * < 0 - error code - * MIGRATEPAGE_SUCCESS - success - */ -static int move_to_new_folio(struct folio *dst, struct folio *src, +static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, enum migrate_mode mode) { int rc = -EAGAIN; @@ -1072,7 +1061,13 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, WARN_ON_ONCE(rc == MIGRATEPAGE_SUCCESS && !folio_test_isolated(src)); } +out: + return rc; +} +static void _move_to_new_folio_finalize(struct folio *dst, struct folio *src, + int rc) +{ /* * When successful, old pagecache src->mapping must be cleared before * src is freed; but stats require that PageAnon be left as PageAnon. @@ -1099,7 +1094,29 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, if (likely(!folio_is_zone_device(dst))) flush_dcache_folio(dst); } -out: +} + + +/* + * Move a page to a newly allocated page + * The page is locked and all ptes have been successfully removed. + * + * The new page will have replaced the old page if this function + * is successful. + * + * Return value: + * < 0 - error code + * MIGRATEPAGE_SUCCESS - success + */ +static int move_to_new_folio(struct folio *dst, struct folio *src, + enum migrate_mode mode) +{ + int rc; + + rc = _move_to_new_folio_prep(dst, src, mode); + + _move_to_new_folio_finalize(dst, src, rc); + return rc; } @@ -1344,29 +1361,9 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, return rc; } -/* Migrate the folio to the newly allocated folio in dst. */ -static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, - struct folio *src, struct folio *dst, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) +static void _migrate_folio_move_finalize1(struct folio *src, struct folio *dst, + int old_page_state) { - int rc; - int old_page_state = 0; - struct anon_vma *anon_vma = NULL; - bool is_lru = !__folio_test_movable(src); - struct list_head *prev; - - __migrate_folio_extract(dst, &old_page_state, &anon_vma); - prev = dst->lru.prev; - list_del(&dst->lru); - - rc = move_to_new_folio(dst, src, mode); - if (rc) - goto out; - - if (unlikely(!is_lru)) - goto out_unlock_both; - /* * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will @@ -1382,8 +1379,12 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, if (old_page_state & PAGE_WAS_MAPPED) remove_migration_ptes(src, dst, 0); +} -out_unlock_both: +static void _migrate_folio_move_finalize2(struct folio *src, struct folio *dst, + enum migrate_reason reason, + struct anon_vma *anon_vma) +{ folio_unlock(dst); set_page_owner_migrate_reason(&dst->page, reason); /* @@ -1403,6 +1404,34 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, put_anon_vma(anon_vma); folio_unlock(src); migrate_folio_done(src, reason); +} + +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, + struct folio *src, struct folio *dst, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) +{ + int rc; + int old_page_state = 0; + struct anon_vma *anon_vma = NULL; + bool is_lru = !__folio_test_movable(src); + struct list_head *prev; + + __migrate_folio_extract(dst, &old_page_state, &anon_vma); + prev = dst->lru.prev; + list_del(&dst->lru); + + rc = move_to_new_folio(dst, src, mode); + if (rc) + goto out; + + if (unlikely(!is_lru)) + goto out_unlock_both; + + _migrate_folio_move_finalize1(src, dst, old_page_state); +out_unlock_both: + _migrate_folio_move_finalize2(src, dst, reason, anon_vma); return rc; out: