From patchwork Fri Jan 3 17:24:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13925731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F126AE7718F for ; Fri, 3 Jan 2025 17:26:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 88D936B0092; Fri, 3 Jan 2025 12:26:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 83B736B0093; Fri, 3 Jan 2025 12:26:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B59A6B0095; Fri, 3 Jan 2025 12:26:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 470566B0092 for ; Fri, 3 Jan 2025 12:26:21 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 090848097F for ; Fri, 3 Jan 2025 17:26:21 +0000 (UTC) X-FDA: 82966817346.07.26C0F9A Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2080.outbound.protection.outlook.com [40.107.101.80]) by imf25.hostedemail.com (Postfix) with ESMTP id DC775A0008 for ; Fri, 3 Jan 2025 17:25:40 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=DbxOdeVi; spf=pass (imf25.hostedemail.com: domain of ziy@nvidia.com designates 40.107.101.80 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735925142; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QuwWf93KZrL6mtGmG8Qf0lVvYao/70tVo+I+7ylzu1w=; b=z/T25OLwNbfsEVFoqNEN3T2daESv8D/bk1TOEhNTLLJBLTFoB9lGspxtB5bv7HA/ZH2Wvp qFg2BmAVZe6ZH0RGKZY3XobZLIwzw0vbqbpsvxwcmVqCZv5+gx0D2G9u9xz9cuQAZ3VN1j WPmymSFl+vsu89+QGdPvFyCdUCUDrlw= ARC-Authentication-Results: i=2; imf25.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=DbxOdeVi; spf=pass (imf25.hostedemail.com: domain of ziy@nvidia.com designates 40.107.101.80 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1735925142; a=rsa-sha256; cv=pass; b=GEj8o6oVpl0LBxLUVH4FJe6trNavKRU7BsoeBvnwn0pUiHypMkWFFlt/jljAyUBFspkmeJ yjlqs2sdZfC7DW+N0ixut4My+TMPZIck8YdK1P+XsUq0LufZxrlGNqGE5hFDYPfrNEgXYn muFTYUuAbB/XhVxC2CymfgC2ikK4ptg= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=fyTjPaIyRr4xGoKRlInmu5u8ds2jZ7Ri/ggF/bb15As09QVPXE9d99GlMkvZ+EYlZmL2Wg0xJRbIwAeciJNrYN8MeHTd5PgXCIqgr1G++YpekTn/uZauT3IkJhDRlrTklYV0pc8YjKicXPiiv9mCgzcgyQ5Xk8RG3dNEDHkfnjZ2/PTSRa5lPPBCxpxFmYo7DMCemoa8zplXOcDtRQjrFNtUjYhX78Frau2SI77/jdGbZ+YElnqHszepBWU6pAexYmB4rlq3LzDttwmPV3wGmB0D+N35KxSayxvramgo/0qs9gedEd3gDwm4z5alyfRTK9a62dYEFLX/L2J99/QlHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QuwWf93KZrL6mtGmG8Qf0lVvYao/70tVo+I+7ylzu1w=; b=cMgsRXBlnhOBqhIq3y3Ad0xBTtxymfvsTmtamj9V9QZKwEgWcWr+KTY/1g6omxiT2komeULobyOsZauk2WZAj2vTZl6KOfCKHPmwG0Ng3GrMb2+5kF7sKVmW6fcFlRK/2r6pPXFWKiFnt2uTFXzK6Du4gHAQTH+NQBiivOwfbYTitRz7uII5R3DtiZjEJpXX2m01LOaf9zMFvOHADGuHzZnq4r/brVVQtCOoOAQjotXgj455Q1s7Lro6o2QBy8Y8HmFB/cL8Dn3rInpzoARgfS6/7Ry6n5/tIN+yCgDOo4q3jWOhqmKqxhOAWrmCL+nSbHgmHowLzho2QbEBtA7N9Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QuwWf93KZrL6mtGmG8Qf0lVvYao/70tVo+I+7ylzu1w=; b=DbxOdeVitYwFpoqI6sSOo3yj/cocRVG5R1UBzV5LTwYj1w8wREAQiIgCX8RZqdkf4H776208QowFdb/iSceeV1FgzQXDFDNqgIQoX3y3EyXK1vy9e53F5IfVNvejOmzIzF6YJfn9ltoChZM0VoCrv/U2VVYq5AVnNdIH3fyWf+yfS8x/5CBw77worx9Fy+VH7mMEkI8YFocA8Oc79+nu7EEvC+RvupLhBU94igZyLQVfBHi9Nesb2FCpbr2gvIzvA5b4U2KyDuV6e140JstXgzCHtMZzXWwaEbpw/1SEjECRjCLflv8YMUDhXVzRdqrMYY5BCCHakboSZ0PjvhVf8A== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY8PR12MB8066.namprd12.prod.outlook.com (2603:10b6:930:70::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.11; Fri, 3 Jan 2025 17:24:34 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%3]) with mapi id 15.20.8314.013; Fri, 3 Jan 2025 17:24:34 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: David Rientjes , Shivank Garg , Aneesh Kumar , David Hildenbrand , John Hubbard , Kirill Shutemov , Matthew Wilcox , Mel Gorman , "Rao, Bharata Bhasker" , Rik van Riel , RaghavendraKT , Wei Xu , Suyeon Lee , Lei Chen , "Shukla, Santosh" , "Grimm, Jon" , sj@kernel.org, shy828301@gmail.com, Liam Howlett , Gregory Price , "Huang, Ying" , Zi Yan Subject: [RFC PATCH 3/5] mm/migrate: add migrate_folios_batch_move to batch the folio move operations Date: Fri, 3 Jan 2025 12:24:17 -0500 Message-ID: <20250103172419.4148674-4-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250103172419.4148674-1-ziy@nvidia.com> References: <20250103172419.4148674-1-ziy@nvidia.com> X-ClientProxiedBy: BL1PR13CA0183.namprd13.prod.outlook.com (2603:10b6:208:2be::8) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY8PR12MB8066:EE_ X-MS-Office365-Filtering-Correlation-Id: 39cb1bff-3ba3-4cb9-98f6-08dd2c1b7df0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: QBawTc3z5iu3lVNttX7q2/k7kThMU3foimtOlqsBcjy/5wtOoj335uxyfcyAEjAajfrtFajdJLdN4cLJZAEWsEXGCAbrmH1gZKKlC7F7rA5vwRws3T9mJCoy+SSmX1y1Oc3NSIP97T/HKvUkeCPSSagRbGzjVX+mOu+xe5vPZHV0FYEfV8UqxA2C32S+pQtADtrlzvzJe490NNn506Jlg42Y8HeKYNmbte6qrpBBp6g8wgmlV9WSwj07WYeNnRWPisNDPrpFH1PqYFRjJyzI+Tb0uNAmdwvDp/VxXCv9WXtHoA6GkAj17b/kucZT+q2ITQUDM0aawaza0W+KgAN+I8o0s9n8XgetSwZ5Sg9kN6Wo5D/6nWikcN+k4wHqNEE+Hzh/9lp1LsEGgG4qxkZvpx2KrPTJM4ACJdQUUk481+cBG9DMcMSQZwcWPuIXniVFu5MoIF1KSeYzEqr/NwMVcnIvn4HGUwAt9zl5Kp0hyZfrvjG32PRaAzmvnFKO3tn+6ZWZCZNIb37CQzVQIDuv5vbF5kdt+M1/MJx4heFBKj3tz+8aAE4tryOg7NYnDCYvvv9S+58gwJQcBUwmz6xyldxL2de9P0jdaWSECLWs9LKYrjNYKbFs1z3keokYW5+LF7weDbWqrwoUe4V9i02wECqXrSXUthMn6E8xmuO41qVHATEWPHlAr9zq3XZt7yrKzjCkn4Q22r8pBXZBEhz2YSNBR/WX7Uur27Az3e63ByeD3aH4q/eSvqxxzmcsHikoMcVYI51k8M6wxlLMC4RvZLk8P6qJ1Cx/xJip32ix2l8tk0bx/y76gzEY+gKGy0Hz7B86Sj844s2PKSQBK2eZvb06RPTngSbyyn7t6Elq4XSLRCRLJb6iEL5Q3KIkvvbojNcILYEF9/olE8mPFUdP/Y9sLYts5fVCz7o6fZMvOWeJCWoBsxbXTeu1aN34BZ9GwZPrt+BjVFZDujdWmuuPtDKPl4csCLz33vNWumP5X1OXT1xytSaO/kQ9yh/Ly7D+bBk6FaVYUG9nQkarRPyA9uc+EFovQ0BgyubcHf925op9/438JlAdzPWieciCfxJs2B6m8lQEZVeOEUR8FTr0KnDIyBcxFkquI21XTzaKvpYhRg/wLZjTLoijQ3FFzQPKI8dt3Y7r5nRTpg8vpa8HjVkVhdfT3y13av5pXkD4wfRxmed4/9cFnpU11hKBpfWdy/gLrO1ChU83vEF8q9YG06TH6bIwqZxOklvqJeSaO4+Qw/gPFYqin7r9M4kqW7cXzE/3VPNsvYPPCksebqGhYTNVhbGPqbGveLwVFzp+RxQb2jY9aKnhs5btRhHTyBCZ7bk5tNNsLXkS+ltyIi7/8fq24UySI1M2+w5Q5qmOT18IesESitWL8Qr2g16KiuMi X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: w4aHQC6JPkAWgk37MakXupaqjjwMcN9OujJ4z4vAjtQRz0lkp0/0fRJIf3wp8VBjadFpytorn2TLg6yd9s2FJMpt5mvzu7SKhbZNpk6AobazWzSaNQDhtGDCUJf8ARJrbOjXEcFOTZ8iUXRl9w0NYMd5Q2t2fGlOxXG7bjHV7iKBUiS4vO7YcdEp/4KMWMyAPoRyKt56MqJwOTUX0pFjv54Ik77yz001dtuyZDJzOUfrzuoJvOA01Lo+luf3vVdqZtmjnGNlsh5ZBhyQVw5frtk5ulEW0CNd97Hem2TM3ilwLeqQ8Tx24Gp0KYSpoRGmv38rEteuG5cgS3DzKQ98469mjmrQiP9ZBtqqRh8hHk3u5Hw/d5QOJtBJsi5d5n+vo8iHkvqKiPptA8Dxpae3IBVPurUG5J9v5PRAsJL/NVwsdG3LLvhVGHc6eq9+3JAlWS8R3pXgg7pTnPquKAjNWxwiaOeW/4jOABf37NK9aVcf0SXUMuQf6dy0g1NetWSnssmhq5TPXfciBo019fxV6wXOSDYJb2SlvquGXlJfEP1ftttGknl2Yd2L1CZb3kqzsVGnPLlD4KVi9gsKzR0CJUFzHlHs33u9G9UK7bglUdmOfA+lIEpsTeJiB7tByzAUP7hyZcrlb6uMhvk1rq8SLhC+C3ObIgPUPgco3V8ox99CyXA+bv6NBzRJctImX0SMm7hg5kokmuXL9kX1zrzwAHP8PpniBexf+AuaB5bwOyjf8XdEwNtLbcuY/9zktzsy1lYNf3lMMbpGGJInqHsoh0PxphH7R7oTfsWgDapJhQMPXpdi5XePXVA5cpA8dAhDjw2l2sjM3OBF5dPfdFk1UkEhLZ/M7g2NosIy8P4UvugwKK5N0dCrvtuXOOAgSTObT2eRKnj5BwCAd3jCMzLU/eYo09yt63oHvCoLkHwFVR78oq7Tl5ZouciuatBIxUOK257SnuHblyUY/LT/HIKYlU6Pb0vsvuo0mdqWAqmk1X9CTemqp5QqUtii1cQM69miSgK1PfoIe9+elATryMg8ki2wTZvXynbAtI8+9VSJVwLhY9oMMZPeb6JNt+p2aewx9e78C6H8Z5wQ6XbMQrb/RLZQKmWM6Ccww8/0QOQjReunhP04bZQSEaX0qiJsV9Mv4+zKlUF/QOmphRfMOjEgDnzWVqZgZUV5jhyvwQeUSjk2QIr0Enp74EMTVj1P4KEiChmn16hgnwcACLH3YSdAP5YF22FdPpJnVor2/BbQlmfucJ4r/1UzkQVZu5wCK08ScJrQ2CDMFs5ob5wf3IygS7FEZdftEu8GAdbUOVSImYeoQqmvO8uxpwPY6v8uYQMZsOMTkdLdyEBUcnxsreDVtrkV05li2DFzf+rUD3D/fZWp0FOSGoApNZ8UVM3nzMVPDdzn/4TiokfGgIhWBsOQa//sdrdH9VRZojipa/1ppvhNGs5LPAvB9h90pr9LtU7fjZMpOOUPRFxBENx+b8FD9grwc7mRZt+tSrzdVKS5yBm/WlKmYsLMzg+SxIOWFNXWbcK7Sbu8iEF2PKiHMYvJC13UNrp2u9Cd95QFjJ1KQGI= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 39cb1bff-3ba3-4cb9-98f6-08dd2c1b7df0 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2025 17:24:34.6361 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: AvuaQzy9chQAOvbB8XdYWftPqM20QrxCDGIjv1uO5y/795S2QpbAx6Msc9m7t7gD X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8066 X-Rspamd-Server: rspam05 X-Stat-Signature: q3dedmoxerd7y8j9cxwjfoybkqjs81ka X-Rspamd-Queue-Id: DC775A0008 X-Rspam-User: X-HE-Tag: 1735925140-976617 X-HE-Meta: U2FsdGVkX1+4O2nPba+Df5KDJY6PSYox7BIp6GgO/jdozLWKfYZB6SC7Tpgu+ZC5ivGnlRRB5I2f7A6CIg5qzKXguV1oXhilU4fEE5dWPOdWTO71Hjtu/y3/HXsbBpGXxBkt2p0D4ai5rw189ZN+gVyhMv/f08lXOGkuL3vI97BG5YuA17MTBL9rNMeWgznxGRLrDZqF8tDQfmQM+LBwoXuUfOfdYjI2YFmc3s+9l8tFX+bsakN24Om01A69yCkLlCN4AJgved8pF/tK42dQ7tWU7eAhVuDUoMgeyv9tGpzFI7aVuM65IuT4ImwTlAXGCBe4+frwGMmUR8CJmRCJC5lJDT2KbRklF0nMKxdFx4O+mDM7cHIpNFuDnzfx30wssZeMG1qRn7SJmUddjV+GHFmCoRHEH0XwaFtnjMpuCz0qTAEGQYiJ12NVCmDoaHHp9t3ngFzlI2jwy4yj6Qqmi3sBiAxKL/K33Sxj3MNhGwsIWPQbknR9zp/ORnvvS7xHabuc5aMdMVSmHHwFWeLvF0cHxO5w20kTrUl9SJVk5Zozb0I36hbeJQmbYy2WBrfzJlMuEA7T6u52TlugUPJze/p2VdW+sQYZ+8NKNGMGuTA/68P69hPJn7beZxQOS9e3aZTZFxeQvj7GYgCe1QvVuhyC91vm92sXq/uFMXO1Vrrnz6j+3SNnaLibEeziPXBBuU1BqcRjnBLQlOg19EEHLi6sBVlDw3Yy61Yxz+vn0S7tQD51kVhJkSflUR3cPcvQmvTiTZAtzewlbTD96WJ+S1006p8jCZfxl1x/ApEHnjLiRbnQwmI1FqdG1/do1NR2s3lUZceh6UZhdBknhgVbD8yRfkk4GfVh7xFkYnkcQ4oP0Zll98Uvnvn/BpeZwNmHhY320F2KcxUM1rr1+iSUQo4kvO1WZhNO79Xpl7V8W3Os7rRhDLDGKoPmdPylJvzzB7XKG7DycZz1ciuuwYL j/2mZDEr Ti4VoYSAvqP7u3VLauTp7JaHpfmos+ui4WYBVVHxt9iSVTwqhCardZn10gEggxIKFW5RMewdPuW30ziwUyaQ2v6TQ2DQWHj+zKyJZabuP6pOdOilDS43DsmEeCV4l+ZegoXeXmp8SFjnnwxTJa1bwZz09tjnxvG1eRFbj4mAO39xtup/7g4dO559QQHztfV1NHAJdqWahlONfXX4UAxAuix1XW7pG4AvCVWp9Q/2TLHBEYYxKOA+4zO6zuCpQnph7sXSRj7ypuUoZNHsrWkMTz0gC7MMTdiKjY7Nd+uf1m0yIcaD3czC2AZhnYmBWZZ6c3VwYoAzeTuFdKnELuKiUlWKeFV8nCTPtmjCVy6BhPcASPR+eO1G0QY1ThKHpa4oXY/8Gw7RbJtVJIghxjtQt+b/AovDZNy6VjY8I90xuIsLIFlZs+olEjDdCB80mS7GhQdx0jXcOES+DmB8aKDZJC3eZbEv6PUvjrbSCQUBqko3WraI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparatory patch that enables batch copying for folios undergoing migration. By enabling batch copying the folio content, we can efficiently utilize the capabilities of DMA hardware or multi-threaded folio copy. It also adds MIGRATE_NO_COPY back to migrate_mode, so that folio copy will be skipped during metadata copy process and performed in a batch later. Currently, the folio move operation is performed individually for each folio in sequential manner: for_each_folio() { Copy folio metadata like flags and mappings Copy the folio content from src to dst Update page tables with dst folio } With this patch, we transition to a batch processing approach as shown below: for_each_folio() { Copy folio metadata like flags and mappings } Batch copy all src folios to dst for_each_folio() { Update page tables with dst folios } dst->private is used to store page states and possible anon_vma value, thus needs to be cleared during metadata copy process. To avoid additional memory allocation to store the data during batch copy process, src->private is used to store the data after metadata copy process, since src is no longer used. Originally-by: Shivank Garg Signed-off-by: Zi Yan --- include/linux/migrate_mode.h | 2 + mm/migrate.c | 207 +++++++++++++++++++++++++++++++++-- 2 files changed, 201 insertions(+), 8 deletions(-) diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h index 265c4328b36a..9af6c949a057 100644 --- a/include/linux/migrate_mode.h +++ b/include/linux/migrate_mode.h @@ -7,11 +7,13 @@ * on most operations but not ->writepage as the potential stall time * is too significant * MIGRATE_SYNC will block when migrating pages + * MIGRATE_NO_COPY will not copy page content */ enum migrate_mode { MIGRATE_ASYNC, MIGRATE_SYNC_LIGHT, MIGRATE_SYNC, + MIGRATE_NO_COPY, }; enum migrate_reason { diff --git a/mm/migrate.c b/mm/migrate.c index a83508f94c57..95c4cc4a7823 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -51,6 +51,7 @@ #include "internal.h" + bool isolate_movable_page(struct page *page, isolate_mode_t mode) { struct folio *folio = folio_get_nontail_page(page); @@ -752,14 +753,19 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst, enum migrate_mode mode) { int rc, expected_count = folio_expected_refs(mapping, src); + unsigned long dst_private = (unsigned long)dst->private; /* Check whether src does not have extra refs before we do more work */ if (folio_ref_count(src) != expected_count) return -EAGAIN; - rc = folio_mc_copy(dst, src); - if (unlikely(rc)) - return rc; + if (mode == MIGRATE_NO_COPY) + dst->private = NULL; + else { + rc = folio_mc_copy(dst, src); + if (unlikely(rc)) + return rc; + } rc = __folio_migrate_mapping(mapping, dst, src, expected_count); if (rc != MIGRATEPAGE_SUCCESS) @@ -769,6 +775,10 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst, folio_attach_private(dst, folio_detach_private(src)); folio_migrate_flags(dst, src); + + if (mode == MIGRATE_NO_COPY) + src->private = (void *)dst_private; + return MIGRATEPAGE_SUCCESS; } @@ -1042,7 +1052,7 @@ static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, mode); else rc = fallback_migrate_folio(mapping, dst, src, mode); - } else { + } else if (mode != MIGRATE_NO_COPY) { const struct movable_operations *mops; /* @@ -1060,7 +1070,8 @@ static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, rc = mops->migrate_page(&dst->page, &src->page, mode); WARN_ON_ONCE(rc == MIGRATEPAGE_SUCCESS && !folio_test_isolated(src)); - } + } else + rc = -EAGAIN; out: return rc; } @@ -1138,7 +1149,7 @@ static void __migrate_folio_record(struct folio *dst, dst->private = (void *)anon_vma + old_page_state; } -static void __migrate_folio_extract(struct folio *dst, +static void __migrate_folio_read(struct folio *dst, int *old_page_state, struct anon_vma **anon_vmap) { @@ -1146,6 +1157,13 @@ static void __migrate_folio_extract(struct folio *dst, *anon_vmap = (struct anon_vma *)(private & ~PAGE_OLD_STATES); *old_page_state = private & PAGE_OLD_STATES; +} + +static void __migrate_folio_extract(struct folio *dst, + int *old_page_state, + struct anon_vma **anon_vmap) +{ + __migrate_folio_read(dst, old_page_state, anon_vmap); dst->private = NULL; } @@ -1771,6 +1789,174 @@ static void migrate_folios_move(struct list_head *src_folios, } } +static void migrate_folios_batch_move(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + enum migrate_mode mode, int reason, + struct list_head *ret_folios, + struct migrate_pages_stats *stats, + int *retry, int *thp_retry, int *nr_failed, + int *nr_retry_pages) +{ + struct folio *folio, *folio2, *dst, *dst2; + int rc, nr_pages = 0, nr_mig_folios = 0; + int old_page_state = 0; + struct anon_vma *anon_vma = NULL; + bool is_lru; + int is_thp = 0; + LIST_HEAD(err_src); + LIST_HEAD(err_dst); + + if (mode != MIGRATE_ASYNC) { + *retry += 1; + return; + } + + /* + * Iterate over the list of locked src/dst folios to copy the metadata + */ + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + is_lru = !__folio_test_movable(folio); + + /* + * dst->private is not cleared here. It is cleared and moved to + * src->private in __migrate_folio(). + */ + __migrate_folio_read(dst, &old_page_state, &anon_vma); + + /* + * Use MIGRATE_NO_COPY mode in migrate_folio family functions + * to copy the flags, mapping and some other ancillary information. + * This does everything except the page copy. The actual page copy + * is handled later in a batch manner. + */ + rc = _move_to_new_folio_prep(dst, folio, MIGRATE_NO_COPY); + + /* + * -EAGAIN: Move src/dst folios to tmp lists for retry + * Other Errno: Put src folio on ret_folios list, remove the dst folio + * Success: Copy the folio bytes, restoring working pte, unlock and + * decrement refcounter + */ + if (rc == -EAGAIN) { + *retry += 1; + *thp_retry += is_thp; + *nr_retry_pages += nr_pages; + + list_move_tail(&folio->lru, &err_src); + list_move_tail(&dst->lru, &err_dst); + __migrate_folio_record(dst, old_page_state, anon_vma); + } else if (rc != MIGRATEPAGE_SUCCESS) { + *nr_failed += 1; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; + + list_del(&dst->lru); + migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + migrate_folio_undo_dst(dst, true, put_new_folio, private); + } else /* MIGRATEPAGE_SUCCESS */ + nr_mig_folios++; + + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + + /* Exit if folio list for batch migration is empty */ + if (!nr_mig_folios) + goto out; + + /* Batch copy the folios */ + { + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && + folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + rc = folio_mc_copy(dst, folio); + + if (rc) { + int old_page_state = 0; + struct anon_vma *anon_vma = NULL; + + /* + * dst->private is moved to src->private in + * __migrate_folio(), so page state and anon_vma + * values can be extracted from (src) folio. + */ + __migrate_folio_extract(folio, &old_page_state, + &anon_vma); + migrate_folio_undo_src(folio, + old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_folio, + private); + } + + switch (rc) { + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + break; + default: + *nr_failed += 1; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; + break; + } + + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + } + + /* + * Iterate the folio lists to remove migration pte and restore them + * as working pte. Unlock the folios, add/remove them to LRU lists (if + * applicable) and release the src folios. + */ + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + /* + * dst->private is moved to src->private in __migrate_folio(), + * so page state and anon_vma values can be extracted from + * (src) folio. + */ + __migrate_folio_extract(folio, &old_page_state, &anon_vma); + list_del(&dst->lru); + + _move_to_new_folio_finalize(dst, folio, MIGRATEPAGE_SUCCESS); + + /* + * Below few steps are only applicable for lru pages which is + * ensured as we have removed the non-lru pages from our list. + */ + _migrate_folio_move_finalize1(folio, dst, old_page_state); + + _migrate_folio_move_finalize2(folio, dst, reason, anon_vma); + + /* Page migration successful, increase stat counter */ + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +out: + /* Add tmp folios back to the list to let CPU re-attempt migration. */ + list_splice(&err_src, src_folios); + list_splice(&err_dst, dst_folios); +} + static void migrate_folios_undo(struct list_head *src_folios, struct list_head *dst_folios, free_folio_t put_new_folio, unsigned long private, @@ -1981,13 +2167,18 @@ static int migrate_pages_batch(struct list_head *from, /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); - retry = 1; + retry = 0; + /* Batch move the unmapped folios */ + migrate_folios_batch_move(&unmap_folios, &dst_folios, put_new_folio, + private, mode, reason, ret_folios, stats, &retry, + &thp_retry, &nr_failed, &nr_retry_pages); + for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; thp_retry = 0; nr_retry_pages = 0; - /* Move the unmapped folios */ + /* Move the remaining unmapped folios */ migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry,