From patchwork Fri Jan 3 17:24:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13925732 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAD89E77188 for ; Fri, 3 Jan 2025 17:27:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 558766B0095; Fri, 3 Jan 2025 12:27:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E18F6B0096; Fri, 3 Jan 2025 12:27:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E66E6B0098; Fri, 3 Jan 2025 12:27:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 067246B0095 for ; Fri, 3 Jan 2025 12:27:35 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B088CC07E4 for ; Fri, 3 Jan 2025 17:27:34 +0000 (UTC) X-FDA: 82966820034.19.BDEF943 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2079.outbound.protection.outlook.com [40.107.100.79]) by imf15.hostedemail.com (Postfix) with ESMTP id 12FB3A000F for ; Fri, 3 Jan 2025 17:25:55 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="Im4T0/jG"; spf=pass (imf15.hostedemail.com: domain of ziy@nvidia.com designates 40.107.100.79 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1735925203; a=rsa-sha256; cv=pass; b=tDq99JV9UTnmH6ConNJ1fTnKJTJxQ4mkmjqlijpGjV0BkIi3EDAe+cYEoAOYxhQ2oK8K5e eCCeqDmbASMw/4UegeSy+DtNAyNyp+G1yHesWF88QZ7LZW1zXPizxkcLnJywCc1C6c+WrL ziHNnHCSX1wSHb217PiLrojqXRupI3o= ARC-Authentication-Results: i=2; imf15.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="Im4T0/jG"; spf=pass (imf15.hostedemail.com: domain of ziy@nvidia.com designates 40.107.100.79 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735925203; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ej3UokVF/XKxua5H3iLTNC9XIyooIX5KbLGnOa5F/Ww=; b=rcFJVrlc043fvqqn6Gomnba9aWgQOfHR5LARPvvpBCscaAHk6tK6huSoSlPtHYck2+b7Q7 t9KfZl8MMaRE8uRwIJYh3EizIKmCLbmRgqPf6C5JluS5Y1bp032FQmKNfMfgm7MUrcsci8 EV27i8TdZSzejFYa0ejGxEDbdFLekbY= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cNWtCDrldY8B92XhGZzCst8DkHwD/N1eYBZi9ASGebWZfl4U8TE+YdWt0rB7KwCoGe3Daz34UKr52+jrwduqhDV+XtbgrxbTlNp76Mda8IeFBUo7+x4YC9+5E8MaFHCIgW6KO1jP7avW1vO7pmOf77qqg6NwzMJ1+VYGXQNmh/fMUnsvBNy7g2iAEnSoyndUXA5YFm1I8atvS/QoLdMprii71jOTkCtIuKKc03tsOOOsK9wIrD9yG/sCPRIzvJ3rI2aa4Gt1SrcBYDgns/vRjQ7d7Z2L//z+QJtraFm3/pvHSgJSE969IufFllmiP79/ick1yrEaEhFV7hLmDE2rnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ej3UokVF/XKxua5H3iLTNC9XIyooIX5KbLGnOa5F/Ww=; b=bI+WfXOKZXemscJlKHxc1IG5HA2xu3C7cHoJXi7fI0cdtj9kk85kpNv331bQ5uZ0YoqaEGxyOkb22Su4GIr+/VseWE5w9diMRcgPB84YFR3qzMZFql+g8stwmA8B7ZEp+Lcs0XUwajE1P1K2v+FzBKAceT0VjIXM/zVRMehNvJV4QakS2k43FgAHCrs5GNI+tUIkkYK13PF60v5N9+gCRHkqzP9DBWlCrlzj6YmPKYQ69jV3TxCam5+2sAL/dmWovSaIxkyXJmUXB1inGsweQztozbUY25URmCJ69ZHSVB5e7lDp7TCAMA009/X0YBZM8rMfsIpGK0r6n7Qlg2Zxyw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ej3UokVF/XKxua5H3iLTNC9XIyooIX5KbLGnOa5F/Ww=; b=Im4T0/jGRQuRLshw5sgs0tuvjadVbez0A+vFN38sraPilVNN321Cwz+IHogS5PeCubIoLMWJf9HctNDpxbKam4Mn18+Y3uRVh9xeYi4LnXvAW42ke3QhYQQ4wEualkj8MOhj5eaLHl58Mbo8aC0oLQlecW++hb1vwvQ6OGAGkm1RCFUoWf0wz+OZMhNAUUwli6+3S2ir86gmfZmeD/B6ag5cMeZGfuYkWDME5nxs+Bj2Hrru3KVqAMFtzECYaGw79l8sd3M3qaQLdphJyCV9nUZv7BoOeTJwvKF6IqMMnhtQ8G0uHfovXqNCRvIlFP5zubIVEm3ZqeuyJ05mE26Lng== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY8PR12MB8066.namprd12.prod.outlook.com (2603:10b6:930:70::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.11; Fri, 3 Jan 2025 17:24:36 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%3]) with mapi id 15.20.8314.013; Fri, 3 Jan 2025 17:24:36 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: David Rientjes , Shivank Garg , Aneesh Kumar , David Hildenbrand , John Hubbard , Kirill Shutemov , Matthew Wilcox , Mel Gorman , "Rao, Bharata Bhasker" , Rik van Riel , RaghavendraKT , Wei Xu , Suyeon Lee , Lei Chen , "Shukla, Santosh" , "Grimm, Jon" , sj@kernel.org, shy828301@gmail.com, Liam Howlett , Gregory Price , "Huang, Ying" , Zi Yan Subject: [RFC PATCH 4/5] mm/migrate: introduce multi-threaded page copy routine Date: Fri, 3 Jan 2025 12:24:18 -0500 Message-ID: <20250103172419.4148674-5-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250103172419.4148674-1-ziy@nvidia.com> References: <20250103172419.4148674-1-ziy@nvidia.com> X-ClientProxiedBy: BL1PR13CA0189.namprd13.prod.outlook.com (2603:10b6:208:2be::14) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY8PR12MB8066:EE_ X-MS-Office365-Filtering-Correlation-Id: cd39796c-9537-46ee-196e-08dd2c1b7eb7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: uTtOAOG77Ry9B2b5HUjMKJ6nHqsgcnsnBTT5Cc4F3lS3DS+FBRVaguDUk4X3yZ0og7ULsG+Hc/3Ptr9NoLdRV31mDAwmXxiziUBUVIPe/m94a/0+yIAkCE8TIUYRMQcT7x3RElWXcugAo0arRcPi6DRWXPZQOTI7iXJ/dutZNm7U3jmfZcsu1PmNzrdaT4OmY1iEkCHca5kvS8DS18BSnh/Mpls96+zMF4qF0PexynF/6qpJCY9/aeB07wdqlYCRjZ124NRJrSdIDVUC6v23Nozp9/zXGZsGccd7WcMkRVThd9C5LPLDCDMoDhFX+M02JAX4u9KBYKyFZX1mDxA+kl6yZ2ykGK8tpPPW01kSgL9b/U6SqkTpm3TlDLCpZnZURy5OHxKCtrxLP+HIbTInicBJqSuZWZcqoY+eqJCwlB2l7d6RNjR3mFerth+e5Q8hKY3tUiLki4sT8/oW0O7IR5xPbOV7qOTjHgMv/HDBGC/YICMR/OtZoLzxpjsPar5XiTiKqlHZXP96PgLgM55Fh7dhPJXRSynQwtoHkpbueUHgymplYWAENIUqOzrJOA+twD+viyxNodB9SK2gu1i/q+NasRLXjGRmCbMRT7f4k78d1hO5RjIr4ZjcpvvR2qohGkOJl4vNtW/LpjqdF3k2Xa97tGeO29/87xE0O0U5Q3V322D7uAy4IrcdvIPdBQfCl7Zal/OvNC3u6LJnjP6pP0sexB+lRKw8BPaMBl/yKIkIwe8Lm3MJoki4h4HlnlItXtWkNQvhlb+yr1Z3OVoyQ9KJt+kN/GZVLxEnrIjoVxKZX2dnwcxHasgdTqrLF2ugANS0ST+2G4g4DhP/peAQVj5aA0+7OiqiZ+v2dnKGEKy0XbBxTn2ihz7rdQzLuEoUHglAjkICxK4ckIWoF6w2mqrei+nX2VgyDcxxdLb8u6TrrnVHQGtSrEeiA4RLPyJR5K5lDLgi6Rl2IYxn/rjwdpLtc391KtGL0CsLL3qTimZ2evytJFhGZcKWRTictmt2Dbm3ygJM2VH3xeLMU0U1V5V9PezjMBos2XwX0F1nXIguSfZKCH0ynO9KF8T0AY+RU14HJtrB4QHuneR7rTrykumU7cU10qlCZKFzkMD5/RatqUZs0ez6fv0pcMeEYfyAQ8V1+ADSTneNdbMRdxQTQGj5OJWag8c+e3eHETpJekwHWO2FdnZVs7ZW4hFAZWCAn8yhrfJKowENW6Z2SCGYPGhktyh0FogItsLM141ztkVLuAisvLbZXzdsYAigC8lYK7FgGPtn7H4X1N3QS0B9hYH3TnzrX6dKnEuvFuKS0mvD1h5Ni54BildRVr1OO08t+tbjWyZ1YkjrZ1Md+Q151hO0BjfMFZtjaX+LMa30C8qU2SuIdJizY8pUiK+S71O8 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 09ygDoo4FCdmLBq91l9X7z+TxlrXK62OxfMfdZXTWIlxC+Z7geJ/o602poj88Hfh9zPw2I2qg0OsQGn1gHfO8s4gxGCAk0F4outSUD9w/F0eI+s9ViQtDCkKNB26QNVbG0vwL8ubG6APOm/lO+EMRqAy+ZOzvg4GxYFj+J9VLgdHnKzjn80lMVWk6OPQWtIedhOnp1hoC9e+TLWGUgiSksieKJnMF7H5lTpBvqOFGa0LC6im6OCXaKuf0tb0iEHld2ZDXNDaER+fSs5Zm2Khi4Eha332K3yJUmzTWGrUEnxAL1oonPOZIU7FZi0j+ZDPUyYnJ205H/0AzpuRnMYxXl0Uh3bEEjS3sYqIR0kPS+xdcvyaf52VOCOfcq+TLxpNr24JOVahSMqejNwB/Z66wEVe7K6YhZmSEEfiJbjOqVIo6jTAOhtjgq6sZZWFY48i/uUuO/IsKLJvTqzLSyHs8sCUwLW6o4vuNNrqFOyJ4m6s47elCiPdjzRQg+5Gn8sVHBqlwl8PMQzFIvspQITLLqxPQT2LzlmbcjBtINns4DtCMfi00dpnJf+8u2+lG3OKuL6cBSKK1ejW2Lb2lR5fcNgH2ZqKXoEgqxzn0s0aYPN2uEs09PTnXmpHVI7MBTnR/SgNZ6D02cI+KDTpOBHhzqNHxlTleQCVCtNY5pv9ZkZbArqeQajnUMmfr4idWu42SrXXA8KjoQD8G5FqhpfFIRUMSQz1ySkBFaQhwCdxlLAmfncVG/GR8sH8yGGVPeucUQxuRftRpE708DpRh8iJDfvYohN00JPVhGuxbh61h9N/ChGcOmWi0jtoLd5PFbhWd7ICbjh1uK59QW/pw/e8VaUsc8nzuTZkRRJ886hRqE+i/1plKn4xm9+ZCtSjddgPaOmgYVV3g70TZklNPenkMJcM0bbhuLAOb+G0/YLNEq8QKxocGNQ5cXtbvKyARt4wpmdLkSU7Ru5aEVYvbqJBYDMne6kMjJ8ZDxSQFU4tS5oI89GlQVU5/jJrbsTmqtPTjA1wQBUNF8GQsygMBObrUsKlX5MndkeeXdjoX0yBOjfRfNqxGnxyR34RFWHMyzqCTg/SHJKvZ5CCXLwDSRfIqtEnLU3PptUp28LkmMVzhsF0XbIG82ApjrwApWFi48/PPM2CkSAoLocPV7hvQHORIsOBdY+bl1vRBbTess/817/oXFdOdmWHwsqO5bKOvnoQqIF93XPxs6Qd9Itao/BGoe/2+sl3/I5VWXPoNWwstErTcUu7asmcAq3CocVaDm6acmkzGg7wuNTeuFSEFquzDbQdHbQtt0MMeoT4y9QvTIeqj5b7VsJv85Uj2sl4sLuuR9/iqaPlxwkZ+KzEZ9cvxEuVfK3dFUfHGOQQ1nnOjIEKHuysjDSzPT/SjEQxDJFevzfqw+9C3laoO/+ZFExwzXKJQzq9+sCtuuRr12sSOJ9tHx2E4QCvkZ1B7QZpk9odOAWHHvTGofA9dGwDQdrJoGEsBEizOjo9xPOfrQ/RKUg/b1N7fRCKao8hP0u1d6nv6kKrGUKv/4GrSqr0jEH6uFNI1OkQE81inR4MFyTOZRQ= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: cd39796c-9537-46ee-196e-08dd2c1b7eb7 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2025 17:24:35.9423 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +5tiddOO9fhidpFg4cbWJ1X1MhrzIED0xV9ZMl9dZ713CI2uyICsgoCvf0DHB3KG X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8066 X-Rspamd-Queue-Id: 12FB3A000F X-Stat-Signature: da1dh67pwge4ie59ezrjpwr7qfrej9pr X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1735925155-790194 X-HE-Meta: U2FsdGVkX186mDNhAmnW6I91pZNKSvgKinNIdpT7SMGS+pqi5fPf14ji+r/3thFLTFSLjZwvaSjyFOeQbimu+Aw4G/xr7a8U+wHeQLtYzuFRZ2jExuNUV5EEN3j8qsIvDDZqPZ6NGxgbXOEZL7+a1iGR9IowKB5Cn86z4BKNVsunet9GMtVw1bE1YHc0YHS96DUFXmrg05BHMV/LJfM2yfPiBFZOHs1p6LzAbfOEtVCrwnZCRHArbgRMe5oa4G2rw41bD1k6LDoQwaen2Xthh+ymdp8og8lKkYt+rLYpLgBS2rOtqYkHHtRTn1/fj6krAvSPBo7o7QIyDF+1n6kpbLKqSiywtQR7rVTSjfbp/0r7huHKl4l5MlTt4cbwN34DSzEzOkxBEHGSmTzTIfteFv68mTINXpPzXLrwTwmWQKnQbamiJLZctae5QUwrBubp09X4Pu1OFx5NAGo6cReJSLmzZ/Vptg5wHbYGJAm2t78oLn+roFX6efsQeznh7k7wLhwbh5X7uB8BFV0PTwtGPgL/nq7vHMvJTtmixZOjD8Ro4+bifHFBBHgA7kbVRYJSuJuX4Plexqq4N9bviSdtUy5Bm+x6kaMpPwhrkAxekXuvUC0HqhXx1yEd2+3pVVV8KNypjnSHpdbH58P1NN98J3tqvaIVabdZZLvbkniYMbFwwp483oytozBEra680FYUXPTseoparuB+TztFq8IDFDt4xr0Wcr0cfxDzKPOmePVw9Rbyn8kqtQWUmz1gi2gNXGNe6Ia5gS1TynvHf0SNJh2kmusBjhZCGFjkKZpflgRA651Fj9k5wVUj4brOA3PGG1ljpW4Sg2KJlSiDLTFOdn8kkclRWGBV0h/+VVnU4osJPLHRnS1cTfClH25mKpMSzB+O97+QprkQj2pIRpCBOX+s+b4yw2soKcqObNuDnqZ8HHcbpOsucUrjvlP443AokakpOfdT8cEd3cdPykE 1qRHCMg0 y8P/GTgttD9uB69/CXgrTsA5WnXBNdumo1hr36ltSDw3Cd+So3kPG7nREgPaGRSdAjHFs3VVfNsWzkww56KOIAOmSmoGapNXSrhzP/8IjA+2Iupe0iXJuy5WZblkOEdbqHY+AGmcAzSIGNqsAfogmMQ83HOd7SmnxkDugnt0O4rYEn037frUEiiRKLDFrKw765fkjeE4JKioWzXmPq4E2QUAd9s3MZr2Ev89RmiKbvWdfFoDggjVudBgEyDj5ztvGaua6H9KhZB9KJHADpNCknsoJFWxTM7yazWBRV03/yS0eBU+jgb5x9TwGsHpzbqialQqemcJedICWoXiBSE0ZO4pD0CZWatrSybq+9LBEf/q67Vj3k82Lwqp7YHxVvYMjI9wxoVbw7BBkJD14OHe1MDqRum0Uzn9YAWfh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now page copies are batched, multi-threaded page copy can be used to increase page copy throughput. Add copy_page_lists_mt() to copy pages in multi-threaded manners. Empirical data show more than 32 base pages are needed to show the benefit of using multi-threaded page copy, so use 32 as the threshold. Signed-off-by: Zi Yan --- include/linux/migrate.h | 3 + mm/Makefile | 2 +- mm/copy_pages.c | 186 ++++++++++++++++++++++++++++++++++++++++ mm/migrate.c | 19 ++-- 4 files changed, 199 insertions(+), 11 deletions(-) create mode 100644 mm/copy_pages.c diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 29919faea2f1..a0124f4893b0 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -80,6 +80,9 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio); int folio_migrate_mapping(struct address_space *mapping, struct folio *newfolio, struct folio *folio, int extra_count); +int copy_page_lists_mt(struct list_head *dst_folios, + struct list_head *src_folios, int nr_items); + #else static inline void putback_movable_pages(struct list_head *l) {} diff --git a/mm/Makefile b/mm/Makefile index 850386a67b3e..f8c7f6b4cebb 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -92,7 +92,7 @@ obj-$(CONFIG_KMSAN) += kmsan/ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o obj-$(CONFIG_MEMTEST) += memtest.o -obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_MIGRATION) += migrate.o copy_pages.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o diff --git a/mm/copy_pages.c b/mm/copy_pages.c new file mode 100644 index 000000000000..0e2231199f66 --- /dev/null +++ b/mm/copy_pages.c @@ -0,0 +1,186 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Parallel page copy routine. + */ + +#include +#include +#include +#include +#include + + +unsigned int limit_mt_num = 4; + +struct copy_item { + char *to; + char *from; + unsigned long chunk_size; +}; + +struct copy_page_info { + struct work_struct copy_page_work; + unsigned long num_items; + struct copy_item item_list[]; +}; + +static void copy_page_routine(char *vto, char *vfrom, + unsigned long chunk_size) +{ + memcpy(vto, vfrom, chunk_size); +} + +static void copy_page_work_queue_thread(struct work_struct *work) +{ + struct copy_page_info *my_work = (struct copy_page_info *)work; + int i; + + for (i = 0; i < my_work->num_items; ++i) + copy_page_routine(my_work->item_list[i].to, + my_work->item_list[i].from, + my_work->item_list[i].chunk_size); +} + +int copy_page_lists_mt(struct list_head *dst_folios, + struct list_head *src_folios, int nr_items) +{ + int err = 0; + unsigned int total_mt_num = limit_mt_num; + int to_node = folio_nid(list_first_entry(dst_folios, struct folio, lru)); + int i; + struct copy_page_info *work_items[32] = {0}; + const struct cpumask *per_node_cpumask = cpumask_of_node(to_node); + int cpu_id_list[32] = {0}; + int cpu; + int max_items_per_thread; + int item_idx; + struct folio *src, *src2, *dst, *dst2; + + total_mt_num = min_t(unsigned int, total_mt_num, + cpumask_weight(per_node_cpumask)); + + if (total_mt_num > 32) + total_mt_num = 32; + + /* Each threads get part of each page, if nr_items < totla_mt_num */ + if (nr_items < total_mt_num) + max_items_per_thread = nr_items; + else + max_items_per_thread = (nr_items / total_mt_num) + + ((nr_items % total_mt_num) ? 1 : 0); + + + for (cpu = 0; cpu < total_mt_num; ++cpu) { + work_items[cpu] = kzalloc(sizeof(struct copy_page_info) + + sizeof(struct copy_item) * max_items_per_thread, + GFP_NOWAIT); + if (!work_items[cpu]) { + err = -ENOMEM; + goto free_work_items; + } + } + + i = 0; + /* TODO: need a better cpu selection method */ + for_each_cpu(cpu, per_node_cpumask) { + if (i >= total_mt_num) + break; + cpu_id_list[i] = cpu; + ++i; + } + + if (nr_items < total_mt_num) { + for (cpu = 0; cpu < total_mt_num; ++cpu) { + INIT_WORK((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + work_items[cpu]->num_items = max_items_per_thread; + } + + item_idx = 0; + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(src, src2, src_folios, lru) { + unsigned long chunk_size = PAGE_SIZE * folio_nr_pages(src) / total_mt_num; + /* XXX: not working in HIGHMEM */ + char *vfrom = page_address(&src->page); + char *vto = page_address(&dst->page); + + VM_WARN_ON(PAGE_SIZE * folio_nr_pages(src) % total_mt_num); + VM_WARN_ON(folio_nr_pages(dst) != folio_nr_pages(src)); + + for (cpu = 0; cpu < total_mt_num; ++cpu) { + work_items[cpu]->item_list[item_idx].to = + vto + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].from = + vfrom + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].chunk_size = + chunk_size; + } + + item_idx++; + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + + for (cpu = 0; cpu < total_mt_num; ++cpu) + queue_work_on(cpu_id_list[cpu], + system_unbound_wq, + (struct work_struct *)work_items[cpu]); + } else { + int num_xfer_per_thread = nr_items / total_mt_num; + int per_cpu_item_idx; + + + for (cpu = 0; cpu < total_mt_num; ++cpu) { + INIT_WORK((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + + work_items[cpu]->num_items = num_xfer_per_thread + + (cpu < (nr_items % total_mt_num)); + } + + cpu = 0; + per_cpu_item_idx = 0; + item_idx = 0; + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(src, src2, src_folios, lru) { + /* XXX: not working in HIGHMEM */ + work_items[cpu]->item_list[per_cpu_item_idx].to = + page_address(&dst->page); + work_items[cpu]->item_list[per_cpu_item_idx].from = + page_address(&src->page); + work_items[cpu]->item_list[per_cpu_item_idx].chunk_size = + PAGE_SIZE * folio_nr_pages(src); + + VM_WARN_ON(folio_nr_pages(dst) != + folio_nr_pages(src)); + + per_cpu_item_idx++; + item_idx++; + dst = dst2; + dst2 = list_next_entry(dst, lru); + + if (per_cpu_item_idx == work_items[cpu]->num_items) { + queue_work_on(cpu_id_list[cpu], + system_unbound_wq, + (struct work_struct *)work_items[cpu]); + per_cpu_item_idx = 0; + cpu++; + } + } + if (item_idx != nr_items) + pr_warn("%s: only %d out of %d pages are transferred\n", + __func__, item_idx - 1, nr_items); + } + + /* Wait until it finishes */ + for (i = 0; i < total_mt_num; ++i) + flush_work((struct work_struct *)work_items[i]); + +free_work_items: + for (cpu = 0; cpu < total_mt_num; ++cpu) + kfree(work_items[cpu]); + + return err; +} diff --git a/mm/migrate.c b/mm/migrate.c index 95c4cc4a7823..18440180d747 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1799,7 +1799,7 @@ static void migrate_folios_batch_move(struct list_head *src_folios, int *nr_retry_pages) { struct folio *folio, *folio2, *dst, *dst2; - int rc, nr_pages = 0, nr_mig_folios = 0; + int rc, nr_pages = 0, total_nr_pages = 0, total_nr_folios = 0; int old_page_state = 0; struct anon_vma *anon_vma = NULL; bool is_lru; @@ -1807,11 +1807,6 @@ static void migrate_folios_batch_move(struct list_head *src_folios, LIST_HEAD(err_src); LIST_HEAD(err_dst); - if (mode != MIGRATE_ASYNC) { - *retry += 1; - return; - } - /* * Iterate over the list of locked src/dst folios to copy the metadata */ @@ -1859,19 +1854,23 @@ static void migrate_folios_batch_move(struct list_head *src_folios, migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, anon_vma, true, ret_folios); migrate_folio_undo_dst(dst, true, put_new_folio, private); - } else /* MIGRATEPAGE_SUCCESS */ - nr_mig_folios++; + } else { /* MIGRATEPAGE_SUCCESS */ + total_nr_pages += nr_pages; + total_nr_folios++; + } dst = dst2; dst2 = list_next_entry(dst, lru); } /* Exit if folio list for batch migration is empty */ - if (!nr_mig_folios) + if (!total_nr_pages) goto out; /* Batch copy the folios */ - { + if (total_nr_pages > 32) { + copy_page_lists_mt(dst_folios, src_folios, total_nr_folios); + } else { dst = list_first_entry(dst_folios, struct folio, lru); dst2 = list_next_entry(dst, lru); list_for_each_entry_safe(folio, folio2, src_folios, lru) {