From patchwork Fri Jan 3 17:24:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13925728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91C19E7718F for ; Fri, 3 Jan 2025 17:24:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BE6066B0082; Fri, 3 Jan 2025 12:24:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B96196B0083; Fri, 3 Jan 2025 12:24:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0F8D6B0088; Fri, 3 Jan 2025 12:24:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 843F46B0082 for ; Fri, 3 Jan 2025 12:24:40 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 94F5AC09BE for ; Fri, 3 Jan 2025 17:24:39 +0000 (UTC) X-FDA: 82966813020.06.100414B Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2076.outbound.protection.outlook.com [40.107.212.76]) by imf29.hostedemail.com (Postfix) with ESMTP id 7508612000C for ; Fri, 3 Jan 2025 17:23:23 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=UYY40W1I; spf=pass (imf29.hostedemail.com: domain of ziy@nvidia.com designates 40.107.212.76 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735925037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=l+CqyGM1L18z+d0UgYNHbrwjXua3cyv/GnYItzfnGWE=; b=cFCxrMCs9RjSYb7y7CbecEmNobj/bCCXhyQPzjPn3/qGeuMp/Ou3FrmPLm3FrroED4/ksb uPFY9yo6Xs4lT/QITRdjh71l1q2mZnjumnFUr1lzVpWsihlHYJ6HDdb3nH1FENVLrFoDnn VY/ArF9WnoadrW6RlSL/1DJeA6EENpQ= ARC-Authentication-Results: i=2; imf29.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=UYY40W1I; spf=pass (imf29.hostedemail.com: domain of ziy@nvidia.com designates 40.107.212.76 as permitted sender) smtp.mailfrom=ziy@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1735925037; a=rsa-sha256; cv=pass; b=a2WlHqa6rnatgzrnPnFAhHTpafyha9uSBgU0tMFinMFnkjlS+HfjlB/UVCTEjC95AVRQee 5y+c6+Ft03rAqvnJXhegzecDQ+tVZDqCximsxwZgLbr0w5McaVK4vCzJjbLkk6I2EFR9Ks Jk3+hWY93lWTFUEmcLgvXS3HFfvnnRw= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=yej9IxTH49aZ/5BqQovo3KcX27YzUR5dHPvmNHIciHnBCyRNxYnfY4JxogtHxVXa0NBYRV51AXXcKndkpu3i1HJaL1M+R66vwj2lICenuQ5/kyqijAWCv18ssvQxai937pu5cMoKg6BLUzcoUi+tjtctlFaslzftCLhJOs01PPLG6nPB9qJ2w34Y20dNaW+i0g1OiumL5YAd2jNpq6xJlmv2H5O9Y56MRhnDovSefVq5rUoA5MI0XXio1suoM4fzRp+E0bjXc1LjjpVgZE6J+6PsQmlgL+etR3KCotrfiof91pkSoEwmQvQVjGe2/SoOPmRikCgmCzGqdpeh0f4WwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=l+CqyGM1L18z+d0UgYNHbrwjXua3cyv/GnYItzfnGWE=; b=h5+Jgn0fCO+lNUevgakJ9N9zyR8L2kjxCSWwz+kkZM/r4YLvfgeZX0ba9bBjtiu5/TGhENmu+7zbqAPTgJPDbulXPfpRsLU4BSxZtWptQkAZYSZdUgtFxYBz5/1jEdBzRL2e7WjlNj2bkRWo052mn9YriHzf0K4GZ7Refb44r1SJD+0+A8dw5HLpWUl7pDkSax7bnB+BjKCp8wLcg5a+7Ieepn5K7aGk0IgTpkZfgLlEEnDf0VSLpnexqU1Y8K9J7IrIOLNlPFxOj0bwfiO2mXmrExfiNiV9qZhykm1ztxsYmkzU04C/bkX7ZI3BO9pfugBDIkaRplLLibjsHLyL8g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=l+CqyGM1L18z+d0UgYNHbrwjXua3cyv/GnYItzfnGWE=; b=UYY40W1IzD+b7Q6Aphh6hSWvzYIFoMwiuF6Q7hegeyzhlDVZg6S5OWK36zi0K2zknk/moXwrZmpMiShB5y4t5W6J2PH2A73dGq3uybkprLJHT04eMexCOJ3rBDFFdeNDotqq6bckPo3qA/ypod0Jle0WIguAvnQFSM4LuQbt74qOf0drbv5nZolCvffjDWQ/yvJ0pZqf4DTiF3A1GpYNz9xI/tyq2atxOg4RbA11ov1zuxbfimLT8p0gp6yPJnlNLvjtbv3E6VqFqlMj5GGCb/mSLZnsTfIWj+E9WgHNAB0OoK2jjJaN1Yw7wgMhmUX49L5SXqE15c0lDSFUWJIZNA== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY8PR12MB8066.namprd12.prod.outlook.com (2603:10b6:930:70::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.11; Fri, 3 Jan 2025 17:24:30 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%3]) with mapi id 15.20.8314.013; Fri, 3 Jan 2025 17:24:30 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: David Rientjes , Shivank Garg , Aneesh Kumar , David Hildenbrand , John Hubbard , Kirill Shutemov , Matthew Wilcox , Mel Gorman , "Rao, Bharata Bhasker" , Rik van Riel , RaghavendraKT , Wei Xu , Suyeon Lee , Lei Chen , "Shukla, Santosh" , "Grimm, Jon" , sj@kernel.org, shy828301@gmail.com, Liam Howlett , Gregory Price , "Huang, Ying" , Zi Yan Subject: [RFC PATCH 0/5] Accelerate page migration with batching and multi threads Date: Fri, 3 Jan 2025 12:24:14 -0500 Message-ID: <20250103172419.4148674-1-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 X-ClientProxiedBy: BL1PR13CA0200.namprd13.prod.outlook.com (2603:10b6:208:2be::25) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY8PR12MB8066:EE_ X-MS-Office365-Filtering-Correlation-Id: d5d4cb9f-928b-4237-9a41-08dd2c1b7b4b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: 7ykU7ipxWM/WE6mKgRSL9Q9y20ja2stxfFrH+Fm91VtWZqIWJGqPFHJFKtbg5OJuPh3OVvtqdtneaniVvPXxpIxyYMttcWOnWU+C2eMRwoqTk8kDyMQCvuUwnTTfL+zzBctLslirQj/rhIUKnw+HtS6qP60sLYzWgc5JQq7svpTA3gOZ7CVJFfHIJcQwxUWNyF6njk+BxnVlkE4s01/ZAaJP2fYYnSP/mPay3MM58PfXTJkkaXVYX6aDKFzFBGD/+wJ8jEoovNyZBpjB/XWV9FIXmBR41Fu9O6W+UYH6TfVB4ONfexPjHqM5NSzDDXO+yFNn6OAhtjgsapUcQRQVumSxXy9hr7lbzTcwDdgh88v+9jpE9y5fLR+fosP4S8GcWLcZK6qu1X6od5s4MK2frgi3OkHbhStUU5LjOImVQtk8iAwXL45Ew7yxSn4660BwIXN/Txb+y7rrFR0YZZPigVC9qv5MO6XfkQW6/ikhX48Rij5RYy8IU3FWWrV+8n98E9iZTYmYOX9vATwuM2nFXCoLbblEyAlPYH/+2MFXhlTZojGWZmlf+LytPtugs+/Xo5SUczXn/eywl2JllfxhEewSTi0TBaplf2a0cPu1OEKLhL67DtL7ngAElNt03Cy3wt2MskWxKmP4VszBI4vlfyh13B6l2ebDY8EaxX8GU4crR4eso48efhy8Dm4wZJBwGGNwgRlQZiYva3QsMDAi7cmh4qhAj1vk5z7zBoKQPBRtXC+juEgI9qTcJaxFYzaWduSmmFD337QRwd3GKK7gOboIUYl2VMyLBqG5I9jb4PXFLXbAfdKT1sOhA0pVwKR28KLCR9fRm4dOOVq9sMKE0sfRNN5ThKpqhXQ5g0rSw7wt+K/FBjEnp01/xnyeVUhp4aaLYVXVtyyIVSv1CbAeglHtg22OZz/AhK/7IwKpP4rREXA+qwmtg4l3mHA9NEVGgZMkZ/pD7wJ2QsYbnsQq/KTP9JpCByaENHo4F7R8EplRAgi5TRCPezveGUKCtYKhG/rYuH7jrrgQThkxXy5CIDOsziP1DZ4n/6GAgKwzYo+X8FF3/VsQMlrsYM90Y6ssgQIJkcO28UHiUjGh0emFrcwVAejMqSk5OeDqxbhRLxcb7MphLu0s2Kg/HmYqpaTvvcb8Fb+O8ucC8wICLZqUeuIoSnmoYJ2l3E6pX4UQr+t9ufnI4TezKNbY5bSlY5351UR8YyOql4auDzlVmq3HJUBv7KwsLAEOWAhr7qT+6rQeCaifm+CTKmtcss31st0qUBnEJvhZoiutJRHUxW5gRok2f5noUf7xb44LyUHxHp7oz0DcjwXKQTVlE6SBbtk554dIhDn0mgvM2mK+iiNinHW29MotKOJUDxWExckK2yv6ZJL2Auz/OWUNt6vjo9Zs X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 60EYopr5AYDuqUzOftoxM/7zJ7oRjWwnkFp30UGq7dKmuosvLrIPKxi+hiMEsGonCqO1hABbkdSuGmxHrJIJgHEgUVhcMNBvWvA15tGbfvphyjCAQA8SLdqxLu5KKqr4q9Zy7Vh0+9AVCa55JC5SplwPy2t+EEgG26raCm9YOhimsRyUfo4mTykmMhm3f2sf40E2pBNkPi189spAGe+O7t68lz88PfViDq9Mwrndhr9u7fCWwa738P0gjRnHZSS0bHmO6evou5kGhmMvJeYKu2agxJ92PO2dhLUYAWUBGJK6T9fdQ0Innz5LpaZKgxqEJ2ipblo9MBXyXZplYO1j+PyeagDoKF59mUrUknGaAuYYBsRitELUGujDD3CW/Aa76YPN9vIMO+qW3dOcG62nvdwne5qsEDSmpjauk9Gu+AOjsdtyiydZyMZi/H1Tzehccfo5D0szZwB0Yj1eMNbUjHHrc925wvICph8OtYzaz4GXyqReGrvhQgkeVEE/d60JxnSHbAUNzTOwa8nxopzFIWksmvmmmX6aOO53nX5Uvn5fxERV3pdWVW7Dx7bev6QFHAoqiS5GqWdpWe8eoZi/nsfsNSTkQOLexGkkV4PZFGg2X6LodubuAxthsG3GzhYMJSas7TMGjf/F7ORgTcR1D1L4uRqaW8iP0cdwWP8NC7sMxihKuyaVTnPNqb++yYHFHrNC8GQQSMxYheqRtONFGzHDpah6hXSjtDRVUREVkLeLFRMQDwtk5TJtY0PSoa+24hnCke72RAjSh+zSSLIWZA9AQXBQnsJZMhSzg8pk/DweOgOzjClI1c1b22iJanXwWJ8WCSdG05eBd7oO7J69fYF8JWIy2PPOERClMgxYWvB6RfIc5xlrWtQyuO95VOxtgIEqb2XXKP4i+sUgOj5PmwwP8nMnCmIFn/c+Lf36Cs2w6CltiQE8d0M9M4o4AC8WmqpXIO5Bu0smBP1QvOltWVE35NhZ6bjyNqLUQ4kaJE+vGkxxe5prQNQHArcYh/aLrQA+2/IRkONPaPo0yCi19p2zfMyIrrJUH8ZtsJgVKmAOKY5o/Mk68RVvADhvPB1NlEx0G1H9GR/EX47/RqRKqV1/WiHHyVOBvqQi41a6/cTP2Ky/1+0LYezzotryCfJoTwlD08CYqfkmOZg1t9LXGGLUjUKUijl45yUETJgNMhDV+GhtPZ0GT+d+vyBY3mCyifrmZci1/hTQAnoMo/mjDxmaSpXaKaekKPRl/6ILhoLSF/6n202DS2FWei1nddlyiCsdVCSb43mmyvOwa/J+AklBHaNCbQrLQOL4BubLBiQiph0pr8uKZ1ZIEXc1jp6rc+7h2bq1LXUOzUiu70MjIWAi6pC+Ld1S8V3WB+cbFQrVP/uR6jAA1XqX09igd+cuSCNlezbvGSELIqifi+rOlSS+2+zijulJgd+nJBY1kpIw1rMtIodXsMBHKiOmDnyOX/vYsZAdsjSJEw5aIJvim3yPcQ50y/8kOcr7KZF3m96Jl8KmWv6Fm+EJRnWMUe62dhnq9Xs88tNTJceVj0T0PUlxT0CIspfJg9oDrk7fB8o= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: d5d4cb9f-928b-4237-9a41-08dd2c1b7b4b X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2025 17:24:30.2042 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ft7hoS74H1+A0VStWnoqjUnTJkE2zYaKM0x5y9CtuQXY42/zVLJQrc2ulMfVKyQ7 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8066 X-Rspamd-Queue-Id: 7508612000C X-Rspamd-Server: rspam12 X-Stat-Signature: 5o7ixr7o3kjtnmp77qu9cothpzodhgqn X-Rspam-User: X-HE-Tag: 1735925003-980893 X-HE-Meta: U2FsdGVkX1++P734abiPGblGchhQ9ESU1P6Rmtr1qLvabNN06QQfj7CccjotPS9MD/iBx1hCodNVls7wqwprOdhkJxEGjpR6JduRZM6FCk9FlXcK61IovCNzILYhAmDy22ekUQ53YnwtHb9CchSwRLUN3AIQs4E7K+kjtkRzRfOCp9DRePoBRbCLQE4KOfUcCC6ZGnXI98S9zbBWXW257MXYJhoE/bB2AaPko2/59U1spIqSXPRSd8doR+snn//Er7BF8ToYF93DjnhPhp/EmEH5X1ENh5Ytp/zEEzeZ5U/JcW+c1yICD5yXVCKnk4vSkFmdxnQbvqPeMaSyqpDfzNWk0qZAK3+z60O5RzFMJqa4YYE8gRgdWs53yuZoIQ/VvyXOnqxICmvhzzWmc2TxNuxGQtD6Dsfzuc61rP2WwhC/eM6Qb39gCEejsspGD/RiwpuBLnICVo/PxSwCRWfRpWux/vIESXJU/bN+ZxeuSIT7rsvYx8gE1yTAAf3D9rfIa6Qk0Ulx1mAtD6Uhj5AlK+OnxQJzFjjIqkUkeXhWK0eWzri2GEHhdtmDQyuLh2o5MxczfaGrYRIXJ6BFuPqOKzYTcEsvBobaCCEOY71haCKo4uVxcsq2lrpxB7GtJbDqWMqhfyXlhUVdxcan4bvKZdlwEI91a0TAMIfMcitZp5B1XDYv2ZGFYV5+1D5jWFGCwZpjbLTer6LkjW8wY02n8SuPeuMyjsaMwaUvdcJLyvTIJZq6l++uleyua5GpRj/ukle8TCjqZxCXv86okyHAFgUYm3w1oII9cxqDURQdlSb6RNctoyMLDoZgdEUn7daVDSXoZTLnRski5bxiK4Ucql5RukoFixJMyWjVEJuQT9WMBy8gefX2owooV3HogVyZO+jV1NNbAiGPFOuCQQhA+z+JwP06bvgsld5bp0YP7GZcVd/CoUHr9MpJ2U5AONMYPzlullb9PGdqYVcRdFn pjkBVqDP hXgFJqaMGlMUSgpDTBYX/w1dE87K3nWWeHnFOdQCW+NPw30NYqxR58oEDafOoBAWZMebXU6HB2jlp2uC9jdKu5p4nZ4+p9T+byc8oQbhZVvlu5Ag0yyv2IlTN3NIPw7Hu0zojvyGKWyb/aoUTdBfVfYyAcezIXTdAqr08X1ZiVevnoiX6f5B/F5nJVqofsjilZMkeMpJRcCMrbZtc+oPPFkj1vvYZggamSAGxBq6Z1XOnZyI7zy6UmHGSdMtc6Uh8J+gAXC0tUG/iAqZjd72SOe7N5EqRZNld2zDPosCw0f6le7OhVAZcfUCsbsjYAtUW5F93XEJ5dqiG1a07+eMHkZChpJ4dclp02W5WpqTpKFKeaLIuE06NEe2cK/QD4+IENIiBvZXUSp0wpAAOXvYR8wJ7CFbp02BrLWUsQ/hCIMk2GYYMWRQwaPrSPRdYudCAou/t2iwkD/MPJkdu3RTQR22SF2yawmoAZ1ssG4gcCnKgxOk8ETh3Z2rqiloksefnDquQVCl9dsrkaj+aXg7X2wuUlrFBNU44tpcHFDqDdeZQloQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi all, This patchset accelerates page migration by batching folio copy operations and using multiple CPU threads and is based on Shivank's Enhancements to Page Migration with Batch Offloading via DMA patchset[1] and my original accelerate page migration patchset[2]. It is on top of mm-everything-2025-01-03-05-59. The last patch is for testing purpose and should not be considered. The motivations are: 1. Batching folio copy increases copy throughput. Especially for base page migrations, folio copy throughput is low since there are kernel activities like moving folio metadata and updating page table entries sit between two folio copies. And base page sizes are relatively small, 4KB on x86_64, ARM64 and 64KB on ARM64. 2. Single CPU thread has limited copy throughput. Using multi threads is a natural extension to speed up folio copy, when DMA engine is NOT available in a system. Design === It is based on Shivank's patchset and revise MIGRATE_SYNC_NO_COPY (renamed to MIGRATE_NO_COPY) to avoid folio copy operation inside migrate_folio_move() and perform them in one shot afterwards. A copy_page_lists_mt() function is added to use multi threads to copy folios from src list to dst list. Changes compared to Shivank's patchset (mainly rewrote batching folio copy code) === 1. mig_info is removed, so no memory allocation is needed during batching folio copies. src->private is used to store old page state and anon_vma after folio metadata is copied from src to dst. 2. move_to_new_folio() and migrate_folio_move() are refactored to remove redundant code in migrate_folios_batch_move(). 3. folio_mc_copy() is used for the single threaded copy code to keep the original kernel behavior. Performance === I benchmarked move_pages() throughput on a two socket NUMA system with two NVIDIA Grace CPUs. The base page size is 64KB. Both 64KB page migration and 2MB mTHP page migration are measured. The tables below show move_pages() throughput with different configurations and different numbers of copied pages. The x-axis is the configurations, from vanilla Linux kernel to using 1, 2, 4, 8, 16, 32 threads with this patchset applied. And the unit is GB/s. The 32-thread copy throughput can be up to 10x of single thread serial folio copy. Batching folio copy not only benefits huge page but also base page. 64KB (GB/s): vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_32 32 5.43 4.90 5.65 7.31 7.60 8.61 6.43 256 6.95 6.89 9.28 14.67 22.41 23.39 23.93 512 7.88 7.26 10.15 17.53 27.82 27.88 33.93 768 7.65 7.42 10.46 18.59 28.65 29.67 30.76 1024 7.46 8.01 10.90 17.77 27.04 32.18 38.80 2MB mTHP (GB/s): vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_32 1 5.94 2.90 6.90 8.56 11.16 8.76 6.41 2 7.67 5.57 7.11 12.48 17.37 15.68 14.10 4 8.01 6.04 10.25 20.14 22.52 27.79 25.28 8 8.42 7.00 11.41 24.73 33.96 32.62 39.55 16 9.41 6.91 12.23 27.51 43.95 49.15 51.38 32 10.23 7.15 13.03 29.52 49.49 69.98 71.51 64 9.40 7.37 13.88 30.38 52.00 76.89 79.41 128 8.59 7.23 14.20 28.39 49.98 78.27 90.18 256 8.43 7.16 14.59 28.14 48.78 76.88 92.28 512 8.31 7.78 14.40 26.20 43.31 63.91 75.21 768 8.30 7.86 14.83 27.41 46.25 69.85 81.31 1024 8.31 7.90 14.96 27.62 46.75 71.76 83.84 TODOs === 1. Multi-threaded folio copy routine needs to look at CPU scheduler and only use idle CPUs to avoid interfering userspace workloads. Of course more complicated policies can be used based on migration issuing thread priority. 2. Eliminate memory allocation during multi-threaded folio copy routine if possible. 3. A runtime check to decide when use multi-threaded folio copy. Something like cache hotness issue mentioned by Matthew[3]. 4. Use non-temporal CPU instructions to avoid cache pollution issues. 5. Explicitly make multi-threaded folio copy only available to !HIGHMEM, since kmap_local_page() would be needed for each kernel folio copy work threads and expensive. 6. A better interface than copy_page_lists_mt() to allow DMA data copy to be used as well. Let me know your thoughts. Thanks. [1] https://lore.kernel.org/linux-mm/20240614221525.19170-1-shivankg@amd.com/ [2] https://lore.kernel.org/linux-mm/20190404020046.32741-1-zi.yan@sent.com/ [3] https://lore.kernel.org/linux-mm/Zm0SWZKcRrngCUUW@casper.infradead.org/ Byungchul Park (1): mm: separate move/undo doing on folio list from migrate_pages_batch() Zi Yan (4): mm/migrate: factor out code in move_to_new_folio() and migrate_folio_move() mm/migrate: add migrate_folios_batch_move to batch the folio move operations mm/migrate: introduce multi-threaded page copy routine test: add sysctl for folio copy tests and adjust NR_MAX_BATCHED_MIGRATION include/linux/migrate.h | 3 + include/linux/migrate_mode.h | 2 + include/linux/mm.h | 4 + include/linux/sysctl.h | 1 + kernel/sysctl.c | 29 ++- mm/Makefile | 2 +- mm/copy_pages.c | 190 +++++++++++++++ mm/migrate.c | 443 +++++++++++++++++++++++++++-------- 8 files changed, 577 insertions(+), 97 deletions(-) create mode 100644 mm/copy_pages.c